Allard Hendriksen | CI @ CWI
Using Sacred with Mongo without root user rights
Jun 1, 2018
Allard Hendriksen
6 minute read
Back to top

Sacred is a framework to “help you configure, organize, log and reproduce experiments”. It is a tool for reproducible research in the same category as sumatra or pigx. Its documentation is very comprehensive and makes it easy to get started.

Installing Sacred

Sacred can be installed using

pip install sacred

For the latest version, with the latest bugfixes, use

pip install git+https://github.com/IDSIA/sacred

Using Sacred together with Mongo

To get the most out of Sacred, you will want to collect all experiment data in a central location. Furthermore, you will want to query that data, visualize it, and use it for further experiments.

To collect all data in a central location, there are multiple database options. I recommend mongodb. This is a database that is easy to work with from Python and is best supported by Sacred.

Installing Mongo

To install mongodb as a non-root user, you need to follow these steps. I assume you have Conda installed and have a working gcc installation. If you have already have a working version of mongo installed, you can skip this section.

  1. get mongo from https://github.com/mongodb/mongo/

  2. use the guide from https://github.com/mongodb/mongo/blob/master/docs/building.md to install.

  3. Create a conda environment named mongo

    conda create -n mongo_build python=2.7

  4. Install requirements

    source activate mongo_build
pip install -r buildscripts/requirements.txt

    If any errors occur, install packages using conda instead of pip. The packages cryptography and cheetah appear to give problems.

  5. Build mongo

    python buildscripts/scons.py -j 7 \
                  --disable-warnings-as-errors \
                  mongod

  6. Install mongo:

    mkdir -p ~/local/
python buildscripts/scons.py -j 7 \
                 --disable-warnings-as-errors \
                 --prefix=~/local/ install

  7. Then, add ~/local/bin to $PATH in ~/.bash_profile.

Configuring mongo

Now that mongo is installed, we can configure the database.

  1. Make a directory to store your mongo database. I use the following path

    mkdir -p /export/scratch2/hendriks/mongo/

  2. Start the mongo database with

    mongod --unixSocketPrefix=/tmp/  \
       --bind_ip 0.0.0.0 \
       --dbpath /export/scratch2/hendriks/mongo/ \
       --auth

    If this does not work, make sure that mongod is in your path:

    source ~/.bash_profile
# or (if you use the module system)
module load mongo

  3. Login to mongo shell using

    mongo

  4. Create admin user:

    use admin
db.createUser({user: "admin",
           pwd: "thisisyouradminpassword",
           roles : [{ role : "root", db : "admin" } ] })
db.auth("admin", "thisisyouradminpassword")
use sacred
db.createUser({user: 'sacred',
           pwd: 'sacredpassword',
           roles : [{role: 'readWrite', db: 'sacred'},
            {role: 'dbOwner', db: 'sacred'}]})

  5. Now run the following python program to test your connection: (in an environment where Sacred is installed)

    from sacred.observers import MongoObserver
from sacred import Experiment

ex = Experiment('Test Sacred')

ex.observers.append(MongoObserver.create(
    url='mongodb://sacred:sacredpassword'
    '@localhost:27017/sacred?authMechanism=SCRAM-SHA-1',
    db_name='sacred'))

@ex.automain
def main():
    print('hello world')

Now comes the nice part: using Sacred!

Setting up Sacred with Mongodb

We could start experiments and store them in the Mongo database right now. We could just expand the previous script and run it. You will not want to do this though. There are two reasons for this:

  1. The code contains your password in plaintext. This makes it more difficult to share your code with others, especially if you use git.
  2. The code absolutely does not work when your mongo server is not running. This can be quite frustrating.

To prevent storing you password in your code, I suggest storing them in your .bash_profile. For me, it contains:

# Used for sacred and mongo:                 #share-aah-cwi
export MONGO_SACRED_ENABLED=true             #share-aah-cwi
export MONGO_SACRED_USER=sacred              #share-aah-cwi
export MONGO_SACRED_PASS=mypass      #share-aah-cwi
export MONGO_SACRED_HOST=sipsey.ci.cwi.nl    #share-aah-cwi

Be sure to change your hostname to the hostname of your current computer. This is especially important if you plan to run experiments on machines other than your current machine. If you plan to use only your current machine, you can use MONGO_SACRED_HOST=localhost. Be sure to change the .bash_profile of any machine that you are planning to run experiments on!

Note that we have set MONGO_SACRED_ENABLED=true. The experiment code (see below) actually checks this environment variable before it tries to store the results in the mongo database. Therefore, if the code is run on a computer where these environment variables are not set up correctly, the code will still run. I use the following standard snippet for all my experiments.

from sacred.observers import MongoObserver
from sacred import Experiment
from os import environ

ex = Experiment('My experiment name', ingredients=[])

mongo_enabled = environ.get('MONGO_SACRED_ENABLED')
mongo_user = environ.get('MONGO_SACRED_USER')
mongo_pass = environ.get('MONGO_SACRED_PASS')
mongo_host = environ.get('MONGO_SACRED_HOST')

if mongo_enabled == 'true':
    assert mongo_user, 'Setting $MONGO_USER is required'
    assert mongo_pass, 'Setting $MONGO_PASS is required'
    assert mongo_host, 'Setting $MONGO_HOST is required'

    mongo_url = 'mongodb://{0}:{1}@{2}:27017/' \
		'sacred?authMechanism=SCRAM-SHA-1'.format(
		    mongo_user, mongo_pass, mongo_host)

    ex.observers.append(MongoObserver.create(url=mongo_url, db_name='sacred'))


@ex.config
def cfg():
    pass


@ex.automain
def main():
    print('hello world')
    return 1.0

If your mongo service is not running and you absolutely want to run an experiment you can run it with

MONGO_SACRED_ENABLED=false python experiment.py

Or you could just reboot your mongo server

mongod --unixSocketPrefix=/tmp/ \
       --bind_ip 0.0.0.0 \
       --dbpath /export/scratch2/hendriks/mongo/ \
       --auth

Visualizing your results with sacredboard

We have just covered how to run experiments with Sacred. We have not covered how to actually visualize the experiments and get an overview. I do this with Sacredboard. Sacredboard is a local webservice that gives an overview of all finished and running experiments. It has support for viewing the parameters, command-line output, and running time of an experiment. What is most exciting, however, is its support for plotting experiment metrics!

Installing sacredboard is quite simple. You could, for instance, follow these steps:

conda create -y -n sacredboard python=3.6.1 anaconda
source activate sacredboard
pip install sacredboard

To start the sacredboard server, run

source activate sacredboard
URL="mongodb://${MONGO_SACRED_USER}:${MONGO_SACRED_PASS}@${MONGO_SACRED_HOST}:27017"
URL="$URL/sacred?authMechanism=SCRAM-SHA-1"
sacredboard -mu $URL sacred

Again, make sure that the environment variables are set up as explained above.

Now you can easily view experiments and view their metrics.

You can run the following experiment to log some metrics into the database.

from sacred.observers import MongoObserver
from sacred import Experiment
from os import environ

ex = Experiment('Experiment with metrics', ingredients=[])

mongo_enabled = environ.get('MONGO_SACRED_ENABLED')
mongo_user = environ.get('MONGO_SACRED_USER')
mongo_pass = environ.get('MONGO_SACRED_PASS')
mongo_host = environ.get('MONGO_SACRED_HOST')

if mongo_enabled == 'true':
    assert mongo_user, 'Setting $MONGO_USER is required'
    assert mongo_pass, 'Setting $MONGO_PASS is required'
    assert mongo_host, 'Setting $MONGO_HOST is required'

    mongo_url = 'mongodb://{0}:{1}@{2}:27017' \
		'/sacred?authMechanism=SCRAM-SHA-1'.format(
		    mongo_user, mongo_pass, mongo_host)

    ex.observers.append(MongoObserver.create(url=mongo_url, db_name='sacred'))


@ex.config
def cfg():
    pass


@ex.automain
def main():
    print('hello world')
    for i in range(10):
	ex.log_scalar("Training error", 10 - i)
    return 0.0

Now we can visit sacredboard at http://localhost:5000 and we see

Summary

I have described how to set up Sacred with a Mongo database. Furthermore, I have explained some best practices with respect to handling passwords in code. Furthermore, I have shown how to use Sacredboard to visualize and compare your experiments.

I have not shown the full feature set of Sacred and all the things you can do with it. Most importantly, I have left out how to configure parameters for your experiments. Another very useful feature is for Sacred to save your files and other “experimental artifacts”. I recommend checking out the Sacred documentation for more!


How to read and write tiffstacks in Python next

Copyright © 2018 Allard Hendriksen
github linkedin email rss