[cityhall]

I've been trying to do this better recently after having some non-reproducible results. I've settled on taking all hyperparameters (including booleans like whether to use batch norm) from a global dict. Instead of commenting and uncommenting lines, I look up a key with a default value, adding the default to the dict if it wasn't there. Then I print and log the dict with the results.

I end up with a bunch of code like:

    if get_param('use_convnet_for_thing1', True):
        convnet1_params = get_param('convnet1_params', None)
        thing1 = build_convnet(thing1_input, convnet1_params)
    elif ...

By logging the hyperparameter dict, source checkpoint, and rand seed, results should be reproducible.

This works well for rapid iteration like in jupyter notebooks. For models that take days to train, you might as well use source control for your scripts.