Tuning a model often requires exploring the impact of changes to many hyperparameters. The best way to approach this is generally not to progressively change your source code, but rather to define external flags for key parameters which you may want to vary. This allows you to easily try out different combinations of parameters without changing your code, and makes your training script easier to integrate with hyperparameter tuning tools.
flags() function to define a set of flags (and default values) for a training script. For example,
library(keras) FLAGS <- flags( flag_integer("dense_units1", 128), flag_numeric("dropout1", 0.4), flag_integer("dense_units2", 128), flag_numeric("dropout2", 0.3), flag_integer("epochs", 30), flag_integer("batch_size", 128), flag_numeric("learning_rate", 0.001) )
These flags would then used within a script as follows:
input <- layer_input(shape = c(784)) predictions <- input %>% layer_dense(units = FLAGS$dense_units1, activation = 'relu') %>% layer_dropout(rate = FLAGS$dropout1) %>% layer_dense(units = FLAGS$dense_units2, activation = 'relu') %>% layer_dropout(rate = FLAGS$dropout2) %>% layer_dense(units = 10, activation = 'softmax') model <- keras_model(input, predictions) %>% compile( loss = 'categorical_crossentropy', optimizer = optimizer_rmsprop(lr = FLAGS$learning_rate), metrics = c('accuracy') ) history <- model %>% fit( x_train, y_train, batch_size = FLAGS$batch_size, epochs = FLAGS$epochs, verbose = 1, validation_split = 0.2 )
Note that instead of literal values for the various parameters we want to vary we now reference members of the
FLAGS list returned from the
When you execute your training script either interactively or using the
source() function then the default values defined for your
FLAGS will be utilized. There are a number of ways to train with alternate flag values, each of which is described below.
If you execute your training script via Rscript, you can vary the values of
FLAGS by passing them on the command line.
For example, to run your training script with alternate dropout values you would do this:
$ Rscript mnist_mlp.R --dropout1=0.2 --dropout2==0.1
To run your training script with a different number of epochs and alternate learning rate:
$ Rscript mnist_mlp.R --epochs=40 --learning-rate=0.005
Note that for the
learning_rate flag we used a dash (“-”) rather than underscore on the command line. Either is valid (dashes are automatically converted to underscores). This accommodation is made because dashes are not valid R identifiers but are commonly used within command line arguments.
You can also create a YAML configuration file (by default “flags.yml”) with alternate flag values. For example:
dropout1: 0.2 dropout2: 0.1 epochs: 40 learning_rate: 0.005
If a “flags.yml” file is located within the current working directory when the
flags() function is called then it’s values will be read and used within the training script.
Note that you can combine the use of a “flags.yml” file and command-line parameters. In that case flags passed as command-line parameters take priority over flags defined in the config file.
You can vary the file used to provide flags using the
file parameter. For example, to use a file named “alt_flags.yml” you would use this in your training script:
FLAGS <- flags(file = "alt_flags.yml", # flag definitions here )
You can also define a “flags.yml” that has named configurations which inherit from each other, for example:
alt: epochs: 50 learning_rate: 0.003 alt2: inherits: alt1 learning_rate; 0.002
You would use the “alt2” config in your training script as follows:
FLAGS <- flags(config = "alt2", # flag definitions here )
See the documentation for the config package for additional details on using named configuration profiles.
The tfruns package provides a suite of tools for tracking, visualizing, and managing TensorFlow training runs. The package includes integrated support for both invoking scripts with flags via the
training_run() function as well as tracking flag values used within each training run.
Here are some example uses of
training_run() that mirror the flag examples from above:
library(tfruns) training_run("mnist_mlp.R", flags = c(dropout1 = 0.2, dropout2 = 0.1)) training_run("mnist_mlp.R", flags = "alt_flags.yml") training_run("mnist_mlp.R", config = "alt2")
If you are a frequent user of flags then training with the tfruns package is strongly recommended.
Flags have a data type which is validated when flag values are parsed. Available flag types correspond to the following functions:
||Integer values (e.g. 1, 2, 3, 4)|
||Floating point values (e.g. 1.1, 2.2)|
||Logical values (e.g. true, false, 1, 0)|
||Character values (e.g. “train.csv”)|
Flag definitions can include an optional description. For example:
FLAGS <- flags( flag_integer("dense_units1", 128, "Dense units in first layer"), flag_numeric("dropout1", 0.4, "Dropout after first layer"), flag_integer("epochs", 30, "Number of epochs to train for") )
This description is used when the
--help argument is used to invoke your training script. For example:
$ Rscript mnist_mlp.R --help Options: --dense-units1 Dense units in first layer (128) --dropout1 Dropout after first layer (0.4) --epochs Number of epochs to train for (30)
Flag descriptions are a nice way to provide basic documentation on the parameters available within training scripts that you plan on sharing with other users.