Custom layers

    We recommend using keras as a high-level API for building neural networks. That said, most TensorFlow APIs are usable with eager execution.

    library(tensorflow)
    library(keras)

    Layers: common sets of useful operations

    Most of the time when writing code for machine learning models you want to operate at a higher level of abstraction than individual operations and manipulation of individual variables.

    Many machine learning models are expressible as the composition and stacking of relatively simple layers, and TensorFlow provides both a set of many common layers as a well as easy ways for you to write your own application-specific layers either from scratch or as the composition of existing layers.

    TensorFlow includes the full Keras API in the keras package, and the Keras layers are very useful when building your own models.

    The full list of pre-existing layers can be seen in the documentation. It includes Dense (a fully-connected layer), Conv2D, LSTM, BatchNormalization, Dropout, and many others.

    ## tf.Tensor(
    ## [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]], shape=(10, 10), dtype=float32)
    ## [[1]]
    ## <tf.Variable 'dense_1/kernel:0' shape=(5, 10) dtype=float32, numpy=
    ## array([[ 0.09410495,  0.446168  , -0.0122596 ,  0.614111  , -0.5365224 ,
    ##         -0.08093637, -0.16290972, -0.55459946, -0.06654024,  0.1030122 ],
    ##        [-0.36125702,  0.48036283, -0.34813565,  0.15942985,  0.06840414,
    ##         -0.16349554,  0.14036733, -0.5154793 ,  0.5000066 , -0.46155727],
    ##        [ 0.26240402,  0.40478462,  0.37507576,  0.22511405,  0.52451235,
    ##          0.19205123,  0.57743555,  0.47449833, -0.15287456, -0.1075297 ],
    ##        [-0.36259723, -0.25658298,  0.10848331, -0.4361353 ,  0.54963666,
    ##          0.3634495 , -0.5338099 ,  0.5618717 ,  0.46066135, -0.03748995],
    ##        [-0.5290042 , -0.17431661,  0.31270218, -0.07373512, -0.06707269,
    ##         -0.29736194,  0.10370523,  0.31938863, -0.5069216 , -0.4542585 ]],
    ##       dtype=float32)>
    ## 
    ## [[2]]
    ## <tf.Variable 'dense_1/bias:0' shape=(10,) dtype=float32, numpy=array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>
    ## <tf.Variable 'dense_1/kernel:0' shape=(5, 10) dtype=float32, numpy=
    ## array([[ 0.09410495,  0.446168  , -0.0122596 ,  0.614111  , -0.5365224 ,
    ##         -0.08093637, -0.16290972, -0.55459946, -0.06654024,  0.1030122 ],
    ##        [-0.36125702,  0.48036283, -0.34813565,  0.15942985,  0.06840414,
    ##         -0.16349554,  0.14036733, -0.5154793 ,  0.5000066 , -0.46155727],
    ##        [ 0.26240402,  0.40478462,  0.37507576,  0.22511405,  0.52451235,
    ##          0.19205123,  0.57743555,  0.47449833, -0.15287456, -0.1075297 ],
    ##        [-0.36259723, -0.25658298,  0.10848331, -0.4361353 ,  0.54963666,
    ##          0.3634495 , -0.5338099 ,  0.5618717 ,  0.46066135, -0.03748995],
    ##        [-0.5290042 , -0.17431661,  0.31270218, -0.07373512, -0.06707269,
    ##         -0.29736194,  0.10370523,  0.31938863, -0.5069216 , -0.4542585 ]],
    ##       dtype=float32)>
    ## <tf.Variable 'dense_1/bias:0' shape=(10,) dtype=float32, numpy=array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)>

    Implementing custom layers

    The best way to implement your own layer is extending the KerasLayer class and implementing:

    • initialize , where you can do all input-independent initialization
    • build, where you know the shapes of the input tensors and can do the rest of the initialization
    • call, where you do the forward computation

    Note that you don’t have to wait until build is called to create your variables, you can also create them in initialize. However, the advantage of creating them in build is that it enables late variable creation based on the shape of the inputs the layer will operate on. On the other hand, creating variables in initialize would mean that shapes required to create the variables will need to be explicitly specified.

    Layer Wrapper Function

    In order to use the custom layer within a Keras model you also need to create a wrapper function which instantiates the layer using the create_layer() function. For example:

    Some important things to note about the layer wrapper function:

    • It accepts object as its first parameter (the object will either be a Keras sequential model or another Keras layer). The object parameter enables the layer to be composed with other layers using the magrittr pipe (%>%) operator.

    • It converts it’s output_dim to integer using the as.integer() function. This is done as convenience to the user because Keras variables are strongly typed (you can’t pass a float if an integer is expected). This enables users of the function to write output_dim = 32 rather than output_dim = 32L.

    • Some additional parameters not used by the layer (name and trainable) are in the function signature. Custom layer functions can include any of the core layer function arguments (input_shape, batch_input_shape, batch_size, dtype, name, trainable, and weights) and they will be automatically forwarded to the Layer base class.

    We can use the defined layer, for example:

    ## tf.Tensor(
    ## [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
    ##  [0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]], shape=(10, 10), dtype=float32)

    Overall code is easier to read and maintain if it uses standard layers whenever possible, as other readers will be familiar with the behavior of standard layers. If you want to use a layer which is not present in tf.keras.layers, consider filing a github issue or, even better, sending us a pull request!

    Models: Composing layers

    Many interesting layer-like things in machine learning models are implemented by composing existing layers. For example, each residual block in a resnet is a composition of convolutions, batch normalizations, and a shortcut. Layers can be nested inside other layers.

    Typically you use keras_model_custom when you need the model methods like: fit,evaluate, and save (see Custom Keras layers and models for details).

    One other feature provided by MOdel (instead of Layer) is that in addition to tracking variables, a Model also tracks its internal layers, making them easier to inspect.

    For examplle here is a ResNet block:

    ## tf.Tensor(
    ## [[[[0. 0. 0.]
    ##    [0. 0. 0.]
    ##    [0. 0. 0.]]
    ## 
    ##   [[0. 0. 0.]
    ##    [0. 0. 0.]
    ##    [0. 0. 0.]]]], shape=(1, 2, 3, 3), dtype=float32)
    ## [[1]]
    ## <tensorflow.python.keras.layers.convolutional.Conv2D>
    ## 
    ## [[2]]
    ## <tensorflow.python.keras.layers.normalization.BatchNormalization>
    ## 
    ## [[3]]
    ## <tensorflow.python.keras.layers.convolutional.Conv2D>
    ## 
    ## [[4]]
    ## <tensorflow.python.keras.layers.normalization.BatchNormalization>
    ## 
    ## [[5]]
    ## <tensorflow.python.keras.layers.convolutional.Conv2D>
    ## 
    ## [[6]]
    ## <tensorflow.python.keras.layers.normalization.BatchNormalization>
    length(block$variables)
    ## [1] 18

    Much of the time, however, models which compose many layers simply call one layer after the other. This can be done in very little code using keras_model_sequential:

    ## tf.Tensor(
    ## [[[[0. 0. 0.]
    ##    [0. 0. 0.]
    ##    [0. 0. 0.]]
    ## 
    ##   [[0. 0. 0.]
    ##    [0. 0. 0.]
    ##    [0. 0. 0.]]]], shape=(1, 2, 3, 3), dtype=float32)
    summary(model)
    ## Model: "sequential"
    ## ___________________________________________________________________________
    ## Layer (type)                     Output Shape                  Param #     
    ## ===========================================================================
    ## conv2d_3 (Conv2D)                multiple                      4           
    ## ___________________________________________________________________________
    ## batch_normalization_3 (BatchNorm multiple                      4           
    ## ___________________________________________________________________________
    ## conv2d_4 (Conv2D)                multiple                      4           
    ## ___________________________________________________________________________
    ## batch_normalization_4 (BatchNorm multiple                      8           
    ## ___________________________________________________________________________
    ## conv2d_5 (Conv2D)                multiple                      9           
    ## ___________________________________________________________________________
    ## batch_normalization_5 (BatchNorm multiple                      12          
    ## ===========================================================================
    ## Total params: 41
    ## Trainable params: 29
    ## Non-trainable params: 12
    ## ___________________________________________________________________________