```
library(tensorflow)
library(keras)
```

# The Sequential model

## Setup

## When to use a Sequential model

A `Sequential`

model is appropriate for **a plain stack of layers** where each layer has **exactly one input tensor and one output tensor**.

Schematically, the following `Sequential`

model:

```
# Define Sequential model with 3 layers
<- keras_model_sequential() %>%
model layer_dense(2, activation = "relu", name = "layer1") %>%
layer_dense(3, activation = "relu", name = "layer2") %>%
layer_dense(4, name = "layer3")
# Call model on a test input
<- tf$ones(shape(3, 3))
x <- model(x) y
```

is equivalent to this function:

```
# Create 3 layers
<- layer_dense(units = 2, activation = "relu", name = "layer1")
layer1 <- layer_dense(units = 3, activation = "relu", name = "layer2")
layer2 <- layer_dense(units = 4, name = "layer3")
layer3
# Call layers on a test input
<- tf$ones(shape(3, 3))
x <- layer3(layer2(layer1(x))) y
```

A Sequential model is **not appropriate** when:

- Your model has multiple inputs or multiple outputs
- Any of your layers has multiple inputs or multiple outputs
- You need to do layer sharing
- You want non-linear topology (e.g. a residual connection, a multi-branch model)

## Creating a Sequential model

You can create a Sequential model by piping a model through a series layers.

```
<- keras_model_sequential() %>%
model layer_dense(2, activation = "relu") %>%
layer_dense(3, activation = "relu") %>%
layer_dense(4)
```

Its layers are accessible via the `layers`

attribute:

`$layers model`

```
[[1]]
<keras.layers.core.dense.Dense object at 0x7f4e0fb298a0>
[[2]]
<keras.layers.core.dense.Dense object at 0x7f4e0fd07880>
[[3]]
<keras.layers.core.dense.Dense object at 0x7f4e0fd078b0>
```

You can also create a Sequential model incrementally:

```
<- keras_model_sequential()
model %>% layer_dense(2, activation = "relu")
model %>% layer_dense(3, activation = "relu")
model %>% layer_dense(4) model
```

Note that there’s also a corresponding `pop()`

method to remove layers: a Sequential model behaves very much like a stack of layers.

```
%>% pop_layer()
model length(model$layers) # 2
```

`[1] 2`

Also note that the Sequential constructor accepts a `name`

argument, just like any layer or model in Keras. This is useful to annotate TensorBoard graphs with semantically meaningful names.

```
<- keras_model_sequential(name = "my_sequential")
model %>% layer_dense(2, activation = "relu", name = "layer1")
model %>% layer_dense(3, activation = "relu", name = "layer2")
model %>% layer_dense(4, name = "layer3") model
```

## Specifying the input shape in advance

Generally, all layers in Keras need to know the shape of their inputs in order to be able to create their weights. So when you create a layer like this, initially, it has no weights:

```
<- layer_dense(units = 3)
layer $weights # Empty layer
```

`list()`

It creates its weights the first time it is called on an input, since the shape of the weights depends on the shape of the inputs:

```
# Call layer on a test input
<- tf$ones(shape(1, 4))
x <- layer(x)
y $weights # Now it has weights, of shape (4, 3) and (3,) layer
```

```
[[1]]
<tf.Variable 'dense_6/kernel:0' shape=(4, 3) dtype=float32, numpy=
array([[ 0.8255856 , -0.6298292 , -0.25837338],
[-0.77226764, -0.4046759 , -0.9097155 ],
[-0.7591605 , -0.06652707, -0.5182005 ],
[ 0.56201947, 0.55342543, -0.6044154 ]], dtype=float32)>
[[2]]
<tf.Variable 'dense_6/bias:0' shape=(3,) dtype=float32, numpy=array([0., 0., 0.], dtype=float32)>
```

Naturally, this also applies to Sequential models. When you instantiate a Sequential model without an input shape, it isn’t “built”: it has no weights (and calling `model$weights`

results in an error stating just this). The weights are created when the model first sees some input data:

```
<- keras_model_sequential() %>%
model layer_dense(2, activation = "relu") %>%
layer_dense(3, activation = "relu") %>%
layer_dense(4)
# No weights at this stage!
# At this point, you can't do this:
try(model$weights)
```

```
Error in py_get_attr_impl(x, name, silent) :
ValueError: Weights for model sequential_3 have not yet been created. Weights are created when the Model is first called on inputs or `build()` is called with an `input_shape`.
See `reticulate::py_last_error()` for details
```

```
# The model summary is also not available:
summary(model)
```

`Model: <no summary available, model was not built>`

```
# Call the model on a test input
<- tf$ones(shape(1, 4))
x <- model(x)
y cat("Number of weights after calling the model:", length(model$weights), "\n") # 6
```

`Number of weights after calling the model: 6 `

Once a model is “built”, you can call its `summary()`

method to display its contents (the `summary()`

method is also called by the default `print()`

method:

`summary(model)`

```
Model: "sequential_3"
____________________________________________________________________________
Layer (type) Output Shape Param #
============================================================================
dense_9 (Dense) (1, 2) 10
dense_8 (Dense) (1, 3) 9
dense_7 (Dense) (1, 4) 16
============================================================================
Total params: 35
Trainable params: 35
Non-trainable params: 0
____________________________________________________________________________
```

However, it can be very useful when building a Sequential model incrementally to be able to display the summary of the model so far, including the current output shape. In this case, you should start your model by passing an `input_shape`

argument to your model, so that it knows its input shape from the start:

```
<- keras_model_sequential(input_shape = c(4))
model %>% layer_dense(2, activation = "relu")
model
model
```

```
Model: "sequential_4"
____________________________________________________________________________
Layer (type) Output Shape Param #
============================================================================
dense_10 (Dense) (None, 2) 10
============================================================================
Total params: 10
Trainable params: 10
Non-trainable params: 0
____________________________________________________________________________
```

Models built with a predefined input shape like this always have weights (even before seeing any data) and always have a defined output shape.

In general, it’s a recommended best practice to always specify the input shape of a Sequential model in advance if you know what it is.

## A common debugging workflow: `%>%`

+ `summary()`

When building a new Sequential architecture, it’s useful to incrementally stack layers and print model summaries. For instance, this enables you to monitor how a stack of `Conv2D`

and `MaxPooling2D`

layers is downsampling image feature maps:

```
<- keras_model_sequential(input_shape = c(250, 250, 3)) # 250x250 RGB images
model
%>%
model layer_conv_2d(32, 5, strides = 2, activation = "relu") %>%
layer_conv_2d(32, 3, activation = "relu") %>%
layer_max_pooling_2d(3)
# Can you guess what the current output shape is at this point? Probably not.
# Let's just print it:
model
```

```
Model: "sequential_5"
____________________________________________________________________________
Layer (type) Output Shape Param #
============================================================================
conv2d_1 (Conv2D) (None, 123, 123, 32) 2432
conv2d (Conv2D) (None, 121, 121, 32) 9248
max_pooling2d (MaxPooling2D) (None, 40, 40, 32) 0
============================================================================
Total params: 11,680
Trainable params: 11,680
Non-trainable params: 0
____________________________________________________________________________
```

```
# The answer was: (40, 40, 32), so we can keep downsampling...
%>%
model layer_conv_2d(32, 3, activation = "relu") %>%
layer_conv_2d(32, 3, activation = "relu") %>%
layer_max_pooling_2d(3) %>%
layer_conv_2d(32, 3, activation = "relu") %>%
layer_conv_2d(32, 3, activation = "relu") %>%
layer_max_pooling_2d(2)
# And now?
model
```

```
Model: "sequential_5"
____________________________________________________________________________
Layer (type) Output Shape Param #
============================================================================
conv2d_1 (Conv2D) (None, 123, 123, 32) 2432
conv2d (Conv2D) (None, 121, 121, 32) 9248
max_pooling2d (MaxPooling2D) (None, 40, 40, 32) 0
conv2d_5 (Conv2D) (None, 38, 38, 32) 9248
conv2d_4 (Conv2D) (None, 36, 36, 32) 9248
max_pooling2d_2 (MaxPooling2D) (None, 12, 12, 32) 0
conv2d_3 (Conv2D) (None, 10, 10, 32) 9248
conv2d_2 (Conv2D) (None, 8, 8, 32) 9248
max_pooling2d_1 (MaxPooling2D) (None, 4, 4, 32) 0
============================================================================
Total params: 48,672
Trainable params: 48,672
Non-trainable params: 0
____________________________________________________________________________
```

```
# Now that we have 4x4 feature maps, time to apply global max pooling.
%>% layer_global_max_pooling_2d()
model
# Finally, we add a classification layer.
%>% layer_dense(10) model
```

Very practical, right?

## What to do once you have a model

Once your model architecture is ready, you will want to:

- Train your model, evaluate it, and run inference. See our guide to training & evaluation with the built-in loops
- Save your model to disk and restore it. See our guide to serialization & saving.
- Speed up model training by leveraging multiple GPUs. See our guide to multi-GPU and distributed training.

## Feature extraction with a Sequential model

Once a Sequential model has been built, it behaves like a Functional API model. This means that every layer has an `input`

and `output`

attribute. These attributes can be used to do neat things, like quickly creating a model that extracts the outputs of all intermediate layers in a Sequential model:

```
<-
initial_model keras_model_sequential(input_shape = c(250, 250, 3)) %>%
layer_conv_2d(32, 5, strides = 2, activation = "relu") %>%
layer_conv_2d(32, 3, activation = "relu") %>%
layer_conv_2d(32, 3, activation = "relu")
<- keras_model(
feature_extractor inputs = initial_model$inputs,
outputs = lapply(initial_model$layers, \(layer) layer$output)
)
# Call feature extractor on test input.
<- tf$ones(shape(1, 250, 250, 3))
x <- feature_extractor(x) features
```

Here’s a similar example that only extract features from one layer:

```
<-
initial_model keras_model_sequential(input_shape = c(250, 250, 3)) %>%
layer_conv_2d(32, 5, strides = 2, activation = "relu") %>%
layer_conv_2d(32, 3, activation = "relu", name = "my_intermediate_layer") %>%
layer_conv_2d(32, 3, activation = "relu")
<- keras_model(
feature_extractor inputs = initial_model$inputs,
outputs = get_layer(initial_model, name = "my_intermediate_layer")$output
)
# Call feature extractor on test input.
<- tf$ones(shape(1, 250, 250, 3))
x <- feature_extractor(x) features
```

## Transfer learning with a Sequential model

Transfer learning consists of freezing the bottom layers in a model and only training the top layers. If you aren’t familiar with it, make sure to read our guide to transfer learning.

Here are two common transfer learning blueprint involving Sequential models.

First, let’s say that you have a Sequential model, and you want to freeze all layers except the last one. In this case, you would simply iterate over `model$layers`

and set `layer$trainable = FALSE`

on each layer, except the last one. Like this:

```
<- keras_model_sequential(input_shape = c(784)) %>%
model layer_dense(32, activation = 'relu') %>%
layer_dense(32, activation = 'relu') %>%
layer_dense(32, activation = 'relu') %>%
layer_dense(10)
# Presumably you would want to first load pre-trained weights.
$load_weights(...)
model
# Freeze all layers except the last one.
for (layer in head(model$layers, -1))
$trainable <- FALSE
layer
# can also just call: freeze_weights(model, to = -2)
# Recompile and train (this will only update the weights of the last layer).
%>% compile(...)
model %>% fit(...) model
```

Another common blueprint is to use a Sequential model to stack a pre-trained model and some freshly initialized classification layers. Like this:

# Load a convolutional base with pre-trained weights

```
<- application_xception(
base_model weights = 'imagenet',
include_top = FALSE,
pooling = 'avg')
# Freeze the base model
$trainable <- FALSE
base_model
# Use a Sequential model to add a trainable classifier on top
<- keras_model_sequential() %>%
model base_model() %>%
layer_dense(1000)
# Compile & train
%>% compile(...)
model %>% fit(...) model
```

If you do transfer learning, you will probably find yourself frequently using these two patterns.

That’s about all you need to know about Sequential models!

To find out more about building models in Keras, see: