R/layer-attention.R

layer_attention

Creates attention layer

Description

Dot-product attention layer, a.k.a. Luong-style attention.

Usage

 
layer_attention( 
  inputs, 
  use_scale = FALSE, 
  causal = FALSE, 
  batch_size = NULL, 
  dtype = NULL, 
  name = NULL, 
  trainable = NULL, 
  weights = NULL 
) 

Arguments

Arguments Description
inputs a list of inputs first should be the query tensor, the second the value tensor
use_scale If True, will create a scalar variable to scale the attention scores.
causal Boolean. Set to True for decoder self-attention. Adds a mask such that position i cannot attend to positions j > i. This prevents the flow of information from the future towards the past.
batch_size Fixed batch size for layer
dtype The data type expected by the input, as a string (float32, float64, int32…)
name An optional name string for the layer. Should be unique in a model (do not reuse the same name twice). It will be autogenerated if it isn’t provided.
trainable Whether the layer weights will be updated during training.
weights Initial weights for layer.

See Also

Other core layers: layer_activation(), layer_activity_regularization(), layer_dense_features(), layer_dense(), layer_dropout(), layer_flatten(), layer_input(), layer_lambda(), layer_masking(), layer_permute(), layer_repeat_vector(), layer_reshape()