Module activation
Expand description
§Activation Layers
Users who desire a selectable activation function should
consider Activation, which provides an abstraction over:
Relu- the default,- [‘PRelu’]
GeluLeakyReluSwiGluSeluSigmoidHardSigmoidHardSwishSoftplusSoftsignTanhEluCeluThresholdedRelu
The activation layer GLU has shape-changing behaviors
not compatible with the common API, and is not included
in the abstraction wrappers.
Structs§
- Celu
- CELU (Continuously Differentiable Exponential Linear Unit) layer.
- Celu
Config - Configuration to create a Celu layer using the init function.
- Elu
- ELU (Exponential Linear Unit) layer.
- EluConfig
- Configuration to create an Elu layer using the init function.
- GLU
- Applies the gated linear unit function.
- Gelu
- Applies the Gaussian Error Linear Units function element-wise.
- Hard
Shrink - Hard Shrink layer.
- Hard
Shrink Config - Configuration to create a HardShrink layer using the init function.
- Hard
Sigmoid - Hard Sigmoid layer.
- Hard
Sigmoid Config - Configuration to create a Hard Sigmoid layer using the init function.
- Hard
Swish - Hard Swish layer.
- Hardtanh
- HardTanh layer, clamping each element to the range
[min_val, max_val]. - Hardtanh
Config - Configuration to create a Hardtanh layer using the init function.
- Leaky
Relu - Leaky ReLu layer.
- Leaky
Relu Config - Configuration to create a Leaky Relu layer using the init function.
- LogSigmoid
- Applies the LogSigmoid activation function element-wise.
- Mish
- Applies the Mish activation function element-wise.
- PRelu
- Parametric Relu layer.
- PRelu
Config - Configuration to create a Parametric Relu layer using the init function.
- RRelu
- Randomized Leaky ReLU (RReLU) layer, from Empirical Evaluation of Rectified Activations.
- RRelu
Config - Configuration to create a RRelu layer using the init function.
- Relu
- Applies the rectified linear unit function element-wise See also relu
- Relu6
- Applies the ReLU6 function element-wise, clamping the output to
[0, 6]. See also relu6 - Selu
- Applies the Scaled Exponential Linear Unit function element-wise. See also selu
- Shrink
- Shrink layer.
- Shrink
Config - Configuration to create a Shrink layer using the init function.
- SiLU
- Applies the SiLU (Sigmoid Linear Unit) activation function element-wise.
- Sigmoid
- Applies the sigmoid function element-wise See also sigmoid
- Soft
Shrink - Soft Shrink layer.
- Soft
Shrink Config - Configuration to create a SoftShrink layer using the init function.
- Softplus
- Softplus layer.
- Softplus
Config - Configuration to create a Softplus layer using the init function.
- Softsign
- Applies the softsign function element-wise See also softsign
- SwiGlu
- Applies the SwiGLU or Swish Gated Linear Unit to the input tensor.
The SwiGLU activation function is defined as:
SwiGLU(x) = Swish(W_inner * x + b_inner) * (W_outer * x + b_outer) - SwiGlu
Config - Configuration to create a SwiGlu activation layer using the init function.
- Tanh
- Applies the tanh activation function element-wise See also tanh
- Tanhshrink
- Applies the Tanhshrink function element-wise,
x - tanh(x). See also tanhshrink - Threshold
- Threshold layer: returns
xwherex > threshold, andvalueotherwise. - Threshold
Config - Configuration to create a Threshold layer using the init function.
- Thresholded
Relu - Thresholded ReLU layer.
- Thresholded
Relu Config - Configuration to create a ThresholdedRelu layer using the init function.
Enums§
- Activation
- Activation Layer Wrapper.
- Activation
Config ActivationConfiguration.