Module norm

Expand description

§Normalization Layers

Users who wish to provide an abstraction over swappable normalization layers can use the Normalization wrapper, with support for:

NormalizationConfig can be used as a generic normalization policy:

Construct a config with arbitrary input features (we suggest 0).
Clone and match that config to the target input layer, using the NormalizationConfig::with_num_features() method.

BatchNorm: Applies Batch Normalization over a tensor.
BatchNormConfig: BatchNorm Configuration.
BatchNormRecord: The record type for the module.
BatchNormRecordItem: The record item type for the module.
GroupNorm: Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization.
GroupNormConfig: Configuration to create a GroupNorm layer using the init function.
GroupNormRecord: The record type for the module.
GroupNormRecordItem: The record item type for the module.
InstanceNorm: Applies Instance Normalization over a tensor as described in the paper Instance Normalization
InstanceNormConfig: Configuration to create a InstanceNorm layer using the init function.
InstanceNormRecord: The record type for the module.
InstanceNormRecordItem: The record item type for the module.
LayerNorm: Applies Layer Normalization over an input tensor as described in the paper Layer Normalization.
LayerNormConfig: Configuration to create a LayerNorm layer using the init function.
LayerNormRecord: The record type for the module.
LayerNormRecordItem: The record item type for the module.
RmsNorm: Applies RMS Normalization over an input tensor along the last dimension.
RmsNormConfig: Configuration to create a RMS Norm layer using the init function.
RmsNormRecord: The record type for the module.
RmsNormRecordItem: The record item type for the module.