Expand description
Neural network module.
Modules§
- activation
- Activation Layers
- loss
- Loss module
- modules
- Neural network modules implementations.
Structs§
- Batch
Norm - Applies Batch Normalization over a tensor.
- Batch
Norm Config BatchNormConfiguration.- BiGru
- The BiGru module. This implementation is for Bidirectional GRU.
- BiGru
Config - Configuration to create a BiGru module using the init function.
- BiLstm
- The BiLstm module. This implementation is for Bidirectional LSTM.
- BiLstm
Config - Configuration to create a BiLstm module using the init function.
- BiRnn
- The BiRnn module. This implementation is for Bidirectional RNN. Should be created with BiRnnConfig.
- BiRnn
Config - Configuration to create a BiRnn module using the init function.
- Celu
- CELU (Continuously Differentiable Exponential Linear Unit) layer.
- Celu
Config - Configuration to create a Celu layer using the init function.
- Cosine
Similarity - Computes the cosine similarity between two tensors along a dimension, following
torch.nn.CosineSimilarity. - Cosine
Similarity Config - Configuration to create a CosineSimilarity layer using the init function.
- Dropout
- Set at random some elements of the input tensor to zero during training.
- Dropout
Config - Configuration to create a Dropout layer using the init function.
- Elu
- ELU (Exponential Linear Unit) layer.
- EluConfig
- Configuration to create an Elu layer using the init function.
- Embedding
- Lookup table to store a fix number of vectors.
- Embedding
Config - Configuration to create an Embedding layer using the init function.
- Fold4d
- Four-dimensional folding, the inverse of Unfold4d.
- Fold4d
Config - Configuration to create a fold 4d layer using the init function.
- GLU
- Applies the gated linear unit function.
- Gate
Controller - A GateController represents a gate in an LSTM cell. An LSTM cell generally contains three gates: an input gate, forget gate, and output gate. Additionally, cell gate is just used to compute the cell state.
- Gaussian
Noise - Add pseudorandom Gaussian noise to an arbitrarily shaped tensor.
- Gaussian
Noise Config - Configuration to create a GaussianNoise layer using the init function.
- Gelu
- Applies the Gaussian Error Linear Units function element-wise.
- Group
Norm - Applies Group Normalization over a mini-batch of inputs as described in the paper Group Normalization.
- Group
Norm Config - Configuration to create a GroupNorm layer using the init function.
- Gru
- The Gru (Gated recurrent unit) module. This implementation is for a unidirectional, stateless, Gru.
- GruConfig
- Configuration to create a gru module using the init function.
- Hard
Shrink - Hard Shrink layer.
- Hard
Shrink Config - Configuration to create a HardShrink layer using the init function.
- Hard
Sigmoid - Hard Sigmoid layer.
- Hard
Sigmoid Config - Configuration to create a Hard Sigmoid layer using the init function.
- Identity
- Identity Module.
- Instance
Norm - Applies Instance Normalization over a tensor as described in the paper Instance Normalization
- Instance
Norm Config - Configuration to create a InstanceNorm layer using the init function.
- Layer
Norm - Applies Layer Normalization over an input tensor as described in the paper Layer Normalization.
- Layer
Norm Config - Configuration to create a LayerNorm layer using the init function.
- Leaky
Relu - Leaky ReLu layer.
- Leaky
Relu Config - Configuration to create a Leaky Relu layer using the init function.
- Linear
- Applies a linear transformation to the input tensor.
- Linear
Config - Configuration to create a
Linearlayer using the init function. - Local
Response Norm - Applies Local Response Normalization as described in ImageNet Classification with Deep Convolutional Neural Networks.
- Local
Response Norm Config - Configuration to create a LocalResponseNorm layer using the init function.
- Lstm
- The Lstm module. This implementation is for a unidirectional, stateless, Lstm.
- Lstm
Config - Configuration to create a Lstm module using the init function.
- Lstm
State - State bundle for LSTM implementations.
- PRelu
- Parametric Relu layer.
- PRelu
Config - Configuration to create a Parametric Relu layer using the init function.
- Pairwise
Distance - Computes the pairwise distance between the rows of two tensors, following
torch.nn.PairwiseDistance. - Pairwise
Distance Config - Configuration to create a PairwiseDistance layer using the init function.
- Pixel
Shuffle - Rearranges a tensor of shape
[N, C * r^2, H, W]into[N, C, H * r, W * r], whereris the upscale factor. Inverse of PixelUnshuffle, matchingtorch.nn.PixelShuffle. - Pixel
Shuffle Config - Configuration to create a PixelShuffle layer using the init function.
- Pixel
Unshuffle - Rearranges a tensor of shape
[N, C, H * r, W * r]into[N, C * r^2, H, W], whereris the downscale factor. Inverse of PixelShuffle, matchingtorch.nn.PixelUnshuffle. - Pixel
Unshuffle Config - Configuration to create a PixelUnshuffle layer using the init function.
- Positional
Encoding - Positional encoding layer for transformer models.
- Positional
Encoding Config - Configuration to create a PositionalEncoding layer using the init function.
- Relu
- Applies the rectified linear unit function element-wise See also relu
- RmsNorm
- Applies RMS Normalization over an input tensor along the last dimension.
- RmsNorm
Config - Configuration to create a RMS Norm layer using the init function.
- Rnn
- The Rnn module. This implementation is for a unidirectional, stateless, Rnn. Should be created with RnnConfig.
- RnnConfig
- Configuration to create a Rnn module using the init function.
- RnnState
- A RnnState is used to store hidden state in RNN.
- Rotary
Encoding - A module that applies rotary positional encoding to a tensor. Rotary Position Encoding or Embedding (RoPE), is a type of position embedding which encodes absolute positional information with rotation matrix and naturally incorporates explicit relative position dependency in self-attention formulation.
- Rotary
Encoding Config - Configuration to create a RotaryEncoding layer using the init function.
- Selu
- Applies the Scaled Exponential Linear Unit function element-wise. See also selu
- Shrink
- Shrink layer.
- Shrink
Config - Configuration to create a Shrink layer using the init function.
- Sigmoid
- Applies the sigmoid function element-wise See also sigmoid
- Soft
Shrink - Soft Shrink layer.
- Soft
Shrink Config - Configuration to create a SoftShrink layer using the init function.
- Softplus
- Softplus layer.
- Softplus
Config - Configuration to create a Softplus layer using the init function.
- Softsign
- Applies the softsign function element-wise See also softsign
- SwiGlu
- Applies the SwiGLU or Swish Gated Linear Unit to the input tensor.
The SwiGLU activation function is defined as:
SwiGLU(x) = Swish(W_inner * x + b_inner) * (W_outer * x + b_outer) - SwiGlu
Config - Configuration to create a SwiGlu activation layer using the init function.
- Tanh
- Applies the tanh activation function element-wise See also tanh
- Thresholded
Relu - Thresholded ReLU layer.
- Thresholded
Relu Config - Configuration to create a ThresholdedRelu layer using the init function.
- Unfold4d
- Four-dimensional unfolding.
- Unfold4d
Config - Configuration to create an unfold 4d layer using the init function.
Enums§
- Initializer
- Enum specifying with what values a tensor should be initialized
- Linear
Layout - The layout in which the linear parameters are stored.
- Padding
Config1d - Padding configuration for 1D operators.
- Padding
Config2d - Padding configuration for 2D operators.
- Padding
Config3d - Padding configuration for 3D operators.
Traits§
- Optional
Initial Lstm State - Extension trait for attaching an initializer to an
Option<LstmState>.
Functions§
- generate_
sinusoids - Returns sinusoids for positional embedding introduced in Attention is all you need.