Struct PositionWiseFeedForward 

pub struct PositionWiseFeedForward {
    pub linear_inner: Linear,
    pub linear_outer: Linear,
    pub dropout: Dropout,
    pub activation: Activation,
}

Applies the position-wise feed-forward network to the input tensor from the paper Attention Is All You Need.

Params

• linear inner: Linear layer with d_model input features and d_ff output features.
• linear outer: Linear layer with d_ff input features and d_model output features.

FFN(x) = max(0, xW1 + b1)W2 + b2

Should be created using PositionWiseFeedForwardConfig

Notes

The activation field is currently marked #[module(skip)] for backward compatibility with records saved before this field was introduced (when the activation was always Gelu and had no state). This means activation state is not persisted when saving or loading records.

For stateless activations (GELU, ReLU, etc.) this has no effect. If you are using SwiGLU, its learnable parameters will not be saved or loaded correctly.

Fields

linear_inner: Linear

Linear layer with d_model input features and d_ff output features.

linear_outer: Linear

Linear layer with d_ff input features and d_model output features.

dropout: Dropout

Dropout layer.

activation: Activation

Activation function.

Implementations

impl PositionWiseFeedForward

pub fn forward<const D: usize>(&self, input: Tensor<D>) -> Tensor<D>

Applies the forward pass on the input tensor.

Shapes

• input: [batch_size, seq_length, d_model]
• output: [batch_size, seq_length, d_model]

Panics

Panics if the last axis of input is not d_model.

Trait Implementations

impl Clone for PositionWiseFeedForward

fn clone(&self) -> PositionWiseFeedForward

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PositionWiseFeedForward

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for PositionWiseFeedForward

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Module for PositionWiseFeedForward

fn num_params(&self) -> usize

Get the number of parameters the module has, including all of its sub-modules, without initializing the ones not initialized yet.

fn visit<Visitor>(&self, visitor: &mut Visitor)

where Visitor: ModuleVisitor,

Visit each parameter and module-owned control value with a visitor.

fn map<Mapper>(self, mapper: &mut Mapper) -> PositionWiseFeedForward

where Mapper: ModuleMapper,

Map each parameter and module-owned control value with a mapper.

fn collect_devices(&self, devices: Vec<Device>) -> Vec<Device>

Return all the devices found in the underneath module tree added to the given vector without duplicates.

fn to_device(self, device: &Device) -> PositionWiseFeedForward

Move the module and all of its sub-modules to the given device.

fn fork(self, device: &Device) -> PositionWiseFeedForward

Fork the module and all of its sub-modules to the given device.

fn valid(&self) -> PositionWiseFeedForward

Returns a validation snapshot without autodiff or active training flags.

fn train(self) -> PositionWiseFeedForward

Enables autodiff and restores configured training state throughout the module.

fn materialize(self) -> PositionWiseFeedForward

Fold parameter reparameterizations into their effective values and remove them.

fn devices(&self) -> Vec<Device>

Return all the devices found in the underneath module tree without duplicates.

fn set_require_grad(self, require_grad: bool) -> Self

Set whether every floating-point tensor parameter in the module tree requires gradients.

fn set_require_grad_group(self, group: ParamGroup, require_grad: bool) -> Self

Set whether matched floating-point tensor parameters require gradients.

fn no_grad(self) -> Self

Disable gradients for every floating-point tensor parameter in the module tree.

fn freeze(self) -> Self

Freeze the whole module tree.

fn unfreeze(self) -> Self

Unfreeze the whole module tree.

fn freeze_group(self, group: ParamGroup) -> Self

Freeze every module-owned value in the given group, leaving the rest of the module untouched.

fn unfreeze_group(self, group: ParamGroup) -> Self

Unfreeze every module-owned value in the given group, leaving the rest of the module untouched.

fn quantize_weights(self, quantizer: &mut Quantizer) -> Self

Quantize the weights of the module.

fn quantize_weights_group( self, quantizer: &mut Quantizer, group: ParamGroup, ) -> Self

Quantize the weights of the given parameter group.

fn apply_reparameterization<R>(self, reparameterizer: R) -> Self

where Self: Sized, R: Reparameterizer,

Attach reparameterizations using the given Reparameterizer.

fn apply_lora(self, lora: Lora) -> Self

where Self: Sized,

Attach LoRA adapters to the module’s 2-D weights, freezing all base tensor parameters.

fn apply_qlora(self, qlora: QLora) -> Self

where Self: Sized,

Apply QLoRA to the module: quantize the (frozen) base tensor parameters and attach trainable LoRA adapters to 2-D weights.

fn into_record(self) -> ModuleRecord

where Self: Sized,

Collect this module’s tensor parameters into a ModuleRecord.

fn into_record_group(self, group: ParamGroup) -> ModuleRecord

where Self: Sized,

Collect the tensor parameters group names into a ModuleRecord.

fn try_load_record(self, record: ModuleRecord) -> Result<Self, RecordError>

where Self: Sized,

Apply a ModuleRecord to this module, returning the loaded module.

fn load_record(self, record: ModuleRecord) -> Self

where Self: Sized,

Apply a ModuleRecord to this module, consuming and returning it.

fn save_file<P>(self, path: P) -> Result<(), RecordError>

where P: AsRef<Path>, Self: Sized,

Save this module’s parameters to a burnpack file on disk.

fn load_file<P>(self, path: P) -> Self

where P: AsRef<Path>, Self: Sized,

Load this module’s parameters from a burnpack file on disk, returning the loaded module.

fn try_load_file<P>(self, path: P) -> Result<Self, RecordError>

where P: AsRef<Path>, Self: Sized,

Fallible variant of load_file.

impl ModuleDisplay for PositionWiseFeedForward

fn custom_settings(&self) -> Option<DisplaySettings>

Custom display settings for the module.

fn custom_content(&self, content: Content) -> Option<Content>

Custom attributes for the module.

fn format(&self, passed_settings: DisplaySettings) -> String

Formats the module with provided display settings.

impl ModuleDisplayDefault for PositionWiseFeedForward

fn content(&self, content: Content) -> Option<Content>

Attributes of the module used for display purposes.

fn num_params(&self) -> usize

Gets the number of the parameters of the module.