Struct PositionalEncoding 

pub struct PositionalEncoding {
    pub sinusoids: Tensor<3>,
    pub max_sequence_size: usize,
    pub max_timescale: usize,
}

Positional encoding layer for transformer models.

This layer adds positional information to the input embeddings, allowing the transformer model to take into account the order of the sequence. The positional encoding is added to the input embeddings by computing a set of sinusoidal functions with different frequencies and phases.

Sinusoids are used for positional embedding introduced in Attention is all you need.

The reference implementation can be found here: LANGUAGE MODELING WITH NN.TRANSFORMER AND TORCHTEXT

Should be created using PositionalEncodingConfig

Fields

sinusoids: Tensor<3>

The sinusoids used to add positional information to the input embeddings.

max_sequence_size: usize

The maximum sequence size to use.

max_timescale: usize

Max time scale to use.

Implementations

impl PositionalEncoding

pub fn forward(&self, input: Tensor<3>) -> Tensor<3>

Applies the forward pass on the input tensor by adding the sinusoids to the input.

Shapes

• input: [batch_size, seq_length, d_model]
• output: [batch_size, seq_length, d_model]

Panics

• Panics if the input sequence length is greater than the maximum sequence size.
• Panics if the input d_model is not equal to the d_model of the sinusoids.

Trait Implementations

impl Clone for PositionalEncoding

fn clone(&self) -> PositionalEncoding

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for PositionalEncoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Display for PositionalEncoding

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Module for PositionalEncoding

fn num_params(&self) -> usize

Get the number of parameters the module has, including all of its sub-modules, without initializing the ones not initialized yet.

fn visit<Visitor>(&self, visitor: &mut Visitor)

where Visitor: ModuleVisitor,

Visit each parameter and module-owned control value with a visitor.

fn map<Mapper>(self, mapper: &mut Mapper) -> PositionalEncoding

where Mapper: ModuleMapper,

Map each parameter and module-owned control value with a mapper.

fn collect_devices(&self, devices: Vec<Device>) -> Vec<Device>

Return all the devices found in the underneath module tree added to the given vector without duplicates.

fn to_device(self, device: &Device) -> PositionalEncoding

Move the module and all of its sub-modules to the given device.

fn fork(self, device: &Device) -> PositionalEncoding

Fork the module and all of its sub-modules to the given device.

fn valid(&self) -> PositionalEncoding

Returns a validation snapshot without autodiff or active training flags.

fn train(self) -> PositionalEncoding

Enables autodiff and restores configured training state throughout the module.

fn materialize(self) -> PositionalEncoding

Fold parameter reparameterizations into their effective values and remove them.

fn devices(&self) -> Vec<Device>

Return all the devices found in the underneath module tree without duplicates.

fn set_require_grad(self, require_grad: bool) -> Self

Set whether every floating-point tensor parameter in the module tree requires gradients.

fn set_require_grad_group(self, group: ParamGroup, require_grad: bool) -> Self

Set whether matched floating-point tensor parameters require gradients.

fn no_grad(self) -> Self

Disable gradients for every floating-point tensor parameter in the module tree.

fn freeze(self) -> Self

Freeze the whole module tree.

fn unfreeze(self) -> Self

Unfreeze the whole module tree.

fn freeze_group(self, group: ParamGroup) -> Self

Freeze every module-owned value in the given group, leaving the rest of the module untouched.

fn unfreeze_group(self, group: ParamGroup) -> Self

Unfreeze every module-owned value in the given group, leaving the rest of the module untouched.

fn quantize_weights(self, quantizer: &mut Quantizer) -> Self

Quantize the weights of the module.

fn quantize_weights_group( self, quantizer: &mut Quantizer, group: ParamGroup, ) -> Self

Quantize the weights of the given parameter group.

fn apply_reparameterization<R>(self, reparameterizer: R) -> Self

where Self: Sized, R: Reparameterizer,

Attach reparameterizations using the given Reparameterizer.

fn apply_lora(self, lora: Lora) -> Self

where Self: Sized,

Attach LoRA adapters to the module’s 2-D weights, freezing all base tensor parameters.

fn apply_qlora(self, qlora: QLora) -> Self

where Self: Sized,

Apply QLoRA to the module: quantize the (frozen) base tensor parameters and attach trainable LoRA adapters to 2-D weights.

fn into_record(self) -> ModuleRecord

where Self: Sized,

Collect this module’s tensor parameters into a ModuleRecord.

fn into_record_group(self, group: ParamGroup) -> ModuleRecord

where Self: Sized,

Collect the tensor parameters group names into a ModuleRecord.

fn try_load_record(self, record: ModuleRecord) -> Result<Self, RecordError>

where Self: Sized,

Apply a ModuleRecord to this module, returning the loaded module.

fn load_record(self, record: ModuleRecord) -> Self

where Self: Sized,

Apply a ModuleRecord to this module, consuming and returning it.

fn save_file<P>(self, path: P) -> Result<(), RecordError>

where P: AsRef<Path>, Self: Sized,

Save this module’s parameters to a burnpack file on disk.

fn load_file<P>(self, path: P) -> Self

where P: AsRef<Path>, Self: Sized,

Load this module’s parameters from a burnpack file on disk, returning the loaded module.

fn try_load_file<P>(self, path: P) -> Result<Self, RecordError>

where P: AsRef<Path>, Self: Sized,

Fallible variant of load_file.

impl ModuleDisplay for PositionalEncoding

fn custom_settings(&self) -> Option<DisplaySettings>

Custom display settings for the module.

fn custom_content(&self, content: Content) -> Option<Content>

Custom attributes for the module.

fn format(&self, passed_settings: DisplaySettings) -> String

Formats the module with provided display settings.

impl ModuleDisplayDefault for PositionalEncoding

fn content(&self, content: Content) -> Option<Content>

Attributes of the module used for display purposes.

fn num_params(&self) -> usize

Gets the number of the parameters of the module.