Function burn::nn::generate_sinusoids

pub fn generate_sinusoids<B>(
    length: usize,
    d_model: usize,
    max_timescale: usize,
    device: &<B as Backend>::Device,
) -> Tensor<B, 2>
where
    B: Backend,

Returns sinusoids for positional embedding introduced in Attention is all you need.

The reference implementation can be found here: LANGUAGE MODELING WITH NN.TRANSFORMER AND TORCHTEXT

Arguments

• length - The length of the sequence.
• d_model - The size of each vector.
• max_timescale - The maximum time scale to use.

Returns

A tensor of shape [length, d_model] containing the sinusoids.