Struct TransformerDecoderConfig
pub struct TransformerDecoderConfig {
pub d_model: usize,
pub d_ff: usize,
pub n_heads: usize,
pub n_layers: usize,
pub dropout: f64,
pub norm_first: bool,
pub quiet_softmax: bool,
pub initializer: Initializer,
}
Expand description
Configuration to create a Transformer Decoder layer using the init function.
Fields§
§d_model: usize
The size of the model.
d_ff: usize
The size of the position-wise feed-forward network.
n_heads: usize
The number of attention heads.
n_layers: usize
The number of layers.
dropout: f64
The dropout rate. Default: 0.1
norm_first: bool
Layer norm will be applied first instead of after the other modules.
quiet_softmax: bool
Use “quiet softmax” instead of regular softmax.
- Usage may improve performance by allowing attention heads to deposit no information (if the sequence contains no information relevant to that head).
- Usage may reduce the entropy of weights in the model, enhancing quantization and compression.
Reference: https://www.evanmiller.org/attention-is-off-by-one.html
initializer: Initializer
The type of function used to initialize neural network parameters
Implementations§
§impl TransformerDecoderConfig
impl TransformerDecoderConfig
pub fn with_dropout(self, dropout: f64) -> TransformerDecoderConfig
pub fn with_dropout(self, dropout: f64) -> TransformerDecoderConfig
The dropout rate. Default: 0.1
pub fn with_norm_first(self, norm_first: bool) -> TransformerDecoderConfig
pub fn with_norm_first(self, norm_first: bool) -> TransformerDecoderConfig
Layer norm will be applied first instead of after the other modules.
pub fn with_quiet_softmax(self, quiet_softmax: bool) -> TransformerDecoderConfig
pub fn with_quiet_softmax(self, quiet_softmax: bool) -> TransformerDecoderConfig
Use “quiet softmax” instead of regular softmax.
pub fn with_initializer(
self,
initializer: Initializer,
) -> TransformerDecoderConfig
pub fn with_initializer( self, initializer: Initializer, ) -> TransformerDecoderConfig
The type of function used to initialize neural network parameters
§impl TransformerDecoderConfig
impl TransformerDecoderConfig
pub fn init<B>(&self, device: &<B as Backend>::Device) -> TransformerDecoder<B>where
B: Backend,
pub fn init<B>(&self, device: &<B as Backend>::Device) -> TransformerDecoder<B>where
B: Backend,
Initialize a new Transformer Decoder module.
Trait Implementations§
§impl Clone for TransformerDecoderConfig
impl Clone for TransformerDecoderConfig
§fn clone(&self) -> TransformerDecoderConfig
fn clone(&self) -> TransformerDecoderConfig
1.0.0 · Source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read more§impl Config for TransformerDecoderConfig
impl Config for TransformerDecoderConfig
§impl<'de> Deserialize<'de> for TransformerDecoderConfig
impl<'de> Deserialize<'de> for TransformerDecoderConfig
§fn deserialize<D>(
deserializer: D,
) -> Result<TransformerDecoderConfig, <D as Deserializer<'de>>::Error>where
D: Deserializer<'de>,
fn deserialize<D>(
deserializer: D,
) -> Result<TransformerDecoderConfig, <D as Deserializer<'de>>::Error>where
D: Deserializer<'de>,
§impl Display for TransformerDecoderConfig
impl Display for TransformerDecoderConfig
§impl Serialize for TransformerDecoderConfig
impl Serialize for TransformerDecoderConfig
§fn serialize<S>(
&self,
serializer: S,
) -> Result<<S as Serializer>::Ok, <S as Serializer>::Error>where
S: Serializer,
fn serialize<S>(
&self,
serializer: S,
) -> Result<<S as Serializer>::Ok, <S as Serializer>::Error>where
S: Serializer,
Auto Trait Implementations§
impl Freeze for TransformerDecoderConfig
impl RefUnwindSafe for TransformerDecoderConfig
impl Send for TransformerDecoderConfig
impl Sync for TransformerDecoderConfig
impl Unpin for TransformerDecoderConfig
impl UnwindSafe for TransformerDecoderConfig
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
Source§impl<T> CloneToUninit for Twhere
T: Clone,
impl<T> CloneToUninit for Twhere
T: Clone,
§impl<T> Instrument for T
impl<T> Instrument for T
§fn instrument(self, span: Span) -> Instrumented<Self>
fn instrument(self, span: Span) -> Instrumented<Self>
§fn in_current_span(self) -> Instrumented<Self>
fn in_current_span(self) -> Instrumented<Self>
Source§impl<T> IntoEither for T
impl<T> IntoEither for T
Source§fn into_either(self, into_left: bool) -> Either<Self, Self>
fn into_either(self, into_left: bool) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left
is true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read moreSource§fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
fn into_either_with<F>(self, into_left: F) -> Either<Self, Self>
self
into a Left
variant of Either<Self, Self>
if into_left(&self)
returns true
.
Converts self
into a Right
variant of Either<Self, Self>
otherwise. Read more§impl<T> Pointable for T
impl<T> Pointable for T
§impl<T> PolicyExt for Twhere
T: ?Sized,
impl<T> PolicyExt for Twhere
T: ?Sized,
§impl<T> ToCompactString for Twhere
T: Display,
impl<T> ToCompactString for Twhere
T: Display,
§fn try_to_compact_string(&self) -> Result<CompactString, ToCompactStringError>
fn try_to_compact_string(&self) -> Result<CompactString, ToCompactStringError>
ToCompactString::to_compact_string()
] Read more§fn to_compact_string(&self) -> CompactString
fn to_compact_string(&self) -> CompactString
CompactString
]. Read more§impl<T> ToStringFallible for Twhere
T: Display,
impl<T> ToStringFallible for Twhere
T: Display,
§fn try_to_string(&self) -> Result<String, TryReserveError>
fn try_to_string(&self) -> Result<String, TryReserveError>
ToString::to_string
, but without panic on OOM.