Struct AdafactorConfig 

pub struct AdafactorConfig { /* private fields */ }

Adafactor configuration.

By default, the supplied learning rate caps the relative step size at min(lr, 1 / sqrt(step)), which is then multiplied by the parameter’s root mean square (RMS). A learning rate of 0.01 gives the schedule proposed in the paper. Disable both relative_step and scale_parameter to use an absolute learning rate controlled entirely by an external scheduler.

Implementations

impl AdafactorConfig

pub fn new() -> AdafactorConfig

Create a new instance of the config.

Arguments

Optional Arguments

grad_clipping

Optional gradient clipping applied before the Adafactor update.

• Defaults to None

Default Arguments

epsilon_1

Positive constant added to squared gradients for numerical stability.

• Defaults to 1e-30

epsilon_2

Positive lower bound on the parameter RMS when scaling the step size.

• Defaults to 1e-3

clip_threshold

Positive upper bound on the RMS of the normalized update before learning rate scaling. This is separate from gradient clipping.

• Defaults to 1.0

decay_rate

Negative exponent in the second moment decay schedule beta_2 = 1 - step^decay_rate.

• Defaults to -0.8

relative_step

Whether to cap the supplied learning rate by 1 / sqrt(step).

• Defaults to true

scale_parameter

Whether to multiply the step size by max(epsilon_2, RMS(parameter)).

• Defaults to true

weight_decay

Decoupled weight decay coefficient, multiplied by the supplied learning rate.

• Defaults to 0.0

impl AdafactorConfig

pub fn with_epsilon_1(self, epsilon_1: f32) -> AdafactorConfig

Sets the value for the field epsilon_1.

Positive constant added to squared gradients for numerical stability.

• Defaults to 1e-30

pub fn with_epsilon_2(self, epsilon_2: f32) -> AdafactorConfig

Sets the value for the field epsilon_2.

Positive lower bound on the parameter RMS when scaling the step size.

• Defaults to 1e-3

pub fn with_clip_threshold(self, clip_threshold: f32) -> AdafactorConfig

Sets the value for the field clip_threshold.

Positive upper bound on the RMS of the normalized update before learning rate scaling. This is separate from gradient clipping.

• Defaults to 1.0

pub fn with_decay_rate(self, decay_rate: f64) -> AdafactorConfig

Sets the value for the field decay_rate.

Negative exponent in the second moment decay schedule beta_2 = 1 - step^decay_rate.

• Defaults to -0.8

pub fn with_relative_step(self, relative_step: bool) -> AdafactorConfig

Sets the value for the field relative_step.

Whether to cap the supplied learning rate by 1 / sqrt(step).

• Defaults to true

pub fn with_scale_parameter(self, scale_parameter: bool) -> AdafactorConfig

Sets the value for the field scale_parameter.

Whether to multiply the step size by max(epsilon_2, RMS(parameter)).

• Defaults to true

pub fn with_weight_decay(self, weight_decay: f32) -> AdafactorConfig

Sets the value for the field weight_decay.

Decoupled weight decay coefficient, multiplied by the supplied learning rate.

• Defaults to 0.0

pub fn with_grad_clipping( self, grad_clipping: Option<GradientClippingConfig>, ) -> AdafactorConfig

Sets the value for the field grad_clipping.

Optional gradient clipping applied before the Adafactor update.

• Defaults to None

impl AdafactorConfig

pub fn build(&self) -> Adafactor

Build the per-parameter Adafactor optimizer.

Use Self::init to construct a whole-module optimizer with the configured gradient clipping, or pass this optimizer to ModuleOptimizer::with_group.

Panics

Panics if either epsilon or the clipping threshold is not positive and finite, the decay exponent is not negative and finite, or weight decay is not nonnegative and finite.

pub fn init(&self) -> ModuleOptimizer

Initialize a whole-module Adafactor optimizer.

Trait Implementations

impl Clone for AdafactorConfig

fn clone(&self) -> AdafactorConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Config for AdafactorConfig

fn save<P>(&self, file: P) -> Result<(), Error>

where P: AsRef<Path>,

Saves the configuration to a file.

fn load<P>(file: P) -> Result<Self, ConfigError>

where P: AsRef<Path>,

Loads the configuration from a file.

fn load_binary(data: &[u8]) -> Result<Self, ConfigError>

Loads the configuration from a binary buffer.

impl Debug for AdafactorConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for AdafactorConfig

fn deserialize<D>( deserializer: D, ) -> Result<AdafactorConfig, <D as Deserializer<'de>>::Error>

where D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for AdafactorConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl Serialize for AdafactorConfig

fn serialize<S>( &self, serializer: S, ) -> Result<<S as Serializer>::Ok, <S as Serializer>::Error>

where S: Serializer,

Serialize this value into the given Serde serializer.