Struct PerplexityMetric

pub struct PerplexityMetric<B>
where
    B: Backend,
{ /* private fields */ }

The perplexity metric.

Perplexity is a measure of how well a probability distribution or probability model predicts a sample. It’s commonly used to evaluate language models. A lower perplexity indicates that the model is more confident in its predictions.

Mathematically, perplexity is defined as the exponentiation of the cross-entropy loss: PPL = exp(H(p, q)) = exp(-1/N * Σ log(p(x_i)))

where:

• H(p, q) is the cross-entropy between the true distribution p and predicted distribution q
• N is the number of tokens
• p(x_i) is the predicted probability of the i-th token

Aggregation

Unlike other metrics, perplexity cannot be simply averaged across batches. This implementation correctly accumulates the total negative log-likelihood and total token count across batches, then computes perplexity as exp(total_nll / total_tokens).

Implementations

impl<B> PerplexityMetric<B>

where B: Backend,

pub fn new() -> PerplexityMetric<B>

Creates the metric.

pub fn with_pad_token(self, index: usize) -> PerplexityMetric<B>

Sets the pad token to exclude from perplexity calculation.

When a pad token is set, predictions for padding tokens are masked out and do not contribute to the perplexity calculation. This is important for variable-length sequences where padding is used.

Trait Implementations

impl<B> Clone for PerplexityMetric<B>

where B: Clone + Backend,

fn clone(&self) -> PerplexityMetric<B>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<B> Default for PerplexityMetric<B>

where B: Backend,

fn default() -> PerplexityMetric<B>

Returns the “default value” for a type.

impl<B> Metric for PerplexityMetric<B>

where B: Backend,

type Input = PerplexityInput<B>

The input type of the metric.

fn update( &mut self, input: &PerplexityInput<B>, _metadata: &MetricMetadata, ) -> MetricEntry

Update the metric state and returns the current metric entry.

fn clear(&mut self)

Clear the metric state.

fn name(&self) -> Arc<String>

The parameterized name of the metric.

impl<B> Numeric for PerplexityMetric<B>

where B: Backend,

fn value(&self) -> NumericEntry

Returns the numeric value of the metric.