Metric

When working with the learner, you have the option to record metrics that will be monitored throughout the training process. We currently offer a restricted range of metrics.

Using Metrics with the Learner

In order to use a metric, the output of your training step must implement the Adaptor trait from burn-train::metric for each metric's corresponding input type. The Adaptor trait simply converts your output struct into the input type the metric expects.

Burn provides four built-in output structs that cover common tasks. Each one already implements Adaptor for a set of metrics, so in many cases you can use them directly without writing any adaptor code yourself.

• ClassificationOutput:
  • Use case: Single-label classification
  • Fields: loss: Tensor<1>, output: Tensor<2>, targets: Tensor<1, Int>
  • Adapted metrics: Accuracy, TopKAccuracy, Perplexity, Precision*, Recall*, FBetaScore*, AUROC*, AUC-PR*, Loss
• MultiLabelClassificationOutput:
  • Use case: Multi-label classification
  • Fields: loss: Tensor<1>, output: Tensor<2>, targets: Tensor<2, Int>
  • Adapted metrics: HammingScore, Precision*, Recall*, FBetaScore*, AUROC*, AUC-PR*, Loss
• RegressionOutput:
  • Use case: Regression tasks
  • Fields: loss: Tensor<1>, output: Tensor<2>, targets: Tensor<2>
  • Adapted metrics: Loss
• SequenceOutput:
  • Use case: Sequence prediction
  • Fields: loss: Tensor<1>, logits: Tensor<3>, predictions: Option<Tensor<2, Int>>, targets: Tensor<2, Int>
  • Adapted metrics: Accuracy, TopKAccuracy, Perplexity, CER, WER, Loss

* Precision, Recall, FBetaScore, AUROC, and AUC-PR all use ConfusionStatsInput as their input type so these metrics are automatically (implicitly) adapted since ConfusionStatsInput is adapted.

If your metric isn't already adapted for the appropriate output struct, you can implement Adaptor yourself. For example, here is how ClassificationOutput adapts to AccuracyInput:

impl Adaptor<AccuracyInput> for ClassificationOutput {
    fn adapt(&self) -> AccuracyInput {
        AccuracyInput::new(self.output.clone(), self.targets.clone())
    }
}

If your task type is not covered by the built-in output structs, you can create an output struct for your data and then adapt your metric for the output struct:

#[derive(new)]
pub struct ClassificationOutput {
    /// The loss.
    pub loss: Tensor<1>,

    /// The output.
    pub output: Tensor<2>,

    /// The targets.
    pub targets: Tensor<1, Int>,
}

impl Adaptor<AccuracyInput> for ClassificationOutput {
    fn adapt(&self) -> AccuracyInput {
        AccuracyInput::new(self.output.clone(), self.targets.clone())
    }
}

You can also open an issue on the GitHub repository when your task type is not covered by the built-in output structs. However, since creating an output struct for your data is simple, it is recommended to try creating your own output struct first.

Custom Metric

Generating your own custom metrics is done by implementing the Metric trait.

/// Metric trait.
///
/// # Notes
///
/// Implementations should define their own input type only used by the metric.
/// This is important since some conflict may happen when the model output is adapted for each
/// metric's input type.
pub trait Metric: Send + Sync + Clone {
    /// The input type of the metric.
    type Input;

    /// The parameterized name of the metric.
    ///
    /// This should be unique, so avoid using short generic names, prefer using the long name.
    ///
    /// For a metric that can exist at different parameters (e.g., top-k accuracy for different
    /// values of k), the name should be unique for each instance.
    fn name(&self) -> MetricName;

    /// A short description of the metric.
    fn description(&self) -> Option<String> {
        None
    }

    /// Attributes of the metric.
    ///
    /// By default, metrics have no attributes.
    fn attributes(&self) -> MetricAttributes {
        MetricAttributes::None
    }

    /// Update the metric state and returns the current metric entry.
    ///
    /// # Errors
    ///
    /// Returns a [`TensorReadError`] when a tensor read fails. In that case, the metric state
    /// could hold part or none of this update.
    fn update(
        &mut self,
        item: &Self::Input,
        metadata: &MetricMetadata,
    ) -> Result<SerializedEntry, TensorReadError>;

    /// Compute the final metric value.
    ///
    /// # Errors
    ///
    /// Returns a [`TensorReadError`] when a tensor read fails.
    fn compute(&mut self) -> Result<SerializedEntry, TensorReadError>;

    /// Clear the metric state.
    fn clear(&mut self);
}

In update and compute, read tensors with the fallible methods (try_into_data, try_into_scalar) and propagate the error with ? to keep the event processor reporting errors to the learner. The processor still updates the other metrics, and reports every metric that failed, each with its name and split, as a EventProcessorError. When update fails, the metric state could hold part or none of that update. This includes tensors the metric keeps across batches, such as running totals: once one of them was computed from a failed input, reading it fails too, until clear resets the state at the end of the epoch.

As an example, let's see how the loss metric is implemented.

/// The loss metric.
#[derive(Clone)]
pub struct LossMetric {
    name: Arc<String>,
    state: NumericMetricState,
}

/// The [loss metric](LossMetric) input type.
#[derive(new)]
pub struct LossInput {
    tensor: Tensor<1>,
}

impl Default for LossMetric {
    fn default() -> Self {
        Self::new()
    }
}

impl LossMetric {
    /// Create the metric.
    pub fn new() -> Self {
        Self {
            name: Arc::new("Loss".to_string()),
            state: NumericMetricState::default(),
        }
    }
}


impl Metric for LossMetric {
    type Input = LossInput;

    fn update(
        &mut self,
        loss: &Self::Input,
        _metadata: &MetricMetadata,
    ) -> Result<SerializedEntry, TensorReadError> {
        let [batch_size] = loss.tensor.dims();
        let loss = loss
            .tensor
            .clone()
            .mean()
            .try_into_data()?
            .iter::<f64>()
            .next()
            .unwrap();

        self.state.update(loss, batch_size);
        Ok(self
            .state
            .compute_update(FormatOptions::new(self.name()).precision(2)))
    }

    fn compute(&mut self) -> Result<SerializedEntry, TensorReadError> {
        Ok(self
            .state
            .compute_final(FormatOptions::new(self.name()).precision(2)))
    }

    fn clear(&mut self) {
        self.state.reset()
    }

    fn name(&self) -> MetricName {
        self.name.clone()
    }

    fn attributes(&self) -> MetricAttributes {
        NumericAttributes {
            unit: None,
            higher_is_better: false,
        }
        .into()
    }
}

When the metric you are implementing is numeric in nature, you may want to also implement the Numeric trait. This will allow your metric to be plotted.

impl Numeric for LossMetric {
    fn value(&self) -> Option<NumericEntry> {
        Some(self.state.current_value())
    }

    fn running_value(&self) -> Option<NumericEntry> {
        Some(self.state.running_value())
    }

    fn final_value(&self) -> NumericEntry {
        self.state.final_value()
    }
}