Evaluation metrics

Loss alone hides whether a model is useful; task metrics drive model selection and early stopping.

By default Trainer.evaluate() reports only eval_loss. Pass a compute_metrics function to get task metrics. It receives an EvalPrediction with:

predictions: the model's logits as NumPy arrays (for classification, shape (n, num_labels)),
label_ids: the true labels.

It returns a dict of floats; the Trainer prefixes keys with eval_, so {"f1": 0.9} is logged as eval_f1. That is the name to use in metric_for_best_model="f1" (with greater_is_better=True).

You can compute metrics with scikit-learn or NumPy directly, or with Hugging Face's evaluate library (evaluate.load("accuracy")).

Tips:

For generation or token-level tasks, logits can be huge. preprocess_logits_for_metrics lets you reduce them (e.g. argmax) on the device before they are accumulated, saving a lot of memory.
For token classification, drop positions whose label is -100 before scoring.
Choose metrics that match the goal: macro F1 for imbalanced classes, not plain accuracy.

The same function works with trainer.predict(test_ds), which returns predictions, labels and metrics (prefixed test_) for a final held-out evaluation.

Example

import numpy as np
from sklearn.metrics import accuracy_score, f1_score
from transformers import EvalPrediction

def compute_metrics(eval_pred: EvalPrediction):
    logits, labels = eval_pred.predictions, eval_pred.label_ids
    preds = np.argmax(logits, axis=-1)
    return {"accuracy": accuracy_score(labels, preds),
            "f1": f1_score(labels, preds, average="macro")}

# what Trainer would pass in after evaluation:
fake = EvalPrediction(predictions=np.array([[2.0, -1.0], [0.1, 0.3], [1.5, 0.2]]),
                      label_ids=np.array([0, 1, 1]))
print(compute_metrics(fake))   # {'accuracy': 0.666..., 'f1': 0.666...}

# Trainer(..., compute_metrics=compute_metrics) with
# TrainingArguments(metric_for_best_model="f1", greater_is_better=True, ...)

Remember

compute_metrics gets logits and labels and returns a dict of floats
Logged keys get an eval_ prefix; refer to them without it in metric_for_best_model
Use preprocess_logits_for_metrics to shrink large logits

Watch out for

Returning tensors or arrays instead of plain floats, which breaks logging.

Check yourself

compute_metrics returns {'f1': 0.91}. Under which key does Trainer.evaluate() report it?

Official docs: huggingface.co/docs/transformers/main_classes/trainer

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks