TopolearnChoosing and computing the right metric is what separates a convincing NLP result from a misleading one.
Accuracy alone hides a lot. NLTK includes the tools to measure systems properly.
Classification metrics in nltk.metrics.scores work on sets of item ids: precision(reference, test), recall, f_measure. Build one set of ids per label for gold and predicted, then compute per class. nltk.ConfusionMatrix(gold, predicted) shows which labels get mixed up and offers .evaluate() for a per-label report.
String distance: nltk.edit_distance(a, b) is the Levenshtein distance (with transpositions=True for Damerau), jaccard_distance(set_a, set_b) compares sets. Useful for spelling correction and fuzzy matching.
Annotation agreement: before trusting labels, measure whether humans agree. nltk.metrics.agreement.AnnotationTask(data) takes (coder, item, label) triples and computes kappa(), pi() and alpha(). Cohen's kappa corrects for chance agreement: 0.8+ is strong, below about 0.6 means the guidelines need work. The model can't be more reliable than its labels.
Machine translation and generation: nltk.translate.bleu_score.sentence_bleu(references, hypothesis) scores n-gram overlap (1 to 4) with a brevity penalty; corpus_bleu aggregates properly over a test set (not the average of sentence scores). Short sentences with no 4-gram matches score 0, so pass a SmoothingFunction().method1 when scoring single sentences. meteor_score and nist_score exist too.
Report metrics with their exact settings, and always on a held-out set.
Example
import nltk
from nltk.metrics import precision, recall, f_measure
from nltk.metrics.agreement import AnnotationTask
from nltk.translate.bleu_score import SmoothingFunction, sentence_bleu
gold = ["pos", "pos", "neg", "neg", "pos", "neg"]
pred = ["pos", "neg", "neg", "neg", "pos", "pos"]
ref = {i for i, g in enumerate(gold) if g == "pos"}
hyp = {i for i, p in enumerate(pred) if p == "pos"}
print(round(precision(ref, hyp), 2), round(recall(ref, hyp), 2), round(f_measure(ref, hyp), 2))
# 0.67 0.67 0.67
print(nltk.ConfusionMatrix(gold, pred)["pos", "neg"]) # 1 gold pos predicted neg
print(nltk.edit_distance("kitten", "sitting")) # 3
data = [("a1", i, g) for i, g in enumerate(gold)] + [("a2", i, p) for i, p in enumerate(pred)]
print(round(AnnotationTask(data=data).kappa(), 2)) # 0.33 weak agreement
reference = ["the cat is on the mat".split()]
smooth = SmoothingFunction().method1
print(round(sentence_bleu(reference, "the cat is on the mat".split()), 2)) # 1.0
print(round(sentence_bleu(reference, "the cat sat on mat".split(), smoothing_function=smooth), 2))
# 0.11 low but non-zero thanks to smoothingRemember
- precision, recall and f_measure take sets of item ids per class
- ConfusionMatrix shows which labels are confused
- Kappa/alpha check that human labels are reliable
- Use corpus_bleu for test sets; smooth sentence-level BLEU
Watch out for
- Averaging sentence_bleu scores over a test set instead of computing corpus_bleu.