TopolearnN-gram LMs are the conceptual ancestor of today's large language models and still teach smoothing, sparsity and evaluation.
A language model assigns probabilities to word sequences. An n-gram model approximates P(word | history) using only the previous n-1 words, estimated from counts.
nltk.lm structures this:
padded_everygram_pipeline(n, sentences)pads each sentence with<s>and</s>and produces all 1..n-grams for training plus a flat vocabulary stream.- Model classes:
MLE(raw relative frequencies),Laplace(add-one),Lidstone(gamma),KneserNeyInterpolatedandWittenBellInterpolated. lm.fit(train, vocab), thenlm.score(word, context),lm.logscore(base 2),lm.generate(n, text_seed=..., random_seed=...).
Sparsity is the core problem. MLE gives probability 0 to any unseen n-gram, so a single new bigram makes a test sentence impossible and perplexity infinite. Smoothing moves some probability mass to unseen events: Laplace does it crudely; Kneser-Ney, which also accounts for how many contexts a word appears in, is the classic best performer.
The Vocabulary maps rare or unseen words to <UNK> (controlled by unk_cutoff), which keeps out-of-vocabulary words from breaking scoring.
Perplexity, lm.perplexity(test_ngrams), is 2 to the power of the average negative log2 probability: lower is better and roughly means "how many choices the model is confused between". Compare perplexities only between models with the same vocabulary.
Example
from nltk.lm import MLE, KneserNeyInterpolated, Laplace
from nltk.lm.preprocessing import padded_everygram_pipeline
from nltk.util import bigrams, pad_sequence
corpus = [s.split() for s in ["the cat sat on the mat", "the dog sat on the log",
"the cat saw the dog", "a dog sat on a mat"]]
models = {}
for name, cls in [("mle", MLE), ("laplace", Laplace), ("kn", KneserNeyInterpolated)]:
train, vocab = padded_everygram_pipeline(2, corpus)
models[name] = cls(2)
models[name].fit(train, vocab)
mle = models["mle"]
print(mle.counts[["the"]]["cat"], round(mle.score("cat", ["the"]), 3)) # 2 0.333
print(mle.score("log", ["a"])) # 0.0 unseen
test = list(bigrams(pad_sequence("a cat sat on the log".split(), 2,
pad_left=True, left_pad_symbol="<s>",
pad_right=True, right_pad_symbol="</s>")))
for name, m in models.items():
print(name, round(m.perplexity(test), 2))
# mle inf (one unseen bigram ruins it)
# laplace 7.04
# kn 4.09
print(mle.generate(5, text_seed=["the"], random_seed=3)) # e.g. ['cat', 'saw', 'the', 'dog', 'sat']Remember
- N-gram LMs estimate P(word | previous n-1 words) from counts
- padded_everygram_pipeline pads sentences and builds training n-grams
- MLE gives zero to unseen n-grams; Laplace and Kneser-Ney smooth
- Perplexity (lower is better) compares models on the same vocabulary
Watch out for
- Comparing perplexities of models trained with different vocabularies or UNK handling.