TopolearnPhrases like 'New York' or 'machine learning' carry meaning that single words lose, and n-grams are the basis of language models.
An n-gram is a run of n consecutive tokens. nltk.ngrams(tokens, n) yields tuples; nltk.bigrams and nltk.trigrams are shortcuts. Counting them with FreqDist shows frequent phrases, but raw counts favour boring pairs such as ("of", "the").
A collocation is a pair that occurs together more often than chance predicts. BigramCollocationFinder.from_words(tokens) collects counts, and an association measure scores each pair:
BigramAssocMeasures.pmi(pointwise mutual information): high when words rarely appear apart, but it overrates rare pairslikelihood_ratioandchi_sq: more robust for typical corporaraw_freq: plain frequency
Before scoring, filter: finder.apply_freq_filter(3) drops pairs seen fewer than 3 times (essential for PMI), and finder.apply_word_filter(lambda w: w in stop) removes pairs containing stopwords. Then finder.nbest(measure, 10) returns the top pairs. Trigram versions exist too.
nltk.Text(tokens).collocation_list() gives a quick default list.
Uses: phrase detection before topic modelling, building keyword lists, and features for classifiers. MWETokenizer can then merge discovered phrases such as ("New", "York") into single tokens.
Example
import nltk
from nltk.collocations import BigramAssocMeasures, BigramCollocationFinder
from nltk.corpus import gutenberg, stopwords
tokens = ["machine", "learning", "is", "fun", "and", "machine", "learning", "is", "hard"]
print(list(nltk.bigrams(tokens))[:3])
# [('machine', 'learning'), ('learning', 'is'), ('is', 'fun')]
print(nltk.FreqDist(nltk.ngrams(tokens, 3)).most_common(1))
# [(('machine', 'learning', 'is'), 2)]
words = [w.lower() for w in gutenberg.words("austen-emma.txt") if w.isalpha()]
stop = set(stopwords.words("english"))
finder = BigramCollocationFinder.from_words(words)
finder.apply_freq_filter(5)
finder.apply_word_filter(lambda w: w in stop)
m = BigramAssocMeasures()
print(finder.nbest(m.likelihood_ratio, 4))
# [('mr', 'knightley'), ('mrs', 'weston'), ('frank', 'churchill'), ('mr', 'elton')]
mwe = nltk.MWETokenizer([("machine", "learning")])
print(mwe.tokenize(tokens)[:2]) # ['machine_learning', 'is']Remember
- nltk.ngrams/bigrams produce tuples of consecutive tokens
- Collocation finders score pairs with PMI, likelihood ratio, chi-square
- Apply frequency and stopword filters before ranking
- MWETokenizer merges known phrases into single tokens
Watch out for
- Ranking by PMI without a frequency filter, so pairs of one-off rare words dominate.