TopolearnWord counts are the first look at any corpus and the basis of keyword extraction and bag-of-words features.
nltk.FreqDist(tokens) counts how often each token occurs. It is a subclass of collections.Counter, so fd["word"], fd.most_common(10) and arithmetic work as usual, plus extras: fd.N() (total tokens), fd.B() (distinct tokens), fd.freq(w) (relative frequency), fd.hapaxes() (words seen once) and fd.plot(20) with matplotlib.
Raw counts are dominated by stopwords: "the", "of", "and", which say little about the topic. nltk.corpus.stopwords.words("english") returns a list of about 180 common English words; put it in a set for fast membership tests. Lists exist for many other languages.
A typical cleanup before counting:
- lowercase tokens so "The" and "the" merge,
- keep only alphabetic tokens (
w.isalpha()) to drop punctuation and numbers, - remove stopwords.
Be deliberate. The stopword list includes negations such as "not", "no" and "nor", and removing them flips meaning for sentiment tasks. Domain texts also have their own uninformative words ("patient" in clinical notes), which you add to the set.
ConditionalFreqDist counts per condition, for example word frequencies per genre or per author: ConditionalFreqDist((genre, w) for genre in genres for w in words(genre)), then cfd["news"].most_common(5).
Example
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
text = '''The cat sat on the mat. The dog sat on the log.
The cat and the dog are not friends, but the cat likes the mat.'''
tokens = [w.lower() for w in word_tokenize(text) if w.isalpha()]
fd = nltk.FreqDist(tokens)
print(fd.most_common(3)) # [('the', 8), ('cat', 3), ('sat', 2)]
print(fd.N(), fd.B()) # 26 13
stop = set(stopwords.words("english"))
print(len(stop), "not" in stop) # 198 True (count varies by version)
content = [w for w in tokens if w not in stop]
print(nltk.FreqDist(content).most_common(3)) # [('cat', 3), ('sat', 2), ('mat', 2)]
cfd = nltk.ConditionalFreqDist((len(w), w) for w in content)
print(sorted(cfd[3])) # ['cat', 'dog', 'log', 'mat', 'sat']Remember
- FreqDist is a Counter with extras: most_common, N, B, freq, hapaxes
- Lowercase and filter non-alphabetic tokens before counting
- Use a set of stopwords; the English list includes negations
- ConditionalFreqDist counts separately per condition
Watch out for
- Removing NLTK stopwords before sentiment analysis, which deletes 'not' and inverts meanings.