TopolearnLemmas give clean, readable vocabulary for features and keywords and handle irregular forms that stemmers miss.
A lemma is the dictionary form of a word: "ran" to "run", "mice" to "mouse", "better" to "good". WordNetLemmatizer().lemmatize(word, pos) looks words up in WordNet and applies its morphology rules (it needs the wordnet data package).
The crucial detail is the `pos` argument. It defaults to noun ("n"), so verbs and adjectives are left alone: lemmatize("running") returns "running", while lemmatize("running", pos="v") returns "run". Valid values are "n", "v", "a" (adjective), "r" (adverb) and "s" (satellite adjective).
In practice you get the POS from a tagger and map Penn Treebank tags to WordNet's letters: tags starting with J are adjectives, V verbs, N nouns, R adverbs.
The lemmatizer expects lowercase input for most words and returns the word unchanged when WordNet doesn't know it, so names and typos pass through.
Compared with stemming, lemmatization is slower and needs the POS, but produces real words, handles irregular forms and doesn't over-merge. Use it when output is shown to people or when vocabulary quality matters, for example topic models and keyword extraction. spaCy's lemmatizer does the tagging step automatically if you are building a fuller pipeline.
Example
from nltk import pos_tag, word_tokenize
from nltk.corpus import wordnet as wn
from nltk.stem import WordNetLemmatizer
lem = WordNetLemmatizer()
print(lem.lemmatize("mice")) # mouse
print(lem.lemmatize("running")) # running (treated as a noun!)
print(lem.lemmatize("running", pos="v")) # run
print(lem.lemmatize("better", pos="a")) # good
def to_wn(tag):
return {"J": wn.ADJ, "V": wn.VERB, "R": wn.ADV}.get(tag[0], wn.NOUN)
sent = "The striped bats were hanging on their feet"
tagged = pos_tag(word_tokenize(sent))
print([lem.lemmatize(w.lower(), to_wn(t)) for w, t in tagged])
# ['the', 'striped', 'bat', 'be', 'hang', 'on', 'their', 'foot']Remember
- WordNetLemmatizer returns dictionary forms such as run, mouse, good
- pos defaults to noun: pass 'v', 'a' or 'r' for other word classes
- Map Penn Treebank tags (J, V, N, R) to WordNet POS from a tagger
- Unknown words are returned unchanged
Watch out for
- Calling lemmatize without a POS on verbs and wondering why nothing changes.