TopolearnStemming merges word variants cheaply, shrinking vocabularies for search and classic bag-of-words models.
A stemmer chops affixes off words using rules, so variants map to one key: "connect", "connected", "connecting" and "connection" all become "connect". The stem need not be a real word: "studies" becomes "studi".
NLTK provides:
PorterStemmer: the classic 1980 algorithm, moderate and widely usedSnowballStemmer("english"): "Porter2", slightly better, and available for many languages (SnowballStemmer.languages)LancasterStemmer: very aggressive, often over-stems ("maximum" to "maxim")RegexpStemmerfor your own suffix rules
Stemmers process one token at a time: stemmer.stem(word). They lowercase by default.
Stemming has two failure modes. Over-stemming merges unrelated words ("universe" and "university" both become "univers"). Under-stemming misses true relatives, especially irregular forms ("ran" stays "ran", "better" stays "better").
When to use it: search indexes and simple text classifiers where speed matters and a slightly odd vocabulary is fine. When you need real dictionary words, or correct handling of irregular forms, use lemmatization instead. Modern transformer models need neither, because their subword tokenizers handle variants themselves.
Example
from nltk.stem import LancasterStemmer, PorterStemmer, SnowballStemmer
words = ["connect", "connected", "connecting", "connection",
"studies", "running", "ran", "university", "universe", "maximum"]
porter = PorterStemmer()
snow = SnowballStemmer("english")
lanc = LancasterStemmer()
for w in words:
print(f"{w:11} {porter.stem(w):9} {snow.stem(w):9} {lanc.stem(w)}")
# connection connect connect connect
# studies studi studi study
# ran ran ran ran
# university univers univers univers
# maximum maximum maximum maxim
print("german" in SnowballStemmer.languages) # True
print(SnowballStemmer("german").stem("Häuser")) # hausRemember
- Stemmers strip affixes by rule; stems need not be real words
- Snowball (Porter2) is the usual default and supports many languages
- Over-stemming merges unrelated words; under-stemming misses irregular forms
- Use lemmatization when you need dictionary words
Watch out for
- Showing stems such as 'studi' to end users, for example as keywords.