TopolearnMost text preprocessing and feature extraction is a filter or map over these token attributes.
Each token exposes lexical and predicted attributes:
lemma_: the base form ("running" to "run"), from the lemmatizerpos_: coarse Universal POS tag (NOUN,VERB,ADJ,PROPN...);tag_: the fine-grained, language-specific tag (Penn TreebankVBGfor English)morph: morphological features such asTense=Past|VerbForm=Finis_stop,is_punct,is_alpha,is_digit,like_num,like_email,like_urlshape_: an abstract shape likeXxxxxordd,lower_,prefix_,suffix_
The trailing underscore matters. token.pos_ is the readable string; token.pos (no underscore) is an integer id or hash. spaCy stores strings as 64-bit hashes internally for speed and memory, so the int form is what you'd use in arrays, and the string form is for humans and most code.
spacy.explain("VBG") describes a tag or label.
Lexical flags like is_stop and like_num come from language data and don't depend on the model; pos_, tag_ and lemma_ are predictions (or rule-based lookups) and can be wrong on unusual text.
A typical cleanup for bag-of-words features is a comprehension: keep tokens that are alphabetic and not stopwords, map them to lowercase lemmas. Unlike NLTK, the context-aware tagger already handles the POS needed for correct lemmas.
Example
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("The striped bats were hanging upside down, eating 12 mangoes.")
for t in doc[:5]:
print(f"{t.text:8} {t.lemma_:6} {t.pos_:5} {t.tag_:4} {t.is_stop}")
# The the DET DT True
# striped striped ADJ JJ False
# bats bat NOUN NNS False
# were be AUX VBD True
# hanging hang VERB VBG False
print(doc[9].text, doc[9].like_num, doc[9].shape_) # 12 True dd
print(doc[4].morph) # Aspect=Prog|Tense=Pres|VerbForm=Part
print(doc[2].pos, doc[2].pos_) # 92 NOUN (id vs string)
print(spacy.explain("VBG")) # verb, gerund or present participle
clean = [t.lemma_.lower() for t in doc if t.is_alpha and not t.is_stop]
print(clean) # ['striped', 'bat', 'hang', 'upside', 'eat', 'mango']Remember
- lemma_, pos_, tag_ and morph are pipeline predictions
- is_stop, is_alpha, like_num, shape_ come from lexical rules
- Underscore = readable string; no underscore = integer id/hash
- spacy.explain decodes tags and labels
Watch out for
- Using
token.lemma(an integer hash) when you meant the stringtoken.lemma_.