TopolearnPOS tags feed lemmatization, chunking, keyword extraction and many rule-based text patterns.
Part-of-speech tagging assigns each token a grammatical class. nltk.pos_tag(tokens) takes a tokenized sentence and returns (word, tag) pairs using a pretrained averaged perceptron model (data package averaged_perceptron_tagger_eng).
Tags follow the Penn Treebank tagset, which is fine-grained:
- nouns:
NN(singular),NNS(plural),NNP(proper) - verbs:
VB(base),VBD(past),VBG(-ing),VBN(past participle),VBZ(3rd person singular) JJadjective,RBadverb,DTdeterminer,INpreposition,PRPpronoun,CDnumber
nltk.help.upenn_tagset("VB.*") explains tags (needs the tagsets_json data). If you only need coarse classes, download universal_tagset and pass tagset="universal" to get NOUN, VERB, ADJ, ADV, DET, ADP and so on.
Tag whole sentences, not isolated words or lists of keywords: the tagger uses context, which is how it can tell that "permit" is a verb in "refuse to permit us" and a noun in "the permit". It is still a statistical model and makes mistakes, especially on words much more common in one role. Tagging after removing stopwords or lowercasing everything hurts accuracy too.
Use pos_tag_sents to tag many sentences efficiently. The built-in tagger is fast and reasonable on news-like English; for other languages, informal text or higher accuracy, a spaCy or transformer pipeline is the usual upgrade. You can also train your own taggers from tagged corpora.
Example
import nltk
from nltk import pos_tag, pos_tag_sents, word_tokenize
print(pos_tag(word_tokenize("They refuse to permit us to obtain the refuse permit")))
# [('They', 'PRP'), ('refuse', 'VBP'), ('to', 'TO'), ('permit', 'VB'), ('us', 'PRP'),
# ('to', 'TO'), ('obtain', 'VB'), ('the', 'DT'), ('refuse', 'NN'), ('permit', 'NN')]
print(pos_tag(word_tokenize("I want to book a flight"))[3])
# ('book', 'NN') -- taggers make mistakes: here it should be VB
nltk.download("universal_tagset", quiet=True) # mapping table
print(pos_tag(word_tokenize("Dogs bark loudly"), tagset="universal"))
# [('Dogs', 'NOUN'), ('bark', 'VERB'), ('loudly', 'ADV')]
sents = [word_tokenize(s) for s in ["Time flies.", "She sold 3 cars."]]
for tagged in pos_tag_sents(sents):
nouns = [w for w, t in tagged if t.startswith("NN")]
print(nouns)
# ['Time', 'flies'] ('flies' tagged as a plural noun here)
# ['cars']Remember
- pos_tag returns (word, Penn Treebank tag) pairs
- Tags encode detail: NNS plural noun, VBD past verb, JJ adjective
- tagset='universal' gives coarse classes
- Tag full sentences in their original form; context drives accuracy
Watch out for
- Tagging a bag of lowercased keywords without context and trusting the tags.