TopolearnChunks such as noun phrases and entities such as people and places are what most information extraction is after.
Chunking (shallow parsing) groups POS-tagged tokens into non-overlapping phrases without building a full parse tree. nltk.RegexpParser(grammar) takes rules written as regular expressions over tags:
NP: {<DT>?<JJ>*<NN.*>+}This says a noun phrase is an optional determiner, any adjectives, then one or more nouns. <NN.*> matches NN, NNS, NNP, NNPS. Rules can also chink (remove) with }...{ and you can define several labels, applied in order.
parser.parse(tagged) returns an nltk.Tree: the root S contains subtrees for chunks and plain (word, tag) tuples for everything else. Iterate with tree.subtrees(filter=lambda t: t.label() == "NP") and get words with t.leaves().
Named entity recognition: nltk.ne_chunk(tagged) uses a pretrained maximum entropy chunker (data maxent_ne_chunker_tab plus words) and labels PERSON, ORGANIZATION, GPE (geo-political entity), LOCATION and others. binary=True collapses them all to NE.
nltk.chunk.tree2conlltags(tree) converts to IOB tags (B-NP, I-NP, O), the standard format for training and evaluating chunkers.
NLTK's NER is dated and noticeably less accurate than spaCy or transformer models; the example shows it splitting a person's name and mislabelling a company. Use it to learn the concepts or for quick experiments; RegexpParser remains handy for precise, rule-based phrase patterns.
Example
import nltk
from nltk import ne_chunk, pos_tag, word_tokenize
tagged = pos_tag(word_tokenize("The little yellow dog barked at the old cat"))
parser = nltk.RegexpParser("NP: {<DT>?<JJ>*<NN.*>+}")
tree = parser.parse(tagged)
for np_ in tree.subtrees(filter=lambda t: t.label() == "NP"):
print(" ".join(w for w, t in np_.leaves()))
# The little yellow dog
# the old cat
print(nltk.chunk.tree2conlltags(tree)[:3])
# [('The', 'DT', 'B-NP'), ('little', 'JJ', 'I-NP'), ('yellow', 'JJ', 'I-NP')]
sent = "Barack Obama visited Microsoft in Seattle last week."
ents = ne_chunk(pos_tag(word_tokenize(sent)))
for t in ents.subtrees(filter=lambda t: t.label() != "S"):
print(t.label(), " ".join(w for w, _ in t.leaves()))
# PERSON Barack
# PERSON Obama
# PERSON Microsoft <- wrong label, and the name above was split in two
# GPE Seattle (output from NLTK 3.10's model)Remember
- RegexpParser rules are regular expressions over POS tags
- Chunk results are nltk.Tree objects; filter subtrees by label
- ne_chunk labels PERSON, ORGANIZATION, GPE and more
- IOB tags (B-, I-, O) are the standard chunk annotation format
Watch out for
- Writing chunk patterns over words instead of tags, e.g. <dog> rather than <NN>.