TopolearnPretrained tokenizers break on legal, medical or technical text full of unusual abbreviations, silently corrupting every later step.
Punkt is an unsupervised algorithm: it learns from raw text which tokens ending in a period are abbreviations, which words commonly start sentences, and which period-word pairs are collocations. The shipped models were trained on news text, so domain abbreviations such as approx., Fig. or cf. can cause false sentence breaks.
Two ways to adapt it:
- Train on a large sample of your own raw text:
PunktTrainer().train(text, finalize=False), callfinalize_training(), then buildPunktSentenceTokenizer(trainer.get_params()). More text gives better statistics; a few hundred kilobytes is a reasonable start. - Patch parameters directly: add lowercase abbreviation types without the period to
tokenizer._params.abbrev_types. Quick and predictable, but it touches a private attribute.
span_tokenize(text) returns character offsets instead of strings, which you need for mapping results back to the original document (highlighting, annotation).
For words, RegexpTokenizer(pattern, gaps=False) gives full control. With gaps=True the pattern describes separators instead of tokens. Write verbose patterns with (?x) to handle tricky cases: abbreviations like U.S.A., hyphenated words, currency and percentages. MWETokenizer then merges multi-word expressions, and TreebankWordDetokenizer turns tokens back into readable text.
Test tokenizers on real examples from your corpus and compare output before and after changes; small changes ripple through every downstream count.
Example
from nltk.tokenize import RegexpTokenizer, sent_tokenize
from nltk.tokenize.punkt import PunktSentenceTokenizer, PunktTrainer
text = "Results are shown in Fig. 3 of the paper. Values were approx. 5 mm."
print(sent_tokenize(text))
# ['Results are shown in Fig.', '3 of the paper.', 'Values were approx.', '5 mm.']
trainer = PunktTrainer()
trainer.INCLUDE_ALL_COLLOCS = True
trainer.train("See Fig. 1 and Fig. 2 here. It was approx. 3 cm long. " * 50, finalize=False)
trainer.finalize_training()
tok = PunktSentenceTokenizer(trainer.get_params())
tok._params.abbrev_types.update({"fig", "approx"}) # make sure
print(tok.tokenize(text))
# ['Results are shown in Fig. 3 of the paper.', 'Values were approx. 5 mm.']
print(list(tok.span_tokenize(text))) # [(0, 41), (42, 67)]
pattern = r'''(?x) (?:[A-Z]\.)+ | \$?\d+(?:\.\d+)?%? | \w+(?:-\w+)* | [^\w\s]'''
print(RegexpTokenizer(pattern).tokenize("The U.S.A. grew 3.5% in well-known $12 deals."))
# ['The', 'U.S.A.', 'grew', '3.5%', 'in', 'well-known', '$12', 'deals', '.']Remember
- Punkt learns abbreviations and sentence starters from raw text
- Train with PunktTrainer on your domain or add abbreviations to the params
- span_tokenize returns character offsets for mapping back
- Verbose (?x) regexes in RegexpTokenizer give precise word rules
Watch out for
- Assuming the default sentence splitter works on technical or legal text without checking a sample.