TopolearnBad token boundaries break every later step: matchers, tags and entities all operate on tokens.
spaCy's tokenizer first splits on whitespace, then processes each substring:
- special cases (exceptions): exact strings with a fixed tokenization, like "don't" to "do" + "n't"
- prefixes split off the start (quotes, brackets, currency), suffixes off the end (punctuation, units like "km"), repeatedly
- infixes split inside the string (hyphens between letters, "/" between words)
token_matchandurl_matchprotect strings that must stay whole
Special cases are the simplest fix: nlp.tokenizer.add_special_case("gimme", [{ORTH: "gim"}, {ORTH: "me"}]). The pieces' ORTH values must concatenate to the original string. They're also the way to keep a term together, such as "C++" or "e.g.".
To change the regex rules, start from the language defaults and compile new ones:
infixes = [p for p in nlp.Defaults.infixes if "-" not in p]
nlp.tokenizer.infix_finditer = compile_infix_regex(infixes).finditercompile_prefix_regex and compile_suffix_regex do the same for prefixes and suffixes. nlp.tokenizer.explain(text) shows which rule produced each token, invaluable for debugging.
Changing tokenization after training means trained components see token boundaries they never learned, which can hurt accuracy; ideally customise before training. Custom tokenizer settings are saved with nlp.to_disk, but a completely custom tokenizer class needs registering to be loadable.
Example
import spacy
from spacy.attrs import ORTH
from spacy.util import compile_infix_regex
nlp = spacy.blank("en")
print([t.text for t in nlp("We use C++ and state-of-the-art 5km runs, gimme!")])
# ['We', 'use', 'C++', 'and', 'state', '-', 'of', '-', 'the', '-', 'art', '5', 'km', 'runs', ',', 'gimme', '!']
nlp.tokenizer.add_special_case("gimme", [{ORTH: "gim"}, {ORTH: "me"}])
infixes = [p for p in nlp.Defaults.infixes if "-|–|—" not in p] # keep hyphenated words
nlp.tokenizer.infix_finditer = compile_infix_regex(infixes).finditer
doc = nlp("We use C++ and state-of-the-art 5km runs, gimme!")
print([t.text for t in doc])
# [..., 'state-of-the-art', '5', 'km', 'runs', ',', 'gim', 'me', '!']
for rule, text in nlp.tokenizer.explain("(5km)"):
print(rule, text)
# PREFIX ( / TOKEN 5 / SUFFIX km / SUFFIX )Remember
- Tokenization = whitespace split + special cases + prefix/suffix/infix rules
- add_special_case fixes specific strings; ORTH pieces must rebuild the text
- Recompile Defaults.prefixes/suffixes/infixes to change regex rules
- tokenizer.explain shows which rule made each token
Watch out for
- Customising tokenization of a trained pipeline and expecting the model to keep its accuracy on the new boundaries.