TopolearnDomain-specific text, offsets for span tasks and multilingual quirks all require looking inside the tokenizer.
A Rust-backed tokenizer runs four stages:
1. Normalization: Unicode normalization, lowercasing or accent stripping (BERT-uncased lowercases; most LLMs do not). 2. Pre-tokenization: splitting into words or byte-level chunks (GPT-style byte-level BPE maps every byte to a symbol, so no input is ever unknown). 3. Model: the subword algorithm. BPE merges the most frequent symbol pairs (GPT, Llama, Qwen); WordPiece is similar but chooses merges by likelihood and marks continuations with ## (BERT); Unigram starts from a large vocabulary and prunes it probabilistically (T5, via SentencePiece). 4. Post-processing: adds special tokens such as [CLS]/[SEP] or BOS.
tok.backend_tokenizer exposes these components. return_offsets_mapping=True returns character spans per token, essential for mapping predictions back to text in NER or extractive QA; enc.word_ids() maps tokens to words.
For a new domain (code, chemistry, a new language), tok.train_new_from_iterator(texts, vocab_size=...) trains a tokenizer with the same pipeline and special tokens on your corpus. A new tokenizer means new token IDs, so the model's embeddings no longer match; this is useful when pretraining from scratch. To keep a pretrained model, instead add a few tokens with tok.add_tokens([...]) and model.resize_token_embeddings(len(tok)).
Watch for whitespace handling: byte-level BPE encodes leading spaces into tokens, so "hello" and " hello" get different IDs.
Example
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, trainers
from transformers import PreTrainedTokenizerFast
corpus = ["the cat sat on the mat", "the cats sat", "a matter of cats"] * 50
raw = Tokenizer(models.WordPiece(unk_token="[UNK]"))
raw.normalizer = normalizers.Lowercase()
raw.pre_tokenizer = pre_tokenizers.Whitespace()
raw.train_from_iterator(corpus, trainers.WordPieceTrainer(
vocab_size=60, special_tokens=["[UNK]", "[PAD]"]))
tok = PreTrainedTokenizerFast(tokenizer_object=raw, unk_token="[UNK]", pad_token="[PAD]")
enc = tok("The Cats sat", return_offsets_mapping=True)
print(tok.convert_ids_to_tokens(enc["input_ids"])) # ['the', 'cats', 'sat']
print(enc["offset_mapping"]) # [(0, 3), (4, 8), (9, 12)]
print(enc.word_ids()) # [0, 1, 2]
print(tok.tokenize("matters")) # ['matter', '##s']
# pretrained = AutoTokenizer.from_pretrained("gpt2")
# new_tok = pretrained.train_new_from_iterator(my_texts, vocab_size=32000)Remember
- Pipeline: normalize, pre-tokenize, subword model (BPE/WordPiece/Unigram), post-process
- Offsets and word_ids map tokens back to character spans and words
- A newly trained tokenizer needs a newly trained model; add tokens to extend an existing one
Watch out for
- Training a new tokenizer for an existing pretrained model, which scrambles the mapping between token IDs and learned embeddings.