Tokenizers: text to input IDs

Models only see integers; wrong tokenization or padding silently ruins results.

A tokenizer splits text into subword tokens and maps them to integer IDs from the model's vocabulary. Always load the tokenizer that matches the checkpoint: AutoTokenizer.from_pretrained(repo). Mixing tokenizers and models produces garbage without any error.

Calling the tokenizer returns a dict-like BatchEncoding:

input_ids: token IDs, including special tokens such as [CLS]/[SEP] for BERT or a BOS token for many LLMs.
attention_mask: 1 for real tokens, 0 for padding, so the model ignores pads.
Some models also return token_type_ids.

For batches, sequences must have the same length:

padding=True pads to the longest sequence in the batch; padding="max_length" pads to max_length.
truncation=True cuts sequences longer than max_length (or the model's limit).
return_tensors="pt" returns PyTorch tensors instead of lists.

tokenizer.tokenize(text) shows the subword strings, and tokenizer.decode(ids, skip_special_tokens=True) turns IDs back into text. Rare words split into several pieces (tokenization becomes token + ##ization in BERT's WordPiece), which is how a fixed vocabulary of about 30k to 150k entries covers any text.

Tokenizers in Transformers are backed by the Rust tokenizers library, so batch tokenization is fast. Token counts, not characters, determine context limits and API costs.

Example

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")

enc = tok("Hello world!")
print(enc["input_ids"])          # [101, 7592, 2088, 999, 102]
print(tok.convert_ids_to_tokens(enc["input_ids"]))
# ['[CLS]', 'hello', 'world', '!', '[SEP]']
print(tok.tokenize("tokenization"))   # ['token', '##ization']

batch = tok(["short one", "a somewhat longer sentence here"],
            padding=True, truncation=True, max_length=16, return_tensors="pt")
print(batch["input_ids"].shape)  # torch.Size([2, 7])
print(batch["attention_mask"][0])  # tensor([1, 1, 1, 1, 0, 0, 0])

print(tok.decode(batch["input_ids"][0], skip_special_tokens=True))  # short one

Remember

Always load the tokenizer from the same checkpoint as the model
Batches need padding and usually truncation; the attention mask marks real tokens
Use return_tensors="pt" to feed the model directly

Watch out for

Forgetting truncation, so long inputs exceed the model's maximum length and raise an index error.

Check yourself

What is the attention_mask for?

Official docs: huggingface.co/docs/transformers/fast_tokenizers

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks