TopolearnEvery other lesson uses these words; knowing them makes the rest of the course easy to follow.
Earlier you saw that a model is a formula with millions of learned weights, and that training those weights takes lots of data and GPU time. Training a big model from nothing can cost a fortune, but you rarely have to: companies and researchers publish their trained models for anyone to download.
Hugging Face is the company behind the main website for sharing them, the Hub. Its Python library Transformers downloads and runs those models in a few lines of code.
Most of these models share one neural-network design, the transformer. Its key layer is attention: while working on one word, the model weighs every other word in the input by how relevant it is. In "the bank of the river", attention is what lets the model read "bank" as a riverbank rather than a place for money.
A model can't read letters, only numbers. So text is first cut into tokens: short, common chunks of text, sometimes a whole word ("the") and sometimes part of one ("token" + "ization"). The model has a fixed list of every token it knows, its vocabulary, and each token is replaced by its position in that list. Those numbers are the token IDs (called input_ids in code).
The software that does this cutting and numbering is a tokenizer. Every model comes with its own, because it learned with that exact vocabulary.
A model published on the Hub is called a checkpoint: a named set of files, such as google-bert/bert-base-uncased, holding the model's settings, its learned weights and its tokenizer. Download one and you have a pretrained model, ready to use as it is or to train a little more on your own examples. That extra training is called fine-tuning.
The best-known transformers are large language models (LLMs) such as Llama or Qwen. They are trained on huge amounts of text to predict the next token, over and over, which is how chatbots write their answers. The same library also runs smaller transformers that sort or tag text, and models for images and audio.
Example
from transformers import AutoTokenizer
# downloads only the tokenizer files of a checkpoint from the Hub
tok = AutoTokenizer.from_pretrained("google-bert/bert-base-uncased")
print(tok.tokenize("Hello world!"))
# ['hello', 'world', '!']
print(tok("Hello world!")["input_ids"])
# [101, 7592, 2088, 999, 102] 101 and 102 mark the start and end
print(tok.tokenize("tokenization"))
# ['token', '##ization'] a rarer word becomes two pieces
print(len(tok))
# 30522 tokens in this model's vocabularyRemember
- Transformers downloads and runs pretrained models shared on the Hugging Face Hub
- Text is cut into tokens, and each token becomes an ID from the model's vocabulary
- A checkpoint is a named model on the Hub: settings, weights and tokenizer
- Fine-tuning is training a pretrained model a little more on your own data
Watch out for
- Loading the model from one checkpoint and the tokenizer from another: the token IDs then point at the wrong vocabulary and the output is nonsense.