Topolearn

Learn Hugging Face Transformers

Pretrained transformer models for text, vision and audio, plus training and generation

24 lessons · 5 hours · 4 levels · covers Hugging Face Transformers 5.x

Transformers is Hugging Face's library of pretrained model architectures (BERT, Llama, Qwen, Whisper, ViT and hundreds more) with a uniform API to download, run, fine-tune and share them. It covers quick inference with pipelines, tokenization, the Trainer training loop, text generation for LLMs, and integrations for LoRA fine-tuning and quantization. It is the default entry point to open-weight models on the Hugging Face Hub.

Sign in free to track progressOfficial docs ↗

Before you start

Hugging Face Transformers builds on these; the tree links to the exact lessons you need.

PythonPractical levelPyTorchEssentials level

How deep do you want to go?

1. EssentialsRead and write simple code, do the everyday basics8 lessons · 2 hours in total
2. PracticalUse it productively at work, with the usual idioms15 lessons · 3 hours in total
3. AdvancedThe less common features, performance and trade-offs20 lessons · 4 hours in total
4. ExpertInternals and the edge cases experts know24 lessons · 5 hours in total

The Hugging Face Transformers tree

Hugging Face TransformersPipelines: inference in three linesPipelines: inferencein three linesThe Hub, checkpoints and cachingThe Hub,checkpoints and ca…Tokenizers: text to input IDsTokenizers: text toinput IDsAuto classes and task headsAuto classes andtask headsRunning a model by handRunning a model byhandChoosing pipeline tasksChoosing pipelinetasksLoading data with DatasetsLoading data withDatasetsGenerating text with causal LMsGenerating text withcausal LMsPreprocessing and data collatorsPreprocessing anddata collatorsFine-tuning with TrainerFine-tuning withTrainerEvaluation metricsEvaluation metricsSaving, loading and pushing to the HubSaving, loading andpushing to the HubChat models and chat templatesChat models and chattemplatesDecoding strategies and GenerationConfigDecoding strategiesand GenerationCon…LoRA fine-tuning with PEFTLoRA fine-tuning withPEFTStrings and f-stringsStrings and f-stringsnn.Module: defining modelsnn.Module: definingmodelsClasses and instancesClasses andinstancesDtypes and devicesDtypes and devicesDictionariesDictionariesDefining functionsDefining functionsThe training loopThe training loop
Essentials and Practical shown (15 of 24 lessons); the full list is below. Click a lesson to read it.

All 24 lessons, in order

Pipelines and inference

  1. Pipelines: inference in three lines · Essentials · 10 min

    Run a pretrained model on raw text, images or audio with pipeline(), which handles preprocessing and postprocessing.

  2. The Hub, checkpoints and caching · Essentials · 10 min

    Find models on the Hugging Face Hub, pin revisions, understand the cache and safetensors, and treat remote code with care.

  3. Choosing pipeline tasks · Essentials · 10 min

    Use zero-shot classification, NER, fill-mask, embeddings and vision or audio tasks, and tune their key options.

Tokenizers and models

  1. Tokenizers: text to input IDs · Essentials · 12 min

    Convert text into token IDs and attention masks with AutoTokenizer, including padding, truncation and decoding.

  2. Auto classes and task heads · Essentials · 12 min

    Load the right architecture and task head with AutoModel and AutoModelFor... classes and from_pretrained.

  3. Running a model by hand · Essentials · 12 min

    Tokenize, run the forward pass under inference mode, and turn logits into probabilities and labels yourself.

  4. Tokenizer internals and training · Expert · 14 min

    Learn the tokenization pipeline (normalize, pre-tokenize, model, post-process), BPE vs WordPiece vs Unigram, and train a new tokenizer.

Fine-tuning with Trainer

  1. Loading data with Datasets · Essentials · 12 min

    Load, inspect, split and transform datasets with the Hugging Face datasets library.

  2. Preprocessing and data collators · Practical · 12 min

    Tokenize a dataset with map and let a data collator pad each batch dynamically.

  3. Fine-tuning with Trainer · Practical · 15 min

    Configure TrainingArguments and run a complete fine-tuning loop with Trainer, including evaluation and checkpoints.

  4. Evaluation metrics · Practical · 10 min

    Report accuracy, F1 and other metrics during training with a compute_metrics function.

  5. Saving, loading and pushing to the Hub · Practical · 10 min

    Save fine-tuned models and tokenizers, reload them, and publish them to the Hub with a model card.

  6. Callbacks and custom training logic · Advanced · 14 min

    Extend Trainer with callbacks, early stopping, a custom loss and your own optimizer without rewriting the loop.

Generation and chat

  1. Generating text with causal LMs · Essentials · 12 min

    Load a decoder-only LLM with AutoModelForCausalLM and generate continuations with model.generate.

  2. Chat models and chat templates · Practical · 12 min

    Format conversations with apply_chat_template so instruct models see exactly the prompt format they were trained on.

  3. Decoding strategies and GenerationConfig · Practical · 12 min

    Control output quality and diversity with greedy, sampling, temperature, top-p, beam search and stopping criteria.

  4. KV cache and streaming output · Advanced · 14 min

    Understand the key-value cache that makes generation fast, choose cache types, and stream tokens as they are produced.

Efficient fine-tuning and quantization

  1. LoRA fine-tuning with PEFT · Practical · 15 min

    Fine-tune large models cheaply by training small low-rank adapters with the PEFT library.

  2. Quantized loading · Advanced · 14 min

    Load models in 8-bit or 4-bit with bitsandbytes and know when other methods such as GPTQ, AWQ or torchao fit better.

  3. QLoRA: LoRA on a 4-bit base · Advanced · 14 min

    Combine 4-bit quantized base weights with LoRA adapters to fine-tune large models on a single GPU.

Scaling and internals

  1. Memory and speed in training · Advanced · 14 min

    Fit bigger models and batches with mixed precision, gradient accumulation, gradient checkpointing and better optimizers.

  2. Attention backends and device placement · Expert · 14 min

    Select SDPA, FlashAttention or custom attention kernels, and control multi-device placement and offloading.

  3. Multi-GPU training: DDP, FSDP, DeepSpeed · Expert · 16 min

    Scale Trainer runs across GPUs and nodes with torchrun or accelerate, using data parallelism, FSDP or DeepSpeed ZeRO.

  4. Model classes, configs and custom models · Expert · 15 min

    See how PreTrainedConfig, PreTrainedModel and task heads fit together, and register your own architecture with the Auto classes.