TopolearnAuto classes let the same code load BERT, Qwen or Llama; choosing the right head determines what the model outputs.
from_pretrained reads config.json, picks the matching architecture and loads weights. Auto classes do the picking for you, so code works across model families:
AutoModel: the bare backbone, returning hidden states (embeddings).AutoModelForSequenceClassification: a classification head on top, returninglogitsper label.AutoModelForTokenClassification(NER),AutoModelForQuestionAnswering,AutoModelForCausalLM(text generation),AutoModelForSeq2SeqLM,AutoModelForImageClassification, and more.
Loading a backbone with a new head is the start of fine-tuning: AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=3) initialises the head randomly and logs a warning listing the newly initialised weights. That warning is expected before training, but alarming if you meant to load a fine-tuned classifier.
Common loading options:
dtype="auto"(default in v5) keeps the checkpoint's precision;dtype=torch.bfloat16halves memory versus float32.device_map="auto"places layers across available GPUs and CPU via Accelerate, needed for models larger than one GPU.id2label/label2idattach readable label names to the config.
The model is a regular torch.nn.Module: call .to("cuda"), .eval(), inspect .config, or count parameters with model.num_parameters().
Example
import torch
from transformers import AutoModel, AutoModelForSequenceClassification, AutoTokenizer
name = "google-bert/bert-base-uncased"
tok = AutoTokenizer.from_pretrained(name)
backbone = AutoModel.from_pretrained(name)
out = backbone(**tok("hello", return_tensors="pt"))
print(out.last_hidden_state.shape) # torch.Size([1, 3, 768])
clf = AutoModelForSequenceClassification.from_pretrained(
name, num_labels=3,
id2label={0: "neg", 1: "neu", 2: "pos"}, label2id={"neg": 0, "neu": 1, "pos": 2},
) # logs: Some weights ... are newly initialized: ['classifier.bias', 'classifier.weight']
print(clf.config.id2label[2]) # pos
print(f"{clf.num_parameters():,}") # 109,484,547
half = AutoModel.from_pretrained(name, dtype=torch.bfloat16)
print(next(half.parameters()).dtype) # torch.bfloat16Remember
- Auto classes pick the architecture from config.json
- The
...For<Task>class decides the head and the outputs - A 'newly initialized' warning means the head still needs training
Watch out for
- Loading a base checkpoint with AutoModelForSequenceClassification and using it for predictions without fine-tuning; the head is random.