TopolearnDoing what a pipeline does manually gives you batching control, hidden states and custom postprocessing.
A manual inference loop has four steps:
1. Tokenize a batch with padding=True, truncation=True, return_tensors="pt" and move tensors to the model's device: inputs = inputs.to(model.device). 2. Put the model in evaluation mode with model.eval() (disables dropout; from_pretrained already does this, but it matters after training). 3. Run the forward pass inside torch.inference_mode() (or torch.no_grad()) so no autograd graph is built, saving memory and time. 4. Postprocess: outputs are a ModelOutput object with attributes like .logits, .hidden_states (with output_hidden_states=True) or .loss (when you pass labels).
For classification, logits.softmax(-1) gives probabilities, and logits.argmax(-1) the predicted class index; map it through model.config.id2label.
For sentence embeddings with an AutoModel, a common approach is mean pooling: average last_hidden_state over real tokens using the attention mask, so padding does not dilute the result. Dedicated embedding models (sentence-transformers checkpoints) are trained for this and work much better than raw BERT.
Process data in batches sized to your memory; sort by length to reduce padding when throughput matters.
Example
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer
name = "distilbert/distilbert-base-uncased-finetuned-sst-2-english"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()
texts = ["What a fantastic movie", "Boring and far too long"]
inputs = tok(texts, padding=True, truncation=True, return_tensors="pt").to(model.device)
with torch.inference_mode():
logits = model(**inputs).logits # shape (2, 2)
probs = logits.softmax(dim=-1)
for text, p in zip(texts, probs):
label = model.config.id2label[int(p.argmax())]
print(f"{label:8} {p.max():.3f} {text}")
# POSITIVE 1.000 What a fantastic movie
# NEGATIVE 1.000 Boring and far too longRemember
- Tokenize a batch, move it to
model.device, run undertorch.inference_mode() - Outputs are ModelOutput objects: use
.logits,.hidden_states,.loss - Mask-aware mean pooling turns hidden states into sentence embeddings
Watch out for
- Leaving inputs on the CPU while the model is on the GPU, which raises a device mismatch error.