TopolearnText generation is the core of chatbots, assistants and most modern LLM use.
A causal language model predicts the next token given the previous ones. Generation repeats that step: predict a distribution, pick a token, append it, and continue until an end-of-sequence token or a length limit.
The typical setup:
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto", device_map="auto")Then tokenize a prompt, call model.generate(**inputs, max_new_tokens=50), and decode. Key points:
- Always set `max_new_tokens`. The default generation length is short (often 20 tokens) and limits the output silently.
generatereturns prompt plus continuation. Slice off the prompt (out[:, inputs["input_ids"].shape[1]:]) before decoding if you only want new text.- By default decoding is greedy unless the checkpoint's
generation_config.jsonenables sampling; many instruct models ship withdo_sample=Trueand a temperature. - For batched generation with decoder-only models, set
tokenizer.padding_side = "left"so all prompts end at the same position, and make sure a pad token exists (tok.pad_token = tok.eos_tokenif missing).
Small models like HuggingFaceTB/SmolLM2-135M-Instruct or Qwen/Qwen2.5-0.5B-Instruct run on a laptop CPU and are ideal for learning. The text-generation pipeline wraps all of this, including chat formatting, when you just need outputs.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "HuggingFaceTB/SmolLM2-135M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, dtype="auto")
inputs = tok("The three primary colors are", return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=20, do_sample=False)
new_tokens = out[0, inputs["input_ids"].shape[1]:] # drop the prompt
print(tok.decode(new_tokens, skip_special_tokens=True))
# e.g. " red, yellow and blue. ..."
# batched prompts need left padding for decoder-only models
tok.padding_side = "left"
if tok.pad_token is None:
tok.pad_token = tok.eos_token
batch = tok(["Hello, my name is", "Python is"], return_tensors="pt", padding=True)
outs = model.generate(**batch, max_new_tokens=10)
print(len(tok.batch_decode(outs, skip_special_tokens=True))) # 2Remember
AutoModelForCausalLM+generateproduces text token by token- Always set
max_new_tokens, and strip the prompt before decoding - Use left padding and a pad token for batched generation
Watch out for
- Using right padding for batched decoder-only generation, which makes the model continue after pad tokens and produce junk.