Decoding strategies and GenerationConfig

The same model gives repetitive, creative or nonsensical text depending on decoding settings.

At each step the model outputs a distribution over the vocabulary; the decoding strategy decides which token to take.

Greedy (do_sample=False, num_beams=1): always the most likely token. Deterministic, good for short factual answers, prone to repetition in long text.
Sampling (do_sample=True): draw from the distribution. temperature below 1 sharpens it, above 1 flattens it. top_p=0.9 (nucleus) samples only from the smallest set of tokens covering 90% of probability; top_k=50 limits to the 50 most likely; min_p keeps tokens above a fraction of the top probability.
Beam search (num_beams=4): keeps several candidate sequences, useful for constrained outputs like translation, rarely for open chat.

Other useful knobs: repetition_penalty (e.g. 1.1), no_repeat_ngram_size, stop_strings=["\n\n"] (requires passing tokenizer=tok to generate), eos_token_id, and num_return_sequences.

Settings live in a GenerationConfig. The checkpoint's generation_config.json provides defaults (many instruct models default to sampling with a specific temperature), arguments to generate override them, and you can save your own with model.generation_config.save_pretrained(dir).

Seed with transformers.set_seed(0) for reproducible sampling. For evaluations, prefer greedy or fixed seeds so differences come from the model, not luck.

Example

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig, set_seed

repo = "HuggingFaceTB/SmolLM2-135M"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)
inputs = tok("Once upon a time", return_tensors="pt")

greedy = model.generate(**inputs, max_new_tokens=30, do_sample=False)

set_seed(0)
creative = model.generate(**inputs, max_new_tokens=30, do_sample=True,
                          temperature=0.8, top_p=0.9, repetition_penalty=1.1)

stop = model.generate(**inputs, max_new_tokens=60, stop_strings=["."], tokenizer=tok)
print(tok.decode(stop[0], skip_special_tokens=True))   # ends at the first period

cfg = GenerationConfig(max_new_tokens=50, do_sample=True, temperature=0.7, top_p=0.95)
out = model.generate(**inputs, generation_config=cfg)
print(model.generation_config.eos_token_id)   # checkpoint default, e.g. 0

Remember

Greedy is deterministic; sampling with temperature/top-p adds diversity
Checkpoint generation_config.json sets defaults that generate() arguments override
Use stop_strings (with tokenizer=) and repetition controls to shape outputs

Watch out for

Setting temperature without do_sample=True, which has no effect under greedy decoding (and logs a warning).

Check yourself

You set temperature=0.3 but leave do_sample=False. What happens?

Official docs: huggingface.co/docs/transformers/generation_strategies

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks