TopolearnHand-built prompts for chat models degrade quality badly; templates make prompting correct and portable.
Instruct and chat models were fine-tuned on conversations rendered in a specific format with special tokens, such as <|im_start|>user ... <|im_end|>. Each model's tokenizer ships a chat template (a Jinja template) that renders a list of messages into that format.
messages = [{"role": "system", "content": "You are concise."},
{"role": "user", "content": "What is a tensor?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")add_generation_prompt=Trueappends the header for the assistant's turn, so the model answers instead of continuing the user's message.tokenize=Falsereturns the rendered string so you can inspect it.- In v5, the tokenized result is a dict (
input_ids,attention_mask) by default, so pass it withmodel.generate(**inputs). tools=[...]passes function schemas to models trained for tool calling;continue_final_message=Truelets the model continue a prefilled assistant message.
The text-generation pipeline applies the template automatically when you pass a list of messages, and returns the conversation with the new assistant message appended.
Do not add the template's special tokens yourself and then tokenize with add_special_tokens=True; you would duplicate BOS tokens. When fine-tuning a chat model, render training data with the same template.
Example
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
repo = "HuggingFaceTB/SmolLM2-135M-Instruct"
tok = AutoTokenizer.from_pretrained(repo)
messages = [{"role": "system", "content": "Answer in one sentence."},
{"role": "user", "content": "What is gravity?"}]
print(tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True))
# <|im_start|>system
# Answer in one sentence.<|im_end|>
# <|im_start|>user ... <|im_start|>assistant
model = AutoModelForCausalLM.from_pretrained(repo)
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=40)
print(tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
chat = pipeline("text-generation", model=repo)
reply = chat(messages, max_new_tokens=40)[0]["generated_text"][-1]
print(reply["role"]) # assistantRemember
- Chat models need their own template; render messages with
apply_chat_template - Use
add_generation_prompt=Truebefore generating a reply - Pipelines accept message lists and apply the template for you
Watch out for
- Formatting prompts as plain 'User: ... Assistant:' text for an instruct model, which ignores its trained special tokens.