TopolearnHow retrieved text becomes an answer affects accuracy, cost and how well long contexts are handled.
After retrieval, a response synthesizer turns the question plus nodes into an answer. The response_mode decides how:
compact(default): packs as many nodes as fit into each prompt, then refines across prompts if needed. Few LLM calls.refine: one call per node; each call improves the previous answer with the next node. Thorough but slow and costly.tree_summarize: answers over groups of nodes, then summarises those answers, repeating until one remains. Good for summaries over many chunks.simple_summarize: truncates everything into one prompt. Cheap, may lose information.accumulate: answers separately per node and joins the results, useful for "list what each document says".no_text: retrieval only, no LLM call. Handy for testing.
Set it with index.as_query_engine(response_mode="tree_summarize") or build one with get_response_synthesizer(response_mode=...).
Custom prompts. The main question-answering prompt is called text_qa_template. It receives {context_str} (the retrieved text) and {query_str} (the question). Replace it to change tone, language, citation style or the "I don't know" behaviour:
qe.update_prompts({"response_synthesizer:text_qa_template": PromptTemplate(...)})qe.get_prompts() lists every prompt key an engine uses.
Prompt rules that pay off: answer only from the context, say clearly when the context doesn't contain the answer, cite sources using metadata, and match the user's language.
Example
from llama_index.core import Document, PromptTemplate, VectorStoreIndex
index = VectorStoreIndex.from_documents([
Document(text="Refunds take 5 business days.", metadata={"source": "faq.md"}),
])
qe = index.as_query_engine(response_mode="compact", similarity_top_k=3)
print(list(qe.get_prompts()))
# ['response_synthesizer:text_qa_template', 'response_synthesizer:refine_template', ...]
qa = PromptTemplate(
"Answer using ONLY the context below. If the answer is not there, reply "
"'I could not find this in our documents.' Cite the source file.\n"
"Context:\n{context_str}\n\nQuestion: {query_str}\nAnswer: "
)
qe.update_prompts({"response_synthesizer:text_qa_template": qa})
print(qe.query("Do you ship to Mars?")) # I could not find this in our documents.
summary_qe = index.as_query_engine(response_mode="tree_summarize")Remember
compact(default),refine,tree_summarize,accumulate,no_texttext_qa_templateuses{context_str}and{query_str}- Use
get_prompts()andupdate_prompts()to inspect and replace prompts
Watch out for
- Using refine mode with a high similarity_top_k, which makes one LLM call per node and multiplies cost and latency.