TopolearnThese two techniques give some of the largest, most reliable quality gains in production RAG.
Plain vector search has two common weaknesses. Its top results are often roughly related rather than exactly right, and it can miss exact terms such as product codes, names or error numbers.
Reranking fixes the first. Retrieve generously (say the top 20), then let a stronger, slower model re-score those candidates and keep the best few:
query_engine = index.as_query_engine(similarity_top_k=20,
node_postprocessors=[SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=4)])A cross-encoder reads the question and a passage together and outputs a relevance score, which is more accurate than comparing two separate embeddings. Options: SentenceTransformerRerank (local model, needs sentence-transformers), LLMRerank (asks your LLM to rank), and hosted rerankers such as Cohere's via integration packages.
Hybrid search fixes the second. Combine vector search with BM25, a classic keyword-ranking method that rewards exact word matches. Then merge the two ranked lists:
BM25Retriever(packagellama-index-retrievers-bm25) does the keyword part.QueryFusionRetriever([vector_retriever, bm25_retriever], mode="reciprocal_rerank")merges them with reciprocal rank fusion: documents ranked high in either list rise to the top. Setnum_queries=1to disable its default behaviour of generating extra query variations with the LLM.
Many vector databases (Qdrant, Weaviate, Elasticsearch, pgvector setups) offer hybrid search natively, which scales better than in-memory BM25.
Measure before and after with a retrieval evaluation; both techniques add latency.
Example
# pip install llama-index-retrievers-bm25 sentence-transformers
from llama_index.core import Document, VectorStoreIndex
from llama_index.core.node_parser import SentenceSplitter
from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import QueryFusionRetriever
from llama_index.retrievers.bm25 import BM25Retriever
docs = [Document(text="Error E-4031 means the printer drum is worn."),
Document(text="Replace toner when prints look faded."),
Document(text="Paper jams: open tray 2 and remove the sheet.")]
nodes = SentenceSplitter(chunk_size=256).get_nodes_from_documents(docs)
index = VectorStoreIndex(nodes)
hybrid = QueryFusionRetriever(
[index.as_retriever(similarity_top_k=10), BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=10)],
similarity_top_k=10, num_queries=1, mode="reciprocal_rerank", use_async=False)
rerank = SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-6-v2", top_n=2)
qe = RetrieverQueryEngine.from_args(hybrid, node_postprocessors=[rerank])
resp = qe.query("What does E-4031 mean?")
print(resp.source_nodes[0].node.text) # Error E-4031 means the printer drum is worn.Remember
- Rerank: retrieve many, re-score with a cross-encoder or LLM, keep few
- Hybrid: BM25 keyword + vector results merged by reciprocal rank fusion
QueryFusionRetriever(..., num_queries=1, mode="reciprocal_rerank")
Watch out for
- Adding a reranker but keeping similarity_top_k at 2, leaving the reranker nothing to choose from.