TopolearnThe vector index is the default and most-used index in LlamaIndex; everything else builds on it.
An index is a data structure that makes your nodes searchable. The most common is `VectorStoreIndex`: it stores an embedding for every node and finds the nodes whose embeddings are closest to the question's embedding.
Two ways to build one:
index = VectorStoreIndex.from_documents(documents) # parses into nodes for you
index = VectorStoreIndex(nodes) # when you already have nodesBuilding calls the embedding model once per node (in batches), so large datasets take time and money. show_progress=True displays a progress bar.
By default the vectors live in memory in a SimpleVectorStore. For anything persistent or large, you plug in a real vector database (a later lesson). Alongside vectors, the index keeps a docstore (the node texts and metadata) and an index store (bookkeeping), all grouped in a StorageContext.
Keeping it up to date:
index.insert(document)adds a new document.index.delete_ref_doc(doc_id, delete_from_docstore=True)removes all nodes of a document.index.refresh_ref_docs(documents)re-indexes only documents whose text changed, which is why stable document ids matter.
From one index you create the objects you use day to day: as_retriever() for search, as_query_engine() for answers and as_chat_engine() for conversations.
Other index types exist, such as SummaryIndex (reads every node, good for summarising a small set) and PropertyGraphIndex (a knowledge graph), but start with the vector index.
Example
from llama_index.core import Document, VectorStoreIndex
docs = [
Document(text="Refunds take 5 business days.", id_="faq-refunds"),
Document(text="Shipping is free over 50 EUR.", id_="faq-shipping"),
]
index = VectorStoreIndex.from_documents(docs, show_progress=True)
index.insert(Document(text="Support is open Mon-Fri.", id_="faq-support"))
print(len(index.docstore.docs)) # 3 nodes
index.delete_ref_doc("faq-shipping", delete_from_docstore=True)
print(len(index.docstore.docs)) # 2
changed = index.refresh_ref_docs([Document(text="Refunds take 3 business days.", id_="faq-refunds")])
print(changed) # [True] (text changed, re-indexed)
retriever = index.as_retriever(similarity_top_k=1)
print(retriever.retrieve("money back")[0].node.text) # Refunds take 3 business days.Remember
VectorStoreIndex.from_documents(docs)orVectorStoreIndex(nodes)- Insert, delete and refresh documents by stable ids
- Create retrievers, query engines and chat engines from the index
Watch out for
- Rebuilding the whole index from scratch on every change instead of inserting or refreshing changed documents.