TopolearnRetrieval quality depends heavily on how documents are loaded and chunked.
A Document (langchain_core.documents.Document) has page_content (text) and metadata (a dict such as source, page, URL, date). Metadata travels with every chunk and is how you cite sources or filter results later.
Document loaders create Documents from files and services: PyPDFLoader, CSVLoader, WebBaseLoader, DirectoryLoader and hundreds more in langchain-community and partner packages. They expose .load() and the lazy .lazy_load() for large sources. You can also build Documents yourself from any data.
Long documents must be split into chunks, because embeddings represent short passages best and prompts have limits. RecursiveCharacterTextSplitter from langchain-text-splitters is the default choice: it tries to split on paragraphs, then lines, then sentences and words, keeping chunks under chunk_size characters with chunk_overlap shared between neighbours so context is not cut mid-thought.
Choosing parameters:
- Start around 500 to 1,000 characters with 10 to 20% overlap, then evaluate retrieval.
- Use
from_tiktoken_encoderor a tokenizer-based length function to size by tokens. - Use structure-aware splitters for Markdown headers, HTML or code (
RecursiveCharacterTextSplitter.from_language). add_start_index=Truerecords each chunk's position in the original text.
Split documents with split_documents(docs) so metadata is copied onto each chunk.
Example
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
text = ("LangChain splits text recursively.\n\n"
"First it tries paragraphs, then lines, then sentences. "
"Overlap keeps context between neighbouring chunks.\n\n"
"Metadata such as the source file is copied to every chunk.")
docs = [Document(page_content=text, metadata={"source": "guide.md"})]
splitter = RecursiveCharacterTextSplitter(chunk_size=80, chunk_overlap=20,
add_start_index=True)
chunks = splitter.split_documents(docs)
for c in chunks:
print(len(c.page_content), c.metadata)
# 34 {'source': 'guide.md', 'start_index': 0}
# 76 {'source': 'guide.md', 'start_index': 36}
# ... every chunk keeps the source plus its own start_index
print(all(len(c.page_content) <= 80 for c in chunks)) # TrueRemember
- Documents = page_content + metadata; keep metadata for citations and filters
- RecursiveCharacterTextSplitter splits on natural boundaries with overlap
- Tune chunk size and overlap by measuring retrieval, not by guesswork
Watch out for
- Splitting raw strings and losing the source metadata, so answers cannot cite where information came from.