TopolearnWhat you load and how determines everything downstream; garbage in means garbage answers.
A reader (also called a data connector) turns a source into a list of Document objects.
`SimpleDirectoryReader` handles local files and is built in. It picks a parser by file extension: plain text and Markdown, PDF, Word, PowerPoint, CSV, images and more (some types need extra packages such as pypdf). Useful options:
input_dir="data"orinput_files=["a.pdf", "b.md"].recursive=Trueto include subfolders.required_exts=[".md", ".pdf"]to load only those types.exclude=["*.tmp"]to skip files.filename_as_id=Truegives each document a stable id based on its path, which matters for updating an index later.num_workers=4loads files in parallel.
Each document gets metadata automatically: file_name, file_path, file_type, size and dates. Metadata travels with every chunk and is used for citations and filters.
For other sources, LlamaHub (the integrations catalogue) offers hundreds of readers, each a separate package: web pages, Notion, Google Drive, Slack, databases and more. They all return Document lists, so the rest of your code stays the same.
Complex PDFs with tables, columns or scanned pages often come out messy with simple parsers. LlamaParse (a later lesson) is LlamaIndex's hosted parser for those.
For huge folders, use reader.iter_data() to process one file at a time instead of loading everything into memory.
Example
from llama_index.core import SimpleDirectoryReader
reader = SimpleDirectoryReader(
input_dir="data",
recursive=True, # include subfolders
required_exts=[".md", ".txt", ".pdf"],
filename_as_id=True, # stable ids from file paths
)
docs = reader.load_data()
print(len(docs)) # e.g. 12
print(docs[0].metadata["file_name"]) # refunds.md
print(docs[0].doc_id) # .../data/refunds.md
print(docs[0].text[:40]) # Refunds are processed within 5 busin
for file_docs in reader.iter_data(): # one file at a time for big folders
passRemember
SimpleDirectoryReaderloads many file types and adds file metadata- Use
recursive,required_extsandfilename_as_id - LlamaHub readers load other sources into the same Document type
Watch out for
- Loading a whole folder including backups and temporary files, which fills the index with duplicates and junk.