TopolearnCorpora give you labelled text to practise on, and corpus readers are how NLTK loads any collection of documents.
NLTK ships access to dozens of corpora through nltk.corpus, each downloaded on first use like other data. A corpus reader exposes a consistent API:
fileids(): the documents in the corpusraw(fileid): the text as one stringwords(fileid): a list of word tokens;sents(fileid)a list of tokenized sentencescategories()andwords(categories="news")for categorised corpora such asbrownandmovie_reviews- tagged corpora add
tagged_words()andtagged_sents()
Some favourites: gutenberg (classic books), brown (a balanced corpus of 15 genres, POS-tagged), movie_reviews (2000 labelled positive/negative reviews), reuters, inaugural and treebank.
Reader methods return lazy corpus views: they look like lists but stream from disk on demand, so even big corpora don't fill memory until you iterate or convert with list().
To use your own documents, point a reader at a folder: PlaintextCorpusReader(root, r".*\.txt") tokenizes plain text files, and CategorizedPlaintextCorpusReader derives categories from file names or folders with a cat_pattern. Since NLTK 3.10, readers only open files under trusted data roots, so add your folder to nltk.data.path (or NLTK_DATA) first. You then get the same words(), sents() and categories() interface, which plugs straight into frequency counts and classifiers.
nltk.Text(words) wraps tokens for exploration: concordance("word") shows each occurrence in context, similar() and collocations() find related words.
Example
import os
import nltk
from nltk.corpus import brown, gutenberg, movie_reviews
from nltk.corpus.reader import PlaintextCorpusReader
print(gutenberg.fileids()[:3]) # ['austen-emma.txt', 'austen-persuasion.txt', 'austen-sense.txt']
print(len(gutenberg.words("austen-emma.txt"))) # 192427
print(brown.categories()[:4]) # ['adventure', 'belles_lettres', 'editorial', 'fiction']
print(brown.tagged_words(categories="news")[:2]) # [('The', 'AT'), ('Fulton', 'NP-TL')]
print(movie_reviews.categories()) # ['neg', 'pos']
print(len(movie_reviews.fileids("pos"))) # 1000
os.makedirs("mydocs", exist_ok=True)
nltk.data.path.append(os.path.abspath("mydocs")) # NLTK 3.10+: trusted roots only
with open("mydocs/a.txt", "w") as f:
f.write("Small corpora are easy. Readers tokenize for you.")
mine = PlaintextCorpusReader("mydocs", r".*\.txt")
print(mine.fileids(), len(mine.sents("a.txt"))) # ['a.txt'] 2
print(mine.words()[:4]) # ['Small', 'corpora', 'are', 'easy']Remember
- Corpus readers share fileids, raw, words, sents and categories
- Results are lazy views that stream from disk
- brown, movie_reviews and treebank come labelled or tagged
- PlaintextCorpusReader turns your own folder into a corpus
Watch out for
- Calling list() on very large corpus views repeatedly instead of iterating once.