TopolearnMost NLTK errors on a new machine are LookupErrors for missing data, not bugs in your code.
NLTK is two things: the Python package (pip install nltk) and a separate collection of data packages such as tokenizer models, tagger weights, word lists and corpora. The code alone can't tokenize a sentence until the relevant data is downloaded.
nltk.download("name") fetches a package into an nltk_data directory, usually in your home folder. Packages you'll need early:
punkt_tab: sentence tokenizer models (replaced the older pickledpunktin NLTK 3.9)stopwords,wordnet,averaged_perceptron_tagger_eng(the POS tagger)vader_lexiconfor sentiment; corpora such asgutenberg,brown,movie_reviews
When something is missing you get a LookupError whose message names the exact package to download. Read it: it is the fix.
nltk.data.path lists the directories searched, in order. Point NLTK at a project-local or shared directory with the NLTK_DATA environment variable, or download to a specific place with nltk.download("stopwords", download_dir=path). In Docker images and CI, download data during the build step, not at request time, so production never depends on network access.
Recent releases harden data loading against path tricks and network abuse. Behind a corporate proxy, newer versions may refuse downloads until you opt in (the error names the setting, such as the NLTK_ALLOW_PROXIED_URLOPEN variable). python -m nltk.downloader stopwords punkt_tab downloads from the command line.
Example
import nltk
for pkg in ["punkt_tab", "stopwords", "wordnet"]:
nltk.download(pkg, quiet=True) # no-op if already present
print(nltk.__version__) # e.g. 3.9.2
print(nltk.data.path[0]) # first directory searched
try:
nltk.data.find("corpora/not_a_real_corpus")
except LookupError as e:
print("LookupError:", "not_a_real_corpus" in str(e)) # True
print(nltk.data.find("corpora/stopwords") is not None) # True: installed
# Shell equivalents:
# python -m nltk.downloader punkt_tab stopwords wordnet
# NLTK_DATA=/opt/nltk_data python app.pyRemember
- The pip package and the data packages are installed separately
nltk.download(name)fetches data; LookupError messages name what is missingNLTK_DATAandnltk.data.pathcontrol where data is found- Download data at build time for servers and containers
Watch out for
- Calling nltk.download inside a web request handler, so production depends on network access and slows down.