TopolearnTokens are the unit every later step counts, tags or classifies, and naive splitting gets punctuation and abbreviations wrong.
Tokenization splits raw text into units. NLTK's two workhorses:
nltk.sent_tokenize(text): splits into sentences using the pretrained Punkt model, which learned thatDr.orU.S.usually don't end a sentence.nltk.word_tokenize(text): splits each sentence into words and punctuation using Treebank-style rules.
Why not text.split()? Splitting on spaces leaves punctuation attached ("world!") and can't separate contractions. word_tokenize gives "Don't" as ["Do", "n't"] and "$3.50" as ["$", "3.50"], the conventions expected by NLTK's taggers and parsers trained on the Penn Treebank.
Both functions accept language= (for example "german") to use other Punkt models.
Other tokenizers for special cases:
TweetTokenizerkeeps emoticons, hashtags and @handles intact and can shorten elongated words (reduce_len=True).RegexpTokenizer(r"\w+")keeps only what matches a pattern, a quick way to drop punctuation.wordpunct_tokenizesplits on every punctuation boundary.
Tokenizers don't change case. Lowercasing, stopword removal and stemming are separate normalisation steps you choose based on the task, because a sentiment model may need "NOT" and "!" that a topic model would throw away.
Example
from nltk.tokenize import (sent_tokenize, word_tokenize,
TweetTokenizer, RegexpTokenizer)
text = "Dr. Smith paid $3.50 in the U.S. yesterday. Don't worry, it's fine!"
print(sent_tokenize(text))
# ['Dr. Smith paid $3.50 in the U.S. yesterday.', "Don't worry, it's fine!"]
print(word_tokenize("Don't worry, it's fine!"))
# ['Do', "n't", 'worry', ',', 'it', "'s", 'fine', '!']
print("Don't worry, it's fine!".split())
# ["Don't", 'worry,', "it's", 'fine!']
tweet = "@ana loving #NLP sooooo much :-)"
print(TweetTokenizer(strip_handles=True, reduce_len=True).tokenize(tweet))
# ['loving', '#NLP', 'sooo', 'much', ':-)']
print(RegexpTokenizer(r"\w+").tokenize("Hello, world! 42 times."))
# ['Hello', 'world', '42', 'times']Remember
- sent_tokenize uses Punkt to handle abbreviations; word_tokenize uses Treebank rules
- word_tokenize splits punctuation and contractions ('n't', "'s")
- TweetTokenizer and RegexpTokenizer handle social text and custom patterns
- Tokenizing doesn't lowercase or filter; that is a separate step
Watch out for
- Using str.split() and then counting 'fine!' and 'fine' as different words.