TopolearnEvery spaCy result lives on these three objects, so navigating them fluently is the core skill.
Calling nlp(text) returns a `Doc`: a sequence of tokens plus all the annotations the pipeline added. Three objects matter:
- `Doc`: the container.
len(doc)counts tokens; iterate or index it like a list. - `Token`:
doc[i], a view of one token.token.text,token.i(index in the doc),token.idx(character offset) andtoken.whitespace_(the trailing space, if any). - `Span`:
doc[start:end], a slice of tokens. Spans are views too, with.text,.start,.end,.start_char,.end_char, and they're how sentences, entities and matches are represented.
spaCy's tokenization is non-destructive: each token remembers its trailing whitespace, so doc.text reproduces the input exactly, and token.idx lets you map results back to character positions for highlighting.
The tokenizer splits on whitespace, then applies language-specific rules for prefixes, suffixes, infixes and exceptions, which is why "U.K." stays one token while "don't" becomes "do" and "n't".
Tokens and spans don't own data; they point into the Doc. Keep the Doc alive while using them, and store token.i or character offsets rather than the objects if you need to persist positions.
To get a span from character offsets, for example from an external annotation, use doc.char_span(start, end). It returns None if the offsets don't align with token boundaries; alignment_mode="expand" snaps to the surrounding tokens.
Example
import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("Don't ship it. Test it first!")
print(len(doc), [t.text for t in doc])
# 10 ['Do', "n't", 'ship', 'it', '.', ' ', 'Test', 'it', 'first', '!']
tok = doc[2]
print(tok.text, tok.i, tok.idx, repr(tok.whitespace_)) # ship 2 6 ' '
print(doc.text == "".join(t.text_with_ws for t in doc)) # True: non-destructive
span = doc[6:9]
print(span.text, span.start, span.end, span.start_char) # Test it first 6 9 16
print(doc.char_span(16, 23)) # Test it
print(doc.char_span(17, 23)) # None: not a token boundary
print(doc.char_span(17, 23, alignment_mode="expand")) # Test itRemember
- nlp(text) returns a Doc; doc[i] is a Token; doc[a:b] is a Span
- Tokens keep trailing whitespace, so the original text is recoverable
- token.i is the token index, token.idx the character offset
- doc.char_span maps character offsets to a Span (or None)
Watch out for
- Storing Token or Span objects long-term after the Doc is gone; store indices or offsets instead.