TopolearnHashes explain the int-versus-string attributes, shared memory between Docs and why the vocab grows in long-running services.
spaCy never stores the same string twice. The `StringStore` (nlp.vocab.strings) maps every string to a 64-bit hash and back: strings["coffee"] gives the hash, strings[hash] the text. Hashes are deterministic, so the same word has the same id in every process and pipeline, which lets data be exchanged as plain integers.
The `Vocab` holds the StringStore, the vectors table, morphology and Lexemes. A Lexeme is the context-independent entry for a word type, with flags like is_stop, is_alpha, like_num, lower_, shape_ that are the same wherever the word appears. A Token points to its Lexeme for these attributes and adds context-dependent ones (tag, dependency, entity).
Because Docs store token data as compact C structs holding hashes and pointers, they're memory-efficient and doc.to_array([LOWER, POS, ENT_TYPE]) exports attributes as a NumPy uint64 array in one call. doc.count_by(attr) counts values quickly.
A consequence: to decode a hash you need a StringStore that has seen the string. Docs deserialised from bytes in another process need the strings shipped too, which DocBin handles.
In a long-running service, every new string (typos, ids, URLs) is added to the StringStore and stays there, so memory grows slowly. Mitigate by periodically reloading the pipeline, or process with a fresh nlp in worker processes that are recycled.
Example
import spacy
from spacy.attrs import IS_ALPHA, LOWER, POS
nlp = spacy.load("en_core_web_sm")
strings = nlp.vocab.strings
h = strings["coffee"]
print(h) # 3197928453018144401
print(strings[h]) # coffee
print(spacy.blank("de").vocab.strings["coffee"] == h) # True: same hash anywhere
doc = nlp("I love coffee and coffee loves me")
print(doc[2].orth == doc[4].orth == h) # True
lex = nlp.vocab["coffee"]
print(type(lex).__name__, lex.is_alpha, lex.shape_) # Lexeme True xxxx
arr = doc.to_array([LOWER, POS, IS_ALPHA])
print(arr.shape, arr.dtype) # (7, 3) uint64
counts = doc.count_by(LOWER)
print(counts[h]) # 2
n = len(strings)
nlp("zxqv-unique-id-91")
print(len(strings) > n) # True: new strings accumulateRemember
- StringStore maps strings to deterministic 64-bit hashes and back
- Lexemes hold context-free word attributes; Tokens add context
- doc.to_array and count_by export attributes efficiently
- The StringStore grows with every new string in long-running services
Watch out for
- Sending hashes to another process and trying to decode them with a StringStore that never saw those strings.