TopolearnVector similarity powers quick semantic search, deduplication and clustering without training anything.
The md and lg pipelines include static word vectors: 300-dimensional embeddings. lg stores several hundred thousand distinct vectors; md keeps a pruned table of 20,000 that many words share, so its similarities are coarse (even unrelated words can end up with identical vectors). token.vector returns it as a NumPy array, token.has_vector and token.is_oov report coverage, and nlp.vocab.vectors.shape shows the table size.
Doc.vector and Span.vector default to the average of their token vectors. .similarity(other) computes cosine similarity between vectors, available on tokens, spans and docs.
Useful, but know the limits:
- Averaging blurs meaning. Word order is lost, and long docs drift towards a generic average. "The dog bit the man" and "The man bit the dog" score as identical.
- Static vectors give one vector per word, mixing senses ("bank" of a river and of money).
- Similar is not synonymous: "good" and "bad", or "cat" and "dog", are close because they appear in similar contexts.
- The
smmodels have no real vectors; similarity there falls back to context tensors, with a warning, and is unreliable.
For serious semantic similarity and search, sentence-embedding models (via sentence-transformers or a transformer pipeline) are far better. spaCy's vectors remain handy as cheap features, for quick prototypes and for finding related terms with nlp.vocab.vectors.most_similar.
Example
import numpy as np
import spacy
nlp = spacy.load("en_core_web_lg") # lg: full table of word vectors
print(nlp.vocab.vectors.shape[1]) # 300 dimensions
cat, dog, car = nlp("cat dog car")
print(cat.vector.shape, cat.has_vector) # (300,) True
print(round(cat.similarity(dog), 2), round(cat.similarity(car), 2))
# cat-dog clearly higher than cat-car
a = nlp("The dog bit the man.")
b = nlp("The man bit the dog.")
print(round(a.similarity(b), 3)) # 1.0 word order ignored!
print(round(nlp("good").similarity(nlp("bad")), 2)) # high: related, not synonyms
keys, _, scores = nlp.vocab.vectors.most_similar(
np.asarray([nlp.vocab["coffee"].vector]), n=4)
print([nlp.vocab.strings[k] for k in keys[0]]) # e.g. coffee-related wordsRemember
- lg ships a full static vector table, md a pruned one, sm none
- Doc and Span vectors average token vectors; similarity is cosine
- Averages ignore word order and conflate senses
- Use sentence-embedding models for serious semantic search
Watch out for
- Computing similarity with en_core_web_sm and trusting the numbers.