TopolearnProcessing large volumes efficiently is mostly about batching and running only the components you use.
nlp.pipe_names lists the components that run, in order, on every nlp(text) call: the tokenizer first (not listed), then for example tok2vec, tagger, parser, attribute_ruler, lemmatizer, ner. nlp.get_pipe("ner") returns a component; nlp.add_pipe, remove_pipe, replace_pipe and rename_pipe edit the pipeline.
Running unnecessary components wastes time. Three ways to skip them:
spacy.load(name, exclude=["parser", "ner"]): never loadedspacy.load(name, disable=[...]): loaded but switched off; re-enable withnlp.enable_pipewith nlp.select_pipes(enable=["tagger"]):temporarily, for one block
Watch dependencies: the lemmatizer needs POS from the tagger and attribute ruler, sentence boundaries come from the parser, and components that listen to a shared tok2vec need it enabled.
For many texts, never loop over nlp(text). `nlp.pipe(texts, batch_size=...)` streams Docs while batching internally, which is much faster, especially for transformer and GPU pipelines. Pass as_tuples=True with (text, context) pairs to carry ids or metadata through. n_process=2 or more uses multiprocessing on CPU, though each process loads its own copy of the model, so it pays off only for large jobs.
If you only need tokens, nlp.make_doc(text) runs just the tokenizer. nlp.analyze_pipes(pretty=True) reports what each component sets and requires.
Example
import time
import spacy
nlp = spacy.load("en_core_web_sm", exclude=["parser"])
print(nlp.pipe_names) # ['tok2vec', 'tagger', 'attribute_ruler', 'lemmatizer', 'ner']
texts = [f"Order {i} was shipped to Berlin on Monday." for i in range(2000)]
t = time.perf_counter()
docs = [nlp(t_) for t_ in texts[:500]]
loop = time.perf_counter() - t
t = time.perf_counter()
docs = list(nlp.pipe(texts[:500], batch_size=256))
batched = time.perf_counter() - t
print(batched < loop) # True: batching is faster
with nlp.select_pipes(enable=["tok2vec", "ner"]): # only what NER needs
ents = [d.ents for d in nlp.pipe(texts[:3])]
print(ents[0]) # (Order 0, Berlin, Monday)
records = [("Paris is lovely.", {"id": 1}), ("I met Ana Lopez.", {"id": 2})]
for doc, ctx in nlp.pipe(records, as_tuples=True):
print(ctx["id"], [e.text for e in doc.ents]) # 1 ['Paris'] / 2 ['Ana Lopez']Remember
- pipe_names shows the components run on each call
- exclude/disable/select_pipes skip components you don't need
- nlp.pipe batches many texts and is much faster than a loop
- as_tuples carries metadata; n_process parallelises big CPU jobs
Watch out for
- Calling nlp(text) in a loop over millions of texts instead of nlp.pipe.