TopolearnRules plus a model is the standard way to recognise domain entities the pretrained NER doesn't know.
The `entity_ruler` is a pipeline component that turns patterns into entities in doc.ents. Patterns are dicts with a label and either a phrase string or a token pattern:
{"label": "ORG", "pattern": "Acme Corp"}
{"label": "TICKET", "pattern": [{"TEXT": {"REGEX": "^[A-Z]{2,5}-\\d+$"}}]}Add it with ruler = nlp.add_pipe("entity_ruler", before="ner") and ruler.add_patterns(patterns). Placement decides who wins:
before="ner": rule entities are set first and the statistical NER respects them, predicting around them. Use this when rules are trustworthy.- after
ner(the default end of the pipeline): rules only fill gaps, unless you setconfig={"overwrite_ents": True}.
An id field in a pattern sets ent.ent_id_, which lets several surface forms (such as "NYC" and "New York City") resolve to one canonical entity. Phrase patterns are tokenized, so they only match text that tokenizes the same way; check odd strings such as product codes with nlp.make_doc.
Patterns can be saved and loaded as JSONL with ruler.to_disk("patterns.jsonl") and ruler.from_disk(...), and they are saved with the pipeline by nlp.to_disk. Phrase patterns use the PhraseMatcher internally, so thousands of names stay fast; set phrase_matcher_attr="LOWER" in the config for case-insensitive matching.
For overlapping or nested annotations that shouldn't go into doc.ents, the `span_ruler` writes to doc.spans["ruler"] instead.
Example
import spacy
nlp = spacy.load("en_core_web_sm")
ruler = nlp.add_pipe("entity_ruler", before="ner",
config={"phrase_matcher_attr": "LOWER"})
ruler.add_patterns([
{"label": "PRODUCT", "pattern": "widgetpro 3000", "id": "wp3000"},
{"label": "PRODUCT", "pattern": "widget pro", "id": "wp3000"},
{"label": "TICKET", "pattern": [{"TEXT": {"REGEX": r"^[A-Z]{2,5}-\d+$"}}]},
])
print(nlp.pipe_names[-2:]) # ['entity_ruler', 'ner']
doc = nlp("Sarah from Acme reported OPS-1432 about the WidgetPro 3000 and the Widget Pro.")
for e in doc.ents:
print(f"{e.text:15} {e.label_:8} {e.ent_id_}")
# Sarah PERSON
# Acme GPE (statistical NER: wrong here, add a rule!)
# OPS-1432 TICKET
# WidgetPro 3000 PRODUCT wp3000
# Widget Pro PRODUCT wp3000Remember
- entity_ruler adds pattern-based entities to doc.ents
- Put it before 'ner' to have rules take priority; after it to fill gaps
- Pattern ids map several spellings to one canonical entity
- span_ruler writes overlapping spans to doc.spans
Watch out for
- Adding the ruler after 'ner' and wondering why rule matches that overlap model entities are ignored.