TopolearnKnowing what the engine does explains which patterns are fast and how to debug confusing ones.
re.compile parses the pattern into a tree (in re._parser), compiles it into opcodes for a C matching engine (_sre), and returns a Pattern object. Module-level calls like re.search(p, s) look up p in an internal cache keyed by pattern, type and flags, so repeated calls don't recompile. The cache holds a few hundred patterns; programs that generate many distinct patterns evict each other and recompile constantly. Use re.compile for patterns built dynamically or used in hot loops, and re.purge() clears the cache.
Pass re.DEBUG to print the parse tree. It's a quick way to see how precedence and quantifiers were really understood, for example whether | split where you thought.
Performance insights from the engine's design:
- A pattern starting with a literal prefix lets the engine skip quickly to candidate positions.
- Alternations of single characters (
a|e|i) are optimised into a character set automatically, but multi-character alternations become a BRANCH whose arms are tried one by one at every position. - An anchored pattern (
^,match,fullmatch) fails quickly instead of retrying at every start position. - Plain string methods (
in,startswith,str.replace) beat regex for fixed text.
The re engine is a backtracking implementation in C and doesn't release the GIL during matching, so CPU-heavy regex work across threads doesn't run in parallel on the default build.
Example
import re
import timeit
re.compile(r"(?:a|e|i)+", re.DEBUG)
# MAX_REPEAT 1 MAXREPEAT
# IN <- single-char alternation became a set
# LITERAL 97
# LITERAL 101
# LITERAL 105
# (followed by the compiled opcode listing)
text = "x" * 10_000 + "needle"
t_regex = timeit.timeit(lambda: re.search(r"needle", text), number=2000)
t_str = timeit.timeit(lambda: "needle" in text, number=2000)
print(t_str < t_regex) # True: fixed text needs no regex
pats = [re.compile(fr"id{i}=(\d+)") for i in range(3)] # compile dynamic ones once
print([p.search("id1=7").group(1) if p.search("id1=7") else None for p in pats])
# [None, '7', None]Remember
- Patterns compile to opcodes for a C backtracking engine; results are cached
- Compile dynamic or hot patterns explicitly; use
re.DEBUGto inspect parsing - Literal prefixes and anchors are fast; multi-character alternations are tried branch by branch
Watch out for
- Generating thousands of distinct patterns at runtime and relying on the module cache.