TopolearnSmall parsers for config formats, query languages and calculators start with a tokenizer.
A tokenizer turns text into a stream of typed tokens such as numbers, names and operators. With re, the classic recipe from the Python docs is:
1. Write one named group per token type. 2. Join them with | into a master pattern, ordered so longer or more specific tokens come first (>= before >, keywords before names). 3. Iterate with finditer; m.lastgroup tells you which alternative matched. 4. Include a catch-all group such as MISMATCH=r"." so unexpected characters are reported instead of silently skipped, and a SKIP group for whitespace.
Because finditer searches rather than matching contiguously, an input character not covered by any alternative would be skipped. The catch-all closes that gap. Alternatively, call pattern.match(text, pos) in a loop, which anchors each token at the current position.
Track positions with m.start() to produce error messages that point at the problem, and line numbers by counting newlines in SKIP or NEWLINE tokens.
For keywords, match a generic name and then look it up in a set of keywords; that's simpler and avoids if being matched inside iffy.
Beyond tokenizing, write a recursive-descent parser on top of the tokens, or reach for a parser library when the grammar grows.
Example
import re
SPEC = [("NUMBER", r"\d+(?:\.\d*)?"), ("NAME", r"[A-Za-z_]\w*"),
("OP", r">=|<=|==|[-+*/=<>()]"), ("SKIP", r"\s+"), ("MISMATCH", r".")]
MASTER = re.compile("|".join(f"(?P<{n}>{p})" for n, p in SPEC))
KEYWORDS = {"if", "then"}
def tokens(text):
for m in MASTER.finditer(text):
kind, value = m.lastgroup, m.group()
if kind == "SKIP":
continue
if kind == "MISMATCH":
raise SyntaxError(f"unexpected {value!r} at {m.start()}")
if kind == "NAME" and value in KEYWORDS:
kind = value.upper()
yield kind, value
print(list(tokens("if x >= 2.5 then y")))
# [('IF', 'if'), ('NAME', 'x'), ('OP', '>='), ('NUMBER', '2.5'), ('THEN', 'then'), ('NAME', 'y')]
try:
list(tokens("a = 3 $ 4"))
except SyntaxError as e:
print(e) # unexpected '$' at 6Remember
- Join named alternatives into one master pattern;
m.lastgroupgives the token type - Order alternatives so longer operators come first
- Add SKIP and MISMATCH groups so nothing is silently ignored
Watch out for
- Omitting a catch-all group, so finditer silently skips characters no rule matches.