Unicode, bytes and case folding

International text and binary protocols break naive patterns in subtle ways.

The re module works on both str and bytes, but pattern and text must be the same type: re.search(rb"\d+", data) for bytes, r"\d+" for strings. Mixing them raises TypeError.

In str patterns, matching is Unicode-aware by default:

\w matches letters and digits from every script, so café and 東京 are words; \d matches any Unicode decimal digit; \s includes non-breaking and other Unicode spaces.
re.IGNORECASE uses Unicode simple case folding, which brings surprises: the Kelvin sign K (U+212A) matches k, and ſ (long s) matches s. But multi-character foldings are not applied: ß does not match ss.
re.ASCII restricts \w \d \s \b (and case-insensitive matching) to ASCII, which is what you want for identifiers, protocols and machine formats.

In bytes patterns, everything is ASCII-only; non-ASCII bytes are never word characters.

Text that looks identical may differ in code points: é can be one character (U+00E9) or e plus a combining accent. Normalise both pattern input and text with unicodedata.normalize("NFC", s) before matching user-provided text.

For grapheme clusters, Unicode properties like \p{Greek} or full case folding, use the third-party regex module.

Example

import re
import unicodedata

print(re.findall(r"\w+", "naïve café 東京"))          # ['naïve', 'café', '東京']
print(re.findall(r"\w+", "naïve café", re.ASCII))     # ['na', 've', 'caf']
print(re.findall(rb"\d+", b"id=42;len=7"))            # [b'42', b'7']

print(bool(re.fullmatch(r"k", "\u212a", re.I)))       # True: Kelvin sign
print(bool(re.fullmatch(r"ss", "ß", re.I)))           # False

decomposed = "cafe\u0301"                             # e + combining accent
print(re.search(r"café", decomposed))                 # None
nfc = unicodedata.normalize("NFC", decomposed)
print(re.search(r"café", nfc) is not None)            # True
try:
    re.search(r"\d", b"1")
except TypeError as e:
    print(type(e).__name__)                           # TypeError

Remember

str patterns are Unicode-aware; bytes patterns are ASCII-only
Use re.ASCII for machine formats like identifiers and protocols
Normalise text with unicodedata before matching user input

Watch out for

Validating usernames with \w+ and accidentally allowing look-alike characters from other scripts.

Check yourself

Why might r'^\w+$' be a poor choice to validate ASCII-only usernames?

Official docs: docs.python.org/3/library/re.html#re.ASCII

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks