Unicode in regular expressions

User names, addresses and emoji break ASCII-only patterns in ways that are easy to miss in testing.

Modern engines operate on code points, but what they treat as a letter or digit depends on flags and flavour:

Python 3 str patterns are Unicode-aware by default: \w matches é and ж, \d matches all decimal digits. re.ASCII restricts them.
JavaScript \w and \d are ASCII-only even with the u flag.
\p{...} property classes select by Unicode category or script: \p{L} any letter, \p{Lu} uppercase, \p{N} numbers, \p{Script=Greek}. They work in PCRE, Java, .NET, Rust, and JavaScript with u or v. Python's re lacks them; the regex package adds them.
JavaScript's u flag makes . match a whole astral code point such as an emoji instead of half a surrogate pair. The v flag (ES2024) adds class set operations like [\p{L}--[a-z]] and string properties like \p{RGI_Emoji}.

A user-perceived character (a grapheme) can be several code points: é may be e plus a combining accent, and a family emoji is several code points joined by zero-width joiners. . matches one code point, so it can split them. PCRE and the Python regex module support \X for a grapheme cluster.

Normalise text first (unicodedata.normalize("NFC", s)) so that visually identical strings compare and match the same way.

Example

import re, unicodedata

print(re.findall(r"\w+", "café naïve Жук"))           # ['café', 'naïve', 'Жук']
print(re.findall(r"\w+", "café", re.ASCII))           # ['caf']

decomposed = "café"                              # e + combining acute
print(len(decomposed), re.fullmatch(r"\w{4}", decomposed))   # 5 None
nfc = unicodedata.normalize("NFC", decomposed)
print(len(nfc), bool(re.fullmatch(r"\w{4}", nfc)))   # 4 True

# JavaScript (node):
#   "Ünïcödé 42".match(/\p{L}+/gu)      -> [ 'Ünïcödé' ]
#   "Ünïcödé".match(/\w+/g)             -> [ 'n', 'c', 'd' ]   ASCII-only \w
#   "😀".match(/./g).length              -> 2   (surrogate halves)
#   "😀".match(/./gu).length             -> 1

Remember

Python's \w and \d are Unicode-aware; JavaScript's are ASCII-only
\p{L} and friends match by Unicode property (JS needs u or v)
Normalise to NFC before matching; graphemes can span several code points

Watch out for

Validating names with [A-Za-z]+, which rejects José, Zoë and most of the world.

Check yourself

In JavaScript, which pattern matches a word in any alphabet?

Official docs: developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Unicode_character_class_escape

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks