Canonical JSON for hashing and signing

Content hashes, cache keys and signatures break if the same data can serialise in different ways.

Two equal dicts can produce different JSON text: key order follows insertion order, whitespace depends on options, and non-ASCII characters may or may not be escaped. Anything that hashes or signs JSON needs a canonical form.

A practical canonical encoding with the stdlib:

json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")

That fixes key order, removes optional whitespace and fixes the text encoding. Remaining sources of ambiguity to handle yourself:

Floats: Python writes the shortest repr (0.1), but other languages may write 1e-7 differently from Python's 1e-07. Avoid floats in signed payloads, or use strings or integers (cents, not dollars).
Unicode normalisation: "é" can be one code point or e plus a combining accent. Normalise with unicodedata.normalize("NFC", s) if inputs come from different sources.
Types: tuples and lists both become arrays; int keys become strings. Canonicalise after converting to JSON-native types.
Sets: sort them.

Standards exist for cross-language interoperability, notably RFC 8785 (JSON Canonicalization Scheme), which specifies number formatting precisely. For signing, sign the exact bytes you send and verify the bytes you receive, rather than re-serialising.

Example

import hashlib
import json

def canonical(obj):
    return json.dumps(obj, sort_keys=True, separators=(",", ":"),
                      ensure_ascii=False).encode("utf-8")

a = {"user": "zoë", "roles": ["admin"], "id": 7}
b = {"id": 7, "roles": ["admin"], "user": "zoë"}

print(json.dumps(a) == json.dumps(b))          # False
print(canonical(a) == canonical(b))            # True
print(canonical(a))
# b'{"id":7,"roles":["admin"],"user":"zo\xc3\xab"}'
digest = hashlib.sha256(canonical(a)).hexdigest()
print(len(digest))                             # 64

print(json.dumps(1e-7))                        # 1e-07

Remember

Use sort_keys=True, compact separators and a fixed encoding for canonical bytes
Floats and Unicode normalisation are remaining sources of ambiguity
Sign or hash the exact bytes; RFC 8785 standardises canonical JSON

Watch out for

Hashing json.dumps(obj) with default options, so equal data built in a different order gets a different hash.

Check yourself

Which source of difference does sort_keys=True NOT remove?

Official docs: docs.python.org/3/library/json.html#json.dumps

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks