TopolearnProduction log tools (ELK, Loki, Datadog, CloudWatch) search fields, not free text.
In production, logs are usually shipped to a system that parses them. Plain text needs fragile regexes; JSON lines are parsed for free: one object per line with fields such as time, level, logger, message, request_id and exc.
Implement it with a Formatter subclass whose format(record) builds a dict and returns json.dumps(...):
record.getMessage()merges message and args.self.formatTime(record)orrecord.createdgives the timestamp; prefer UTC ISO 8601.self.formatException(record.exc_info)renders a traceback when there is one.- Copy custom attributes that came from
extra=or filters. One approach: comparerecord.__dict__against the attributes of a blank LogRecord.
Keep the message template constant and put variable data in fields (extra={"order_id": 42}), so the platform can group by message and filter by field.
Third-party packages such as python-json-logger and structlog do this for you; structlog adds bound context and processors. Whatever you choose, ensure json.dumps(..., default=str) so a non-serializable value never kills the log line.
Example
import json, logging, sys
from datetime import datetime, timezone
STD = set(vars(logging.makeLogRecord({}))) | {"message"}
class JsonFormatter(logging.Formatter):
def format(self, record):
out = {
"time": datetime.fromtimestamp(record.created, timezone.utc).isoformat(timespec="seconds"),
"level": record.levelname,
"logger": record.name,
"message": record.getMessage(),
}
out |= {k: v for k, v in vars(record).items() if k not in STD}
if record.exc_info:
out["exc"] = self.formatException(record.exc_info)
return json.dumps(out, default=str)
h = logging.StreamHandler(sys.stdout)
h.setFormatter(JsonFormatter())
log = logging.getLogger("shop"); log.addHandler(h); log.propagate = False
log.warning("payment declined", extra={"order_id": 42})
# {"time": "2026-...+00:00", "level": "WARNING", "logger": "shop",
# "message": "payment declined", "order_id": 42}Remember
- One JSON object per line makes every field searchable
- A Formatter subclass builds the dict; use getMessage() and formatException()
- Keep messages constant and put variables in fields
Watch out for
- json.dumps without default=str, so one datetime or Decimal in extra breaks the log line.