Prometheus 3.x: UTF-8 names and OTLP

Most tutorials still describe 2.x; knowing the breaking changes avoids silent query differences and failed upgrades.

Prometheus 3.0 (November 2024) was the first major release in seven years. The headline changes, continued in 3.1 to 3.5 (the 3.5 LTS release):

UTF-8 names. Metric and label names may contain any UTF-8 characters. In PromQL and the exposition format quote them: {"service.name"="api"}. The old restriction is kept for targets and clients that can't cope via global.metric_name_validation_scheme: legacy and per-target escaping negotiated in the Accept header.
OTLP receiver. With --web.enable-otlp-receiver, Prometheus accepts OpenTelemetry metrics on /api/v1/otlp/v1/metrics, so collectors and SDKs can push directly. The otlp: config block controls translation_strategy (UnderscoreEscapingWithSuffixes, the default, rewrites http.server.duration to http_server_duration_seconds; NoUTF8EscapingWithSuffixes keeps dots; NoTranslation keeps the names untouched) and promote_resource_attributes to turn chosen resource attributes such as service.name into labels instead of leaving them in target_info.
Left-open range selectors. A [5m] window now excludes a sample exactly at its left edge, so a window holds a predictable number of samples; rate results can differ slightly from 2.x.
Strict scrape protocols. Responses without Content-Type fail unless fallback_scrape_protocol is set; scrape_protocols orders the negotiation.
New UI (React), with the old one gone in later 3.x; --enable-feature=old-ui was temporary.
Agent mode and the @ and negative-offset modifiers are no longer feature flags; holt_winters became double_exponential_smoothing behind promql-experimental-functions; Remote Write 2.0 can be enabled per endpoint.
Removed flags and settings: deprecated --storage.tsdb.retention, several --alertmanager.* and --storage.remote.* flags; check the release notes with promtool check config.

Upgrading: go to 2.55 first, then 3.x; 3.x TSDB data can still be read by 2.55 if you must roll back. Regex matchers with . no longer match newlines in some edge cases, and alerts that relied on sample-at-edge behaviour may need their windows widened.

Example

# prometheus.yml additions for OpenTelemetry data
otlp:
  promote_resource_attributes:
    - service.name
    - service.namespace
    - k8s.pod.name
  translation_strategy: UnderscoreEscapingWithSuffixes   # or NoUTF8EscapingWithSuffixes
storage:
  tsdb:
    out_of_order_time_window: 30m    # OTLP pushes may arrive late

# prometheus --config.file=prometheus.yml --web.enable-otlp-receiver
# OTel collector / SDK endpoint: http://prometheus:9090/api/v1/otlp

# PromQL after NoUTF8EscapingWithSuffixes (quoted UTF-8 names):
#   histogram_quantile(0.9, sum by (le) (
#     rate({"http.server.request.duration_seconds_bucket", "service.name"="api"}[5m])))
# Default strategy gives the classic form instead:
#   http_server_request_duration_seconds_bucket{service_name="api"}

Remember

UTF-8 names with quoted syntax; legacy validation available
--web.enable-otlp-receiver plus the otlp: block for OpenTelemetry push
Left-open ranges and strict Content-Type can change behaviour
Upgrade via 2.55; keep rollback possible

Watch out for

Upgrading straight from an old 2.x release without passing through 2.55, which may not read the TSDB correctly.

Check yourself

After enabling the OTLP receiver, service.name shows up only in target_info, not on your metrics. What fixes it?

Official docs: prometheus.io/docs/prometheus/latest/migration/

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks