TopolearnEvery growing Prometheus deployment eventually hits limits; knowing the options avoids costly redesigns.
Cardinality (the number of active series) is the main scaling limit. Find offenders on the Status, TSDB page (top metrics and label pairs by series count), with promtool tsdb analyze, or with queries like topk(10, count by (__name__) ({__name__=~".+"})). Fix them at the source (remove unbounded labels) or drop them with metric_relabel_configs. Per-scrape safety nets such as sample_limit and label_limit fail a scrape that suddenly explodes instead of taking the server down.
Scaling options, roughly in order of complexity:
- Functional sharding: separate Prometheus servers per team, cluster or job. Simple and robust.
- hashmod sharding: several servers split one large job's targets via relabeling.
- Federation: a global Prometheus scrapes
/federateon others for selected, preferably pre-aggregated, series (match[]parameters). Fine for summaries, not for copying everything. - Remote write: stream samples to a long-term, horizontally scalable backend such as Thanos, Grafana Mimir, Cortex or VictoriaMetrics, which provides a global query view, deduplication of HA pairs and years of retention on object storage. Prometheus 3.x can also send Remote Write 2.0, a more compact protocol that carries metadata and native histograms.
- Agent mode (
--agent): Prometheus only scrapes and remote-writes, without local querying, using far less memory.
For high availability run two identical Prometheus servers scraping the same targets. Each evaluates the same rules and sends alerts to a clustered Alertmanager, which deduplicates. Give them the same external_labels plus one replica label so the remote backend can dedup samples. Watch prometheus_remote_storage_* metrics (pending samples, failed sends) for remote write backlogs.
Example
# prometheus.yml (excerpt) for an HA pair writing to Mimir or Thanos Receive
global:
external_labels:
cluster: eu-1
__replica__: prom-a # prom-b on the twin; backend dedups
scrape_configs:
- job_name: api
sample_limit: 50000 # fail the scrape if it explodes
static_configs: [{targets: ["api:8000"]}]
remote_write:
- url: https://mimir.example.com/api/v1/push
queue_config:
max_samples_per_send: 5000
write_relabel_configs: # don't ship debug metrics
- source_labels: [__name__]
regex: debug_.*
action: drop
# Find the biggest metrics (PromQL):
# topk(10, count by (__name__) ({__name__=~".+"}))Remember
- Active series count is the key limit; find it with TSDB status and topk count
- Shard by function first; federate only aggregates
- Remote write to Thanos/Mimir for global view, HA dedup and long retention
Watch out for
- Federating all raw series into one global server, recreating the original scaling problem.