TopolearnWhen Grafana becomes the place where incidents are handled, it can't be a single SQLite file on one VM.
A default Grafana stores everything in SQLite on local disk. That's fine for one instance, but high availability needs:
- A shared MySQL or PostgreSQL database (
[database]section) for dashboards, users, sessions and alert state. - Several stateless Grafana instances behind a load balancer, all with the same configuration and secret key.
- Alerting HA: instances must coordinate so each alert notifies only once. Configure
[unified_alerting]withha_peers(memberlist gossip between instances) or a Redis backend. Rule evaluation and notifications are then deduplicated. - An optional remote cache (Redis or Memcached) for sessions and caching.
Performance at scale is mostly about queries, since every dashboard viewer triggers them:
- Limit panels per dashboard and auto-refresh frequency; a 30-panel dashboard at 5s refresh on a wall screen generates constant load.
- Use recording rules or downsampled data for long ranges.
- Query caching (Enterprise and Cloud) and data source-level caching reduce repeated queries.
- Dashboards in Git (Git Sync or provisioning) make rebuilding an instance quick.
Monitor Grafana itself: it exposes Prometheus metrics at /metrics, including HTTP request durations and alerting metrics. Back up the database, and keep dashboards in code so a rebuild is quick.
Example
# grafana.ini for each of several instances behind a load balancer
[database]
type = postgres
host = grafana-db.internal:5432
name = grafana
user = grafana
password = $__env{GF_DB_PASSWORD}
[security]
; must be identical on all instances
secret_key = $__env{GF_SECRET_KEY}
[unified_alerting]
enabled = true
ha_listen_address = $__env{POD_IP}:9094
ha_peers = grafana-0.grafana:9094,grafana-1.grafana:9094,grafana-2.grafana:9094
[remote_cache]
type = redis
connstr = addr=redis.internal:6379,pool_size=100,db=0Remember
- HA needs a shared MySQL/PostgreSQL database and identical secret keys
- ha_peers lets alerting instances deduplicate notifications
- Query load from dashboards is the main scaling factor
Watch out for
- Running multiple instances on separate SQLite files; each shows different dashboards and alerts fire several times.