TopolearnMost dashboards fail not for lack of data but because nobody can answer "is it broken, and where?" from them in thirty seconds.
Good dashboards answer a question for an audience. Three well-known frameworks tell you which panels belong on them:
- RED for request-driven services: Rate (requests per second), Errors (failed requests per second or error ratio), Duration (latency distribution, p50/p95/p99 or a heatmap). One row of RED per service, split by endpoint below.
- USE for resources (CPU, memory, disk, network, queues): Utilization (how busy), Saturation (how much queued or over capacity), Errors.
- The four golden signals from Google's SRE book: latency, traffic, errors and saturation, which is RED plus saturation.
Organise dashboards as a hierarchy with links between the levels: an overview with one stat row per service (traffic, error ratio, p95, with thresholds), a per-service dashboard with the RED rows and dependencies, and detail dashboards per instance, pod or database. Data links and tag-based dashboard links let you drill down without losing the time range.
Layout conventions:
- Top-left is the most important thing; summary stats first, time series below, tables and logs last.
- Same unit, colour and Y-axis scale for the same quantity everywhere; green/yellow/red thresholds only where they mean something, taken from your SLOs.
- Short titles, a description on every non-obvious panel (shown on hover), and legend names that fit.
- One dashboard per audience and purpose; a wall screen, an on-call view and a capacity planning dashboard are different dashboards.
Finally, use variables for environment and service so you maintain one dashboard, and keep it in version control.
Example
# Service overview row (per $service), RED
# Rate (req/s)
sum(rate(http_requests_total{service="$service"}[$__rate_interval]))
# Error ratio (percentunit, thresholds at SLO: orange 0.1%, red 1%)
sum(rate(http_requests_total{service="$service", status=~"5.."}[$__rate_interval]))
/ sum(rate(http_requests_total{service="$service"}[$__rate_interval]))
# Duration p95 (seconds)
histogram_quantile(0.95, sum by (le) (
rate(http_request_duration_seconds_bucket{service="$service"}[$__rate_interval])))
# Resource row, USE (CPU)
# Utilization
1 - avg(rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[$__rate_interval]))
# Saturation (run queue per core)
node_load1{instance=~"$instance"} / count without (cpu, mode) (node_cpu_seconds_total{mode="idle", instance=~"$instance"})Remember
- RED for services, USE for resources, golden signals = RED + saturation
- Build a drill-down hierarchy: overview, service, instance, linked with time range preserved
- Consistent units, colours and thresholds tied to SLOs
Watch out for
- A dashboard that is a dump of every metric the exporter has, instead of the handful that answer "is it healthy?".