TopolearnDashboards show totals per service or instance, and joining metrics needs explicit label matching.
Aggregation operators reduce many series to fewer: sum, avg, min, max, count, stddev, topk, bottomk, quantile, count_values, group.
sum by (job, status) (...)keeps only the listed labels.sum without (instance) (...)drops the listed labels and keeps the rest; handy when you don't know every label.topk(5, ...)returns the 5 largest series, with their labels intact.
Binary operators between two vectors match series with identical label sets by default. When the label sets differ, say how to match:
on(label, ...): match only on these labels.ignoring(label, ...): match on all labels except these.group_left/group_right: allow many-to-one matching, and copy extra labels from the "one" side. The classic case is joining an info metric:* on(instance) group_left(version) app_build_infoadds theversionlabel to every series.
Without these, mismatched labels produce an empty result, with no error. If a many-to-many match occurs, the query fails with an error.
Aggregation removes the metric name. Also note that avg of per-instance ratios is not the same as the ratio of sums: compute ratios from summed rates, not averages of ratios.
Example
# Requests per second by status code, all instances summed
sum by (status) (rate(http_requests_total{job="api"}[5m]))
# Same, but keep every label except instance and pod
sum without (instance, pod) (rate(http_requests_total[5m]))
# The 3 busiest endpoints
topk(3, sum by (path) (rate(http_requests_total[5m])))
# CPU usage percentage per instance
100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))
# Attach the version label from an info metric (value 1) to each instance's rate
sum by (instance) (rate(http_requests_total[5m]))
* on(instance) group_left(version) app_build_infoRemember
- by (...) keeps labels; without (...) drops them
- Binary ops match identical label sets unless on/ignoring says otherwise
- group_left joins many-to-one and copies labels from info metrics
Watch out for
- Dividing two vectors with different labels and getting an empty result instead of an error.