Instrumenting a Go service

Go is the native language of the Prometheus ecosystem; most exporters and cloud-native services are instrumented this way.

github.com/prometheus/client_golang is the official library. The pieces:

prometheus.NewCounterVec(prometheus.CounterOpts{Name, Help}, []string{"route", "code"}) and the Gauge, Histogram and Summary equivalents. The Vec variants take label names and give a child per label-value set via .WithLabelValues(...).
The registry. promauto registers metrics on the default registry as they are created; prometheus.MustRegister does it explicitly, and prometheus.NewRegistry() gives you a clean one when you don't want the built-in go_* and process_* collectors.
promhttp.Handler() serves /metrics for the default registry; promhttp.HandlerFor(reg, opts) for a custom one.

For HTTP servers the promhttp package has middleware: promhttp.InstrumentHandlerDuration, InstrumentHandlerCounter and InstrumentHandlerInFlight wrap a handler and fill a metric you define, with method and code labels derived automatically. Add a route label yourself with a prometheus.Labels curried vec: dur.MustCurryWith(prometheus.Labels{"route": "/orders"}).

Declare metrics as package-level variables so they are registered once; registering the same name twice panics. Call .WithLabelValues() on the hot path (it's a fast lookup) but avoid building a new label value per request.

To expose data you already hold (a cache size, a pool's state) implement the Collector interface (Describe and Collect) and compute the values on each scrape instead of polling in a goroutine. The collectors package has ready-made ones for database/sql and the Go runtime.

Histograms accept NativeHistogramBucketFactor for native histograms, covered later.

Example

package main

import (
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
	reqs = promauto.NewCounterVec(prometheus.CounterOpts{
		Name: "shop_http_requests_total", Help: "HTTP requests handled.",
	}, []string{"route", "method", "code"})
	dur = promauto.NewHistogramVec(prometheus.HistogramOpts{
		Name: "shop_http_request_duration_seconds", Help: "Request latency.",
		Buckets: []float64{0.05, 0.1, 0.25, 0.5, 1, 2.5},
	}, []string{"route", "method", "code"})
)

func instrument(route string, h http.Handler) http.Handler {
	l := prometheus.Labels{"route": route}
	return promhttp.InstrumentHandlerDuration(dur.MustCurryWith(l),
		promhttp.InstrumentHandlerCounter(reqs.MustCurryWith(l), h))
}

func main() {
	http.Handle("/orders", instrument("/orders", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("ok"))
	})))
	http.Handle("/metrics", promhttp.Handler()) // go_* and process_* come for free
	http.ListenAndServe(":8080", nil)
}
// curl -s localhost:8080/metrics | grep shop_http_requests_total
// shop_http_requests_total{code="200",method="get",route="/orders"} 3

Remember

promauto + package-level vars: create once, register once
promhttp.Handler() exposes /metrics; Instrument* middleware fills request metrics
Implement Collector to compute values at scrape time

Watch out for

Creating metrics inside a handler, which panics with 'duplicate metrics collector registration'.

Check yourself

You want to expose the current size of an in-memory cache without a background goroutine. Best approach?

Official docs: prometheus.io/docs/guides/go-application/

Example

Remember

Watch out for

Check yourself

Builds on

Unlocks