TopolearnBy default anyone who can reach port 9090 can read every metric and, with lifecycle enabled, reload or shut the server down.
Prometheus assumes a trusted network. Out of the box the API has no authentication, and the metrics it stores often reveal hostnames, versions and business numbers. Three layers to consider:
Serving side. The flag --web.config.file=web.yml enables TLS and authentication for the UI and API (same mechanism in Alertmanager and the exporters built on the exporter toolkit). tls_server_config takes cert_file and key_file (optionally client_auth_type: RequireAndVerifyClientCert with a client_ca_file for mTLS). basic_auth_users maps usernames to bcrypt hashes; generate one with htpasswd -nBC 10 "" | tr -d ':\n'. The file is re-read on every request, so password changes need no restart.
Scraping side. Per job you can set scheme: https, tls_config (ca_file, cert_file, key_file, insecure_skip_verify for dev only), basic_auth or authorization: {type: Bearer, credentials_file: ...}. Prefer the *_file variants so secrets stay out of prometheus.yml, which the Status page displays with secrets masked.
Dangerous endpoints. --web.enable-lifecycle exposes /-/reload and /-/quit; --web.enable-admin-api allows deleting series and taking snapshots. Both are off by default; keep them off unless the server is behind auth.
What Prometheus does not have is authorization: any authenticated user can run any query, read the config and see all targets. Per-team or per-tenant access control means a reverse proxy, or a multi-tenant backend such as Mimir or Thanos with its own tenancy model. Remote write endpoints should likewise sit behind TLS and a credential, since anyone who can write can poison your data.
Example
# web.yml (prometheus --web.config.file=web.yml)
tls_server_config:
cert_file: /etc/prometheus/tls/server.crt
key_file: /etc/prometheus/tls/server.key
basic_auth_users:
# htpasswd -nBC 10 "" | tr -d ':\n' -> paste the bcrypt hash
grafana: $2y$10$3ZkE9bQx1t5tJ7mXl0u9IuK0M9QW0qVb0CGz2nL9Vn6wYd6wL1Y3e
# prometheus.yml: scrape an exporter protected the same way
scrape_configs:
- job_name: node
scheme: https
tls_config:
ca_file: /etc/prometheus/tls/ca.crt
basic_auth:
username: prometheus
password_file: /etc/prometheus/secrets/node_exporter.pw
static_configs: [{targets: ["web1:9100"]}]
# Check it: promtool check web-config web.yml
# curl -u grafana:... --cacert ca.crt https://prom.example.com:9090/api/v1/query?query=upRemember
- --web.config.file gives TLS and basic auth to the UI/API
- Scrape jobs set scheme, tls_config and credential files
- No authorization exists; use a proxy or a multi-tenant backend
- Keep lifecycle and admin APIs disabled unless protected
Watch out for
- Setting insecure_skip_verify: true in production 'temporarily' and never removing it.