TopolearnOn Kubernetes nobody edits prometheus.yml by hand; the Operator turns CRDs into configuration and ships a full default setup.
The Prometheus Operator manages Prometheus and Alertmanager as Kubernetes resources. The kube-prometheus-stack Helm chart bundles it with Prometheus, Alertmanager, Grafana, node-exporter, kube-state-metrics (object-level metrics such as deployment replicas and pod phases), plus a large set of recording rules, alerts and dashboards for the cluster itself.
After installing, you describe what to scrape with custom resources instead of scrape_configs:
- ServiceMonitor: selects Services by label and names the port to scrape on their Endpoints. The Operator generates a
kubernetes_sd_configsjob with the right relabeling, includingnamespace,serviceandpodlabels. - PodMonitor: the same for pods without a Service (jobs, DaemonSets).
- Probe: a blackbox_exporter target.
- ScrapeConfig: an escape hatch for arbitrary scrape configuration (static targets, cloud discovery).
- PrometheusRule: a rule group, mounted into Prometheus automatically.
- AlertmanagerConfig: per-namespace routing merged into the Alertmanager config.
The Prometheus resource itself holds replicas, retention, resources, storage (a PVC template) and the selectors that decide which monitors and rules it picks up. The number one gotcha: the chart's serviceMonitorSelectorNilUsesHelmValues default makes Prometheus select only monitors labelled release: <helm release name>. A ServiceMonitor without that label is silently ignored. Set the value to false or add the label. The same applies to ruleSelector and podMonitorSelector.
Debug with the Prometheus UI's Status pages (port-forward svc/<release>-kube-prometheus-stack-prometheus 9090) and kubectl get servicemonitors -A. The Operator's logs report invalid monitors.
Applications still need a Service exposing a named metrics port and the /metrics endpoint.
Example
# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
# helm install mon prometheus-community/kube-prometheus-stack -n monitoring --create-namespace \
# --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: checkout
namespace: shop
labels: {release: mon} # needed unless the Nil... value is false
spec:
selector:
matchLabels: {app: checkout} # Services with this label
namespaceSelector: {matchNames: [shop]}
endpoints:
- port: metrics # the Service port *name*
interval: 30s
path: /metrics
---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata: {name: checkout, namespace: shop, labels: {release: mon}}
spec:
groups:
- name: checkout
rules:
- alert: CheckoutDown
expr: absent(up{job="checkout"}) or up{job="checkout"} == 0
for: 5m
labels: {severity: page}Remember
- The chart ships Operator, Prometheus, Alertmanager, Grafana, exporters and default rules
- ServiceMonitor selects Services by label and a named port; PodMonitor for bare pods
- Match the Prometheus resource's selectors (release label) or the monitor is ignored
Watch out for
- A ServiceMonitor that points at a port number instead of the Service port name, so nothing is discovered.