TopolearnProbes are what make rollouts safe and self-healing real; wrong probes cause outages.
The kubelet runs three kinds of probe against each container:
- readinessProbe: may this container receive traffic? Failing removes the pod from Service endpoints; nothing is restarted. Runs for the container's whole life.
- livenessProbe: is it alive? Failing
failureThresholdtimes in a row makes the kubelet restart the container. Use it for deadlocks, not slowness. - startupProbe: has it finished starting? While it hasn't succeeded, the other probes are disabled. Use it for slow starters instead of a huge
initialDelaySeconds.
Each probe is an httpGet (2xx/3xx is success), tcpSocket, exec command, or grpc check, with periodSeconds, timeoutSeconds (default 1s), failureThreshold and successThreshold.
Good practice:
- Readiness checks what this pod needs to serve (warm caches loaded, can accept connections).
- Liveness should be cheap and check only the process itself. Never make liveness depend on a database or another service: if the DB blips, every pod restarts at once and turns a small incident into an outage.
- Give liveness a generous threshold; a restart is a heavy action.
- Many apps don't need a liveness probe at all if they exit on fatal errors.
Example
containers:
- name: api
image: ghcr.io/acme/api:2.1.0
ports: [{name: http, containerPort: 8080}]
startupProbe: # up to 30 x 5s = 150s to start
httpGet: {path: /healthz, port: http}
periodSeconds: 5
failureThreshold: 30
readinessProbe: # may include dependency checks
httpGet: {path: /ready, port: http}
periodSeconds: 5
livenessProbe: # process-only check, generous
httpGet: {path: /healthz, port: http}
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 6Remember
- Readiness: in or out of load balancing; liveness: restart; startup: protects slow starts
- Never make liveness depend on external services
- Probe failures show up as events in describe
Watch out for
- A liveness probe that checks the database, so a DB outage restarts every pod in a loop.