PodDisruptionBudgets and node drains

Kubernetes · Advanced · 8 min

Cluster upgrades and autoscaler scale-downs evict pods; without a PDB they can take down a whole service.

Voluntary disruptions are evictions someone chose: kubectl drain before node maintenance, cluster upgrades, the autoscaler removing a node. Involuntary ones are crashes, kernel panics and hardware failure.

A PodDisruptionBudget limits voluntary disruptions for pods matching a selector, with either minAvailable (e.g. 2 or "50%") or maxUnavailable. The Eviction API, used by drain and autoscalers, refuses an eviction that would break the budget, so the drain waits and retries until other pods become ready elsewhere.

Points to know:

  • PDBs don't protect against involuntary failures, and they don't stop a direct kubectl delete pod or a Deployment rollout (which has its own maxUnavailable).
  • A PDB that can never be satisfied (minAvailable: 1 on a single-replica Deployment, or maxUnavailable: 0) blocks drains forever, a classic cause of stuck node upgrades. Run at least two replicas for anything with a PDB.
  • unhealthyPodEvictionPolicy: AlwaysAllow lets already-unhealthy pods be evicted, so a broken app can't block maintenance.

kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data cordons the node (no new pods), then evicts its pods respecting PDBs. kubectl uncordon node-3 returns it to service.

Example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: api}
spec:
  minAvailable: 2
  selector: {matchLabels: {app: api}}
  unhealthyPodEvictionPolicy: AlwaysAllow
# kubectl get pdb api
# NAME  MIN AVAILABLE  MAX UNAVAILABLE  ALLOWED DISRUPTIONS
# api   2              N/A              1      (3 healthy replicas)
#
# kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data
# evicting pod shop/api-6f-x1
# error when evicting ... would violate the pod's disruption budget (retrying)

Remember

  • PDBs limit voluntary evictions (drains, upgrades, scale-down)
  • They don't cover crashes or direct deletes
  • An unsatisfiable PDB blocks node drains

Watch out for

  • A PDB with minAvailable: 1 on a single-replica Deployment, which makes every node drain hang.

Check yourself

  1. Which action does a PodDisruptionBudget protect against?

Sign in to answer, see the explanations, and get this lesson back as a quick review later.

Official docs: kubernetes.io/docs/concepts/workloads/pods/disruptions/