TopolearnUnderstanding reconciliation explains Kubernetes behaviour under failure and is the basis for writing operators.
Every controller follows the same pattern: observe desired and actual state, diff, act, repeat. It's level-triggered: it reacts to the current state, not to individual events, so a missed event or a controller restart loses nothing; the next reconcile sees the same state and does the right thing.
Mechanics:
- Controllers list objects once, then watch the API server for changes from that
resourceVersion. client-go's informers keep a local cache and call handlers on add/update/delete, which push object keys (namespace/name) onto a work queue. Workers pop keys and runReconcile(key), retrying with back-off on errors. Duplicate keys collapse, so bursts don't overload. - Optimistic concurrency: every write includes the object's
resourceVersion; if someone else changed it, the API returns 409 Conflict and the controller re-reads and retries. - Owner references link children to parents (ReplicaSet to Deployment, Pod to ReplicaSet). The garbage collector deletes dependents when the owner goes, in background or foreground mode.
- Finalizers on an object block its deletion until a controller has cleaned up external resources and removed its finalizer. A stuck finalizer is why objects hang in Terminating.
metadata.generationincrements on spec changes; controllers writestatus.observedGenerationso clients know status reflects the latest spec.
Leader election via Lease objects ensures only one replica of a controller acts at a time.
Example
# Watch the raw event stream a controller sees
kubectl get deploy web --watch --output-watch-events -o name
# ADDED deployment.apps/web
# MODIFIED deployment.apps/web
kubectl get pod web-7c6d9f8b5-abcde -o jsonpath='{.metadata.ownerReferences[0].kind}'
# ReplicaSet
kubectl get deploy web -o jsonpath='{.metadata.generation} {.status.observedGeneration}'
# 7 7
# Why is this namespace stuck Terminating?
kubectl get ns old -o jsonpath='{.spec.finalizers} {.status.conditions[*].message}'
kubectl get lease -n kube-system # leader election locksRemember
- Level-triggered reconcile: act on current state, tolerate missed events
- Informers cache; work queues dedupe and retry
- resourceVersion conflicts, owner references, finalizers, observedGeneration
Watch out for
- Removing finalizers by hand to 'fix' a stuck delete, leaking the external resources they guarded.