TopolearnWebhooks, policy engines and apply conflicts all live on this path; it explains many confusing errors.
Every write to the API server passes through stages:
1. Authentication: client certificates, bearer tokens (service account JWTs, OIDC) or webhooks establish who you are. 2. Authorization: RBAC (plus Node and webhook authorizers) checks the verb on the resource. 3. Mutating admission: built-in plugins (defaulting ServiceAccounts, LimitRanger) and MutatingAdmissionWebhooks (sidecar injection, defaults) and MutatingAdmissionPolicies may change the object. 4. Schema validation. 5. Validating admission: ResourceQuota, PodSecurity, ValidatingAdmissionPolicy (CEL, in-process) and ValidatingAdmissionWebhooks can reject it. 6. Persisted to etcd, which assigns the new resourceVersion; watchers are notified.
Webhooks are on the critical path: a down webhook with failurePolicy: Fail can block all pod creation. Scope them with selectors, set short timeouts, and exclude kube-system.
Server-side apply (kubectl apply --server-side) moves the merge into the API server. It tracks field ownership in metadata.managedFields: each manager (kubectl, an HPA, Helm, a controller) owns the fields it set. Applying a different value for a field someone else owns is a conflict; you either drop the field from your config or --force-conflicts to take ownership. This is how a GitOps tool and an HPA can share one Deployment safely when your manifest omits replicas.
Example
kubectl apply --server-side --field-manager=ci -f deploy.yaml
kubectl get deploy web --show-managed-fields -o yaml | grep -E 'manager:|operation:'
# manager: ci operation: Apply
# manager: kube-controller-manager operation: Update
# Conflict example:
kubectl apply --server-side --field-manager=other -f deploy-replicas-5.yaml
# error: Apply failed with 1 conflict: conflict with "ci": .spec.replicas
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl get --raw /metrics | grep apiserver_admission_webhook_rejection_count | head -3Remember
- authn, authz, mutating admission, validation, validating admission, etcd
- Webhooks can block the cluster: scope them and set timeouts
- Server-side apply tracks field owners and reports conflicts
Watch out for
- A cluster-wide failing webhook with failurePolicy: Fail that blocks even kube-system pods.