Engineers often state “automate the boring stuff.” The theory is: define a task once and let the automation system repeat it at scale, with equal precision each time. Automation can be a force multiplier – proportional to the strength of the automation system involved. However, multipliers are neutral – they amplify whatever they’re given, good or bad, and increased capacity for automation effectiveness scales with increased risk for disaster. When the process is right, automation gives speed and consistency; when it isn’t, it can wreak havoc, quickly and predictably.
This is a three-part series on managing the risks of automation in Kubernetes. In Part 1, we examine the characteristics that make Kubernetes automation uniquely powerful and dangerous, then cover practical pitfalls, ownership boundaries, and protective practices for critical resources. In Part 2, we present a lightweight admission control framework using native ValidatingAdmissionPolicies to enforce explicit approval for high-consequence changes. In Part 3, we address RBAC and authorization in detail, with a practical framework for scoping controller permissions and enforcing ownership boundaries beyond Kubernetes-native capabilities.
Automation in Kubernetes
Kubernetes makes the automation tradeoff unique. It is well known as a workload orchestrator, but its strength is reconciliation: a closed-form automation or control loop that continuously compares desired state to observed state and acts to close the gap. Kubernetes is more than scheduling, it is a self-correcting system whose goal is to achieve a declared state. In practice, it runs sets of controllers that observe, compare, and act, all driven by the declarative configuration administrators provide it. This control loop enables workload survivability during hardware failures, horizontal scaling during increased demand, and ultimately gives the platform the reliability necessary for a highly available distributed system.
Declarative configuration and reconciliation are not unique to Kubernetes – many systems continuously align running state with declared configuration. What distinguishes Kubernetes is scope and indirection. Configuration rarely targets a single, specific resource; a single object often governs dozens or hundreds of downstream resources. Resources define other resources, creating layers where changes at one level cascade through every layer below. Multiple independent controllers reconcile concurrently against the same API server, each acting on its own view of desired state without coordinating with the others.
This combination of broad scope, layered indirection, and concurrent independent reconcilers is what creates the risk. The control loop’s goal is only to achieve the provided state, with no concern about whether that state is desirable or catastrophic. Once a bad configuration enters the cluster it becomes desired state; controllers treat it as fact until it’s withdrawn. Procedural automation engines typically operate serially and fail fast – update a thousand devices one at a time, and upon the first failure, halt immediately. In Kubernetes, reconciliation is concurrent and continuous. A bad configuration doesn’t fail on one resource and stop; it can propagate across thousands of resources simultaneously, with no single point to interrupt the cascade. There is no wiggle room to stop its inherent automation – the only option is changing the state it desires to achieve. The system is relentless in pursuing the dictated state but has no mechanism to distinguish productive changes from destructive ones.
Unified Declarative Model
Everything in Kubernetes operates on a shared declarative foundation. The same API defines applications, networking, storage, credentials, and even infrastructure. There are no hard boundaries between layers of responsibility; everything is expressed as resources in one unified model. This unification simplifies management but also creates exposure between domains that may have had natural separation before. A malformed manifest or an over-broad CRD doesn’t stay local – it is applied wherever controllers see it. A typo can affect any component that consumes that definition, with no practical bound in how far or wide.
Shared Control Plane
Pre-Kubernetes, isolation often came from the shape of the infrastructure or the topology itself: separate database servers, separate web servers, separate networks, distinct machines and management planes that limited how far a mistake could travel. Kubernetes collapses much of that separation into a single control surface and scheduler. Distinct teams, workloads, and services share a common API and reconciliation fabric. Isolation that used to be physical or topological is now primarily enforced by configuration and policy.
Transparency and Configurability
Kubernetes does not hide complexity, it exposes it for maximum flexibility and composability. That flexibility comes at the cost of increased visibility and configurability across multiple disciplines, and as a result, an increased surface for the introduction of errors. The exposed surface area is vast – networking, storage, scheduling, security, infrastructure and more – and being proficient in all areas is a challenge, let alone achieving expertise. This makes footguns inevitable. Configuration is predominately YAML, which means even well-intentioned changes by knowledgeable engineers are prone to typos, column-shifting errors, and subtle mistakes that pass review but fail at runtime. To keep up with the volume, it’s highly desirable to automate how configuration is built and delivered, creating an outer automation layer that feeds the inner reconciliation loop.
Layered Automation
To cope with scale and gain versioning, review, and repeatability, teams add an automated outer delivery loop: controllers that inject configuration into a cluster that is already self-reconciling. This outer loop automates the definitions themselves, applying the same precision and persistence to configuration that Kubernetes applies to state. When a bad configuration is introduced, both the automation layer and the reconciliation layer will enforce it with the same reliability they apply to correct configurations. Automation controllers extend the same reconciliation model upward – resources that exist to generate and manage other resources, placing the definitions themselves inside the control loop. In doing so, they move the source of truth outside the cluster to Git, OCI registries, or other sources. The result is a recursive form of automation: the system now reconciles the state of what defines state, with the same persistence that Kubernetes applies to workloads. A mistake at the source doesn’t stay contained – it propagates to every resource the automation manages, turning a single bad commit or misconfigured artifact into a cluster-wide event.
Taken together, these concerns form the full risk profile of Kubernetes automation:
- A continuously self‑healing core that never stops enforcing declared intent.
- A unified declarative model that links every domain of configuration.
- A consolidated, shared control plane that increases blast radius and speed of propagation.
- A stack of external controllers that replicate and amplify whatever truth they’re given.
That combination makes Kubernetes automation uniquely powerful and also unforgiving. It’s a system that actively works to push a cluster into state consistent with a defined configuration, and simultaneously push the defined configuration into a state consistent with external sources of truth. The danger lies in the fact that it will do so even when that source of truth is destructive.
Pitfalls and Footguns
1. Controllers
Kubernetes functions through a system of independent controllers, each tasked with reconciling a provided configuration into a desired state. A typical cluster runs dozens of controllers: the built-in workload controllers, automation controllers syncing from external sources, and various operators managing databases, certificates, networking, and more. Each controller enforces its view of desired state without sharing context or timing with the others. The result is not a hierarchy but a set of concurrent decisions that meet at the API server. Administrators rarely have a complete understanding of what every controller does or how it will behave in edge cases – yet each one is capable, with the right permissions, of creating, updating, or deleting resources in response to changes it observes.
Understanding how each and every controller behaves – its deletion semantics, its failure modes, its interactions with other controllers – is difficult in practice. Controllers evolve across versions, new operators are added regularly, and the interactions between them are not always documented. Expecting every engineer to internalize the behavior of every controller is impractical. Compounding this, RBAC for controller service accounts tends to be more permissive than ideal – controllers are often given broad permissions to function when their RBAC requirements are unclear, and administrators grant them without fully understanding the scope. This is why declarative safety measures – admission policies that enforce explicit approval for high-consequence changes – become valuable. They provide guardrails that do not depend on perfect knowledge of every controller’s behavior.
2. Ownership Model in Kubernetes
Kubernetes has a built-in ownership and garbage collection system managed by the kube-controller-manager. When a resource is created, it can declare an owner through the ownerReferences field in its metadata. This creates a parent-child relationship with orphan cleanup: when the parent is deleted, the garbage collector automatically deletes all children. A Deployment owns its ReplicaSets, a ReplicaSet owns its Pods, and deleting the Deployment cascades deletion down the entire chain.
This behavior is often invisible. When you delete a Deployment, you don’t see the ReplicaSet and Pod deletions as separate events – they happen automatically as part of garbage collection. The same applies to custom resources: deleting a PostgreSQL cluster resource causes the operator’s finalizer to run, which may delete StatefulSets, Services, PVCs, and Secrets. Each deletion is correct in isolation, but the combined effect can remove far more than the single object you intended to delete when the dependencies between them are not fully understood.
Understanding this model is important because it operates independently of automation controllers. Even without any external automation, Kubernetes itself will cascade deletions through ownership chains. The automation layer adds another dimension of cascading (covered in the next section), but the ownership model is native to Kubernetes and runs regardless of how resources were created.
3. Automation Deletion Cascades
Automation controllers add their own layer of cascading deletion on top of the Kubernetes-native ownership model. When pruning is enabled, these controllers track every resource they apply and delete any that are no longer present in the source. The trigger for deletion is absence rather than an explicit delete request – a file removed during refactoring, a path filter changed, or a merge conflict that silently drops lines can all cause resources to disappear from the rendered manifests, which the controller interprets as intent to delete.
Most declarative automation setups are hierarchical: a root configuration applies child configurations, and those apply application resources. Removing a child from the source or deleting the root triggers a cascade through the hierarchy – the child configurations are deleted and their finalizers prune everything they managed. In a cluster defined entirely through declarative automation, removing the root configuration can delete the entire stack.
These two cascade mechanisms also compound. When automation prunes a resource, that deletion can trigger Kubernetes-native ownership cascades underneath it. Automation removes a HelmRelease, the HelmRelease finalizer deletes the release contents, and those deletions cascade through ownerReferences to downstream resources. By the time reconciliation completes, both layers have executed and the blast radius extends further than either mechanism would suggest in isolation.
4. RBAC
RBAC is how Kubernetes limits where automation can act. Automation controllers and operators do not ask for permission at runtime; they use the credentials you give them and reconcile until the API accepts or denies the write. The pitfall is that RBAC for controller service accounts tends to be more permissive than necessary. Controllers often request broad permissions to function, and administrators may grant them without fully understanding the scope. It is tempting to over-provision because controllers reconcile many kinds and debugging permission errors can be tedious.
The result is that most incidents happen when a controller or user has the right permissions but applies them to the wrong resources. Broad authorization makes it hard to trace a change – a controller reconciles something unexpected, a script patches the wrong object, or someone runs kubectl against the wrong namespace. When everything has cluster-admin, nothing is contained and attribution becomes difficult.
An extensible framework for scoping controller permissions and enforcing ownership boundaries will be covered in Part 3: RBAC and Authorization [coming soon].
5. Critical Resources
Certain Kubernetes resources are high-value targets for protection from automation, but for different reasons. Some are dangerous because of blast radius – a single automated deletion cascades to hundreds or thousands of dependent objects instantly. Others are dangerous because they represent state that cannot be recovered by declarative configuration alone. Both types deserve explicit safeguards, but the mitigation strategies differ based on whether the problem is scope or irreversibility.
CRDs
CustomResourceDefinitions define kinds for whole groups of objects. Removing a CRD removes the type from the cluster, and every Custom Resource of that kind is deleted along with it. What looks like a routine cleanup in Git can cause a major outage: monitoring controllers vanish, certificates stop renewing, and databases lose orchestration context. The usual cause is unclear ownership of the definition that other controllers depend on.
Namespaces
Namespaces are scope boundaries, and deleting one removes everything within that scope – Pods, Secrets, PVCs, ServiceAccounts, and any other namespaced resources. The API processes the deletion immediately with no recovery mechanism. There is no way to partially delete a namespace or protect specific objects within it; the deletion is all-or-nothing.
Cluster API / Infrastructure
Cluster API introduces an additional layer of irreversible consequence. A Cluster object represents real infrastructure, not an abstraction of it. When the object is deleted, the API machinery downstream will dutifully decommission machines, disks, and networks. This is not an accident – it is the designed behavior of declarative infrastructure – but it means that deletion cannot be treated as routine. The declarative model that makes infrastructure reproducible is the same model that makes it deletable with a single API call.
PVCs
PersistentVolumeClaims appear to be simple declarative objects, but their binding to PersistentVolumes is unique and machine-determined. Once a PVC is deleted, its relationship to the underlying volume is permanently severed. Recreating a claim with the same name does not restore that bond. Depending on the StorageClass reclaim policy, the volume may be destroyed (Delete) or left orphaned (Retain). Either outcome is problematic: destruction means data loss, and orphaned volumes require manual intervention to reattach – a process that is rarely tested until it’s needed.
Mitigations and Techniques
Flux
Several automation controllers exist in the Kubernetes ecosystem – FluxCD, ArgoCD, Rancher Fleet, and others – each with their own reconciliation model and configuration patterns. While the pitfalls described earlier apply broadly, the mitigations in this section focus specifically on FluxCD. The concepts translate to other tools, but the implementation details and configuration options here are Flux-specific.
Pruning
Flux’s kustomize-controller includes a pruning feature that removes resources from the cluster when they disappear from the source. When spec.prune: true is set on a Kustomization, the controller tracks every resource it applies and deletes any that are no longer present in the rendered manifests. This is powerful for keeping clusters clean, but it makes absence a signal for deletion. A bad commit, a moved file, or a misconfigured path can cause the controller to prune resources that were never meant to be removed.
Disabling pruning at root and platform layers keeps the blast radius contained. Pruning works best in leaf scopes where the resource set is fully owned and understood.
Dependencies
Flux’s dependsOn field controls the order in which resources are applied. Both Kustomizations and HelmReleases support this field, though dependencies can only reference resources of the same kind – a Kustomization can depend on other Kustomizations, and a HelmRelease can depend on other HelmReleases, but not across kinds. A resource with dependsOn will wait for the named dependencies to reach a Ready state before reconciling, which prevents resources from being applied before their prerequisites exist.
However, dependsOn only waits for the Flux resource itself to report success – it does not verify that the underlying resources are actually ready to use. A Kustomization can complete successfully while CRDs are still being established, webhooks are registered but not yet serving, or controllers are starting but not yet processing events. Dependent applications that reconcile immediately after may encounter transient failures. Pairing dependsOn with health checks verifies that resources are not just created but actually functional.
Health Checks
Flux health checks extend the dependency model by verifying that resources reach a healthy state before the Kustomization reports Ready. Specifying spec.healthChecks on a Kustomization blocks dependent reconciliations until Deployments are available, CRDs report as Established, or custom resources reach their expected conditions.
Health checks close the gap between “applied” and “ready.” Without them, dependent applications may reconcile against services that are still initializing, creating noise in logs and masking real problems.
OCI-Versioned Artifacts
The term “GitOps” implies Git as the source of truth, but the reconciliation model does not require Git at all. Automation controllers can pull configuration from many sources – Git repositories, Helm registries, or OCI registries. OCI registries in particular offer immutable digest-based references, native support for signatures and attestations, and integration with existing container infrastructure.
Flux supports OCI artifacts as sources through the OCIRepository resource, enabling what might be called “gitless” GitOps. Configuration manifests are packaged as OCI artifacts, pushed to a registry, and referenced by the cluster. The reconciliation model remains the same – the controller watches the source, detects changes, and applies them – but the source is an OCI registry rather than a Git repository. This decouples configuration delivery from Git’s semantics and allows teams to use the same artifact pipelines for both application images and configuration.
Immutability is the foundation of reliable rollbacks. When artifacts are stored in OCI registries and referenced by digest, you can reference a specific version and trust that it will not change. Tags can be overwritten, but digests cannot. This makes rollback deterministic – the previous state still exists exactly as it was. By referencing artifacts by digest rather than mutable tags, you ensure that reconciliation targets a known, fixed configuration. If a bad artifact is pushed, you can roll back to the previous digest with confidence. Without immutability, rollback becomes unreliable because the previous tag may have been overwritten or the artifact may have changed since it was last deployed.
In production environments, digest references can be enforced via admission or CI checks. Keeping previous release artifacts available enables deterministic rollback, and promotion workflows where artifacts are tested in lower environments before their digests reach production add another layer of confidence.
RBAC per Artifact
Flux controllers run with their own service account credentials, but individual Kustomizations and HelmReleases can specify a different service account to use when applying resources. By setting spec.serviceAccountName on each Flux object, you control what permissions that reconciler has at apply time. The controller impersonates the specified ServiceAccount when creating, updating, or pruning resources, so RBAC enforcement happens at reconciliation time rather than relying solely on the controller’s own permissions.
This allows fine-grained scoping: each reconciler can be limited to the namespaces and kinds it owns, even if a bad commit or overwritten artifact is introduced. Scoping each Kustomization or HelmRelease with its own ServiceAccount, RBAC bindings, repository path, and namespace confines blast radius. Pinning ServiceAccount limits the impact of configuration errors – the reconciler cannot change resources outside its scope regardless of what the source contains.
Multi-Tenancy
Flux supports a multi-tenancy model that restricts cross-namespace references. By default, a Kustomization or HelmRelease can reference sources (GitRepository, OCIRepository, HelmRepository) in any namespace. When Flux controllers are configured with the --no-cross-namespace-refs flag, this is disabled – resources can only reference sources within their own namespace.
This enforces tenant isolation at the controller level. Each team or tenant operates in their own namespace with their own sources, and cannot reference or be affected by sources in other namespaces. Combined with RBAC per artifact, this creates strong boundaries: a tenant’s reconcilers can only access their own sources and can only apply resources where their ServiceAccount has permissions. A misconfiguration or compromise in one tenant’s namespace cannot propagate to another.
Suspend
Flux provides a spec.suspend: true field on Kustomizations and HelmReleases that immediately halts reconciliation. When suspended, the controller stops reconciling the target, giving operators time to investigate, remediate, or coordinate without racing against the reconciliation loop.
Suspend is particularly valuable during incidents. If a bad configuration has been merged and is causing failures, suspending the affected Kustomization prevents the controller from continuously retrying and creating noise. It also provides a window to fix the source without the pressure of live reconciliation. For planned maintenance – CRD migrations, cluster upgrades, or coordinated rollouts – suspending reconcilers ensures that changes happen on your schedule rather than the controller’s.
A nuance: if a Kustomization manages another Flux resource (for example, a parent Kustomization that applies child Kustomizations), directly patching the child’s suspend field with kubectl may not stick. Kubernetes tracks field ownership through server-side apply, and when the parent reconciles, it will overwrite the child’s spec – including the suspend field – because it owns that field. The suspend you set manually gets reverted. To avoid this, either suspend the parent instead, use the Flux CLI (flux suspend kustomization <name>), or modify the suspend field at the source level in Git. Understanding field ownership matters when resources are managed by other resources.
Suspend fits naturally into incident response – when something goes wrong, halting the relevant reconcilers stops the bleeding while the root cause is investigated.
Dry-Run and Diff Previews
Before changes are applied, it helps to see what will happen. Flux’s CLI provides flux diff kustomization to preview what resources would be created, updated, or deleted without actually applying them. This server-side diff compares the rendered manifests against the current cluster state and shows the delta.
Integrating diff previews into CI pipelines catches destructive changes before they merge. A pull request that removes a manifest will show the resulting deletion in the diff output, making the consequence visible during review rather than after merge. For high-risk changes, diff output as part of the approval process makes consequences visible before merge.
Diff previews are not a substitute for admission control or other guardrails – they rely on humans noticing problems in the output. But they add a layer of visibility that makes unintended changes harder to miss.
Helm
CRD Handling
Helm treats CRDs differently from other resources. When CRDs are placed in the crds/ directory of a chart, Helm installs them on the first helm install but will not upgrade or delete them during subsequent helm upgrade or helm uninstall operations. This behavior exists specifically because CRD deletion has cluster-wide consequences that extend beyond the chart’s scope.
Charts that include CRDs benefit from placing them in the crds/ folder. Managing CRDs in a platform-only repository and reconciler – separate from application trees – keeps them out of prune scope. Changing or removing a CRD is a cluster-wide schema change that benefits from pausing dependent reconcilers, inventorying remaining Custom Resources, and testing migrations in staging before production.
Resource Policy Keep
When CRDs must live in the templates/ directory rather than crds/ (for example, to support templating or conditional installation), annotate them with helm.sh/resource-policy: keep. This annotation tells Helm to skip deletion of the resource during helm uninstall. The CRD remains in the cluster even when the release is removed, preventing accidental deletion of all Custom Resources of that type.
This annotation is also useful for other resources that should survive release deletion, such as PVCs or namespaces that contain data you want to preserve. This annotation is worth considering for any resource where deletion would have consequences beyond the release itself.
Note that Helm only respects this annotation if it was present at install or upgrade time. Adding the annotation to an existing resource after deployment has no effect – Helm tracks resource metadata from when it was applied, not from the live cluster state.
Namespace Handling
Namespaces should be kept out of the Helm reconciliation loop where possible. When a namespace is included as a resource in a Helm chart, it becomes subject to the same lifecycle as the release – including deletion during helm uninstall. This creates the same risk as CRDs: a routine release cleanup can remove the namespace and everything under it.
The recommended approach is to use helm install --namespace (or the equivalent in automation, such as spec.targetNamespace in a HelmRelease) to deploy into a namespace without managing its lifecycle. The namespace is referenced but not owned by the release. This keeps namespace creation and deletion as separate, deliberate operations outside the chart’s scope.
If a chart must create its own namespace, annotating it with helm.sh/resource-policy: keep prevents deletion during uninstall. But the cleaner pattern is to treat namespace provisioning as a platform concern, separate from application deployment.
Progressive Delivery
Progressive delivery strategies – canary releases, blue/green deployments, and similar patterns – reduce blast radius by gradually rolling out changes rather than applying them all at once. Instead of updating every replica simultaneously, a canary deployment routes a small percentage of traffic to the new version first. If metrics degrade or errors spike, the rollout halts or rolls back before the change affects the full workload.
This approach provides a window to detect problems before they propagate. Automated analysis can compare error rates, latency, and custom metrics between the canary and baseline, promoting the release only when thresholds are met. Blue/green deployments take a different approach: the new version runs in a parallel environment, and traffic is switched over only after validation.
Tools like Flagger (part of the Flux ecosystem, though separate from FluxCD itself) and Argo Rollouts provide progressive delivery capabilities for Kubernetes. These integrate with service meshes and ingress controllers to manage traffic shifting and automated rollback. The specific implementation varies by tool, but the principle is consistent: limit exposure to new changes until they prove safe.
RBAC
RBAC is the primary mechanism for limiting where automation can act. Automation controllers and operators do not ask for permission at runtime – they use the credentials you give them and reconcile until the API accepts or denies the write. Scoping those credentials correctly keeps mistakes local and manageable. When complete understanding of every controller’s behavior is impractical, RBAC provides boundaries that limit what any single controller can affect – even if you don’t fully understand what it does.
Aligning namespaces, groups, and service accounts with team boundaries maps ownership to RBAC. Limiting write scope by default and avoiding cross-namespace writes except through explicit platform APIs keeps boundaries clear. Many incidents happen when a controller or user has the right permissions but applies them to the wrong resources. Tight, intentional scopes make it clear who can touch what, which makes debugging and attribution faster.
Cluster-admin is best treated as an exception – most humans and most controllers can operate with namespaced roles. Platform operators and admission tooling may need cluster scope, but day-to-day identities rarely do. A break-glass role with short-lived credentials and mandatory post-use review covers emergencies without normalizing broad access. Binding controller ServiceAccounts to namespaced roles with only the verbs and kinds they need, and using a dedicated ServiceAccount for CRD management, keeps permissions tight. Audit logging for write verbs in shared namespaces helps with attribution and review.
RBAC has limitations. It is additive-only – there are no deny rules and no way to subtract permissions. You cannot say “grant everything except delete on CRDs”; you must enumerate exactly what is allowed. Permissions accumulate across role bindings, which makes it easy to inadvertently grant more access than intended when multiple roles apply to the same subject. These constraints sometimes require creative structuring of roles and namespaces, or pairing RBAC with admission control to enforce restrictions that RBAC alone cannot express.
For a complete RBAC framework that addresses these limitations, see Part 3: RBAC and Authorization [coming soon].
Finalizers
Finalizers provide an artificial delay between deletion request and actual removal for most resources. When a resource with finalizers is deleted, Kubernetes sets the deletionTimestamp but does not remove the object until all finalizers are cleared. This delay creates a window for operators to detect and react to unintended deletions before data loss occurs.
Adding custom finalizers to critical resources (namespaces, PVCs, platform configuration objects) can provide enough time to halt a bad automated prune or catch an accidental deletion before it completes. Monitoring for resources with deletionTimestamp set and finalizers remaining – and alerting when critical resources enter deletion state – provides early warning.
Finalizers work well in combination with admission policies: the admission check rejects deletion attempts, and if bypassed, the finalizer provides a second line of defense. Note that --force --grace-period=0 bypasses finalizer protection entirely.
Backups and Disaster Recovery
For resources that carry state outside version control, backups and disaster recovery strategies provide the only path to recovery. When automation deletes a PVC or triggers infrastructure decommissioning, the data is gone regardless of what exists in Git.
For PVCs, CSI snapshots on a schedule provide point-in-time recovery. For databases, streaming backups (WAL archiving for PostgreSQL, binlog replication for MySQL) reduce the window of potential data loss. Storing backups outside the cluster – a separate account, region, or provider – ensures a cluster-wide incident doesn’t take the backups with it.
Restore procedures that haven’t been exercised in a non-production environment tend to fail when needed. Retain on critical StorageClasses means deleted PVCs leave their volumes behind rather than destroying them. Having the reattach process documented (matching claimRef, storageClassName, accessModes) before it’s needed saves time during recovery.
For Cluster API and infrastructure objects, snapshotting control plane etcd or disks before destructive actions provides a rollback path. Keeping Cluster API resources in a dedicated repository and reconciler – separate from application pipelines where they might be pruned – reduces the risk of accidental deletion.
Admission Control
Admission control is the last line of defense before a change becomes part of the desired state. Every request flows through the API server’s admission layer before it is persisted, which means rules written here have uniform effect across all clients – humans, CI, and reconcilers alike.
The goal is to make high-consequence changes explicit and gated. Admission policies can require an approval annotation before allowing deletion, block operations unless specific conditions are met, or enforce that certain identities are the only ones permitted to perform destructive actions. Unlike RBAC, which determines whether an operation is ever permitted, admission evaluates whether a specific request should be accepted right now – allowing for context-sensitive rules that RBAC cannot express.
For a complete admission control framework using native ValidatingAdmissionPolicy, see Part 2: Admission Policies as Guardrails.
Summary
Kubernetes automation is powerful and unforgiving. The same reconciliation loops that provide reliability will execute destructive changes just as efficiently as productive ones. Controllers do not evaluate whether a change is sensible – they enforce declared state regardless of consequence. A bad commit, an unclear ownership boundary, or a mistimed deletion can cascade across the platform faster than manual intervention can stop it.
The practices covered in this post address these risks through layered controls. Scoped RBAC and dedicated service accounts limit where automation can act. Understanding controller ownership and deletion cascades prevents accidental pruning of resources you didn’t intend to remove. Protected resource classes – CRDs, namespaces, PVCs, and Cluster API objects – require deliberate handling because of their blast radius or because they carry state that Git cannot recover. Backups, snapshots, and tested restore procedures provide continuity where declarative models cannot. Immutable OCI artifacts referenced by digest (rather than mutable tags) ensure that rollbacks target a known, fixed configuration.
Across these cases, the problem is unclear scope, not bad intent. Destructive actions happen when ownership is loose or missing. The solution is clear ownership, explicit approval for destructive changes, and traceability for who did what and when. With these controls in place, the cluster becomes more predictable and easier to run.
These practices reduce risk, but they don’t prevent a bad change from entering the cluster in the first place. In Part 2: Admission Policies as Guardrails, we present a lightweight admission control framework using native ValidatingAdmissionPolicy to enforce explicit approval for high-consequence changes – stopping destructive operations at the API server before they become part of the desired state. In Part 3: RBAC and Authorization, we provide a practical framework for scoping controller permissions and enforcing ownership boundaries at the authorization layer.
One Response
Loved this line, “A Cluster object represents real infrastructure, not an abstraction of it.” There’s still real metal under there, the kind we Gray Beards had to master before Cloud Native. The abstraction just makes it easier to forget, and easier to destroy. 😭