This is Part 2 of a three-part series on preventing destructive automation in Kubernetes. If you haven’t read it yet, start with Part 1: Pitfalls, Gotchas, and Practices.
In Part 1, we examined why Kubernetes automation carries unique risks: reconciliation loops that never stop, a unified declarative model that links all configuration, and layered automation controllers that amplify whatever truth they’re given. We then covered practical mitigations – scoping controller ownership, tightening RBAC boundaries, and protecting critical resources like CRDs, namespaces, and PVCs from automated deletion cascades.
Those practices reduce risk, but they don’t prevent a bad change from entering the cluster in the first place. Admission control is the last line of defense – the final gate before a change becomes part of the desired state. This section describes a small, native framework for preventing high‑consequence changes from entering the cluster without an explicit signal of intent. The aim is practical: keep throughput and autonomy while ensuring that destructive operations occur only when a human or a designated identity makes that intent unambiguous. The mechanism is admission control at the Kubernetes API server, built on ValidatingAdmissionPolicy (VAP), its Binding, and a thin parameter CRD for local policy data.
API Server Overview
All changes flow through one process – the API server. A request arrives; the server authenticates the caller, authorizes the operation in principle, and then evaluates admission before persisting the change. This ordering means that controllers only act on objects that have already passed admission checks, making admission the last opportunity to prevent a problematic change from becoming part of the desired state. Because every client (humans, CI, reconcilers) uses the same path, small rules written here have uniform effect without introducing a new controller or service to maintain.
Admission Behavior
Admission evaluates the request and the object against a set of rules. Mutating admission may rewrite the object; validating admission decides to accept or reject. The framework here uses validating admission. Multiple validating policies can apply to the same request, and all must pass for the request to be accepted – if any policy denies, the request is rejected. A minor but important detail: for DELETE, the live object is available to the policy as oldObject; for CREATE and UPDATE, the proposed object is available as object.
RBAC vs Admission and VAPs
RBAC decides who may perform an operation; admission decides whether a specific request is accepted right now. RBAC remains the primary mechanism for permissions – admission is not a replacement for it. The guardrails in this framework address a different gap: authorization can be correct while the action is still badly timed or unintended. A controller may have permission to delete a PVC, but that doesn’t mean this particular deletion was deliberate. Admission can require an explicit signal – an approval annotation or a qualified caller – before high-consequence actions proceed.
ValidatingAdmissionPolicies (VAP) are Kubernetes‑native and evaluated inside the API server. Unlike webhook‑based policy engines (e.g., Kyverno or OPA Gatekeeper), there is no network call or external service in the critical path; evaluation is in‑process and predictable. VAPs are less flexible (no mutation; simpler logic), but for this use – requiring explicit authorization or intent before deletes – they are sufficient and operationally lighter.
(For background, see the Kubernetes documentation on Admission Control and ValidatingAdmissionPolicy.)
ValidatingAdmissionPolicy (VAP)
A VAP is a CEL expression evaluated inside the API server. It has three useful levers:
- matchConstraints: select the kinds, API groups, verbs, and scope to which the policy applies.
- matchConditions: fast preconditions that gate evaluation (for example, whether protection is enabled, whether an object is in a protected class, or whether the request matches a label‑based rule).
- validations: the decision expressions and the human‑readable messages that accompany denials.
VAPs can read small, structured policy data passed by reference through paramKind. Externalizing the variable parts of policy into parameter objects keeps CEL simple and predictable while allowing different teams to change scope without editing the policy code.
ValidatingAdmissionPolicyBinding
A policy does nothing until a ValidatingAdmissionPolicyBinding connects it to real objects. The Binding supplies three decisions per scope:
- validationActions determine the mode (Audit to observe, Deny to enforce). Changing between audit and enforcement requires only a binding update without modifying the policy code.
- paramRef selects the parameter objects (by name for tight control, or by label selector for classes of resources).
- matchResources / objectSelector reduces the surface further when needed (for example, to PVC deletes only, or to objects labeled as platform‑managed).
One policy may have many bindings; each binding may select many parameter objects. This allows the framework to remain compact while still targeting specific resources at scale.
Operational Benefits
Most operations pass through without notice. For protected resources, the policy blocks the action until an approval annotation is present on the object – someone has to explicitly mark it for deletion before automation can proceed. A break-glass identity exists for emergencies. Denied requests surface in controller events and logs, making blocked actions visible rather than silent.
Framework Overview
The framework consists of one policy pattern (shown here in its namespaced form), a small parameter type, and bindings that express scope. In practice we run both a namespaced guard and a cluster‑scoped variant for platform objects. Only the namespaced policy is shown; the cluster policy follows the same structure with different matchConstraints.
- parameter CRD: carries local policy data – feature flags, allow lists, and label selectors that define protected classes.
- ValidatingAdmissionPolicy: evaluates requests for UPDATE and DELETE within namespaced scope; it recognizes an approval annotation, allows explicit identities, and enforces label‑ and kind‑based protection rules.
- bindings: map the policy to classes of objects (by labels or kinds) and set the enforcement mode per environment.
Namespaced Policy
(We include finalizers here as a protective convention, though behavior varies by cluster version and should be verified in your environment.)
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
name: guardrailz-resource-protect
finalizers:
- guardrailz.io/protect
spec:
paramKind:
apiVersion: guardrailz.io/v1
kind: GuardrailParams
failurePolicy: Ignore
matchConstraints:
resourceRules:
- apiGroups: ["*"]
apiVersions: ["*"]
resources: ["*"]
operations: ["UPDATE", "DELETE"]
scope: "Namespaced"
matchConditions:
- name: enabled
expression: "params != null && params.spec.enabled"
- name: not-break-glass
expression: >
request.userInfo.username != "system:serviceaccount:kube-system:break-glass"
- name: operation-relevant
expression: >
request.operation == 'DELETE' ||
(
has(object.metadata.annotations) &&
'guardrailz.io/approved-by' in object.metadata.annotations &&
(
!has(oldObject.metadata.annotations) ||
!('guardrailz.io/approved-by' in oldObject.metadata.annotations) ||
oldObject.metadata.annotations['guardrailz.io/approved-by'] !=
object.metadata.annotations['guardrailz.io/approved-by']
)
)
- name: in-protect-resources
expression: >
!has(params.spec.?protect.resources) ||
params.spec.protect.resources.exists(r,
r.group == request.resource.group &&
r.resource == request.resource.resource &&
(has(r.names) ? r.names.exists(nn, nn == request.name) : true)
)
- name: not-in-exclude-resources
expression: >
!(
has(params.spec.?exclude.resources) &&
params.spec.exclude.resources.exists(r,
r.group == request.resource.group &&
r.resource == request.resource.resource &&
(has(r.names) ? r.names.exists(nn, nn == request.name) : true)
)
)
- name: matches-protect-labels
expression: >
!has(params.spec.?protect.labels.matchLabels) ||
(
has(oldObject.metadata.labels) &&
params.spec.protect.labels.matchLabels.all(key,
key in oldObject.metadata.labels &&
oldObject.metadata.labels[key] == params.spec.protect.labels.matchLabels[key]
)
)
- name: not-in-exclude-labels
expression: >
!(
has(params.spec.?exclude.labels.matchLabels) &&
has(oldObject.metadata.labels) &&
params.spec.exclude.labels.matchLabels.all(key,
key in oldObject.metadata.labels &&
oldObject.metadata.labels[key] == params.spec.exclude.labels.matchLabels[key]
)
)
validations:
- message: "Guardrails: delete blocked (protected object; not an allowed SA and no annotated approval)."
expression: >
request.operation != 'DELETE' ||
(
has(params.spec.?allow.serviceAccounts) &&
params.spec.allow.serviceAccounts.exists(sa,
request.userInfo.username ==
'system:serviceaccount:' + sa.namespace + ':' + sa.name
)
)
||
(
has(params.spec.?allow.users) &&
params.spec.allow.users.exists(u, u == request.userInfo.username)
)
||
(
has(params.spec.?allow.groups) &&
params.spec.allow.groups.exists(g,
request.userInfo.groups.exists(gg, gg == g)
)
)
||
(
has(oldObject.metadata.annotations) &&
'guardrailz.io/approved-by' in oldObject.metadata.annotations &&
oldObject.metadata.annotations['guardrailz.io/approved-by'] != ''
)
- message: "guardrailz.io/approved-by annotation may only be set to own username"
expression: >
request.operation == 'DELETE' ||
(
object.metadata.annotations['guardrailz.io/approved-by'] == request.userInfo.username &&
authorizer.requestResource.check('delete').allowed()
)
The policy operates only when parameters indicate protection is enabled. It recognizes three paths past the guard: an explicitly allowed caller (user, group, or ServiceAccount), an emergency identity (break-glass), or a prior approval written to the object. On updates, the approval may be set only by the caller setting it for themself, and only if the caller is authorized to delete; on deletes, the policy verifies that the approval is present on the live object.
Binding Examples
Bindings apply the policy with different scopes and parameter selections. The examples below show one cluster binding (component label class) and one namespaced binding limited to PVC deletes. Validation actions include both Audit and Deny to support staged rollout.
Cluster-scoped (component class)
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: guardrailz-cluster-managed-protect
finalizers:
- guardrailz.io/protect
spec:
policyName: guardrailz-cluster-resource-protect
validationActions: ["Deny", "Audit"]
paramRef:
parameterNotFoundAction: Allow
name: managed-guard
matchResources:
objectSelector:
matchLabels:
guardrailz.io/component: "managed"
Namespaced (PVC deletes; params by class label)
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
name: guardrailz-pvc-protect
finalizers:
- guardrailz.io/protect
spec:
policyName: guardrailz-resource-protect
validationActions: ["Deny", "Audit"]
paramRef:
parameterNotFoundAction: Allow
selector:
matchLabels:
guardrailz.io/class.pvc: "true"
matchResources:
resourceRules:
- apiGroups: [""]
apiVersions: ["*"]
resources: ["persistentvolumeclaims"]
operations: ["DELETE"]
Parameter CRD Schema
<!– TODO: CRD schema –>
Parameter Examples
The following objects illustrate the parameter surface used by the policy.
apiVersion: guardrailz.io/v1
kind: ClusterGuardrailParams
metadata:
name: critical-guard
spec:
enabled: true
protect:
self: true
apiVersion: guardrailz.io/v1
kind: GuardrailParams
metadata:
name: keycloak-db-pvc-guard
labels:
guardrailz.io/class.pvc: "true"
spec:
enabled: true
protect:
self: true
labels:
matchLabels:
cnpg.io/cluster: keycloak-database-cluster
allow:
serviceAccounts:
- name: cloudnative-pg-operator
namespace: cnpg-system
Operational Model
Day-to-day Use
During normal operation, most changes pass through without interaction. When a change does target a protected object and lacks an approval annotation or an allowed identity, the request is denied with a message explaining what’s missing. The engineer either adds the approval annotation and retries, or routes the change through an identity that’s permitted to perform it. Starting in Audit mode allows observing which requests would be blocked before switching to Deny.
Two-step Approval
The two-step pattern separates “should this be deleted” from “delete it now.” The approval annotation captures who decided the deletion was intentional, and that record stays with the object until the action completes. Once the approval is present, the actual deletion proceeds normally. This pattern works well for namespace deletion, CRD removal, and PVCs tied to stateful services – cases where an accidental delete has outsized consequences.
Rollout Strategy
A typical rollout starts with a narrow binding in a non-production environment, where the team can observe behavior before expanding scope. The early adjustments are usually parameter selection and messaging. Once audits are quiet, the same binding can be promoted to Deny in development, then to staging, then to production. Exceptions are expressed as parameter excludes rather than policy forks. Changes in enforcement are binding changes; the policy itself remains stable.
Performance and Reliability
The most important performance consideration is keeping resource matching out of CEL expressions. Bindings (matchResources, objectSelector, resourceRules) filter which requests reach the CEL evaluation stage, and resource matching in selectors is far less expensive than running CEL expressions. In this framework, that means separate bindings per resource type or label class rather than complex matching logic inside CEL. CEL expressions themselves work best when they’re simple – equality checks, membership tests over small lists, and basic field comparisons. API server admission latency is worth monitoring during adoption, and the number of protected resource types per binding can be adjusted if latency increases.
Limitations
- Policies cannot fully protect themselves; finalizers raise the bar but do not create a closed system. Storing policy and binding YAML in protected repositories, and limiting who can change bindings and parameters, provides the external layer of protection that the policies themselves cannot guarantee.
- Reconcilers will retry denied requests, so starting in Audit mode is advisable; some transient log volume is expected when switching to Deny.
- API version and feature behavior vary by cluster version. The highest stable VAP and Binding versions available in the control plane are preferred, and behavior should be verified during upgrades.
- Parameter selection by label implies a default for missing parameters. The parameterNotFoundAction setting deserves deliberate consideration, and monitoring for gaps helps catch missing parameter objects.
Summary
The framework presented here is intentionally minimal. It uses only built-in Kubernetes primitives – ValidatingAdmissionPolicies, CEL expressions, and parameter CRDs – to create enforceable guardrails that run in-process at the API server. There are no additional services to maintain, no webhook latency, and no external policy engines in the critical path. The approach scales across environments by externalizing scope decisions into bindings and parameters rather than hardcoding them into policy logic.
Normal deployments and updates pass through without interruption, but when automation attempts to delete a protected resource – whether from a bad merge, a misconfigured pruning rule, or a change that wasn’t fully understood – the admission layer blocks it until someone explicitly approves the deletion. This catches the kind of mistakes that would otherwise manifest into major outages or data loss.
Combined with the practices from Part 1 – scoped RBAC, protected resource classes, and tested recovery procedures – admission guardrails complete a layered defense. Together, they provide the context that reconciliation loops cannot infer on their own, ensuring that irreversible operations carry clear attribution and occur within agreed boundaries. Teams can adopt this incrementally, starting with audit mode in non-production environments and promoting to enforcement as confidence builds.
In Part 3: RBAC and Authorization [coming soon], we address the authorization layer in detail, with a practical framework for scoping controller permissions and enforcing ownership boundaries.
