Skip to content
UNRESOVED
Completed· 20 Aug 2026

Can a Kubernetes security control prevent an action without generating useful evidence?

KubernetesSecurityObservability

Question

When a validating admission webhook blocks a request, does the cluster retain enough evidence to reconstruct what was attempted, by whom, and how many times?

Hypothesis

A denied admission request generates a Kubernetes API audit event, so I expected the default audit configuration to give us everything we'd need for a post-incident timeline without any extra instrumentation.

Setup

A kind cluster running a validating admission webhook (built on Kyverno) that blocks any Pod spec requesting hostNetwork: true. Kubernetes audit logging enabled at the default "Metadata" level, the level most managed clusters ship with out of the box.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-host-network
spec:
  validationFailureAction: Enforce
  rules:
    - name: no-host-network
      match:
        resources:
          kinds: [Pod]
      validate:
        message: "hostNetwork is not permitted"
        pattern:
          spec:
            =(hostNetwork): "false"

Experiment

I attempted to deploy 12 pods requesting hostNetwork: true from three different service accounts over a 20-minute window, simulating a mix of a misconfigured deploy pipeline and a deliberate privilege-escalation attempt. Then I tried to answer four questions using only what the cluster had recorded: who tried this, how many times, from where, and whether the attempts were increasing in frequency.

Result

The Kyverno policy worked exactly as designed: all 12 pods were rejected at admission. But at the default "Metadata" audit level, the retained audit event recorded the verb, the resource type, the user, and the response code (denied), and nothing about the request body. There was no record of hostNetwork: true specifically, no way to distinguish this denial from any other validation failure, and no record of which container image or namespace was involved beyond what "Metadata" level exposes. Reconstructing intent from the audit trail alone was not possible. Raising audit logging to "RequestResponse" level made every field available (including the full pod spec) but increased audit log volume by roughly 40x in this test, which is the reason most clusters don't run at that level by default.

What I learned

A prevention control and a detection control are not the same investment, and enabling one doesn't give you the other for free. The admission webhook did its job completely: nothing was allowed to run. But if this had been a real privilege-escalation attempt instead of a test, security response would have known an attack was stopped and almost nothing else about it: not the technique, not whether it was automated, not whether it was still being retried. Getting real evidence required deliberately raising audit verbosity for the resources that matter, which is a cost/signal trade-off worth making on purpose rather than discovering during an incident.