Kubernetes Audit Logging#

Kubernetes audit logging records every request to the API server: who made the request, what they asked for, and what happened. Without audit logging, you have no visibility into who accessed secrets, who changed RBAC roles, or who exec’d into a production pod. It is the foundation of security monitoring in Kubernetes.

Audit Policy#

The audit policy defines which events to record and at what detail level. There are four levels:

None – Do not log the event.
Metadata – Log request metadata (user, timestamp, resource, verb) but not the request or response body.
Request – Log metadata plus the request body.
RequestResponse – Log metadata, request body, and response body.

A Production Audit Policy#

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Do not log read requests to endpoints, events, or health checks
  - level: None
    resources:
      - group: ""
        resources: ["endpoints", "events"]
    verbs: ["get", "list", "watch"]

  - level: None
    nonResourceURLs:
      - "/healthz*"
      - "/readyz*"
      - "/livez*"

  # Log secrets access at Metadata level (never log secret values)
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]

  # Log all RBAC changes with full request details
  - level: Request
    resources:
      - group: "rbac.authorization.k8s.io"
        resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  # Log pod exec and attach at RequestResponse
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # Log service account token requests
  - level: Request
    resources:
      - group: ""
        resources: ["serviceaccounts/token"]

  # Log authentication-related resources
  - level: Request
    resources:
      - group: "authentication.k8s.io"
        resources: ["tokenreviews"]
      - group: "authorization.k8s.io"
        resources: ["subjectaccessreviews"]

  # Log namespace creation and deletion
  - level: Request
    resources:
      - group: ""
        resources: ["namespaces"]
    verbs: ["create", "delete"]

  # Log node operations
  - level: Metadata
    resources:
      - group: ""
        resources: ["nodes"]
    verbs: ["create", "delete", "patch", "update"]

  # Default: log everything else at Metadata
  - level: Metadata
    omitStages:
      - "RequestReceived"

Critical design choice: Never log secrets at the Request or RequestResponse level. This would write secret values into the audit log, moving the problem rather than solving it. Metadata level records that someone accessed a secret without capturing its value.

Enabling the Audit Policy#

Save the policy to /etc/kubernetes/audit/audit-policy.yaml on control plane nodes and configure the API server:

# In /etc/kubernetes/manifests/kube-apiserver.yaml
spec:
  containers:
    - command:
        - kube-apiserver
        - --audit-policy-file=/etc/kubernetes/audit/audit-policy.yaml
        - --audit-log-path=/var/log/kubernetes/audit/audit.log
        - --audit-log-maxage=30
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
      volumeMounts:
        - name: audit-policy
          mountPath: /etc/kubernetes/audit
          readOnly: true
        - name: audit-log
          mountPath: /var/log/kubernetes/audit
  volumes:
    - name: audit-policy
      hostPath:
        path: /etc/kubernetes/audit
        type: DirectoryOrCreate
    - name: audit-log
      hostPath:
        path: /var/log/kubernetes/audit
        type: DirectoryOrCreate

Audit Backends#

Log File Backend#

The simplest backend writes structured JSON to a file. The flags above configure log rotation. Each line is a JSON object:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/secrets/db-password",
  "verb": "get",
  "user": {
    "username": "system:serviceaccount:ci:deploy-bot",
    "groups": ["system:serviceaccounts"]
  },
  "sourceIPs": ["10.0.1.45"],
  "objectRef": {
    "resource": "secrets",
    "namespace": "production",
    "name": "db-password"
  },
  "responseStatus": {
    "code": 200
  },
  "requestReceivedTimestamp": "2026-02-22T10:15:30.000000Z"
}

Webhook Backend#

For real-time processing, send audit events to a webhook:

# In kube-apiserver flags
- --audit-webhook-config-file=/etc/kubernetes/audit/webhook-config.yaml
- --audit-webhook-batch-max-wait=5s

# webhook-config.yaml
apiVersion: v1
kind: Config
clusters:
  - name: audit-webhook
    cluster:
      server: https://audit-receiver.monitoring.svc:8443/audit
      certificate-authority: /etc/kubernetes/audit/ca.crt
contexts:
  - context:
      cluster: audit-webhook
    name: default
current-context: default

The webhook receives batches of audit events as JSON. This integrates with Elasticsearch, Splunk, or any log aggregation system.

What to Audit and Why#

Secrets access – Detect unauthorized reading of secrets. Alert when a service account reads secrets outside its namespace or when an unexpected user accesses database credentials.

RBAC changes – Any creation or modification of Roles, ClusterRoles, RoleBindings, or ClusterRoleBindings. Unauthorized RBAC changes are the most common path to privilege escalation.

Pod exec and attach – Someone executing commands inside a running container. Legitimate in development, a red flag in production. Alert on every exec into a production namespace.

Service account token creation – The kubectl create token API. An attacker with access to create tokens can impersonate service accounts.

Namespace creation and deletion – Attackers create new namespaces to deploy crypto miners or exfiltrate data.

Detecting Suspicious Activity#

Secrets Accessed from Unexpected Sources#

# Find all secret access not from the API server itself
jq 'select(.objectRef.resource == "secrets" and
    .verb == "get" and
    (.user.username | startswith("system:serviceaccount:") | not))' \
  /var/log/kubernetes/audit/audit.log

Privilege Escalation Attempts#

# Detect creation of ClusterRoleBindings to cluster-admin
jq 'select(.objectRef.resource == "clusterrolebindings" and
    .verb == "create" and
    .requestObject.roleRef.name == "cluster-admin")' \
  /var/log/kubernetes/audit/audit.log

Unusual Exec Activity#

# All exec events in production namespaces
jq 'select(.objectRef.resource == "pods" and
    .objectRef.subresource == "exec" and
    .objectRef.namespace == "production")' \
  /var/log/kubernetes/audit/audit.log

Integration with Falco#

Falco complements Kubernetes audit logging by monitoring system calls inside containers. Feed audit events to Falco for rule-based detection:

# In Falco configuration
- --audit-webhook-config-file=/etc/kubernetes/audit/falco-webhook.yaml

Falco ships with rules for common threats: container escape attempts, sensitive file access, unexpected network connections, and shell spawning in containers. Combine Falco’s syscall monitoring with Kubernetes audit events for full-stack visibility.

Performance Considerations#

RequestResponse is expensive. Logging full request and response bodies for high-volume resources (pods, deployments) generates massive log volumes and increases API server latency. Use it only for security-critical, low-volume resources like pod exec and RBAC changes.

Metadata is cheap. It adds minimal overhead and should be the default level for most resources.

None for noisy resources. Endpoints, events, and health checks generate thousands of events per minute. Excluding them dramatically reduces log volume without losing security visibility.

Batch webhook delivery. The --audit-webhook-batch-max-wait flag controls how often events are flushed to the webhook. A 5-second interval balances latency with throughput. Do not set this to 0 in production – it sends an HTTP request for every single audit event.