Kubernetes Audit Logging: Tracking API Activity for Security and Compliance#

Audit logging records every request to the Kubernetes API server. Every kubectl command, every controller reconciliation, every kubelet heartbeat, every admission webhook call – all of it can be captured with the requester’s identity, the target resource, the timestamp, and optionally the full request and response bodies. Without audit logging, you have no record of who did what in your cluster. With it, you can trace security incidents, satisfy compliance requirements, and debug access control issues.

Audit Stages#

Each API request passes through four stages. Audit events can be generated at any stage:

Most audit configurations generate events at ResponseComplete since this captures the outcome (response code) along with the request details.

Audit Levels#

The audit level controls how much detail is captured for each event:

The trade-off is clear: RequestResponse gives you complete forensic detail but generates enormous log volume. Metadata is compact and sufficient for most operational and compliance needs. Request captures what was submitted (useful for seeing exactly what RBAC change was made) without the response bloat.

Audit Policy#

The audit policy is a YAML file that defines which requests are logged at which level. Rules are evaluated in order – the first match wins. Requests that match no rule are logged at the Metadata level by default (or not at all if there is a catch-all None rule).

apiVersion: audit.k8s.io/v1
kind: Policy
# Do not log requests to these endpoints (high volume, low value)
omitStages:
  - "RequestReceived"
rules:
  # Do not log health checks, readiness probes, or API discovery
  - level: None
    nonResourceURLs:
    - /healthz*
    - /livez*
    - /readyz*
    - /metrics
    - /openapi/v2*

  # Do not log events from the system:nodes group (kubelet heartbeats)
  - level: None
    users: ["system:apiserver"]
    resources:
    - group: ""
      resources: ["endpoints", "services", "services/status"]

  # Log Secret access at Metadata level only (do NOT log Secret bodies)
  - level: Metadata
    resources:
    - group: ""
      resources: ["secrets"]

  # Log RBAC changes with full request body
  - level: Request
    resources:
    - group: "rbac.authorization.k8s.io"
      resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]

  # Log pod exec and attach with full request
  - level: Request
    resources:
    - group: ""
      resources: ["pods/exec", "pods/attach", "pods/portforward"]

  # Log token creation
  - level: Request
    resources:
    - group: ""
      resources: ["serviceaccounts/token"]

  # Log all other write operations at Metadata level
  - level: Metadata
    verbs: ["create", "update", "patch", "delete", "deletecollection"]

  # Log everything else at Metadata level
  - level: Metadata

The policy above represents a practical starting point: verbose logging for security-sensitive operations (RBAC, exec, tokens), metadata-only for routine operations, and silence for high-volume noise (health checks, system heartbeats). Never log Secret bodies – Request or RequestResponse level on Secrets means Secret values appear in your audit logs, creating a security hole in your audit infrastructure.

Audit Backends#

Log Backend#

Writes audit events as JSON lines to a file on disk. The API server handles file rotation via built-in flags.

Configure on the API server:

--audit-policy-file=/etc/kubernetes/audit/policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=30          # days to retain old log files
--audit-log-maxbackup=10       # number of old log files to keep
--audit-log-maxsize=100        # max size in MB before rotation
--audit-log-format=json        # json or legacy

For kubeadm clusters, add these to the API server static pod manifest:

# /etc/kubernetes/manifests/kube-apiserver.yaml (relevant excerpts)
spec:
  containers:
  - command:
    - kube-apiserver
    - --audit-policy-file=/etc/kubernetes/audit/policy.yaml
    - --audit-log-path=/var/log/kubernetes/audit/audit.log
    - --audit-log-maxage=30
    - --audit-log-maxbackup=10
    - --audit-log-maxsize=100
    volumeMounts:
    - mountPath: /etc/kubernetes/audit
      name: audit-policy
      readOnly: true
    - mountPath: /var/log/kubernetes/audit
      name: audit-log
  volumes:
  - hostPath:
      path: /etc/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-policy
  - hostPath:
      path: /var/log/kubernetes/audit
      type: DirectoryOrCreate
    name: audit-log

Webhook Backend#

Sends audit events to an external HTTP endpoint in real-time. This integrates with SIEM systems (Splunk, Elasticsearch, Datadog) or dedicated audit analysis tools.

# /etc/kubernetes/audit/webhook-config.yaml
apiVersion: v1
kind: Config
clusters:
- name: audit-webhook
  cluster:
    server: https://audit-collector.monitoring.svc:8443/audit
    certificate-authority: /etc/kubernetes/audit/ca.crt
contexts:
- context:
    cluster: audit-webhook
  name: default
current-context: default

--audit-webhook-config-file=/etc/kubernetes/audit/webhook-config.yaml
--audit-webhook-batch-max-wait=5s
--audit-webhook-batch-max-size=100

The webhook backend batches events for efficiency. Tune batch-max-wait and batch-max-size based on your latency and throughput requirements. If the webhook endpoint is unreachable, events are buffered in memory and eventually dropped – monitor the apiserver_audit_event_total and apiserver_audit_error_total metrics.

Managed Kubernetes Audit Logging#

Managed services handle audit logging differently since you do not have access to the API server flags.

Amazon EKS. Enable audit logging through the cluster logging configuration. Audit logs go to CloudWatch Logs under the /aws/eks/<cluster-name>/cluster log group.

aws eks update-cluster-config --name my-cluster \
  --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

Azure AKS. Configure diagnostic settings to send the kube-audit log category to a Log Analytics workspace, storage account, or Event Hub.

az monitor diagnostic-settings create \
  --name audit-logs \
  --resource $(az aks show -g my-rg -n my-cluster --query id -o tsv) \
  --workspace $(az monitor log-analytics workspace show -g my-rg -n my-workspace --query id -o tsv) \
  --logs '[{"category":"kube-audit","enabled":true}]'

Google GKE. Admin Activity audit logs are enabled by default at no charge and cannot be disabled. Data Access audit logs (which include read operations) must be enabled separately and incur logging charges.

Analyzing Audit Logs#

Key Security Events to Monitor#

# Find unauthorized access attempts (403 responses)
jq 'select(.responseStatus.code == 403)' audit.log | \
  jq '{user: .user.username, verb: .verb, resource: .objectRef.resource, ns: .objectRef.namespace}'

# Track who accessed Secrets
jq 'select(.objectRef.resource == "secrets" and .verb == "get")' audit.log | \
  jq '{user: .user.username, secret: .objectRef.name, ns: .objectRef.namespace, time: .requestReceivedTimestamp}'

# Find RBAC modifications
jq 'select(.objectRef.apiGroup == "rbac.authorization.k8s.io" and
    (.verb == "create" or .verb == "update" or .verb == "patch" or .verb == "delete"))' audit.log | \
  jq '{user: .user.username, verb: .verb, kind: .objectRef.resource, name: .objectRef.name}'

# Track exec into pods (potential lateral movement)
jq 'select(.objectRef.subresource == "exec")' audit.log | \
  jq '{user: .user.username, pod: .objectRef.name, ns: .objectRef.namespace, time: .requestReceivedTimestamp}'

# Find privilege escalation attempts (creating pods with hostPath or privileged)
jq 'select(.objectRef.resource == "pods" and .verb == "create" and .responseStatus.code == 201)' audit.log | \
  jq 'select(.requestObject.spec.containers[].securityContext.privileged == true) |
      {user: .user.username, pod: .objectRef.name, ns: .objectRef.namespace}'

Audit Event Structure#

A single audit event in JSON format:

{
  "kind": "Event",
  "apiVersion": "audit.k8s.io/v1",
  "level": "Metadata",
  "auditID": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
  "stage": "ResponseComplete",
  "requestURI": "/api/v1/namespaces/production/secrets/db-credentials",
  "verb": "get",
  "user": {
    "username": "alice@example.com",
    "groups": ["dev-team", "system:authenticated"]
  },
  "sourceIPs": ["10.0.1.50"],
  "objectRef": {
    "resource": "secrets",
    "namespace": "production",
    "name": "db-credentials",
    "apiVersion": "v1"
  },
  "responseStatus": {
    "code": 200
  },
  "requestReceivedTimestamp": "2026-02-22T10:30:45.123456Z",
  "stageTimestamp": "2026-02-22T10:30:45.125678Z"
}

Compliance Frameworks#

Audit logging is not optional in regulated environments:

CIS Kubernetes Benchmark. Section 3.2 requires audit logging to be enabled with an appropriate policy. The benchmark recommends logging at Metadata level minimum for all requests, with Request level for sensitive resources.

SOC 2 (Trust Services Criteria). CC6.1 requires logging of user activities and CC7.2 requires monitoring for anomalies. Kubernetes audit logs satisfy these when combined with alerting on suspicious patterns.

PCI-DSS. Requirement 10 mandates tracking access to cardholder data environments. If your Kubernetes cluster processes card data, you must log and monitor all API access, especially to namespaces containing payment workloads.

HIPAA. Requires audit controls for information systems containing protected health information. Access to PHI-related Kubernetes resources must be logged and reviewable.

Falco Integration#

Falco complements audit logging by monitoring at the syscall level inside containers. While audit logs tell you who called kubectl exec, Falco tells you what commands were run inside the container after exec.

# Falco can consume Kubernetes audit events directly
# In falco.yaml:
webserver:
  enabled: true
  listen_port: 8765

# Configure the API server to send audit events to Falco
# --audit-webhook-config-file pointing to Falco's endpoint

Falco rules can trigger alerts on both audit events and runtime behavior:

• A user execs into a production pod (audit event)
• A process inside a container reads /etc/shadow (syscall event)
• A container spawns an unexpected shell process (syscall event)

Storage Considerations#

Audit logs grow rapidly. A moderately active cluster with 50 nodes and common controller activity can generate 1-5 GB of audit logs per day at Metadata level. At RequestResponse level, expect 10-50x more.

Strategies for managing volume:

1. Filter aggressively. Use None level for high-frequency, low-value events (health checks, leader election, node heartbeats).
2. Use Metadata level as the default. Reserve Request and RequestResponse for security-critical resources only.
3. Set retention policies. Keep detailed logs for 30-90 days, archive summarized data longer for compliance.
4. Stream to external storage. Use the webhook backend to send events to a scalable log store (Elasticsearch, S3, cloud-native logging) rather than relying on local disk.

Common Gotchas#

RequestResponse for everything causes API server performance degradation. Serializing every response body (especially large list operations) adds latency and memory pressure to the API server. A kubectl get pods -A response body can be several megabytes. Multiply this by every controller’s periodic list operation, and the API server spends significant resources just generating audit data. Start with Metadata and selectively increase.

No audit policy means no audit logs. Many clusters, especially development and self-managed installations, ship with no audit policy configured. The API server does not generate audit events unless --audit-policy-file is explicitly set. Check your API server flags – if audit is not configured, you have zero visibility into API activity.

Audit logs containing Secrets. If you set Request or RequestResponse level for Secrets, the actual Secret values (base64-encoded, trivially decoded) appear in your audit logs. Anyone with access to the audit log storage can read every Secret in the cluster. Always use Metadata level for Secrets.