SIEM and Security Log Correlation#

A SIEM collects logs from across your infrastructure, normalizes them, and applies correlation rules to detect threats that no single log source would reveal. A brute force attempt is visible in auth logs. Lateral movement after successful brute force requires correlating auth logs with network flow data and process execution logs. The SIEM makes that correlation possible.

Log Sources#

The value of a SIEM depends entirely on the logs you feed it. Missing a log source means missing the attacks that source would reveal.

Operating System Audit Logs#

Linux audit logs (auditd) record system calls, file access, user authentication, and privilege changes. These are the foundation for detecting host-level compromise.

# Key audit rules for security monitoring
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/sudoers -p wa -k privilege_escalation
-a always,exit -F arch=b64 -S execve -k process_execution
-a always,exit -F arch=b64 -S connect -k network_connections

On Kubernetes nodes, auditd captures container escapes and privilege escalation that container-level monitoring misses. Windows Security Event Log provides equivalent data: logon events (4624/4625), privilege use (4672), and process creation (4688).

Kubernetes API Server Audit Logs#

Every interaction with the Kubernetes API is an auditable event. API audit logs capture who did what, when, and from where:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  - level: Metadata
    resources:
      - group: ""
        resources: ["secrets", "configmaps"]
  - level: RequestResponse
    resources:
      - group: ""
        resources: ["pods/exec", "pods/portforward"]
  - level: Metadata
    verbs: ["create", "delete", "patch", "update"]

Critical events to monitor: pods/exec (someone executing commands inside a running pod), secrets access (credential theft), clusterrolebindings creation (privilege escalation), and serviceaccounts/token requests (identity theft).

Network Flow Data#

Network flows record which hosts communicated, on which ports, for how long, and how much data was transferred. They do not capture packet contents, but they reveal communication patterns.

Zeek produces structured network metadata from packet captures (connection records, DNS queries, HTTP transactions). VPC Flow Logs in AWS, GCP, and Azure capture network flows without packet capture infrastructure. Suricata combines network intrusion detection with flow logging, inspecting traffic against rule sets (EmergingThreats, ET Pro).

Application Logs#

Application-level logs capture authentication events, authorization decisions, and data access that infrastructure logs cannot see. Standardize formats across services with structured JSON including timestamp, service name, user identifier, action, target resource, and outcome:

{
  "timestamp": "2026-02-22T14:30:00Z",
  "service": "auth-api",
  "level": "warn",
  "event": "login_failed",
  "user": "admin@example.com",
  "source_ip": "203.0.113.45",
  "reason": "invalid_password",
  "attempt_count": 4
}

Cloud Provider Logs#

AWS CloudTrail, GCP Cloud Audit Logs, and Azure Activity Logs record every API call across their respective platforms. These detect IAM changes, security group modifications, storage policy changes, and other control plane actions indicating account compromise or insider threat.

Correlation Rules#

Individual log events are noise. Correlation rules combine events across sources and time windows to produce meaningful alerts.

Brute Force Detection#

A single failed login is normal. Fifty failed logins from the same IP in five minutes is an attack:

# Sigma rule format
title: Brute Force Attempt
logsource:
  category: authentication
detection:
  selection:
    event_type: login_failed
  condition: selection | count(source_ip) by target_user > 20
  timeframe: 5m
level: high

Lateral Movement#

A successful login from a source IP that previously failed authentication against multiple hosts suggests credential compromise and lateral movement:

Sequence:
  1. Multiple failed logins from IP-A to hosts B, C, D within 10 minutes
  2. Successful login from IP-A to host E within 30 minutes of step 1
  3. New outbound connections from host E to internal hosts
Alert: Possible lateral movement after brute force

Privilege Escalation#

A web server process spawning a shell and escalating to root is not normal:

Sequence:
  1. Web server process spawns child process (/bin/sh or /bin/bash)
  2. Child process executes privilege escalation (sudo, su, pkexec)
  3. New process runs as root
Alert: Web shell with privilege escalation

Data Exfiltration#

Correlate data access with network flows:

Sequence:
  1. Database query returns > 10,000 rows (application log)
  2. Outbound transfer > 100MB to external IP (network flow)
  3. Destination IP not in known-good list
Alert: Possible data exfiltration

MITRE ATT&CK Mapping#

The MITRE ATT&CK framework catalogs adversary tactics and techniques. Mapping your detection rules to ATT&CK identifies coverage gaps.

Key tactics and the log sources that detect them:

Build a detection coverage matrix. For each ATT&CK technique relevant to your environment, list which detection rules cover it and which log sources feed those rules. Gaps in the matrix are blind spots.

Sigma Rules#

Sigma is the open standard for detection rules, analogous to Snort/Suricata rules for network detection but applicable to any log source. Write detection logic once in Sigma format and convert it to your SIEM’s native query language.

title: Kubernetes Pod Exec
id: a]f8e4b2-1234-5678-9abc-def012345678
status: stable
description: Detects kubectl exec into a running pod
logsource:
  product: kubernetes
  service: audit
detection:
  selection:
    verb: create
    objectRef.resource: pods
    objectRef.subresource: exec
  condition: selection
level: medium
tags:
  - attack.execution
  - attack.t1609

Convert Sigma rules to SIEM-specific formats:

# Convert to Elasticsearch query
sigma convert -t elasticsearch rule.yml

# Convert to Splunk SPL
sigma convert -t splunk rule.yml

# Convert to OpenSearch
sigma convert -t opensearch rule.yml

The Sigma community maintains thousands of detection rules. Start with these rather than writing everything from scratch.

Open-Source SIEM Options#

Wazuh#

Wazuh combines host-based intrusion detection, log analysis, file integrity monitoring, vulnerability detection, and compliance reporting. Agents run on monitored hosts and send data to the Wazuh server, which stores events in OpenSearch.

Strengths: built-in CIS benchmark scanning, file integrity monitoring, active response, Docker/Kubernetes monitoring, and pre-built detection rules. Deploys on Kubernetes via the official Helm chart.

Limitations: default rules produce high false positive rates – expect significant tuning time. OpenSearch requires 16GB+ memory for production.

Security Onion#

Security Onion bundles Suricata, Zeek, Elasticsearch, and Kibana for network security monitoring with pre-built threat hunting dashboards.

Strengths: full packet capture, network flow analysis, and IDS alerting in a single deployment with an integrated analyst workflow.

Limitations: network-focused only. Best deployed alongside Wazuh rather than instead of it.

ELK/OpenSearch Stack#

Running Elasticsearch or OpenSearch with Kibana gives you a log aggregation platform you can build SIEM functionality on top of. Maximum flexibility but significant engineering investment – you must build detection rules, correlation logic, alerting, and case management yourself. Consider this only if Wazuh or Security Onion do not fit your requirements.

Cloud-Native Options#

AWS Security Hub aggregates findings from GuardDuty, Inspector, and Macie, mapping them to CIS and PCI-DSS standards. GuardDuty handles threat detection for CloudTrail, VPC Flow Logs, and DNS logs.

Google Cloud SCC centralizes security findings across GCP. Chronicle provides petabyte-scale log analysis with sub-second search.

Microsoft Sentinel is Azure’s cloud-native SIEM with hundreds of pre-built connectors, MITRE ATT&CK-mapped detection rules, and automated response playbooks via Logic Apps.

For hybrid environments, use cloud-native tools for cloud-specific logs and an open-source SIEM (Wazuh) for on-premises and Kubernetes infrastructure. Forward cloud findings to the central SIEM for cross-environment correlation.

Implementation Priorities#

Start with the log sources covering the most common attack patterns: (1) authentication logs from all systems, (2) Kubernetes API audit logs, (3) network flow data from VPC flow logs or Zeek, and (4) cloud provider audit logs. Add application logs, file integrity monitoring, and process execution logging as your detection engineering matures. Each new log source creates correlation opportunities but increases storage costs – add sources intentionally, with specific detection rules planned for each.