What Cardinality Means#

In Prometheus, cardinality is the number of unique time series. Every unique combination of metric name and label key-value pairs constitutes one series. The metric http_requests_total{method="GET", path="/api/users", status="200"} is one series. Change any label value and you get a different series. http_requests_total{method="POST", path="/api/users", status="201"} is a second series.

A single metric name can produce thousands or millions of series depending on its labels. A metric with no labels is exactly one series. A metric with one label that has 10 possible values is 10 series. A metric with three labels, each having 100 possible values, is up to 1,000,000 series (100 x 100 x 100), though in practice not every combination occurs.

Why Cardinality Matters#

Prometheus memory usage scales linearly with the number of active series. The in-memory head block holds all currently active series, their labels, and recent samples. Rough guidelines for RAM:

• 1 million active series: 3-4 GB RAM
• 5 million active series: 15-20 GB RAM
• 10 million active series: 30-40 GB RAM, queries start slowing down
• 50 million active series: requires a distributed system (Thanos, Mimir, VictoriaMetrics)

Beyond memory, high cardinality affects query performance (more series to scan), compaction time (more blocks to merge), WAL replay on restart (longer startup time), and scrape duration (more samples per scrape).

The danger is that cardinality problems are often invisible until they become critical. A deployment works fine with 500K series. Someone adds a label with user IDs to a metric. Over weeks, as more users interact with the system, series count climbs silently. One day Prometheus OOMs and you discover 8 million series where you expected 500K.

Detecting Cardinality Problems#

Built-in Metrics#

Prometheus exposes its own internal metrics that reveal cardinality issues:

# Total active series in the head block
prometheus_tsdb_head_series

# Rate of new series creation (churn)
rate(prometheus_tsdb_head_series_created_total[1h])

# Samples appended per second
rate(prometheus_tsdb_head_samples_appended_total[5m])

A steadily increasing prometheus_tsdb_head_series indicates new label combinations are appearing continuously. This is the signature of a high-cardinality label.

TSDB Status API#

The /api/v1/status/tsdb endpoint provides detailed breakdowns:

curl -s http://prometheus:9090/api/v1/status/tsdb | jq .

This returns the top 10 metrics by series count, the top 10 label names by number of unique values, and the top 10 label name-value pairs by series count. This is your starting point for any cardinality investigation.

PromQL Queries for Investigation#

# Top 10 metrics by series count
topk(10, count by (__name__)({__name__=~".+"}))

# Series count for a specific metric
count(http_requests_total)

# Series count by label for a specific metric
count by (path) (http_requests_total)

# Find metrics with more than 1000 series
count by (__name__)({__name__=~".+"}) > 1000

# Cardinality of a specific label across all metrics
count(group by (instance) ({__name__=~".+"}))

The first query is expensive on large Prometheus instances. Run it during low-traffic periods or use the TSDB status API instead.

Grafana Cardinality Management Dashboard#

Grafana provides a dedicated cardinality management dashboard (available as a plugin or built into Grafana Cloud) that visualizes series count over time, identifies high-cardinality labels, and shows which metrics contribute the most series. If you use Grafana, this dashboard should be part of your standard monitoring setup.

Common Causes#

Unbounded label values: the most frequent cause. Labels containing user IDs, email addresses, request IDs, UUIDs, session tokens, or IP addresses create a new series for every unique value. A metric with a user_id label and 100,000 users generates 100,000 series from that one metric alone.

Dynamic URL paths: using the full HTTP request path as a label value. A path like /users/12345/orders/67890 creates a unique series for every user-order combination. URLs with query strings (/search?q=unique-term) are even worse – effectively unbounded.

Label explosion: multiple labels with moderately high cardinality multiplied together. Five labels with 100 values each is 10 billion theoretical combinations. Even if only 0.1% of combinations occur, that is still 10 million series.

Uncontrolled instrumentation libraries: some instrumentation libraries or middleware automatically add labels for every dimension they observe. An HTTP middleware that adds path, method, status, content_type, and user_agent labels creates far more series than one that adds only method and status_class.

Series churn: short-lived containers or pods that create new series on every restart. Each restart produces a new instance label value. With high pod churn (frequent deployments, autoscaling), old series accumulate until the retention period expires.

Prevention Strategies#

Label Guidelines#

Establish rules before cardinality becomes a problem:

• Maximum 5-7 labels per metric
• Each label should have bounded cardinality, ideally fewer than 100 unique values
• Labels must represent dimensions you actually query and aggregate by
• If you would never write sum by (label_name), the label should not exist

Path Normalization#

Normalize dynamic URL segments before using them as label values:

// Bad: raw path as label
requestsTotal.WithLabelValues(r.URL.Path).Inc()
// Creates: http_requests_total{path="/users/12345"}
//          http_requests_total{path="/users/67890"}

// Good: normalized path template
requestsTotal.WithLabelValues(normalizePath(r.URL.Path)).Inc()
// Creates: http_requests_total{path="/users/:id"}

Most HTTP router frameworks provide the matched route pattern. Use that instead of the raw URL.

Drop Labels at Scrape Time#

Use metric_relabel_configs in the Prometheus scrape config to remove problematic labels before they are stored:

scrape_configs:
  - job_name: "myapp"
    metric_relabel_configs:
      # Drop specific high-cardinality labels
      - action: labeldrop
        regex: "request_id|trace_id|session_id"

      # Drop entire metrics that are too expensive
      - source_labels: [__name__]
        regex: "myapp_debug_.*"
        action: drop

      # Keep only specific metrics from a verbose exporter
      - source_labels: [__name__]
        regex: "myapp_(requests_total|latency_seconds_bucket|errors_total)"
        action: keep

Recording Rules for Pre-Aggregation#

Instead of storing high-cardinality raw metrics and querying them, aggregate at write time with recording rules:

groups:
  - name: cardinality_reduction
    rules:
      # Aggregate per-path metrics down to per-service
      - record: job:http_requests:rate5m
        expr: sum by (job, method, status_code) (rate(http_requests_total[5m]))

      # Pre-compute latency percentiles without the path dimension
      - record: job:http_request_duration:p99_5m
        expr: |
          histogram_quantile(0.99,
            sum by (le, job) (rate(http_request_duration_seconds_bucket[5m])))

Dashboards and alerts query the recorded metric (job:http_requests:rate5m) instead of the raw metric. The raw metric can then have a shorter retention or be dropped entirely.

Reduction Strategies#

When cardinality is already out of control and Prometheus is under pressure, act in this order:

Step 1: Identify the offending metric. Check the TSDB status API or run topk(10, count by (__name__)({__name__=~".+"})). Find which metric has the most series. Then drill into its labels with count by (label_name) (offending_metric) to find which label is causing the explosion.

Step 2: Drop the metric entirely if it is not critical. The fastest way to reduce cardinality is to stop storing the metric:

metric_relabel_configs:
  - source_labels: [__name__]
    regex: "offending_metric_name"
    action: drop

Step 3: Drop specific labels. If the metric is needed but a specific label is the problem:

metric_relabel_configs:
  - action: labeldrop
    regex: "problematic_label"

Step 4: Bucket label values. Replace high-cardinality label values with grouped categories:

metric_relabel_configs:
  - source_labels: [path]
    regex: "/api/users/.*"
    target_label: path
    replacement: "/api/users/:id"
  - source_labels: [path]
    regex: "/api/orders/.*"
    target_label: path
    replacement: "/api/orders/:id"

Step 5: Create recording rules and drop the original. Pre-aggregate into recording rules, update dashboards and alerts to use the recorded metrics, then drop the original high-cardinality metric from scraping.

Monitoring Cardinality Continuously#

Set up alerts that fire before cardinality becomes an emergency:

groups:
  - name: cardinality-alerts
    rules:
      - alert: HighSeriesCount
        expr: prometheus_tsdb_head_series > 2000000
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Prometheus has {{ $value }} active series"
          description: "Series count exceeds 2M threshold."

      - alert: HighSeriesChurn
        expr: rate(prometheus_tsdb_head_series_created_total[1h]) > 1000
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "High series churn: {{ $value }} new series/sec"
          description: "New series are being created at an unusually high rate."

      - alert: PrometheusHighMemory
        expr: process_resident_memory_bytes{job="prometheus"} / 1e9 > 30
        for: 10m
        labels:
          severity: critical
        annotations:
          summary: "Prometheus using {{ $value | humanize }}GB of memory"

Adjust the thresholds to match your environment. The series churn alert is particularly useful – it catches cardinality creep early, before total series count triggers the upper threshold.

Cardinality in the Pipeline#

Labels enter the system at multiple stages, and you need awareness of each:

Application code: the developer instruments a metric with labels. This is where cardinality problems originate and where they should ideally be prevented through code review and instrumentation guidelines.

Service discovery: Kubernetes SD, Consul SD, and other discovery mechanisms add __meta_* labels. relabel_configs controls which of these are promoted to target labels. Labels like __meta_kubernetes_pod_name become series dimensions if promoted.

Relabeling (pre-scrape): relabel_configs runs before scraping and determines the target label set. Overly broad labelmap rules can promote many meta-labels into stored labels.

Relabeling (post-scrape): metric_relabel_configs runs after scraping and before storage. This is your last chance to drop or modify labels before they create series in the TSDB.

Common Gotchas#

Fixing cardinality in the Prometheus configuration does not retroactively fix stored data. When you drop a label via metric_relabel_configs, new scrapes no longer create series with that label, but existing series remain in the TSDB until the retention period expires. If you need immediate relief, you can delete specific series via the admin API (POST /api/v1/admin/tsdb/delete_series), but this requires the --web.enable-admin-api flag and is a manual operation.

The metric_relabel_configs directive runs after scraping. The data has already been transferred over the network from the target to Prometheus. For high-volume metrics, the scrape itself consumes bandwidth and target resources even if Prometheus drops the data afterward. If a target exposes millions of samples per scrape and you only keep a few, it is more efficient to filter at the source – modify the application to stop exposing those metrics, or use a filtering proxy between the target and Prometheus.