Elasticsearch and OpenSearch: Indexing, Queries, Cluster Management, and Performance#

Elasticsearch and OpenSearch are distributed search and analytics engines built on Apache Lucene. They excel at full-text search, log aggregation, metrics storage, and any workload that benefits from inverted indices. Understanding index design, mappings, query mechanics, and cluster management separates a working setup from one that collapses under production load.

Elasticsearch vs OpenSearch#

OpenSearch is the AWS-maintained fork of Elasticsearch, created after Elastic changed its license from Apache 2.0 to the Server Side Public License (SSPL) in early 2021. For the vast majority of use cases, the two are interchangeable. APIs are compatible, concepts are identical, and most tooling works with both. OpenSearch Dashboards replaces Kibana. This guide applies to both unless explicitly noted.

If you are on AWS, OpenSearch is the default choice (it is what Amazon OpenSearch Service runs). If you are self-hosting or using Elastic Cloud, Elasticsearch is the native option. The core knowledge transfers completely between them.

Core Concepts#

Indices are the primary unit of organization, analogous to a database table. An index contains documents that share a similar structure.

Documents are individual JSON objects stored within an index. Each document has a unique _id within its index.

Mappings define the schema for documents in an index – what fields exist, their data types, and how they are indexed. Mappings can be defined explicitly or created dynamically when the first document is indexed.

Shards are the mechanism for distributing an index across multiple nodes. An index is divided into N primary shards, and each shard is an independent Lucene index. Shard count is set at index creation and cannot be changed afterward (without reindexing).

Replicas are copies of primary shards. They provide fault tolerance (if a node holding a primary shard goes down, a replica can be promoted) and read scaling (search requests can be served by replicas).

Index Design#

Shard count is the most consequential index design decision because it cannot be changed after creation.

Too few shards means the index cannot be distributed across enough nodes, limiting horizontal scaling and creating hot spots. A single shard can hold a lot of data, but search parallelism is limited.

Too many shards creates overhead. Each shard consumes file descriptors, memory, and CPU. Hundreds of tiny shards degrade cluster performance more than a few large ones.

Rule of thumb: 20-50GB per shard. For an index expected to hold 200GB, start with 4-10 primary shards. For time-series data (logs, metrics) where indices roll over daily, size based on daily volume.

Set replicas based on your fault tolerance needs: 1 replica means the cluster can lose one node without data loss. For production, number_of_replicas: 1 is the standard starting point.

PUT /my-index
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}

Mappings: Field Types and Analysis#

Mappings define how fields are stored and searched. The two most important field types are text and keyword.

text fields are analyzed: the content is tokenized, lowercased, and processed through an analyzer before being stored in the inverted index. Use text for fields that need full-text search – descriptions, titles, log messages.

keyword fields are stored as-is, without analysis. Use keyword for exact-match filtering, sorting, and aggregations – status codes, email addresses, tags, identifiers.

A common pattern is to map a field as both:

PUT /my-index
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "status": { "type": "keyword" },
      "created_at": { "type": "date" },
      "price": { "type": "float" },
      "tags": { "type": "keyword" },
      "metadata": { "type": "object" },
      "comments": { "type": "nested" }
    }
  }
}

Nested vs Object. The object type flattens inner objects, which can produce incorrect results when querying across fields of inner objects. nested maintains the relationship between fields within each object but uses more memory and requires special nested queries.

CRUD Operations#

Index a document:

POST /my-index/_doc
{
  "title": "Redis Deep Dive",
  "status": "published",
  "created_at": "2026-02-21"
}

Get by ID: GET /my-index/_doc/abc123

Update: POST /my-index/_update/abc123 { "doc": { "status": "archived" } }

Delete: DELETE /my-index/_doc/abc123

Bulk operations are essential for performance. Never index documents one at a time in production:

POST /_bulk
{ "index": { "_index": "my-index" } }
{ "title": "Document 1", "status": "draft" }
{ "index": { "_index": "my-index" } }
{ "title": "Document 2", "status": "published" }

Batch size of 1000-5000 documents per bulk request is a reasonable starting point. Tune based on document size and cluster capacity.

Query DSL#

The Query DSL is the primary interface for searching. Understanding the distinction between queries (scored, full-text) and filters (yes/no, exact match, cached) is critical for performance.

Match query – full-text search on analyzed text fields:

GET /my-index/_search
{
  "query": {
    "match": { "title": "redis caching" }
  }
}

Term query – exact match on keyword fields:

{ "query": { "term": { "status": "published" } } }

Bool query – combine multiple conditions:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "title": "redis" } }
      ],
      "filter": [
        { "term": { "status": "published" } },
        { "range": { "created_at": { "gte": "2026-01-01" } } }
      ],
      "must_not": [
        { "term": { "status": "archived" } }
      ]
    }
  }
}

Use filter instead of must for exact-match and range conditions. Filters do not calculate relevance scores and are cached by Elasticsearch, making them significantly faster.

Aggregations compute analytics over search results:

{
  "size": 0,
  "aggs": {
    "status_counts": {
      "terms": { "field": "status" }
    },
    "avg_price": {
      "avg": { "field": "price" }
    }
  }
}

Performance Tuning#

Bulk indexing. Always use the _bulk API. Single-document indexing creates massive overhead from individual HTTP requests and per-document index refreshes.

Refresh interval. By default, Elasticsearch refreshes indices every 1 second, making new documents searchable. For write-heavy workloads (log ingestion, bulk imports), increase the refresh interval: "index.refresh_interval": "30s". After bulk loading is complete, reset to 1s or trigger a manual refresh.

Disable replicas during bulk load. Set "number_of_replicas": 0 before a large bulk import, then restore replicas afterward. This avoids the overhead of replicating every document during the initial load.

Force merge after bulk load. POST /my-index/_forcemerge?max_num_segments=1 consolidates Lucene segments, improving search performance. Only do this on indices that are no longer being written to (for example, after rolling over a time-series index).

Avoid deep pagination. from + size pagination is expensive beyond 10,000 results. Use search_after with a sort value for deep pagination, or the scroll API for batch processing large result sets.

Cluster Architecture#

A production cluster has distinct node roles:

Master-eligible nodes manage cluster state – index creation, shard allocation, node membership. Run 3 dedicated master nodes for high availability. Master nodes need minimal CPU and storage but should have reliable, low-latency networking.

Data nodes store shards and execute search and indexing operations. These are the workhorses and need fast storage (SSDs), ample memory, and sufficient CPU. Elasticsearch uses the OS filesystem cache heavily, so allocate no more than 50% of available memory to the JVM heap (the rest benefits the filesystem cache).

Coordinating nodes (also called client nodes) route requests to the appropriate data nodes and aggregate results. They are optional but useful for large clusters where query aggregation is a bottleneck.

Ingest nodes run ingest pipelines that transform documents before indexing (parsing, enriching, converting formats). For most clusters, data nodes can handle ingest unless pipeline processing is CPU-intensive.

Index Lifecycle Management (ILM)#

ILM automates the management of time-series indices through phases:

Hot phase: Actively written to. Fast storage, full replicas. Index rolls over when it reaches a size or age threshold (for example, 50GB or 1 day).

Warm phase: No longer written to, still queried. Can be moved to slower storage, force-merged, and shrunk (reduce shard count).

Cold phase: Rarely queried. Can be moved to cheapest storage, replicas reduced.

Delete phase: Index is deleted after the retention period.

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50gb",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "forcemerge": { "max_num_segments": 1 },
          "shrink": { "number_of_shards": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Monitoring#

Cluster health: GET _cluster/health returns green (all shards allocated), yellow (primary shards allocated but some replicas are not), or red (some primary shards are unallocated – data may be unavailable).

Index status: GET _cat/indices?v shows index size, document count, health, and shard counts.

Node status: GET _cat/nodes?v&h=name,heap.percent,ram.percent,cpu,disk.used_percent shows resource usage per node.

Shard allocation: GET _cat/shards?v shows where each shard lives and its status. Unassigned shards are the most common source of yellow or red cluster health.

Pending tasks: GET _cluster/pending_tasks shows cluster state changes waiting to be applied. A growing queue indicates the master node is overloaded.

Common Gotchas#

Too many small indices and shards. Each shard is a Lucene index with its own file handles, memory overhead, and cluster state. A cluster with 10,000 shards from hundreds of tiny indices will perform worse than one with 100 well-sized shards. Use ILM with rollover to consolidate time-series data, and use index templates to set sensible defaults.

Mapping explosion. With dynamic mapping enabled (the default), every unique JSON key in indexed documents creates a new field in the mapping. If your documents contain arbitrary keys (user-generated metadata, flattened nested structures), the mapping can grow to thousands of fields, degrading performance and consuming memory. Use dynamic: strict or dynamic: false on indices where the schema should be controlled.

Full-text search on keyword fields returns nothing. This is the most common query mistake. A match query on a keyword field will not find partial matches because the field is not analyzed. Conversely, a term query on a text field will not find matches because the stored tokens are lowercased and tokenized. Match the query type to the field type: match for text, term for keyword.

JVM heap sizing. Never set the JVM heap above 50% of available memory, and never above 31GB (above this threshold, Java compressed oops are disabled, increasing memory usage). The remaining memory serves the OS filesystem cache, which Lucene relies on heavily for search performance. A node with 64GB of RAM should have a 31GB heap maximum, leaving 33GB for the filesystem cache.