Running Elasticsearch in Production: Security, Capacity, Monitoring and Common Problems

Version note: The examples in this post were run with Elasticsearch 9.5.6 (basic license), .NET 10 and Elastic.Clients.Elasticsearch 9.5.3, in the single-node lab from post 2 and, for TLS, in a separate single-node container. A three-node cluster, Elastic Cloud and Amazon OpenSearch Service were not run; what is said about them comes from the Elastic and AWS documentation.

Guess First

Before going to production, the search we have been building for Bookstore, the online bookstore used as the shared example across this site's series, got a request: Gül's customer service screen should see only Turkish books, and only the ISBN, title and price. Elasticsearch can do this inside a role: document level security (DLS) limits which documents are visible with a query, and field level security (FLS) limits which fields are visible. We created an API key with these restrictions in the lab:

POST /_security/api_key
{
  "name": "bookstore-dls-test",
  "expiration": "1h",
  "role_descriptors": {
    "tr-books-only": {
      "indices": [
        {
          "names": ["books"],
          "privileges": ["read"],
          "query": { "term": { "language": "tr" } },
          "field_security": { "grant": ["isbn", "title", "price"] }
        }
      ]
    }
  }
}

Is the key created? If it is, what does a search with this key return for "Dune": the Turkish Dune with three fields, nothing, or an error?

Network and TLS

The first production rule is about the network: Elasticsearch's port 9200 must not be open to the internet. The compose.yaml from post 2 binds the port to 127.0.0.1 only; in production the cluster also sits on a network that only the application servers can reach. Security has been on by default since Elasticsearch 8.0, but clusters with security turned off and left open to the internet have caused many data leaks, and the mistake is still being made.

The second rule is TLS. In the lab we turned TLS off on the HTTP layer (xpack.security.http.ssl.enabled=false), which means API keys travel the network in plain text. The official image, on the other hand, generates its own CA on first start without any configuration and puts both HTTP and node-to-node traffic on TLS. I tried it in a second container:

docker run -d --name bookstore-es-tls -p 127.0.0.1:9201:9200 -m 1500m `
  -e discovery.type=single-node -e ELASTIC_PASSWORD=<password> -e ES_JAVA_OPTS="-Xms512m -Xmx512m" `
  docker.elastic.co/elasticsearch/elasticsearch:9.5.6
docker cp bookstore-es-tls:/usr/share/elasticsearch/config/certs/http_ca.crt .

curl.exe -sS https://localhost:9201
curl: (60) SSL certificate problem: self signed certificate in certificate chain

The server's certificate is signed by this CA, and the CA is not in any operating system's trust store. There are two ways for a client to trust the certificate: add the CA to the machine's trust store, or give the client the CA's fingerprint. For the client, the second is narrower: you trust only this CA. ElasticsearchOptions got a new field:

// Post 12: SHA-256 fingerprint of the cluster's HTTP CA certificate when the cluster uses a self-signed CA.
public string? CertificateFingerprint { get; set; }

if (!string.IsNullOrWhiteSpace(options.CertificateFingerprint))
{
    // Trusts exactly this CA instead of turning certificate validation off.
    settings.CertificateFingerprint(options.CertificateFingerprint);
}

To get the fingerprint on Windows without installing openssl, the lab got a small command (X509Certificate2.GetCertHashString(HashAlgorithmName.SHA256)), plus tls-check, which sends a single request:

dotnet run --project src/Bookstore.SearchLab -- ca-fingerprint http_ca.crt
Subject: CN=Elasticsearch security auto-configuration HTTP CA
SHA-256: <fingerprint>

$env:BOOKSTORE_ES_URL = "https://localhost:9201"
dotnet run --project src/Bookstore.SearchLab -- tls-check
https://localhost:9201: failed, HttpRequestException: The SSL connection could not be established, see inner exception. The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot

$env:BOOKSTORE_ES_CA_FINGERPRINT = "<fingerprint>"
dotnet run --project src/Bookstore.SearchLab -- tls-check
https://localhost:9201: HTTP 404, books exists: False

The 404 is the right answer: this is an empty cluster with no books; what matters is that the connection works. Going to the same address with http:// gives The response ended prematurely; you cannot speak plain HTTP to a port that expects TLS. The fingerprint was accepted both in .NET's uppercase form without colons and in openssl's form with colons. This approach is for a self-signed CA; with a certificate signed by the company's own CA or a public CA (as on Elastic Cloud) you do not need a fingerprint. Certificate lifetimes and renewal are the subject of the SSL/TLS Certificates and HTTPS series. CertificateValidations.AllowAll, which comes to mind while debugging, is acceptable for the throwaway container in the Testcontainers test from post 10; on a real cluster it cancels out what TLS is for.

Roles and API Keys: One to Read, One to Write

The answer to the guess: the key is created and the search fails.

{
  "error": {
    "type": "security_exception",
    "reason": "current license is non-compliant for [field and document level security]",
    "license.expired.feature": "field and document level security",
    "indices_with_dls_or_fls": ["books-v2", "books"]
  },
  "status": 403
}

The lab runs a basic license (GET /_license → "type": "basic"); DLS and FLS come with paid subscriptions. The error appears not when the key is created but on the first search; if a test does not catch it before the deployment, you find out in production as a 403. With a basic license there are two ways to meet the same need: a separate index for Gül's screen (or a separate alias with a filter), or filtering fields and books in the API layer. A filtered alias is not a security boundary: if the same key can reach the underlying index, the filter can be bypassed.

What the basic license does include is index-level roles. Since post 2, the API and the worker shared one key: read, write and create index on books*. For production we split them and granted the privileges on the alias:

POST /_security/api_key
{
  "name": "bookstore-api-prod",
  "expiration": "90d",
  "role_descriptors": {
    "bookstore-search-read": {
      "cluster": [],
      "indices": [ { "names": ["books"], "privileges": ["read", "view_index_metadata"] } ]
    }
  }
}

POST /_security/api_key
{
  "name": "bookstore-search-sync-prod",
  "expiration": "90d",
  "role_descriptors": {
    "bookstore-search-write": {
      "cluster": [],
      "indices": [ { "names": ["books"], "privileges": ["write"] } ]
    }
  }
}

GET /books/_count                     (bookstore-api-prod)
{"count":20036, ...}

GET /books-v2/_count                  (bookstore-api-prod)
action [indices:data/read/search] is unauthorized for API key id [<id>] of user [elastic] on indices [books-v2], this action is granted by the index privileges [read,all]

POST /books/_doc                      (bookstore-api-prod)
action [indices:data/write/index] is unauthorized for API key id [<id>] of user [elastic] on indices [books], this action is granted by the index privileges [create_doc,create,index,write,all]

GET /books/_count                     (bookstore-search-sync-prod)
action [indices:data/read/search] is unauthorized for API key id [<id>] of user [elastic] on indices [books], ...

I ran the API with the read key; /books/{isbn}, /books/search and /books/suggest returned 200. I ran the worker with the write key and changed a book's stock: Synced 1 changes (1 indexed, ...). Because the privileges are granted on the alias name, a switch like the one in post 11 does not change the keys; when the alias moved to books-v3, the API kept working with the same key. Even if the read key leaks, it cannot write to the index, read the old indices behind the alias, or create a new index. We created these keys in the lab as the elastic user; in production elastic is used only for the initial setup, and after that keys are issued by an admin role with the manage_api_key privilege. Where to keep secrets and how to rotate them is the subject of the Configuration and Secret Management series.

Capacity: Heap, Memory and Node Roles

GET /_cat/nodes?v&h=name,node.role,heap.max,heap.percent
name         node.role   heap.max heap.percent
bookstore-es cdfhilmrstw    512mb           63

The lab node carries every role on its own (cdfhilmrstw: master, data, ingest, machine learning and others), and its heap is 512 MB; we set -Xms512m -Xmx512m in compose.yaml. According to Elastic's JVM settings documentation, the best option in production is to leave the heap to Elasticsearch: it chooses the size from the node's roles and the container's memory. If you set it yourself, Xms and Xmx must be equal, stay at or below half of the total memory, and stay under the compressed object pointer threshold (the documentation says 26 GB is safe on most systems). The other half of memory is not idle: Lucene reads index files through the operating system's file cache, and search speed depends heavily on that cache. In a container, "total memory" means the container's limit; in a container without a limit, Elasticsearch sees the whole machine.

A single node is not production. In a one-node cluster, search stops when that node stops, replicas cannot be assigned (the yellow status from post 2), and any disk failure means data loss. Elastic's resilience recommendation is at least three master-eligible nodes: decisions about the cluster state need a majority, and when one of three nodes goes away the remaining two keep it; a two-node cluster cannot. Large clusters give the master role to separate, small nodes and the data to other nodes. Since I run at most two containers at a time in the lab, I did not try a three-node setup.

Disk Watermarks

GET /_cluster/settings?include_defaults=true&filter_path=defaults.cluster.routing.allocation.disk.watermark*
{"defaults":{"cluster":{"routing":{"allocation":{"disk":{"watermark":{"low":"85%","high":"90%","flood_stage":"95%",
 "low.max_headroom":"200GB","high.max_headroom":"150GB","flood_stage.max_headroom":"100GB", ...}}}}}}}

When a disk is 85% full, Elasticsearch stops placing new shards on that node; at 90% it tries to move shards to other nodes; at 95% it puts a read_only_allow_delete block on every index with a shard on that node. The max_headroom values cap the percentage on large disks: on a 10 TB disk the 95% threshold would mean 500 GB free, and the headroom brings that down to 100 GB. To see what the last threshold looks like in practice, I lowered the thresholds in the lab to 5%, 6% and 7% (the lab disk is fuller than that) and changed a book's stock while the worker was running:

fail: Bookstore.SearchSync.SearchSyncWorker[0]
      Sync batch failed; retrying from RowVer 0x0000000000005671
      System.InvalidOperationException: Item 9781000000221 failed: 429 cluster_block_exception index [books-v3] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block; ...]

GET /_health_report/disk
{"indicators":{"disk":{"status":"red","symptom":"8 indices are not allowed to be updated. 1 node is out of disk or running low on disk space."}}}

Searches keep working, and writes are rejected with a 429. Thanks to the design from post 5, the worker did not move its checkpoint and retried on every poll. After I restored the default thresholds, the block was lifted on its own (Elasticsearch removes it once disk usage falls below the high watermark), and the next attempt logged Synced 1 changes ... up to RowVer 0x0000000000005672. In production this is an alerting matter: you want to know when you pass 85%, not wait for 95%. Also account for the two indices that coexist during a switch like the one in post 11.

Circuit Breakers and Timeouts

GET /_nodes/stats/breaker
"request":   {"limit_size":"307.1mb", "estimated_size":"0b",   "tripped":0}
"fielddata": {"limit_size":"204.7mb", "estimated_size":"368b", "tripped":0}
"parent":    {"limit_size":"486.3mb", "estimated_size":"327mb", "tripped":0}

Circuit breakers keep a single request from filling the heap and bringing the node down with an OutOfMemoryError: a request estimates how much memory it will use and is rejected before it runs if it would exceed the limit. With a 512 MB heap, the request breaker is 307.1 MB (60%), fielddata 204.7 MB (40%), and the parent 486.3 MB (95%, based on real memory usage). To see what the error looks like, I lowered the request breaker to 1 MB for a moment and sent a search to the API:

fail: Bookstore.Api.BookSearchEndpoints[0]
      Elasticsearch search failed: HTTP 429 search_phase_execution_exception all shards failed
      ... CircuitBreakingException: [request] Data too large, data for [query] would be [1186436/1.1mb], which is larger than the limit of [1048576/1mb]

The API returned a 503; the decision from post 10 pays off here: we do not retry a 429 in a user request. On a real cluster this error most often comes from very large terms aggregations like those in post 9, deep paging (that is why the 10,000 limit from post 6 exists), and fielddata enabled on a text field. Raising the breaker is not the fix; make the request smaller. On the API side, the 5-second RequestTimeout we gave the client in post 10 drops the connection but does not stop the work in Elasticsearch. To limit the search itself you can pass a timeout in the request; according to the documentation, when it expires Elasticsearch returns the results gathered so far with "timed_out": true. On the lab's small index I could not trigger this even with a timeout of 1 nanosecond ("timed_out": false); short queries finish before they reach the timeout checkpoints.

Monitoring: the Health Report, _cat APIs and the Slow Log

GET /_health_report?filter_path=status,indicators.*.status
{"status":"green","indicators":{"master_is_stable":{"status":"green"},"repository_integrity":{"status":"green"},
 "disk":{"status":"green"},"shards_capacity":{"status":"green"},"shards_availability":{"status":"green"},
 "ilm":{"status":"green"},"slm":{"status":"green"}, ...}}

_health_report tells you more than the colour-only answer of _cluster/health: which indicator is red and why, what it affects and what to do. It was the one that said "8 indices are not allowed to be updated" in the disk experiment above. It is the first endpoint to connect to an alerting system; after it come _cat/nodes (heap, disk), _cat/indices (size, health) and _cat/shards (unassigned shards). _cluster/allocation/explain from post 2 explains, one by one, why a shard is not assigned.

For slow queries there is the slow log. I set the threshold to 0 for a moment and ran a search through the API:

PUT /books/_settings
{ "index.search.slowlog.threshold.query.warn": "0ms", "index.search.slowlog.include.user": true }

{
  "log.level": "WARN",
  "apikey.name": "bookstore-api-prod",
  "elasticsearch.slowlog.message": "[books-v2][0]",
  "elasticsearch.slowlog.source": "{\"from\":0,\"size\":1,\"query\":{\"bool\":{\"must\":[{\"function_score\":{\"query\":{\"bool\":{\"should\":[{\"multi_match\":{\"query\":\"kurk mantolu\", ...",
  "elasticsearch.slowlog.took": "14.2ms",
  "elasticsearch.slowlog.total_hits": "2 hits",
  "event.dataset": "elasticsearch.index_search_slowlog"
}

The threshold is per index and measured at the shard level; in production a reasonable starting point for warn is a few hundred milliseconds or a few seconds. With include.user you can see which key the slow query came from; that is another benefit of separate keys. Note that the slow log writes the user's search text ("kurk mantolu") as is. Where the logs are kept and who can read them is a personal data question. After the experiment I removed the settings by setting them to null. Collecting metrics and logs in a central system is the subject of the Observability series.

Backups: SLM and How Long a Reindex Takes

In post 11 we took a snapshot by hand. In production SLM, Elasticsearch's snapshot lifecycle management, does it; it worked on the lab's basic license:

PUT /_slm/policy/nightly-books
{
  "schedule": "0 30 1 * * ?",
  "name": "<books-{now/d}>",
  "repository": "lab-backups",
  "config": { "indices": ["books-v*"], "include_global_state": false, "feature_states": ["synonyms"] },
  "retention": { "expire_after": "30d", "min_count": 5, "max_count": 50 }
}

POST /_slm/policy/nightly-books/_execute
{"snapshot_name":"books-2026.11.14-<suffix>"}

GET /_cat/snapshots/lab-backups?v&h=id,status,indices,duration
id                         status indices duration
books-2026.11.14           SUCCESS       2    201ms
books-2026.11.14-<suffix>  SUCCESS       2       0s

The second snapshot took 0 seconds because snapshots are incremental: most segments were already in the first one. The schedule is a cron expression: every night at 01:30.

So is "the data is already in SQL Server, we can always reindex" enough? I measured both. Restoring books-v2 from the snapshot under a new name:

POST /_snapshot/lab-backups/books-2026.11.14-<suffix>/_restore?wait_for_completion=true
{ "indices": "books-v2", "rename_pattern": "books-(.+)", "rename_replacement": "restored-books-$1", "include_aliases": false }

GET /_cat/recovery/restored-books-v2?v&h=index,type,stage,time,files_total,bytes_total
index             type     stage time  files_total bytes_total
restored-books-v2 snapshot done  163ms 11          5.5mb

From SQL Server, from scratch: I created an empty books-v3, switched the alias to it, reset the worker's checkpoint and measured the time until the book count reached 20,036:

UPDATE dbo.SearchSyncState SET LastRowVer = 0x0000000000000000 WHERE Name = 'books';

info: Bookstore.SearchSync.SearchSyncWorker[0]
      Synced 1000 changes (1000 indexed, 0 deleted, 0 already up to date) up to RowVer 0x0000000000005642
info: Bookstore.SearchSync.SearchSyncWorker[0]
      Synced 36 changes (36 indexed, 0 deleted, 0 already up to date) up to RowVer 0x000000000000566C

The worker wrote 21 batches, and _count reached 20,036 in 5.3 seconds (including the worker's start-up; I measured it in the script that started the worker by querying _count every half second). The restore took 0.16 seconds (a copy of 5.5 MB of files). Both are short in the lab; what matters is how they grow. A restore grows with the index's size in bytes and the speed of the repository; a reindex grows with the number of books, the query in SQL Server, the cost of the analysis chain and Elasticsearch's indexing rate. At this rate a million books take minutes, a hundred million documents take hours, and during that time search either returns incomplete results or waits in a separate index as in post 11. Measure both times with your own data and compare them with your disaster recovery target; without a measurement, "we'll reindex" is not a plan. The two do not replace each other either: a snapshot is a copy of a moment behind the checkpoint in SQL Server; after a restore, the worker has to replay from the RowVer at the time of the snapshot (the external versions from post 11 make that safe).

In Production

Managed or self-managed? Elastic Cloud, Elastic's own service, manages the cluster, version upgrades, snapshots and TLS, and paid subscription features (such as DLS/FLS) come with the plan. Amazon OpenSearch Service runs OpenSearch, not Elasticsearch; the differences are in post 13. In a self-managed setup, every heading of this post (network, TLS, roles, heap, disk, monitoring, backups, upgrades) is your job; count that work in addition to the cost of the machines. I did not run any of the managed services for this post.

Plan your upgrades. Major versions are not skipped: before moving from 8.x to 9.x you move to the last 8.x release (8.19 today) and clear the deprecation warnings. The details and what was removed are in post 13.

A checklist before going to production: 9200 closed to the internet; TLS on for HTTP and between nodes; API keys per application, with an expiration and granted on the alias; at least three master-eligible nodes and at least one replica; heap left to Elasticsearch or below half of memory; an alert at 85% disk; _health_report monitored; slow log thresholds set and access to the logs restricted; SLM running and a restore tested; debug mode off in the client (post 10).

Common Pitfalls

Treating a single node as production. When the node stops, search stops, there are no replicas, and every disk problem is data loss.

Leaving the test-only AllowAll certificate validation in production. Give the fingerprint for a self-signed CA, and use a trusted CA where possible.

Assuming a paid feature on a basic license. DLS and FLS return a 403 on the first search, not when the role is created.

Letting the API and the worker share one broad key. The read key should not be able to write, the write key should not be able to read, and both should be granted on the alias, not on books*.

Waiting until the disk is full. At the flood stage writes stop; the alert should fire at the low watermark.

Relying on "we'll reindex from SQL Server" as the only backup and never measuring how long it takes.

Not treating the slow log as personal data. The search text is written to the logs in plain form.

A Small Experiment

★☆☆ In the lab, invalidate the bookstore-api-prod key with DELETE /_security/api_key and send a search to the API. Which HTTP status and error type are in the log? What does the API return to the user?

★★☆ Grant the read key access through a filtered alias that shows only Turkish books (books-tr, "filter": { "term": { "language": "tr" } }). Can this key read books? How does it differ from DLS?

★★★ Measure restore and reindex times again with 200,000 books (seed-sql 200000, the data from post 1). Which one grows linearly? Write down the steps that make the worker replay from the RowVer at the time of the snapshot after a restore.

The last post of the series looks at versions: what was removed and changed between Elasticsearch 8 and 9, what arrived from 9.1 to 9.5, where OpenSearch 3 diverges, and which one to choose.