Elasticsearch Lab: start-local, Kibana Dev Tools, Cluster Health and the First .NET Connection

Version note: The examples in this post were run with Elasticsearch 9.5.6 and Kibana 9.5.6 (official Docker images), .NET 10 and Elastic.Clients.Elasticsearch 9.5.3; the description of the start-local script is based on the source of version 0.14.0.

Guess First

In the first post we used a one-off container with security turned off. This time we build a permanent lab for Bookstore, the online bookstore used as the shared example across this site's series: Elasticsearch with security on, Kibana next to it, and a small API that connects from .NET. Once the container is up we will try three things:

curl localhost:9200

PUT /books
GET /_cluster/health

# with an API key that may only access books* indices:
GET /

The first is a request without a password; what comes back? In the second we create a new index on a single-node cluster; does health stay green? In the third we send a request to Elasticsearch's root path with the API key we give the application; that is exactly what the .NET client's InfoAsync() does. Does it work?

What Does start-local Do?

Elastic's recommended path for local development is a one-line script: curl -fsSL https://elastic.co/start-local | sh. I downloaded and read version 0.14.0 of the script; this is what it does:

It creates a folder called elastic-start-local in the current directory and writes .env, docker-compose.yml and start.sh, stop.sh, uninstall.sh into it. .env holds a randomly generated password for elastic (ES_LOCAL_PASSWORD), the password and encryption key for Kibana's internal user, the heap setting (-Xms128m -Xmx2g) and the license expiry date. The compose file starts Elasticsearch with xpack.security.enabled=true but with TLS off on HTTP (xpack.security.http.ssl.enabled=false) and with xpack.license.self_generated.type=trial, and publishes the ports on 127.0.0.1 only. Once Elasticsearch is ready, the script creates an API key named elastic-start-local and appends it to .env as ES_LOCAL_API_KEY. The trial license lasts 30 days; after it expires, start.sh switches the cluster to the free Basic license.

There are few options: -v 9.5.6 pins the version (without it the script picks the latest release from Elastic's release list), --esonly installs only Elasticsearch without Kibana, and --edot adds an OpenTelemetry collector. Elastic's documentation says plainly that this is for local development only and must not be used in production. On Windows the script does not run directly; the documentation asks you to install WSL.

This series does not use start-local; instead we use a small compose file that spells out the same settings. The reasons: it should work on Windows without WSL, the version and heap should be pinned in the repo, we want the Basic license from the start rather than a trial so behaviour does not change after 30 days, and every step of the setup should be visible in the post. If you just want to try something quickly, start-local itself is a good option; the Dev Tools and .NET steps below are the same with it.

Our Own Lab: compose.yaml

The file is lab/compose.yaml in the repo:

# Bookstore search lab: Elasticsearch (always) and Kibana (profile "kibana").
# Copy .env.example to .env and set ELASTIC_PASSWORD before the first "docker compose up".
name: bookstore-search-lab

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION:-9.5.6}
    environment:
      - node.name=bookstore-es
      - discovery.type=single-node
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:?Set ELASTIC_PASSWORD in lab/.env}
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
    ports:
      - "127.0.0.1:9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD-SHELL", "curl -s -o /dev/null -w '%{http_code}' -u elastic:$$ELASTIC_PASSWORD http://localhost:9200 | grep -q 200"]
      interval: 5s
      timeout: 5s
      retries: 40

  kibana:
    image: docker.elastic.co/kibana/kibana:${ES_VERSION:-9.5.6}
    profiles: ["kibana"]
    depends_on:
      elasticsearch:
        condition: service_healthy
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_SERVICEACCOUNTTOKEN=${KIBANA_SERVICE_TOKEN:-}
    ports:
      - "127.0.0.1:5601:5601"

volumes:
  es-data:

A few lines deserve an explanation. discovery.type=single-node lets the node form a cluster on its own without looking for other nodes. ELASTIC_PASSWORD is the password of the built-in superuser elastic; thanks to :?, compose refuses to start if it is empty. There is no license setting, so the default Basic license applies. The heap is 512 MB; enough for lab data, while real data needs the heap and the container memory grown together (post 12). I put Kibana in a profile because we first need to create a token for Kibana in Elasticsearch. From the repo root:

# Linux/macOS, instead of Copy-Item: cp lab/.env.example lab/.env
Copy-Item lab/.env.example lab/.env
# in lab/.env, set ELASTIC_PASSWORD to a long random value
docker compose -f lab/compose.yaml up -d --wait

lab/.env is in .gitignore; only .env.example goes into the repo. Now the first request from the guess:

curl localhost:9200

{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\", charset=\"UTF-8\"","ApiKey"]}}], ... ,"status":401}

A cluster with security on returns 401 to an anonymous request and lists two methods in the WWW-Authenticate header: username/password (Basic) and API key. With curl -u elastic:<password> localhost:9200 you get the version information from the first post.

Kibana and a Service Account Token

Kibana needs an identity to connect to Elasticsearch on its own behalf. There are two ways: set a password for the built-in kibana_system user (start-local does this) or create a token for the elastic/kibana service account. I chose the second: a single HTTP request is enough, and the token belongs to the elastic/kibana service account whose privileges Elasticsearch defines and fixes; we are not sharing a user's password. We send the request from inside the Elasticsearch container; the password comes from the container's environment variable and never appears in the command:

docker compose -f lab/compose.yaml exec elasticsearch sh -c 'curl -s -u elastic:$ELASTIC_PASSWORD -X POST http://localhost:9200/_security/service/elastic/kibana/credential/token/bookstore-lab'

{
  "created": true,
  "token": {
    "name": "bookstore-lab",
    "value": "<token>"
  }
}

The single quotes matter: neither PowerShell nor bash touches the $ELASTIC_PASSWORD inside them; the sh in the container expands it. Put value into KIBANA_SERVICE_TOKEN in lab/.env and start Kibana:

docker compose -f lab/compose.yaml --profile kibana up -d

Kibana takes about a minute to come up. Sign in at http://localhost:5601 as elastic and open Dev Tools under Management in the menu (direct address: http://localhost:5601/app/dev_tools#/console). You will see warnings about TLS and encryption keys in the Kibana logs (such as xpack.encryptedSavedObjects.encryptionKey); Dev Tools does not need them, but features like alerting do. Elasticsearch and Kibana together used about 2 GB of memory on my machine.

First Requests in Dev Tools

The Dev Tools console wants each request as a method, a path and a JSON body; the host and credentials come from Kibana. First, the state of the cluster:

GET /_cluster/health

{
  "cluster_name" : "docker-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 51,
  "active_shards" : 51,
  "unassigned_shards" : 0,
  ...
}

Almost all of the 51 shards belong to Kibana's own system indices; before Kibana started the number was 2. Now let's create Bookstore's index without any settings:

PUT /books

GET /_cluster/health?filter_path=status,unassigned_shards
{
  "status" : "yellow",
  "unassigned_shards" : 1
}

GET /_cat/indices/books?v&h=health,status,index,pri,rep,docs.count
health status index pri rep docs.count
yellow open   books   1   1          0

The situation from the first post: the index has one primary and one replica shard, and the replica cannot be assigned. We can ask Elasticsearch why:

GET /_cluster/allocation/explain?filter_path=can_allocate,node_allocation_decisions.node_name,node_allocation_decisions.deciders
{ "index": "books", "shard": 0, "primary": false }

{
  "can_allocate" : "no",
  "node_allocation_decisions" : [
    {
      "node_name" : "bookstore-es",
      "deciders" : [
        {
          "decider" : "same_shard",
          "decision" : "NO",
          "explanation" : "a copy of this shard is already allocated to this node [[books][0], node[<node-id>], [P], s[STARTED], ...]"
        }
      ]
    }
  ]
}

The same_shard rule prevents two copies of the same shard from living on the same node. The health colours are simple: green means every shard is assigned, yellow means all primaries are assigned but at least one replica is not (data complete, redundancy missing), red means at least one primary is unassigned (part of the data is unreachable). On a single-node lab yellow is expected; to fix it we set replicas to zero:

PUT /books/_settings
{ "index": { "number_of_replicas": 0 } }

GET /_cluster/health?filter_path=status,unassigned_shards
{
  "status" : "green",
  "unassigned_shards" : 0
}

In production, setting replicas to zero turns yellow into green but means data loss when a node goes; there the right answer is a second node. Now let's write and read a document. We use the ISBN as the _id:

PUT /books/_doc/9781000000016
{ "isbn": "9781000000016", "title": "İnce Memed", "authors": [ { "name": "Yaşar Kemal", "role": "author" } ], "language": "tr", "price": 385.00 }

{
  "_index" : "books",
  "_id" : "9781000000016",
  "_version" : 1,
  "result" : "created",
  "_shards" : { "total" : 1, "successful" : 1, "failed" : 0 },
  "_seq_no" : 0,
  "_primary_term" : 1
}

GET /books/_doc/9781000000016

{
  "_index" : "books",
  "_id" : "9781000000016",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "isbn" : "9781000000016",
    "title" : "İnce Memed",
    ...
    "price" : 385.0
  }
}

Sending the same PUT again with "price": 370.00 returns "result" : "updated", "_version" : 2 and "_seq_no" : 1: writing to the same _id replaces the whole document. _seq_no and _primary_term are used for optimistic concurrency (post 5). One more difference: GET /books/_doc/{id} is real-time and returns the latest write without waiting for a refresh, while _search waits for the refresh, as we saw in the first post.

API Keys: Only the Access the Application Needs

elastic is a superuser; it can do anything on the cluster. Connecting the application with that password means that if the application has a hole, an attacker can delete the cluster. For the Bookstore API we create an expiring API key that can only access indices starting with books:

POST /_security/api_key
{
  "name": "bookstore-api-dev",
  "expiration": "30d",
  "role_descriptors": {
    "bookstore-books": {
      "cluster": [],
      "indices": [
        { "names": ["books*"], "privileges": ["read", "write", "create_index", "view_index_metadata"] }
      ]
    }
  }
}

{
  "id" : "<id>",
  "name" : "bookstore-api-dev",
  "expiration" : <epoch-ms>,
  "api_key" : "<api_key>",
  "encoded" : "<encoded>"
}

expiration shows the moment 30 days from now in epoch milliseconds. encoded is the Base64 form of id:api_key; clients send it in the Authorization: ApiKey <encoded> header. The value appears only in this response and cannot be retrieved from Elasticsearch later; store it somewhere safe right away. An API key's privileges are the intersection of its creator's privileges and its role_descriptors; we created it as elastic, but it can only work on books*:

curl -H "Authorization: ApiKey <encoded>" "localhost:9200/books/_count?pretty"
{
  "count" : 1,
  ...
}

curl -H "Authorization: ApiKey <encoded>" "localhost:9200/_cluster/health?pretty"
"reason" : "action [cluster:monitor/health] is unauthorized for API key id [<id>] of user [elastic], this action is granted by the cluster privileges [monitor,manage,all]"
"status" : 403

curl -H "Authorization: ApiKey <encoded>" -X DELETE "localhost:9200/books?pretty"
"reason" : "action [indices:admin/delete] is unauthorized for API key id [<id>] of user [elastic] on indices [books], this action is granted by the index privileges [delete_index,manage,all]"

The error messages also say which privilege is missing; read that line before widening access, a single privilege is usually enough. The answer to the third question from the guess is here too: GET / is a cluster-level action (cluster:monitor/main) and this key has no cluster privileges.

The First Connection from .NET

The official client is Elastic.Clients.Elasticsearch; its latest stable version on NuGet is 9.5.3 (28 September 2026). The version number follows the server: a 9.x client is designed to work with 9.x and 10.x servers, and minor releases may contain breaking changes; details in post 10. We add it to the shared Bookstore.Search project:

dotnet add src/Bookstore.Search package Elastic.Clients.Elasticsearch --version 9.5.3

public sealed class ElasticsearchOptions
{
    public const string SectionName = "Elasticsearch";

    public string Url { get; set; } = "http://localhost:9200";

    public string ApiKey { get; set; } = "";
}

public static class SearchClientFactory
{
    public const string BooksIndex = "books";

    // One client per application: it owns the connection pool and is thread-safe
    public static ElasticsearchClient Create(ElasticsearchOptions options)
    {
        if (string.IsNullOrWhiteSpace(options.ApiKey))
        {
            throw new InvalidOperationException("Elasticsearch:ApiKey is empty. Create an API key (post 02) and set it.");
        }

        ElasticsearchClientSettings settings = new ElasticsearchClientSettings(new Uri(options.Url))
            .Authentication(new ApiKey(options.ApiKey))
            .DefaultMappingFor<Book>(m => m.IndexName(BooksIndex).IdProperty(b => b.Isbn));

        return new ElasticsearchClient(settings);
    }
}

DefaultMappingFor<Book> states two things in one place: documents of type Book go to the books index, and the Isbn property is used as the _id. So we do not have to pass the index name on every call. The first-connection command in the lab console connects, indexes Dune and reads it back:

InfoResponse info = await client.InfoAsync();
Console.WriteLine(info.IsValidResponse
    ? $"Connected: {info.Name}, Elasticsearch {info.Version.Number}"
    : $"Info failed: HTTP {info.ApiCallDetails.HttpStatusCode}, {info.ElasticsearchServerError?.Error?.Type}");

IReadOnlyList<Book> books = await BookCatalog.LoadAsync(Path.Combine("data", "books.json"));
Book dune = books.First(b => b.Title == "Dune" && b.Language == "en");

IndexResponse indexed = await client.IndexAsync(dune);
Console.WriteLine($"Index: {indexed.Result}, _id {indexed.Id}, _version {indexed.Version}, _seq_no {indexed.SeqNo}");

GetResponse<Book> got = await client.GetAsync<Book>(dune.Isbn);
Console.WriteLine(got.Found
    ? $"Get: {got.Source!.Title} ({got.Source.Language}), {got.Source.Authors[0].Name}, {got.Source.Price} {got.Source.Currency}, published {got.Source.PublishedOn:yyyy-MM-dd}"
    : "Get: not found");

GetResponse<Book> missing = await client.GetAsync<Book>("9780000000000");
Console.WriteLine($"Missing: IsValidResponse {missing.IsValidResponse}, Found {missing.Found}, HTTP {missing.ApiCallDetails.HttpStatusCode}");

$env:BOOKSTORE_ES_API_KEY = "<encoded>"     # bash: export BOOKSTORE_ES_API_KEY=...
dotnet run --project src/Bookstore.SearchLab -- first-connection

Info failed: HTTP 403, security_exception
Index: Created, _id 9781000000221, _version 1, _seq_no 2
Get: Dune (en), Frank Herbert, 520.0 TRY, published 2019-08-01
Missing: IsValidResponse True, Found False, HTTP 404

Each of the four lines teaches something. InfoAsync got a 403 but no exception was thrown: by default the client puts the error state on the response, and checking it is your job (IsValidResponse). If you want to test the connection with GET /, give the key the monitor privilege; if you would rather not, test it against an index you do have access to (with _count, for example). IndexAsync took the index name and the _id from DefaultMappingFor. The source serializer uses System.Text.Json and writes properties in camelCase; DateOnly went out as "2019-08-01". The last line is subtle: for a missing document IsValidResponse is true, because 404 is a valid answer for this API; Found answers whether it was found.

On the API side the client goes into DI as a singleton. Creating a new client per request means a new connection pool and new TCP connections every time:

WebApplicationBuilder builder = WebApplication.CreateBuilder(args);

ElasticsearchOptions elasticsearch = builder.Configuration.GetSection(ElasticsearchOptions.SectionName).Get<ElasticsearchOptions>()
    ?? new ElasticsearchOptions();

// Singleton: the client keeps the connection pool and node state; never create one per request
builder.Services.AddSingleton<ElasticsearchClient>(_ => SearchClientFactory.Create(elasticsearch));

WebApplication app = builder.Build();

app.MapBookEndpoints();

app.Run();

app.MapGet("/books/{isbn}", async (string isbn, ElasticsearchClient client, ILoggerFactory loggerFactory, CancellationToken cancellationToken) =>
{
    GetResponse<Book> response = await client.GetAsync<Book>(isbn, cancellationToken);

    if (response.IsValidResponse)
    {
        return response.Found ? Results.Ok(response.Source) : Results.NotFound();
    }

    // Transport errors and server errors both end up here; log the reason, not the request headers
    ILogger logger = loggerFactory.CreateLogger("Bookstore.Api.BookEndpoints");
    logger.LogError("Elasticsearch get failed: HTTP {StatusCode} {ErrorType} {Reason}",
        response.ApiCallDetails.HttpStatusCode,
        response.ElasticsearchServerError?.Error?.Type,
        response.ElasticsearchServerError?.Error?.Reason ?? response.ApiCallDetails.OriginalException?.Message);
    return Results.Problem("Search is unavailable.", statusCode: StatusCodes.Status503ServiceUnavailable);
});

appsettings.json only holds Elasticsearch:Url; ApiKey is empty. On the development machine we put the key in user-secrets, so it never enters the repo:

dotnet user-secrets set "Elasticsearch:ApiKey" "<encoded>" --project src/Bookstore.Api
dotnet run --project src/Bookstore.Api

curl -i localhost:5080/books/9781000000221
HTTP/1.1 200 OK
{"isbn":"9781000000221","title":"Dune","authors":[{"name":"Frank Herbert","role":"author"}],"categories":["Fiction","Science Fiction"],"language":"en","price":520.0,"currency":"TRY","stockQuantity":30,"publishedOn":"2019-08-01","averageRating":4.7,"description":"A desert planet, a spice everyone wants and a young heir who must survive among the Fremen."}

curl -i localhost:5080/books/9780000000000
HTTP/1.1 404 Not Found

What happens when the cluster is unreachable? I started the API with Elasticsearch__Url=http://localhost:9299, pointing it at an empty port:

HTTP/1.1 503 Service Unavailable

fail: Bookstore.Api.BookEndpoints[0]
      Elasticsearch get failed: HTTP (null) (null) Connection refused (localhost:9299)

A transport error has no HTTP status code and no server error; the reason is in OriginalException. Even when the search box is down, orders and the cart should keep working; that is why the endpoint returns 503 instead of throwing. The approach from the ASP.NET Core Web API series post "Global Error Handling in ASP.NET Core: Exception Middleware and ProblemDetails" applies here as well.

If you see the NEST package in older projects: it was the client for the 7.x line, and its last version on NuGet is 7.17.5. Elastic announced NEST's deprecation with 8.13; it could run against 8.x servers in compatibility mode, but for new projects the right package is Elastic.Clients.Elasticsearch. Migration notes are in post 10.

Running OpenSearch in the Same Lab

If you want to try OpenSearch, you can add a separate service with the opensearchproject/opensearch image to the same compose layout (the latest release on Docker Hub is 3.9.0); Kibana's counterpart is OpenSearch Dashboards, and the .NET client is a separate package, OpenSearch.Client. I did not run OpenSearch in this post; setup differences such as the image asking for an admin password on first start, and the places where the APIs diverge, are in post 13.

In Production

Two shortcuts in this lab are not acceptable in production. First, TLS is off on HTTP: API keys and passwords travel in plain text. In the lab the port is bound to 127.0.0.1 only, so the risk is low, but on a server TLS is mandatory. Second, a single node: if the node's disk goes, the index goes with it. A production setup with at least three master-eligible nodes, TLS, role-based access and snapshot backups is the subject of post 12.

Manage API keys like an inventory. Create a separate key per application and environment, put the application and environment in its name (like bookstore-api-dev), set an expiration and plan the rotation. If a key leaks you can invalidate it by id with DELETE /_security/api_key; for that you need to have recorded the id.

If you use Elastic Cloud, the connection details may come as a Cloud ID instead of a URL, and the client has a separate constructor for it. I did not run that path, since this series uses no cloud account; client-side settings are in post 10.

Common Pitfalls

Writing the password or the API key into the repo. A key put into appsettings.Development.json because "it's only local" becomes everyone's key when the repo goes public. Use user-secrets in development and an environment variable or a secret store on servers.

Connecting the application as elastic. It works, but a single hole in the application puts the whole cluster at risk. Give the application an API key that only has access to its own indices.

Raising an alarm for yellow health on a single node, or the other way round, ignoring yellow in production. In the lab it is expected; in production it says you are running without redundancy.

Not checking the response. By default the client does not throw on errors; code that ignores IsValidResponse shows both a 403 and a 404 to the user as "no results".

Creating a new ElasticsearchClient per request. The client is thread-safe and designed to live as a single instance for the whole application.

A Small Experiment

★☆☆ Delete the books index and create it again, this time with "settings": { "number_of_replicas": 0 } in the PUT /books body. Does health ever drop to yellow? Then create a books-test index with "number_of_shards": 3 and look at the output of GET /_cat/shards/books-test?v.

★★☆ Create a second API key named books-reader with only the read privilege. Run the first-connection command with it and watch the Index: line: why does it say Created even though no document was written? Fix the code to check IsValidResponse, and use the privilege list in the error message to find which privilege the key is missing.

★★★ While the API is running, invalidate the bookstore-api-dev key through the "ids" list of DELETE /_security/api_key and call /books/9781000000221 again. What HTTP status code and error type show up in the log line? How could the application switch to a new key without a restart, and what does rebuilding the client through IOptionsMonitor cost?

In the next post we build the books index deliberately: the guesses dynamic mapping makes for ISBN and date fields, the difference between text and keyword, multi-fields, scaled_float for prices, object versus nested for the author list, and why a mapping cannot be changed afterwards.