Elastic.Clients.Elasticsearch 9 in .NET: the Typed API, Fluent vs Object Initializer, Serialization, Error Handling and Testing

Version note: The examples in this post were run with Elasticsearch 9.5.6, .NET 10, Elastic.Clients.Elasticsearch 9.5.3 (which brings Elastic.Transport 1.1.0 and Elastic.Esql 0.11.0), Testcontainers.Elasticsearch 4.16.0 and xunit.v3 4.0.3, in the lab we built in post 2.

Guess First

The search code we wrote for Bookstore, the online bookstore used as the shared example across this site's series, builds queries with two different syntaxes: the endpoint in post 6 used the fluent (lambda) syntax, and from post 7 on we used object initializers. The syntax command we added to the lab writes the same query both ways and sends it:

// Fluent: lambdas and descriptors, close to the JSON shape.
SearchResponse<Book> fluent = await client.SearchAsync<Book>(s => s
    .Size(5)
    .Query(q => q.Bool(b => b
        .Must(m => m.Match(mt => mt.Field(f => f.Title).Query("dune")))
        .Filter(f => f.Term(t => t.Field(x => x.Language).Value("tr"))))));

// Object initializer: plain objects that can be built step by step, stored and tested.
SearchRequest<Book> request = new()
{
    Size = 5,
    Query = new BoolQuery
    {
        Must = [new MatchQuery { Field = "title", Query = "dune" }],
        Filter = [new TermQuery { Field = "language", Value = "tr" }]
    }
};
SearchResponse<Book> initializer = await client.SearchAsync<Book>(request);

Do the two requests produce the same JSON? Do they return the same result? Remember that in the client settings we said, with DefaultMappingFor<Book>, that Book's index is books.

Registering the Client: Options, Validation and Debug Mode

First let's tidy up how the client is created. Since post 2, the API and the sync worker built the client by hand in Program.cs. Now there is a single extension method in Bookstore.Search:

public static IServiceCollection AddBookstoreSearch(this IServiceCollection services, IConfiguration configuration, bool isDevelopment)
{
    services.AddOptions<ElasticsearchOptions>()
        .Bind(configuration.GetSection(ElasticsearchOptions.SectionName))
        .Validate(o => !string.IsNullOrWhiteSpace(o.ApiKey), "Elasticsearch:ApiKey is empty.")
        .ValidateOnStart();

    // Singleton: the client keeps the connection pool and node state; never create one per request.
    services.AddSingleton<ElasticsearchClient>(provider =>
    {
        ElasticsearchOptions options = provider.GetRequiredService<IOptions<ElasticsearchOptions>>().Value;
        ILogger? debugLogger = isDevelopment
            ? provider.GetRequiredService<ILoggerFactory>().CreateLogger("Bookstore.Search.Elasticsearch")
            : null;
        return SearchClientFactory.Create(options, debugLogger);
    });

    return services;
}

// Options, validation and the singleton client (post 10); debug mode only in Development.
builder.Services.AddBookstoreSearch(builder.Configuration, builder.Environment.IsDevelopment());

ValidateOnStart makes the application fail at startup, not on the first request, if the API key is empty. A RequestTimeout was added to ElasticsearchOptions too; it is 5 seconds in the API's appsettings.json. The client's default timeout is too long for a search endpoint: when Elasticsearch slows down, requests pile up and threads wait. The worker has no setting and its batch jobs run with the default.

SearchClientFactory turns on debug mode in development:

if (debugLogger is not null)
{
    // Development only: buffers every request and response body and logs them.
    settings.EnableDebugMode(details => debugLogger.LogDebug("{DebugInformation}", details.DebugInformation));
}

dbug: Bookstore.Search.Elasticsearch[0]
      Successful (200) low level call on POST: /books/_search?pretty=true&error_trace=true

      # Audit trail of this API call:
       - [1] HealthyResponse: Node: http://localhost:9200/ Took: 00:00:00.0352720
      # Request:
      {
        "collapse": {
          "field": "title.keyword"
        },
        "query": {
          "bool": {
            "filter": [],
      ...

Nothing beats this while debugging: the JSON sent, the response received, which node was used. The log has no API key in it (I checked), but it does have the user's search text and the returned books. Debug mode also buffers every response in memory and adds pretty=true and error_trace=true to requests. Leaving it on in production is both a performance and a personal data problem; that is why the environment check lives in code, not in a config flag.

If you use Elastic Cloud, the ElasticsearchClientSettings constructor that takes a Cloud ID accepts that identifier instead of a URL; there is no Elastic Cloud in the lab, so I did not run it.

Fluent and Object Initializer

The answer to the guess:

dotnet run --project src/Bookstore.SearchLab -- syntax
Fluent:      POST /_search {"query":{"bool":{"filter":{"term":{"language":{"value":"tr"}}},"must":{"match":{"title":{"query":"dune"}}}}},"size":5} -> 1 hit(s)
Initializer: POST /books/_search {"query":{"bool":{"filter":{"term":{"language":{"value":"tr"}}},"must":{"match":{"title":{"query":"dune"}}}}},"size":5} -> 1 hit(s)
Same request body: True

The bodies are identical but the addresses are not. SearchRequest<Book> takes the index from DefaultMappingFor<Book> and goes to /books/_search; the lambda-based SearchAsync<Book>(s => ...) does not infer the index in 9.5.3 and goes to /_search, that is, to every index it can reach. GetAsync<Book> went to /books/_doc/...; the inconsistency is only in the search descriptor. I could not find this clearly in the 9.0 breaking change list; I found it by trying. The result is the same because the application's API key only sees books* indices and the lab has no other books* index right now. As soon as there is one, the difference shows:

PUT /books-load/_doc/x1?refresh=true
{ "title": "Dune Messiah", "language": "tr" }

dotnet run --project src/Bookstore.SearchLab -- syntax
Fluent:      POST /_search {"query":{"bool":{"filter":{"term":{"language":{"value":"tr"}}},"must":{"match":{"title":{"query":"dune"}}}}},"size":5} -> 2 hit(s)
Initializer: POST /books/_search {"query":{"bool":{"filter":{"term":{"language":{"value":"tr"}}},"must":{"match":{"title":{"query":"dune"}}}}},"size":5} -> 1 hit(s)

DELETE /books-load

That is why we deleted the books-load test index from post 5 at the start of post 6: the search command from post 4 and the /books/suggest endpoint from post 8 used descriptors, and with that index around they would have returned its books too. We added .Indices(SearchClientFactory.BooksIndex) to both. The rule: wherever you search with a descriptor, name the index explicitly; with object initializers, prefer SearchRequest<Book> over the non-generic SearchRequest (which also goes to /_search). Keeping the API key narrow limits the impact of mistakes like this, but does not replace the fix.

Which syntax? Fluent is short and readable because it resembles the JSON in Dev Tools; in 9.0 descriptors changed from classes to structs, and query types live as properties of the Query container. Object initializers suit queries that are built step by step from conditions, stored in a variable and tested. Our rule at Bookstore: one-off queries carried over from Dev Tools are fluent, queries with business logic are object initializers.

Gathering Queries in a Builder

At the end of post 9, the /books/search endpoint built the query, post_filter, aggregations, sorting, highlighting and the suggester itself. We moved all of that into BookSearchRequestBuilder in Bookstore.Search; the builder knows nothing about HTTP, needs no client, and returns a single SearchRequest<Book>:

public static SearchRequest<Book> Build(BookSearchInput input, SearchTuning tuning)
{
    string? text = input.Filter.Text;
    bool hasText = !string.IsNullOrWhiteSpace(text);
    return new SearchRequest<Book>
    {
        From = (input.Page - 1) * input.PageSize,
        Size = input.PageSize,
        TrackTotalHits = new TrackHits(true),
        Query = BookSearchQuery.Build(new BookSearchFilter(text), tuning),
        PostFilter = BookFacets.PostFilter(input.Filter),
        Aggregations = BookFacets.Aggregations(input.Filter),
        Sort = BuildSort(input.Sort),
        Highlight = hasText ? BuildHighlight() : null,
        Suggest = hasText ? BuildDidYouMean(text!) : null
    };
}

BookSearchInput input = new(new BookSearchFilter(q, category, language, minPrice, maxPrice, minRating), sort, currentPage, size);
SearchRequest<Book> request = BookSearchRequestBuilder.Build(input, SearchTuning.Tuned);

SearchResponse<Book> response = await client.SearchAsync<Book>(request, cancellationToken);

The endpoint now only validates parameters, makes the call and maps the response. The real gain is testing: we can turn the request the builder produces into JSON and check it without going to a real Elasticsearch (below).

Serialization: System.Text.Json and the Source Serializer

The client uses two serializers: a request/response serializer for the requests and responses themselves (queries, aggregations), and a source serializer for your documents (_source). Both are System.Text.Json; you should not touch the first, while the second is for your types and can be configured. The default source serializer uses camelCase property names, does not write nulls, and writes enums by name. Suppose we want an enum such as a book format to be stored in lowercase in the index:

// Documents (_source) are serialized with System.Text.Json; enums are written in lowercase kebab case ("e-book").
// Insert, not Add: the default options already contain an enum converter, and the first matching converter wins.
public static Serializer CreateSourceSerializer(Serializer builtIn, IElasticsearchClientSettings settings) =>
    new DefaultSourceSerializer(settings, options => options.Converters.Insert(0, new JsonStringEnumConverter(JsonNamingPolicy.KebabCaseLower)));

dotnet run --project src/Bookstore.SearchLab -- serialization
Default source serializer:   {"isbn":"9781000000030","format":"EBook","price":165.75}
Bookstore source serializer: {"isbn":"9781000000030","format":"e-book","price":165.75}

This factory is passed with new ElasticsearchClientSettings(nodePool, CreateSourceSerializer). My first attempt was options.Converters.Add(...), and the output did not change: the default options already contain a JsonStringEnumConverter, System.Text.Json uses the first suitable converter in the list, and ours, appended at the end, never kicks in. Always verify a setting like this by serializing and looking at the output. Changing the source serializer has a price: existing documents in the index stay in the old format; changing how enum values are written silently breaks filters on keyword fields and needs a reindex.

Error Handling: IsValidResponse, Server and Transport Errors, 429

By default the client does not throw; you have to check IsValidResponse on every response. The errors command shows three kinds of failure: an index that does not exist, a port nobody listens on, and a small stub returning 429 (producing a 429 on a real cluster is hard; the stub is a 20-line HttpListener that returns Elasticsearch's error body and the X-Elastic-Product header):

dotnet run --project src/Bookstore.SearchLab -- errors
Missing index: IsValidResponse False, HTTP 404
  server error: index_not_found_exception no such index [books-missing]
  exception:    TransportException Request failed to execute. Call: Status code 404 from: POST /books-missing/_search. ServerError: Type: index_not_found_exception Reason: "no such index [books-missing]"
Unreachable node: IsValidResponse False, HTTP none
  server error:
  exception:    HttpRequestException Connection refused (localhost:9399)
429 from the cluster: IsValidResponse False, HTTP 429
  server error: es_rejected_execution_exception rejected execution of search
  exception:    TransportException Request failed to execute. Call: Status code 429 from: POST /_search. ServerError: Type: es_rejected_execution_exception Reason: "rejected execution of search"
  requests the stub received: 1

There are three sources of information: ElasticsearchServerError is the error Elasticsearch returned (type and reason), ApiCallDetails.HttpStatusCode is the HTTP status (absent if no connection was made), and ApiCallDetails.OriginalException is the transport-level exception. Since post 6 our endpoints have logged these three and returned 503. With the ThrowExceptions() setting the client can throw a TransportException in these cases; at Bookstore we prefer checking the response, because some "errors", such as a 404, are part of business logic.

It matters that the stub received a single request: the client does not retry a 429 on its own. A 429 is the cluster saying "I can't keep up right now"; since Elasticsearch 9.0, operations that time out also return 429 instead of 5xx (it is in the 9.0 breaking change list). Decide by the type of work. In a user-facing search, retrying adds load to a cluster that is already struggling; return a 503 quickly and let the client try again later. In batch jobs (the sync from post 5, an initial load), retry with increasing waits; BulkAll's BackOffRetries and BackOffTime settings do this. The general design of retries is the subject of the External Service Integration and Resilience series.

Calling ES|QL from .NET

You can call the ES|QL queries we ran in Dev Tools in post 9 from the client in two ways: with a query string or with LINQ. Elastic.Clients.Elasticsearch 9.5.3 brings in the Elastic.Esql package (0.11.0), which translates LINQ to ES|QL, as a dependency; the package is still at 0.x and its API may change.

using ElasticsearchStreamResponse raw = await client.Esql.QueryAsync(q => q
    .Query("FROM books | WHERE language == \"tr\" | STATS books = COUNT(*), avg_price = AVG(price) BY categories | SORT books DESC | LIMIT 3"));
using JsonDocument result = await JsonDocument.ParseAsync(raw.Body);

await foreach (CheapBook book in client.Esql.QueryAsync<Book, CheapBook>(q => q
    .Where(b => b.Language == "tr" && b.Price < 170)
    .OrderBy(b => b.Price)
    .Take(3)
    .Select(b => new CheapBook(b.Title, b.Price))))
{
    Console.WriteLine($"  {book.Title}: {book.Price}");
}

dotnet run --project src/Bookstore.SearchLab -- esql
Raw ES|QL: IsValidResponse True, HTTP 200
  books | avg_price | categories
  21 | 275.9047619047619 | Fiction
  17 | 268.47058823529414 | Classics
  2 | 245.0 | Fantasy
  POST /_query {"columnar":false,"query":"FROM books\n| WHERE (language == "tr" AND price < 170)\n| SORT price\n| LIMIT 3\n| KEEP title, price"}
  Dokuzuncu Hariciye Koğuşu: 120.0
  Aylak Adam: 150.0
  Yaban: 155.0

The string-based call returns the response as a stream (ElasticsearchStreamResponse); you read the JSON yourself and pair up columns and values. When I tried format=txt, IsValidResponse was false and the body empty even though HTTP 200 came back; ask the client for JSON. On the LINQ side I printed the generated ES|QL with the debug client's OnRequestCompleted callback: the index name again comes from DefaultMappingFor<Book>, and Select becomes KEEP title, price. I added the projection later: my first attempt read Book directly and failed with The JSON value is not in a supported DateOnly format, because ES|QL returns dates with a time, in ISO 8601 format. Selecting only the columns you need is the right habit for reports and admin screens anyway.

Integration Tests with Testcontainers

The new tests/Bookstore.Search.Tests project contains two kinds of tests. The first are unit tests for the builder: the request is built and turned into JSON, nothing is sent.

[Fact]
public void Post_filter_is_omitted_when_no_filter_is_selected()
{
    JsonElement json = Build(new BookSearchFilter("roman"));

    // Post 09: a null BoolQuery once became "post_filter": {} and every search failed with HTTP 400.
    Assert.False(json.TryGetProperty("post_filter", out _));
}

This test is the regression test for the bug from post 9. When I reverted the fix and ran it:

dotnet test --project tests/Bookstore.Search.Tests -- --filter-class Bookstore.Search.Tests.BookSearchRequestBuilderTests
BookFacets.cs(51,16): warning CS8604: Possible null reference argument for parameter 'value' in 'Query.implicit operator Query(BoolQuery value)'.
failed Bookstore.Search.Tests.BookSearchRequestBuilderTests.Post_filter_is_omitted_when_no_filter_is_selected (0ms)
  Assert.False() Failure
  Expected: False
  Actual:   True
Test run summary: Failed!
  total: 5
  failed: 1
  succeeded: 4

The compiler warns as well: nullable analysis sees a null BoolQuery going into the implicit operator. In post 9 I only looked at errors, so I missed this warning. We added a Directory.Build.props at the repo root and turned nullable warnings into errors (<WarningsAsErrors>nullable</WarningsAsErrors>); this bug now stops the build.

The second kind are integration tests. The analyzers in the mapping, the ICU plugin and the synonyms set cannot be tested without a real Elasticsearch; mocking them removes the very thing you want to test. On every test run, Testcontainers builds the image from the lab's Dockerfile, starts a temporary container, and loads search/synonyms.json, search/books.index.json and data/books.json:

_image = new ImageFromDockerfileBuilder()
    .WithDockerfileDirectory(Path.Combine(Root, "lab", "elasticsearch"))
    .WithBuildArgument("ES_VERSION", "9.5.6")
    .WithName("bookstore-search-es-test:9.5.6")
    .Build();
await _image.CreateAsync();

// TEST_CONTAINER_PREFIX lets CI give containers a recognisable name.
string prefix = Environment.GetEnvironmentVariable("TEST_CONTAINER_PREFIX") ?? "bookstore-search-test";
_container = new ElasticsearchBuilder(_image)
    .WithName($"{prefix}-{Guid.NewGuid():N}"[..(prefix.Length + 9)])
    .WithEnvironment("ES_JAVA_OPTS", "-Xms512m -Xmx512m")
    .Build();
await _container.StartAsync();

// The container uses HTTPS with a self-signed certificate and the elastic user: fine for a throwaway test node only.
Uri uri = new(_container.GetConnectionString());
string[] credentials = Uri.UnescapeDataString(uri.UserInfo).Split(':', 2);
ElasticsearchClientSettings settings = new ElasticsearchClientSettings(
        new SingleNodePool(new UriBuilder(uri) { UserName = "", Password = "" }.Uri), SearchClientFactory.CreateSourceSerializer)
    .Authentication(new BasicAuthentication(credentials[0], credentials[1]))
    .ServerCertificateValidationCallback(CertificateValidations.AllowAll)
    .DefaultMappingFor<Book>(m => m.IndexName(SearchClientFactory.BooksIndex).IdProperty(b => b.Isbn));

[Fact]
public async Task Search_without_turkish_characters_finds_the_book_first()
{
    SearchRequest<Book> request = BookSearchRequestBuilder.Build(new BookSearchInput(new BookSearchFilter("kurk mantolu")), SearchTuning.Tuned);

    SearchResponse<Book> response = await _client.SearchAsync<Book>(request, TestContext.Current.CancellationToken);

    Assert.True(response.IsValidResponse, response.DebugInformation);
    Assert.Equal("9781000000030", response.Hits.First().Source!.Isbn);
}

xunit.v3 4.x runs on Microsoft.Testing.Platform, and for dotnet test to use it on the .NET 10 SDK the repo root needs a global.json ("test": { "runner": "Microsoft.Testing.Platform" }); without it, dotnet test stops with "Testing with VSTest target is no longer supported". The result (output shortened):

dotnet test --project tests/Bookstore.Search.Tests --output detailed
passed Bookstore.Search.Tests.BookSearchRequestBuilderTests.Every_sort_ends_with_the_isbn_tie_breaker(sort: "price_asc") (63ms)
passed Bookstore.Search.Tests.BookSearchRequestBuilderTests.Every_sort_ends_with_the_isbn_tie_breaker(sort: null) (0ms)
passed Bookstore.Search.Tests.BookSearchRequestBuilderTests.Every_sort_ends_with_the_isbn_tie_breaker(sort: "newest") (0ms)
passed Bookstore.Search.Tests.BookSearchRequestBuilderTests.Each_facet_ignores_its_own_filter (6ms)
passed Bookstore.Search.Tests.BookSearchRequestBuilderTests.Post_filter_is_omitted_when_no_filter_is_selected (0ms)
passed Bookstore.Search.Tests.BooksIndexTests.Strict_mapping_rejects_an_unknown_field (18ms)
passed Bookstore.Search.Tests.BooksIndexTests.Autocomplete_analyzer_folds_before_splitting_prefixes (15ms)
passed Bookstore.Search.Tests.BooksIndexTests.Search_without_turkish_characters_finds_the_book_first (258ms)
passed Bookstore.Search.Tests.BooksIndexTests.Selected_category_keeps_the_other_options_in_its_facet (62ms)
Test run summary: Passed!
  total: 9
  failed: 0
  succeeded: 9
  skipped: 0
  duration: 31s 251ms

The tests themselves take milliseconds; almost all of the 31 seconds is preparing the image and starting the container. That is why the container is started once per test class (IClassFixture) and the tests do not modify the data. For integration and end-to-end tests that run against real dependencies, the Integration and E2E Testing series post "Real SQL Server with Testcontainers: Container Lifecycle, Migrations and Cleanup Between Tests" covers the same approach for SQL Server in detail. These tests, together with rank-eval from post 7, should run on every pull request that changes an analyzer or the mapping.

Production Notes

Manage the client version together with the server. According to Elastic's documentation, a 9.x client works with 9.x and 10.x servers but not with an older major; compatibility does not mean feature parity either, and an older client does not know a newer minor's features. The client does not strictly follow semantic versioning; minor and patch releases can contain breaking changes too (in 9.3.5 the return type of the ES|QL methods changed to ElasticsearchStreamResponse). Read the release notes before upgrading and run the tests from this post.

Plan a migration from NEST as a project of its own. NEST's last release is 7.17.5; the new client is a separate package, so both can be installed side by side in the same application during the migration. System.Text.Json instead of JSON.NET, types generated from the specification and a changed aggregation API mean most of the code gets rewritten. Do the migration with tests that compare the JSON output of critical queries from the old and new clients.

Watch what you log. In an error log, the HTTP status, error type and reason are enough; the request body can contain the user's search text, and the response personal data. Debug mode belongs in development only.

Plan API key rotation. The client is a singleton and the API key is fixed in its settings; changing the key means restarting the application. For rotation without downtime, keep two keys valid for a while (revoke the old one after the new deployment is done).

Common Pitfalls

Not naming the index when searching with a descriptor. In 9.5.3, SearchAsync<T>(s => ...) does not use DefaultMappingFor and goes to /_search; results come from every index the API key can see.

Creating a new client per request. The client holds the connection pool, node state and serializer caches; creating one per request uses up connections and warms up from scratch every time.

Leaving debug mode on in production. Every body is buffered in memory, personal data is written to logs, and pretty is added to requests.

Adding a converter to the source serializer with Add. The default converter comes first, so yours never runs; verify by serializing.

Blindly retrying a 429 in a user request. It adds more load to a struggling cluster; leave retries to batch jobs with increasing waits.

Testing analyzers and mappings with mocks. What you are testing is Elasticsearch's behavior; without a real container, the test only confirms your own assumptions.

A Small Experiment

★☆☆ Add a test to BooksIndexTests that checks the search "ince" returns only the two İnce Memed books (a regression test for the keyword_marker rule from post 4). Remove "ince" from tr_keywords and watch the test break.

★★☆ Create a second client with ThrowExceptions() turned on and run the three scenarios from the errors command with it. Which exception types come back? Which approach would you prefer in an endpoint, and why?

★★★ Move search/relevance/judgments.json into a Testcontainers test: write a test that computes nDCG@5 against the 36 books in the container with a threshold of 0.9. Change one of the boosts from post 7 and watch the test break.

In the next post we move on to the index lifecycle: binding the application to an alias instead of the real index name, zero-downtime reindexing by building books-v2 with a new mapping and swapping the alias atomically, index templates, the shard count decision and a short look at ILM.