Elasticsearch Mapping: text vs keyword, Multi-Fields, Dynamic Mapping, nested and Why Mappings Can't Change

Version note: The examples in this post were run with Elasticsearch 9.5.6, Kibana 9.5.6 Dev Tools, .NET 10 and Elastic.Clients.Elasticsearch 9.5.3, in the lab we built in post 2.

Guess First

A developer in a hurry opens the search index for Bookstore, the online bookstore used as the shared example across this site's series, and sends the first book without any mapping. The serializer wrote the ISBN as a number, and the price happens to be a whole number:

PUT /books-dynamic/_doc/1?refresh=true
{ "isbn": 9781000000016, "title": "İnce Memed", "edition": "1955", "publishedOn": "2021-03-15", "price": 385, "stockQuantity": 42, "authors": [ { "name": "Yaşar Kemal", "role": "author" } ] }

PUT /books-dynamic/_doc/2
{ "isbn": "978-1-00-000002-3", "title": "İnce Memed 2", "price": 395.50 }

PUT /books-dynamic/_doc/3?refresh=true
{ "isbn": "9781000000030", "title": "Kürk Mantolu Madonna", "price": 165.75 }

GET /books-dynamic/_search
{ "query": { "range": { "price": { "gte": 165.5 } } } }

Which types does Elasticsearch pick for isbn, edition, publishedOn and price? Does the second book get indexed? The third costs 165.75; does the last search find it?

What Dynamic Mapping Guesses

The mapping is the schema that tells an index the type of each field and how to index it. If you don't give one, Elasticsearch guesses from the first value it sees. After the first document:

GET /books-dynamic/_mapping

"isbn"          : { "type" : "long" },
"title"         : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
"edition"       : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } },
"publishedOn"   : { "type" : "date" },
"price"         : { "type" : "long" },
"stockQuantity" : { "type" : "long" },
"authors"       : { "properties" : { "name" : { "type" : "text", ... }, "role" : { "type" : "text", ... } } }

The rules are simple: a whole JSON number becomes long, a decimal one float; a string that looks like a date becomes date, otherwise text with a keyword sub-field; a nested JSON object becomes object. "1955" is a string, so it became text; numeric_detection, which detects numbers inside strings, is off by default. What the guesses lead to:

PUT /books-dynamic/_doc/2   →
"type" : "document_parsing_exception",
"reason" : "[1:10] failed to parse field [isbn] of type [long] in document with id '2'. Preview of field's value: '978-1-00-000002-3'"

PUT /books-dynamic/_doc/3   →   "result" : "created"

GET /books-dynamic/_search   (price >= 165.5)
→ 1 hit: İnce Memed (price 385)

The second book was rejected: the field type was locked to long by the first document, and a hyphenated ISBN is no longer a number. The third is sneakier: it was indexed but is missing from the search. docvalue_fields shows why:

GET /books-dynamic/_search
{ "query": { "term": { "_id": "3" } }, "docvalue_fields": ["price", "isbn"] }

"_source" : { "isbn" : "9781000000030", "title" : "Kürk Mantolu Madonna", "price" : 165.75 },
"fields"  : { "price" : [ 165 ], "isbn" : [ 9781000000030 ] }

_source is exactly the JSON you sent; that is what you see when you read the document back. Searching, sorting and aggregations, however, use the indexed value, and that value is 165: the long field silently truncated 165.75 (this behaviour is called coerce). The price looks right on screen while the filter works wrongly; this kind of bug can take days to notice. In the same way, a document sending "publishedOn": "unknown" is rejected with failed to parse field [publishedOn] of type [date].

A Controlled Schema with dynamic: strict

Dynamic mapping is fine for experiments, not for a production index. If you write the mapping yourself and set "dynamic": "strict", a document with an unknown field is rejected:

PUT /books-strict
{ "mappings": { "dynamic": "strict", "properties": { "title": { "type": "text" } } } }

PUT /books-strict/_doc/1
{ "title": "Dune", "titel": "typo" }

"type" : "strict_dynamic_mapping_exception",
"reason" : "[1:25] mapping set to strict, dynamic introduction of [titel] within [_doc] is not allowed"

There are three options: true (the default, adds the new field), false (keeps the new field in _source but does not index it, so it is not searchable) and strict (rejects it). Bookstore uses strict: a property added to the Book record in C# should not reach the index before it is added to the mapping, and a misspelled field should not silently create a new one.

text and keyword

The pair of fields dynamic mapping gives strings shows the most important distinction in Elasticsearch. A text field goes through an analyzer and is split into terms; it is for full-text search. A keyword field stores the value as is, as a single term; it is for exact matching, filters, sorting and aggregations.

GET /books-dynamic/_search
{ "query": { "term": { "title": "İnce Memed" } } }            → 0 hit
{ "query": { "term": { "title.keyword": "İnce Memed" } } }    → 1 hit
{ "query": { "term": { "title.keyword": "ince memed" } } }    → 0 hit
{ "query": { "match": { "title": "ince memed" } } }           → 1 hit
{ "query": { "term": { "title": "memed" } } }                 → 1 hit

A term query does not analyze the value it searches for. The title field has no term "İnce Memed"; the analyzer stored it as ince and memed, which is why term "memed" finds it. title.keyword holds a single term and is case-sensitive. A match query runs the search text through the field's analyzer, so it gives the natural result on a text field. The rule: match on text fields, term on keyword fields; the other way round is almost always a bug. The Query DSL in detail comes in post 6.

If you try sorting or aggregating on a text field:

GET /books-dynamic/_search
{ "aggs": { "titles": { "terms": { "field": "title" } } } }

"reason" : "Fielddata is disabled on [title] in [books-dynamic]. Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. ..."

The message suggests two ways, and the second (fielddata=true) is a trap: it un-inverts the inverted index on the heap and uses a lot of memory. The right way is to make the field you sort or group on a keyword (or a number, or a date). Fields like category, language and currency are not free text typed by the user but fixed values; they should be keyword from the start.

Multi-Fields: Indexing the Same Value Several Ways

The title has to be searchable, sortable and answer users who type without Turkish characters. A single field cannot do all of that; with fields we can index the same value in different ways. In Bookstore's mapping the title looks like this:

"title": {
  "type": "text",
  "fields": {
    "keyword": { "type": "keyword", "ignore_above": 256 },
    "folded": { "type": "text", "analyzer": "folded" }
  }
}

folded is a small custom analyzer we define in the settings: the standard tokenizer, followed by lowercase and asciifolding, which reduces Turkish characters to their ASCII counterparts. _analyze shows the effect:

POST /books/_analyze
{ "field": "title.folded", "text": "Kürk Mantolu Madonna, Çalıkuşu, İnce Memed, Işık" }

kurk | mantolu | madonna | calikusu | ince | memed | isik

GET /books/_search
{ "query": { "match": { "title": "kurk" } } }          → 0 hit
{ "query": { "match": { "title.folded": "kurk" } } }   → 1 hit: Kürk Mantolu Madonna

You send the document once and Elasticsearch builds three separate index structures; the value is stored only once in _source. The cost is disk and indexing time, so we don't add every variant to every field. ignore_above: 256 keeps long values out of the keyword field entirely; for a title longer than 256 characters title.keyword stays empty, the document is not rejected. folded is a stopgap here: asciifolding also turns "ı" into "i" and knows nothing about Turkish lowercase rules. We set this up properly in post 4 with the turkish analyzer and ICU.

Sorting on keyword has a consequence for Turkish too. When we sort the Turkish books by title.keyword, the list ends with "Aylak Adam, Benim Adım Kırmızı, …, Yaprak Dökümü, Çalıkuşu, İnce Memed, İnce Memed 2, İçimizdeki Şeytan": keyword sorts by byte order, so Ç and İ fall to the end of the alphabet. Turkish alphabetical order needs ICU collation (post 4).

Numbers, Dates and Prices

The choice between number types depends on the range and precision of the value: integer for stock, half_float for values that need little precision, like the average rating. Price is a separate matter. Storing a price that is a decimal in C# in a float or double field carries floating-point rounding into search. scaled_float multiplies the value by a factor (scaling_factor) and stores it as a long; with 100, the price becomes a whole number of kuruş (cents):

PUT /price-test
{ "mappings": { "properties": { "price": { "type": "scaled_float", "scaling_factor": 100 } } } }

PUT /price-test/_doc/1   { "price": 165.755 }
PUT /price-test/_doc/2   { "price": 165.75 }

GET /price-test/_search
{ "docvalue_fields": ["price"], "sort": ["_doc"] }

"_id" : "1", "_source" : { "price" : 165.755 }, "fields" : { "price" : [ 165.76 ] }
"_id" : "2", "_source" : { "price" : 165.75 },  "fields" : { "price" : [ 165.75 ] }

The three-decimal value was rounded to the nearest kuruş; a term query for 165.76 finds this document. If you already keep prices with two decimals, nothing is lost. The system of record for the price is decimal(10, 2) in SQL Server; the value in Elasticsearch is a copy for filtering and sorting.

For dates, spelling out the format is a good habit: "publishedOn": { "type": "date", "format": "yyyy-MM-dd" }. .NET's DateOnly goes out in exactly this format (we saw "2019-08-01" in post 2). Elasticsearch stores dates internally as UTC milliseconds; for values with a time zone (DateTimeOffset) choose the format accordingly.

object, nested and flattened

Crime and Punishment (Suç ve Ceza) has two people: the author Fyodor Dostoyevski and the translator Ada Demir. What happens when a user searches for "books where Dostoyevski is the translator"? First with authors as the default object type:

PUT /authors-object/_doc/1?refresh=true
{ "title": "Suç ve Ceza", "authors": [ { "name": "Fyodor Dostoyevski", "role": "author" }, { "name": "Ada Demir", "role": "translator" } ] }

GET /authors-object/_search
{ "query": { "bool": { "must": [
  { "match": { "authors.name": "dostoyevski" } },
  { "term": { "authors.role": "translator" } }
] } } }

→ 1 hit: Suç ve Ceza

Wrong result. Lucene has no such thing as a nested object; an object field is flattened, and inside, the document is stored as two independent lists like authors.name: [fyodor, dostoyevski, ada, demir] and authors.role: [author, translator]. Which name came with which role is lost. The nested type stores each object as a separate, hidden Lucene document, and a nested query looks for the conditions inside the same object. In Bookstore's books index, authors is nested:

GET /books/_search
{ "query": { "nested": { "path": "authors", "query": { "bool": { "must": [
  { "match": { "authors.name": "dostoyevski" } },
  { "term": { "authors.role": "translator" } }
] } } } } }

→ 0 hit

The same query in .NET, in the lab console's translators command:

SearchResponse<Book> response = await client.SearchAsync<Book>(s => s
    .Query(q => q.Nested(n => n
        .Path("authors")
        .Query(nq => nq.Bool(b => b.Must(
            m => m.Match(mt => mt.Field("authors.name").Query(name).Operator(Operator.And)),
            m => m.Term(t => t.Field("authors.role").Value("translator"))))))));

dotnet run --project src/Bookstore.SearchLab -- translators ada demir
Translated by 'ada demir': 2
  Suç ve Ceza (tr) - Fyodor Dostoyevski [author], Ada Demir [translator]
  Hobbit (tr) - J. R. R. Tolkien [author], Ada Demir [translator]

dotnet run --project src/Bookstore.SearchLab -- translators dostoyevski
Translated by 'dostoyevski': 0

The cost of nested is visible too:

GET /books/_count                          → "count" : 36
GET /_cat/indices/books?h=index,docs.count → books 78

There are 36 books but 78 documents at the Lucene level: each book plus 42 author entries. Nested queries have to join these hidden documents with the book, so they cost more than ordinary queries, and updating one author means reindexing the whole book. So use nested only when the question "must several fields match inside the same object" is really asked. If all you search for is "books where one of the authors is Tolkien", object is enough.

The third option is flattened: it puts an object whose keys are not known in advance (like attributes that vary per book) into the mapping as a single field, without opening a new field for every new key:

PUT /attr-test
{ "mappings": { "properties": { "title": { "type": "text" }, "attributes": { "type": "flattened" } } } }

PUT /attr-test/_doc/1   { "title": "Dune", "attributes": { "format": "paperback", "pages": "624", "series": "Dune Chronicles" } }
PUT /attr-test/_doc/2   { "title": "The Hobbit", "attributes": { "format": "hardcover", "illustrated": "yes", "pages": "310" } }

{ "query": { "term": { "attributes.format": "paperback" } } }       → 1 hit: Dune
{ "query": { "range": { "attributes.pages": { "gte": "400" } } } }  → 1 hit: Dune
{ "query": { "range": { "attributes.pages": { "gte": "1000" } } } } → 2 hit: Dune, The Hobbit

The last line shows the limit of flattened: all values are stored as strings, like keyword, so a range query works alphabetically rather than numerically and "624" comes out greater than "1000". No analysis, no scoring, no numbers; only exact matching and simple filters.

Bookstore's books Mapping

All of these decisions are in search/books.index.json in the repo:

{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "analysis": {
      "analyzer": {
        "folded": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "isbn": { "type": "keyword" },
      "title": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 },
          "folded": { "type": "text", "analyzer": "folded" }
        }
      },
      "authors": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "text",
            "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } }
          },
          "role": { "type": "keyword" }
        }
      },
      "categories": { "type": "keyword" },
      "language": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "currency": { "type": "keyword" },
      "stockQuantity": { "type": "integer" },
      "publishedOn": { "type": "date", "format": "yyyy-MM-dd" },
      "averageRating": { "type": "half_float" },
      "description": {
        "type": "text",
        "fields": {
          "folded": { "type": "text", "analyzer": "folded" }
        }
      }
    }
  }
}

ISBN is keyword: it looks like a number, but we do no arithmetic on it, we look it up exactly. categories is a list of strings; every field in Elasticsearch can hold several values, so there is no separate array type. Category and language are keyword because they will become facets and filters (post 9). The lab console creates the index: it sends the JSON as is through Elastic.Transport's low-level API and then loads the 36 books with a single bulk request:

// The mapping lives in a JSON file; the low-level transport sends it as is
string definition = await File.ReadAllTextAsync(Path.Combine("search", "books.index.json"));
StringResponse created = await client.Transport.RequestAsync<StringResponse>(
    HttpMethod.PUT, $"/{indexName}", PostData.String(definition));
Console.WriteLine($"PUT /{indexName}: HTTP {created.ApiCallDetails.HttpStatusCode} {created.Body}");
if (!created.ApiCallDetails.HasSuccessfulStatusCode)
{
    return;
}

IReadOnlyList<Book> books = await BookCatalog.LoadAsync(Path.Combine("data", "books.json"));
BulkResponse bulk = await client.BulkAsync(b => b
    .Index(indexName)
    .IndexMany(books)
    .Refresh(Elastic.Clients.Elasticsearch.Refresh.WaitFor));
Console.WriteLine($"Bulk: {books.Count} books, errors: {bulk.Errors}, took {bulk.Took} ms");

# In Dev Tools, to delete the test index from post 2:
DELETE /books

dotnet run --project src/Bookstore.SearchLab -- create-index
PUT /books: HTTP 200 {"acknowledged":true,"shards_acknowledged":true,"index":"books"}
Bulk: 36 books, errors: False, took 1005 ms

The reason for keeping the mapping as JSON is that it is a contract you try in Dev Tools, read in review and version. The client also has a typed mapping API; we compare building mappings in C# in post 10. Almost all of took 1005 ms is Refresh.WaitFor waiting for the next refresh; bulk and refresh are the topic of post 5. The API key from post 2 has the create_index privilege, so the application could create the index itself; it cannot delete it, so we run DELETE in Dev Tools as elastic.

Why Can't a Mapping Change Later?

Try changing a field's type or analyzer:

PUT /books/_mapping
{ "properties": { "language": { "type": "text" } } }
"reason" : "mapper [language] cannot be changed from type [keyword] to [text]"

PUT /books/_mapping
{ "properties": { "title": { "type": "text", "analyzer": "turkish" } } }
"reason" : "Mapper for [title] conflicts with existing mapper:\n\tCannot update parameter [analyzer] from [default] to [turkish]"

PUT /books/_mapping
{ "properties": { "publisher": { "type": "keyword" } } }
"acknowledged" : true

Adding a new field is allowed, changing an existing one is not. The reason is the inverted index itself: existing documents sit on disk in segments written for the old type and the old analyzer. Lucene segments are immutable; "language is text now" means regenerating every term already written. The only way is to create a new index with the new mapping and write the data into it again (reindex). So that the application is not tied to an index name directly, an alias sits in between: the application reads and writes books, and books actually points to books-v1, later to books-v2. The zero-downtime switch is in post 11. In this series we change the analyzers in post 4, so we will rebuild the index there; simple in the lab, something to plan in production.

The other risk of dynamic mapping is mapping explosion: if keys coming from users or external systems are added as fields, the number of fields grows out of control and each field takes up room in the cluster state. Elasticsearch limits an index to 1000 fields by default:

PUT /explode-test/_doc/1
{ "attr0": 0, "attr1": 1, ..., "attr1000": 1000 }

document_parsing_exception [1:15794] failed to parse: Limit of total fields [1000] has been exceeded while adding new fields [1001]

Raising the limit with index.mapping.total_fields.limit is possible but does not fix the problem; for data with unknown keys, flattened or a nested structure of key/value pairs is the right tool.

In Production

Version the mapping with the code. books.index.json lives in the repo and changes to it show up in pull requests; a change made by hand in Dev Tools gets forgotten in the next environment. A release that needs a mapping change requires the new index to be created and filled before the application is deployed; put that in the deployment plan.

Don't forget index templates. In production, creating indices through an index template instead of by hand makes every new index whose name matches a pattern get the same mapping (post 11). Without a template, the first document sent to a mistyped index name creates a new index with dynamic mapping.

Watch the field count. Every new field that grows the output of GET /books/_mapping takes memory and cluster state; especially in multi-tenant scenarios (like each institution's own fields in Institutional Orders), flattened or a separate-index strategy should be considered from the start.

Common Pitfalls

Leaving everything as text. Filtering by category, sorting by price and faceting by language either do not work or work wrongly on text fields. Every string that is not free text should be keyword.

Making values like ISBNs, postcodes or phone numbers numeric. Leading zeros disappear, hyphens are rejected, and nobody adds them up anyway; use keyword.

Letting the first document decide the type. The first book with a whole-number price caused every later price to be truncated, and nobody saw an error.

Using nested when it is not needed. Every nested object is a separate Lucene document; queries slow down and the index grows. If the question "must two fields of the same object match together" does not exist, object is enough.

A term query on a text field. It compares analyzed terms with an unanalyzed search value and usually comes back empty.

A Small Experiment

★☆☆ Send three documents to the books-dynamic index in this order: { "title": "Huzur", "pageCount": 420 }, { "title": "Yaban", "pageCount": "abc" } and { "title": "Yaban", "pageCount": "312" }. Which one is rejected, and why is the string "312" accepted? What changes when you send the same three documents in reverse order to a new index?

★★☆ Run a terms aggregation on title.keyword in the books index, then try the same for authors.name.keyword. Why does the second return an empty bucket list? Fix it with a nested aggregation and find the people whose names appear in the most books; does doc_count in the buckets count books or author entries?

★★★ Write an alternative mapping that makes authors an object instead of nested, but adds a keyword field called authorsWithRole with values like "translator:Ada Demir". Answer the translator question with this field and compare the Lucene document count (_cat/indices) between the two designs. Which queries become harder with this design?

In the next post we drop folded and handle Turkish text properly: the parts of an analyzer, following each step with _analyze, the apostrophe and turkish_lowercase filters of the turkish analyzer, where the stemmer cuts too much, Turkish sorting with ICU, and synonyms.