Analyzers and Turkish: Tokenizers, Token Filters, the turkish Analyzer, ASCII Folding and Synonyms

Version note: The examples in this post were run with Elasticsearch 9.5.6 (with the analysis-icu plugin), Kibana 9.5.6 Dev Tools, .NET 10 and Elastic.Clients.Elasticsearch 9.5.3, in the lab we built in post 2.

Guess First

For the catalog of Bookstore, the online bookstore used as the shared example across this site's series, the shortest way to fix Turkish search seems to be the built-in turkish analyzer. We put the titles and descriptions of the 36 books into a turkish-test index where both fields use "analyzer": "turkish", and run four searches:

GET /turkish-test/_search
{ "query": { "multi_match": { "query": "ince", "fields": ["title", "description"] } } }

# the same request for "öğretmenler", "kasaba" and "roman"

Does the search for "ince" put İnce Memed first? Çalıkuşu's description contains "kasabalarına öğretmen olarak" ("to its towns as a teacher"); do "öğretmenler" (teachers) and "kasaba" (town) find it? In the first post the standard analyzer found 6 of 9 books for "roman" (novel); how many does turkish find?

Anatomy of an Analyzer

An analyzer has three parts and text passes through them in this order: zero or more char filters (character-level changes on the raw text, such as stripping HTML tags), exactly one tokenizer (splitting the text into tokens) and zero or more token filters (changing, dropping or adding tokens). We can see each step with the explain option of the _analyze API. We build the chain of the built-in turkish analyzer by hand and feed it "Kürk Mantolu Madonna'nın":

POST /_analyze
{
  "tokenizer": "standard",
  "filter": [
    "apostrophe",
    { "type": "lowercase", "language": "turkish" },
    { "type": "stop", "stopwords": "_turkish_" },
    { "type": "stemmer", "language": "turkish" }
  ],
  "text": "Kürk Mantolu Madonna'nın",
  "explain": true
}

tokenizer standard                ['Kürk', 'Mantolu', "Madonna'nın"]
apostrophe                        ['Kürk', 'Mantolu', 'Madonna']
__anonymous__lowercase            ['kürk', 'mantolu', 'madonna']
__anonymous__stop                 ['kürk', 'mantolu', 'madonna']
__anonymous__stemmer              ['kürk', 'mantol', 'madonna']

(I condensed the output to one line per step for readability; the real response also gives positions and offsets for every token.) An analyzer runs in two places: when a document is indexed (index time) and on the query text (search time). By default both are the same; a match query runs the search text through the field's analyzer and looks up the resulting terms in the index. Both sides have to produce the same terms; if "mantolu" sits in the index as mantol, the search must also turn "mantolu" into mantol. With search_analyzer in the mapping you can use a different analyzer at search time; we will use that for synonyms.

The Parts of the turkish Analyzer

Elastic's documentation rebuilds the turkish analyzer from these filters, in this order: apostrophe, Turkish lowercase, Turkish stop, keyword_marker and the Turkish stemmer. Let's see what each one does.

apostrophe drops the apostrophe and everything after it. In Turkish, suffixes on proper names are separated by an apostrophe, so this one filter fixes the "Memed'in" problem from the first post:

POST /_analyze
{ "tokenizer": "standard", "filter": ["apostrophe"], "text": "Madonna'nın Memed'in İstanbul'da" }

Madonna | Memed | İstanbul

Turkish lowercase fixes the dotted/dotless i problem. The same text with the standard lowercase and with "language": "turkish":

{ "tokenizer": "standard", "filter": ["lowercase"], "text": "IŞIK ILIK İSTANBUL Iron" }
işik | ilik | istanbul | iron

{ "tokenizer": "standard", "filter": [{ "type": "lowercase", "language": "turkish" }], "text": "IŞIK ILIK İSTANBUL Iron" }
ışık | ılık | istanbul | ıron

The standard lowercase turns "IŞIK" (light) into "işik", which is wrong for Turkish text. Turkish lowercase gives the correct "ışık", but now it turns the English word "Iron" into "ıron". A field's analyzer has to match the language of its content; for Bookstore, which keeps Turkish and English books in one catalog, that means several analyzers per field, as we will see below.

The stop filter drops frequent words that carry no meaning on their own:

{ "tokenizer": "standard", "filter": [{ "type": "lowercase", "language": "turkish" }, { "type": "stop", "stopwords": "_turkish_" }], "text": "Ve bir de bu kitap için çok güzel ama" }

kitap | güzel

The stemmer tries to reduce a word to its stem; keyword_marker protects the words in its list from the stemmer. This is where the real problems start.

The Limits of the Stemmer

Elasticsearch's Turkish stemmer is a rule-based Snowball stemmer. It does not know what a word means in the dictionary; it cuts suffixes off the end by rules. Let's give it a few words on their own:

{ "tokenizer": "standard", "filter": ["apostrophe", { "type": "lowercase", "language": "turkish" }, { "type": "stemmer", "language": "turkish" }],
  "text": "İnce Memed'in kasabalarına öğretmenler kitaplarımızdan romanı romanlar roman kitap Mantolu madonna" }

i | memedi | kasaba | öğretmen | kitap | roma | roman | roma | kitap | mantol | madonna

{ ..., "text": "öğretmenler kasaba kasabalar öğretmen" }
öğretmen | kasap | kasaba | öğretme

"kitaplarımızdan" (from our books) correctly became "kitap". But "İnce" dropped to "i" (over-stemming: cutting too much), "öğretmen" (teacher) on its own became "öğretme" and "kasaba" became "kasap" (butcher); while "öğretmenler" became "öğretmen" and "kasabalarına" became "kasaba". So the bare form of a word and its suffixed form end up as different terms (under-stemming: keeping apart forms that should match). "roman" and "romanı" become roma while "romanlar" stays roman. "Memed" turning into "memedi" is not a bug but the stemmer's rule that appends a vowel to stems ending in d; since index and query apply the same rule, matching is not broken. The PostgreSQL series post "PostgreSQL Full-Text Search: tsvector, tsquery, the Turkish Configuration, unaccent and pg_trgm" shows the searches for "kasaba" and "öğretmenler" slipping the same way with the same Snowball stemmer family; the problem is not specific to Elasticsearch.

The results of the searches from the guess:

ince:         3 hit  [Kuyucaklı Yusuf 3.47, İnce Memed 2.71, İnce Memed 2 2.26]
öğretmenler:  0 hit
kasaba:       0 hit
roman:        9 hit

Recall for "roman" went from 6/9 to 9/9; the stemmer helped here. But the search for "ince" put Kuyucaklı Yusuf ahead of İnce Memed: the word "ileri" in its description also drops to "i", and a short term in a short field scores high. "öğretmenler" and "kasaba" could not find Çalıkuşu because of the inconsistency above.

The practical fix is keyword_marker: you put words that matter in the catalog and that the stemmer breaks into the list, and the stemmer leaves them alone. In Bookstore's tr_text analyzer the list has three words for now:

"tr_keywords": { "type": "keyword_marker", "keywords": ["ince", "kasaba", "öğretmen"] }

POST /books/_analyze
{ "field": "description.tr", "text": "öğretmenler kasaba kasabalarına öğretmen İnce" }

öğretmen | kasaba | kasaba | öğretmen | ince

"öğretmenler" still drops to "öğretmen" by its own rule, and "öğretmen" is now protected; both forms meet in the same term. To map a specific suffixed form to the stem you want there is also the stemmer_override filter. These lists should stay small and be fed by real search logs; trying to fix every word by hand does not scale. There are morphological analysis tools that solve Turkish morphology with a dictionary (such as Zemberek), but the plugins that bring them to Elasticsearch are third-party and their maintenance status is unclear; check version compatibility before putting one in production. In this series, instead of leaving the stemmer alone, we balance it with several fields.

Users Who Don't Type Turkish Characters

Some users type "calikusu" instead of "Çalıkuşu" and "yasar kemal" instead of "Yaşar Kemal". We fix the folded analyzer from post 3 for Turkish and call it tr_folded: first apostrophe, then Turkish lowercase, and finally asciifolding:

{ "tokenizer": "standard", "filter": ["apostrophe", { "type": "lowercase", "language": "turkish" }, "icu_folding"], "text": "Kürk Çalıkuşu IŞIK İnce ılık Öğretmen Madonna'nın" }
kurk | calikusu | isik | ince | ilik | ogretmen | madonna

{ "tokenizer": "standard", "filter": ["lowercase", "asciifolding"], "text": "Kürk Çalıkuşu IŞIK İnce ılık Öğretmen" }
kurk | calikusu | isik | ince | ilik | ogretmen

On Turkish text the ICU plugin's icu_folding and the built-in asciifolding gave the same result; icu_folding covers more Unicode characters, but for Turkish characters there is no difference. The mapping uses asciifolding, which needs no plugin. This field gives up precision: "ılık" (lukewarm) and "ilik" (marrow) are now the same term. That is why we query it not on its own but together with the fields that keep Turkish characters, and with a lower weight.

GET /books/_search
{ "query": { "match": { "title.folded": { "query": "kurk mantolu", "operator": "and" } } } }
→ 1 hit: Kürk Mantolu Madonna

{ "query": { "match": { "title.folded": "calikusu" } } }
→ 1 hit: Çalıkuşu

{ "query": { "nested": { "path": "authors", "query": { "match": { "authors.name.folded": "yasar kemal" } } } } }
→ 2 hit: İnce Memed, İnce Memed 2

Turkish Sorting with ICU

In post 3 we saw that sorting by title.keyword pushes books starting with Ç and İ to the end of the list. Language-aware sorting needs the icu_collation_keyword type from the ICU plugin. The plugin is not in the official image; we add a small Dockerfile to the lab:

# Official image plus the ICU analysis plugin (post 04: icu_folding, icu_collation_keyword).
ARG ES_VERSION=9.5.6
FROM docker.elastic.co/elasticsearch/elasticsearch:${ES_VERSION}
RUN bin/elasticsearch-plugin install --batch analysis-icu

  elasticsearch:
    # The official image with the analysis-icu plugin added (lab/elasticsearch/Dockerfile, post 04).
    build:
      context: ./elasticsearch
      args:
        ES_VERSION: ${ES_VERSION:-9.5.6}
    image: bookstore-search-es:${ES_VERSION:-9.5.6}

docker compose -f lab/compose.yaml --profile kibana up -d --build --wait

GET /_cat/plugins?v
name         component    version
bookstore-es analysis-icu 9.5.6

The plugin version must match the Elasticsearch version exactly; when you upgrade Elasticsearch you have to rebuild the image. Even though the container is recreated, the data stays in the es-data volume, so no index is lost. With a sort sub-field on the title, Turkish books line up alphabetically:

"sort": { "type": "icu_collation_keyword", "language": "tr" }

GET /books/_search
{ "_source": ["title"], "sort": [{ "title.sort": "asc" }], "query": { "term": { "language": "tr" } } }

Aylak Adam, Benim Adım Kırmızı, Bin Dokuz Yüz Seksen Dört, Çalıkuşu, Dokuzuncu Hariciye Koğuşu, Dune, Hobbit, Huzur, İçimizdeki Şeytan, İnce Memed, İnce Memed 2, Kuyucaklı Yusuf, Kürk Mantolu Madonna, Puslu Kıtalar Atlası, Saatleri Ayarlama Enstitüsü, Savaş ve Barış, Sinekli Bakkal, Suç ve Ceza, Tutunamayanlar, Yaban, Yaprak Dökümü

Çalıkuşu now comes after B, İçimizdeki Şeytan and İnce Memed after H, and Kürk Mantolu Madonna after Kuyucaklı Yusuf. In a small experiment I also saw ICU separate I and İ correctly: the order was "Ilık, Işıklar, İçimizdeki Şeytan, İnce Memed". The value returned as the sort key is not readable text but a collation key; show the title from _source on screen.

Synonyms: the Synonyms API

Turkish users type "roman", English users type "novel"; the catalog holds both languages. Embedding synonyms in the analyzer as a fixed list meant reloading the analyzer on every change. Elasticsearch's Synonyms API keeps synonyms as a separate resource in the cluster and reloads the analyzers itself when they change:

PUT /_synonyms/bookstore-synonyms
{
  "synonyms_set": [
    { "id": "novel", "synonyms": "roman, novel" },
    { "id": "sci-fi", "synonyms": "bilim kurgu, science fiction, sci-fi" }
  ]
}

{ "result" : "created", "reload_analyzers_details" : { ... } }

In the mapping, a synonym_graph filter points to this set and is used only at search time, in the search_analyzer of description:

"bookstore_synonyms": { "type": "synonym_graph", "synonyms_set": "bookstore-synonyms", "updateable": true }

"search_synonyms": {
  "type": "custom",
  "tokenizer": "standard",
  "filter": ["lowercase", "bookstore_synonyms"]
}

updateable: true is required: when I tried to use a filter that reads synonyms from a set in an index-time analyzer, Elasticsearch returned Can't apply [synonyms_set]! Loading synonyms from index is supported only for search time synonyms!. It makes sense: index-time synonyms write terms to disk, so changing a rule would mean reindexing all the data. At search time a rule change takes effect on the next query. Let's watch it happen. A search for "translated" first finds only one English book; after adding a single rule to the set it also finds the Turkish translations:

dotnet run --project src/Bookstore.SearchLab -- search translated
"translated": 1 hits
    3.26  Crime and Punishment (en)

PUT /_synonyms/bookstore-synonyms/translation
{ "synonyms": "çeviri, translated, translation" }

"result" : "created",
"reload_details" : [ { "index" : "books", "reloaded_analyzers" : [ "search_synonyms" ], ... } ]

dotnet run --project src/Bookstore.SearchLab -- search translated
"translated": 5 hits
    3.26  Crime and Punishment (en)
    2.43  Savaş ve Barış (tr)
    2.35  Suç ve Ceza (tr)
    2.35  Hobbit (tr)
    2.13  Dune (tr)

We didn't touch the index, and there was no reindex. The final state of the set is in search/synonyms.json in the repo. Managing the set is a cluster privilege (manage_search_synonyms); the application's API key doesn't have it and shouldn't, so we update the set in Dev Tools or in the deployment pipeline with an admin identity.

Bookstore's New Mapping and Multi-Field Search

All of these decisions are in the new version of search/books.index.json. The analysis section:

"filter": {
  "tr_lowercase": { "type": "lowercase", "language": "turkish" },
  "tr_stop": { "type": "stop", "stopwords": "_turkish_" },
  "tr_keywords": { "type": "keyword_marker", "keywords": ["ince", "kasaba", "öğretmen"] },
  "tr_stemmer": { "type": "stemmer", "language": "turkish" },
  "bookstore_synonyms": { "type": "synonym_graph", "synonyms_set": "bookstore-synonyms", "updateable": true }
},
"analyzer": {
  "tr_text": {
    "type": "custom",
    "tokenizer": "standard",
    "filter": ["apostrophe", "tr_lowercase", "tr_stop", "tr_keywords", "tr_stemmer"]
  },
  "tr_folded": {
    "type": "custom",
    "tokenizer": "standard",
    "filter": ["apostrophe", "tr_lowercase", "asciifolding"]
  },
  "search_synonyms": {
    "type": "custom",
    "tokenizer": "standard",
    "filter": ["lowercase", "bookstore_synonyms"]
  }
}

Title and description are now indexed with several sub-fields: the main field with standard (the word as written), tr (Turkish stem), en (the built-in english analyzer, for English books), folded (without Turkish characters) and, for the title, sort (ICU collation):

"title": {
  "type": "text",
  "fields": {
    "keyword": { "type": "keyword", "ignore_above": 256 },
    "tr": { "type": "text", "analyzer": "tr_text" },
    "en": { "type": "text", "analyzer": "english" },
    "folded": { "type": "text", "analyzer": "tr_folded" },
    "sort": { "type": "icu_collation_keyword", "language": "tr" }
  }
},
"description": {
  "type": "text",
  "search_analyzer": "search_synonyms",
  "fields": {
    "tr": { "type": "text", "analyzer": "tr_text" },
    "en": { "type": "text", "analyzer": "english" },
    "folded": { "type": "text", "analyzer": "tr_folded" }
  }
}

Every book is written to all sub-fields, whatever its language. A cleaner alternative is separate fields per language (title_tr, title_en) and filling only the one matching the book's language; the price is choosing the language in the application or in an ingest pipeline. With 36 books the disk difference is negligible; for a large catalog, measure before deciding. Since the analyzers changed, the index has to be rebuilt; as we saw in post 3, a mapping does not change in place. In the lab: DELETE /books in Dev Tools, then dotnet run --project src/Bookstore.SearchLab -- create-index. Don't forget to create the synonyms set before the index.

On the query side we need to search all of these fields at once. We added a search command to the lab console:

// Every analyzed variant of title and description; the raw title weighs the most.
private static readonly string[] SearchFields =
[
    "title^3", "title.tr^2", "title.folded^2", "title.en",
    "description", "description.tr", "description.folded", "description.en"
];

SearchResponse<Book> response = await client.SearchAsync<Book>(s => s
    .Size(5)
    .Query(q => q.MultiMatch(m => m
        .Query(text)
        .Fields(Fields.FromStrings(SearchFields)))));

"kurk mantolu": 1 hits
   11.28  Kürk Mantolu Madonna (tr)
"öğretmenler": 1 hits
    3.60  Çalıkuşu (tr)
"ince": 2 hits
    8.47  İnce Memed (tr)
    7.11  İnce Memed 2 (tr)
"novel": 7 hits
    3.26  Nineteen Eighty-Four (en)
    1.94  İnce Memed (tr)
    1.94  Yaprak Dökümü (tr)
    1.94  Suç ve Ceza (tr)
    1.88  Yaban (tr)
"memedin": 2 hits
    5.43  İnce Memed (tr)
    4.52  İnce Memed 2 (tr)

Every problem from the guess is gone: "ince" only finds the İnce Memed books, "öğretmenler" finds Çalıkuşu, typing without Turkish characters finds the right book, and "memedin" typed without the apostrophe finds İnce Memed. The field weights (boosts) ^3 and ^2 are guesses for now; the types of multi_match and how scores combine come in post 6, and tuning the weights by measuring in post 7.

In Production

An analyzer change is a schema change. If a new filter, a new keyword_marker word or a different stemmer touches the index-time analyzer, existing documents keep their old terms and a reindex is needed. Changes that stay at search time (like the synonyms set) are the exception. So put rules that change often at search time and the slow-changing ones at index time.

Measure the change. Every exception you add to the stemmer can fix one search and break another. Keep the 100 most frequent queries from your search logs and their expected top results in a set and run it on every analyzer change (with _rank_eval in post 7).

Manage the plugin together with the version. analysis-icu is built separately for each Elasticsearch version; build the image in CI and keep it with a version tag. On managed services (Elastic Cloud, Amazon OpenSearch Service) which plugins can be installed depends on the service; check before making a design depend on ICU.

Common Pitfalls

Leaving index and search analyzers inconsistent. If you use a stemmer at index time and an analyzer without one at search time, a search for "kitaplar" (books) will not find kitap in the index. search_analyzer should only be used for a deliberate difference (like synonyms).

Using the stemmer blindly on short fields. Titles are short and full of proper names; when a name like "İnce" drops to "i", ranking breaks. Query the stemmed title field together with the field as written, and with a lower weight, not on its own.

Applying Turkish lowercase to English text. "Iron" becomes "ıron" and a search for "iron" does not find it. Use a field per language or a sub-field per language.

Forgetting to reindex after an analyzer change. A deploy that changes the analyzer in the mapping starts searching old documents with new rules, and results quietly get worse.

Applying synonyms at index time. Every rule change means a full reindex; use search time with the Synonyms API.

A Small Experiment

★☆☆ Run "Saatleri Ayarlama Enstitüsü'nün" and "Puslu Kıtalar Atlası'ndaki" through _analyze, first with turkish, then with the tr_text and tr_folded analyzers of the books index (with "analyzer": "tr_text"). Which words get over-stemmed?

★★☆ Add the rule "polisiye, cinayet, mystery" to the bookstore-synonyms set and run search mystery. Which books come back, and why? Remove the rule with DELETE /_synonyms/bookstore-synonyms/<id> and compare the result again.

★★★ Instead of adding to the keyword_marker list, define a new analyzer that uses stemmer_override with the rule "romanlar => roma" and try it in a separate index called books-test. Do "roman", "romanı" and "romanlar" now reduce to the same term? What is the maintenance cost of this approach compared with keyword_marker?

In the next post the catalog starts flowing from SQL Server into Elasticsearch: the Bulk API and BulkAll in .NET, handling failures item by item, the cost of refresh, optimistic concurrency with if_seq_no, and a BackgroundService that moves books whose price or stock changed into the index.