Redis JSON and the Query Engine: JSON Documents, FT.CREATE, FT.SEARCH, FT.AGGREGATE and NRedisStack

Version note: The examples in this post were run with Redis 8.10.3 (redis:8), Valkey 9.1.3 (valkey/valkey-bundle:9), .NET 10, StackExchange.Redis 3.4.35 and NRedisStack 1.8.0.

Guess First

Imagine that for the catalog search of Bookstore, the online bookstore used as the shared example across this site's series, we keep every book in Redis as a JSON document. The descriptions are in Turkish:

JSON.SET bookstore:catalog:9780141439518 $ '{"isbn":"9780141439518","title":"Pride and Prejudice","author":"Jane Austen","year":1813,"priceKurus":15990,"stock":8,"categories":["romance","classic"],"description":"Bennet ailesinin kızları, evlilik ve önyargılar üzerine bir İngiliz klasiği."}'

Middlemarch's description reads "Bir İngiliz taşra kasabasında evlilikler, hırslar ve reformlar." We build two indexes over the same eight books: idx:catalog with LANGUAGE turkish, and idx:catalog-en without a language. Then we search the description for "evlilikler", "ingiliz" and "İngiliz": Turkish for "marriages" and, twice, for "English" as an adjective. How many books does each query find?

127.0.0.1:6379> FT.SEARCH idx:catalog "@description:evlilikler" RETURN 1 title
1) (integer) 2
2) "bookstore:catalog:9780141439549"
3) 1) "title"
   2) "Middlemarch"
4) "bookstore:catalog:9780141439518"
5) 1) "title"
   2) "Pride and Prejudice"
127.0.0.1:6379> FT.SEARCH idx:catalog-en "@description:evlilikler" RETURN 1 title
1) (integer) 1
2) "bookstore:catalog:9780141439549"
3) 1) "title"
   2) "Middlemarch"
127.0.0.1:6379> FT.SEARCH idx:catalog "@description:ingiliz" RETURN 1 title
1) (integer) 0
127.0.0.1:6379> FT.SEARCH idx:catalog "@description:İngiliz" RETURN 1 title
1) (integer) 2
2) "bookstore:catalog:9780141439549"
3) 1) "title"
   2) "Middlemarch"
4) "bookstore:catalog:9780141439518"
5) 1) "title"
   2) "Pride and Prejudice"

The index with Turkish stemming found the book that says "evlilik", the singular form, for the query "evlilikler"; the default English index found only the exact word. But the lowercase "ingiliz" found nothing in either index, because "İngiliz" is written with the dotted capital İ, and lowercasing it does not produce "ingiliz". This post covers Redis's JSON type and the Query Engine (formerly RediSearch): JSON documents and partial updates, indexes with FT.CREATE, filtering, sorting and paging with FT.SEARCH, FT.AGGREGATE, NRedisStack in .NET, the limits of Turkish search, the Valkey side, and when Redis is not the right search engine.

The JSON Type: Storing Documents and Partial Updates

With Redis 8 the JSON type and the Query Engine come without installing a separate module; in the redis:8 image MODULE LIST shows both. In post 04 we kept the cart as a hash: a hash is a flat list of field-value pairs and cannot hold nested structures or arrays. The JSON type stores a document as a tree and reaches its parts one by one with JSONPath:

docker compose -f lab/compose.yaml exec -T redis redis-cli < lab/scripts/11-json.txt

127.0.0.1:6379> JSON.SET bookstore:catalog:9780441013593 $ '{"isbn":"9780441013593","title":"Dune","author":"Frank Herbert","year":1965,"price":299.90,"stock":12,"categories":["science-fiction","classic"]}'
OK
127.0.0.1:6379> JSON.GET bookstore:catalog:9780441013593 $.title $.price
"{\"$.title\":[\"Dune\"],\"$.price\":[299.9]}"
127.0.0.1:6379> JSON.NUMINCRBY bookstore:catalog:9780441013593 $.stock -2
"[10]"
127.0.0.1:6379> JSON.SET bookstore:catalog:9780441013593 $.price 279.90
OK
127.0.0.1:6379> JSON.ARRAPPEND bookstore:catalog:9780441013593 $.categories '"bestseller"'
1) (integer) 3
127.0.0.1:6379> JSON.GET bookstore:catalog:9780441013593 '$.categories.length()'
"[3]"
127.0.0.1:6379> JSON.GET bookstore:catalog:9780441013593 '$.price * 100'
"[27989.999999999996]"
127.0.0.1:6379> MEMORY USAGE bookstore:catalog:9780441013593
(integer) 368

Decrementing stock, changing the price and adding a category each took a single command on the server, without reading and writing the whole document. Had we stored the JSON as a string, every change would have meant reading the document, changing it in C# and writing it back, with the same GET + SET race we saw in post 03. Expressions such as length() and $.price * 100 are Redis 8.10's JSONPath extensions; the release notes list functions, arithmetic operators and membership tests such as in/nin.

Look at the last JSON.GET: 279.90 × 100 = 27,989.999999999996. Numbers in JSON are stored as doubles, and 279.90 cannot be represented exactly in binary. Do not keep money in JSON as a fractional number; use an integer number of kuruş (priceKurus), as we did in the price hash in post 09. That is why the rest of the catalog documents are written with a priceKurus field (lab/scripts/11-catalog.txt).

Building an Index: FT.CREATE and Field Types

Storing JSON documents is not enough for search: to ask "classics under 200 TL" you need an index. In the Query Engine an index is a structure that follows the keys starting with a given prefix:

FT.CREATE idx:catalog ON JSON PREFIX 1 bookstore:catalog: LANGUAGE turkish SCHEMA $.title AS title TEXT WEIGHT 2 $.author AS author TEXT $.description AS description TEXT $.categories[*] AS category TAG $.priceKurus AS price NUMERIC SORTABLE $.year AS year NUMERIC SORTABLE $.stock AS stock NUMERIC

Each field is defined with a JSONPath and an alias, and queries use the alias. TEXT is split into words, stemmed and scored (with WEIGHT 2 a word in the title counts twice as much as one in the description). TAG is for values that are not split and match exactly: category, status, language code. $.categories[*] turns every element of the array into a separate tag. NUMERIC gives range queries, and SORTABLE fast sorting on that field. For coordinates such as the warehouse locations in post 07 there is also a GEO type.

According to the documentation, keys that exist when the index is created are indexed in the background (percent_indexed in FT.INFO), while documents written or changed afterwards enter the index before the write command returns. So when JSON.SET or JSON.NUMINCRBY returns, search already sees the new value; there is no need for a separate synchronization job. The index itself is not a key: it is listed with FT._LIST and removed with FT.DROPINDEX (without touching the documents).

FT.SEARCH: Filtering, Sorting and Paging

docker compose -f lab/compose.yaml exec -T redis redis-cli < lab/scripts/11-search.txt

127.0.0.1:6379> FT.SEARCH idx:catalog "@category:{classic} @price:[-inf 20000]" SORTBY price ASC LIMIT 0 2 RETURN 2 title price
1) (integer) 3
2) "bookstore:catalog:9780141439587"
3) 1) "price"
   2) "14990"
   3) "title"
   4) "Emma"
4) "bookstore:catalog:9780141439518"
5) 1) "price"
   2) "15990"
   3) "title"
   4) "Pride and Prejudice"
127.0.0.1:6379> FT.SEARCH idx:catalog "austen" RETURN 1 title
1) (integer) 2
2) "bookstore:catalog:9780141439587"
3) 1) "title"
   2) "Emma"
4) "bookstore:catalog:9780141439518"
5) 1) "title"
   2) "Pride and Prejudice"
127.0.0.1:6379> FT.SEARCH idx:catalog "@category:{science\\-fiction} -@category:{dystopia}" RETURN 1 title
1) (integer) 2
2) "bookstore:catalog:9780553293357"
3) 1) "title"
   2) "Foundation"
4) "bookstore:catalog:9780441013593"
5) 1) "title"
   2) "Dune"
127.0.0.1:6379> FT.SEARCH idx:catalog "@category:{science-fiction}" RETURN 1 title
(error) SEARCH_SYNTAX Syntax error at offset 26 near fiction

The query language in short: @field:{value} is a tag, @field:[min max] a numeric range (open-ended with -inf/+inf), and a word without a field is searched in every TEXT field (the author name "austen" found two books). A space means AND, | OR, and a leading - NOT. The 3 at the top of the first reply is the total number of results; LIMIT 0 2 returned only the first page. Without RETURN the whole document comes back, which is wasted traffic for a listing page.

The last line is the most common mistake: a hyphen in a tag value is an operator in the query language, so science-fiction is a syntax error. Punctuation inside tags and words has to be escaped with a backslash. Putting user input straight into a query causes the same problem; it is not as dangerous as SQL injection, but a hyphen or an @ typed by a user can change what the query means.

FT.AGGREGATE and Multi-Value Fields

For the counts next to a search page, the facets such as "Classic (6), Science fiction (3)", the first thing that comes to mind is FT.AGGREGATE with GROUPBY:

127.0.0.1:6379> FT.AGGREGATE idx:catalog "*" GROUPBY 1 @category REDUCE COUNT 0 AS books REDUCE AVG 1 @price AS avgPrice SORTBY 2 @books DESC
1) (integer) 5
2) 1) "category"
   2) "science-fiction"
   3) "books"
   4) "2"
   5) "avgPrice"
   6) "26490"
3) 1) "category"
   2) "dystopia"
   3) "books"
   4) "2"
   5) "avgPrice"
   6) "19490"
4) 1) "category"
   2) "romance"
   3) "books"
   4) "2"
   5) "avgPrice"
   6) "15490"
5) 1) "category"
   2) "classic"
   3) "books"
   4) "1"
   5) "avgPrice"
   6) "21990"
6) 1) "category"
   2) "fantasy"
   3) "books"
   4) "1"
   5) "avgPrice"
   6) "24990"
127.0.0.1:6379> FT.SEARCH idx:catalog "@category:{classic}" LIMIT 0 0
1) (integer) 6

Six of the eight books are classics, yet GROUPBY counted 1. Each book was grouped only by the first category in its array: Dune is ["science-fiction","classic"], so it was counted only as science fiction. The JSON index limitations section of the documentation says that for a JSONPath returning several values, SORTBY sorts by the first value only; it says nothing about GROUPBY, but in the measurement it used the first value as well. Loading $.categories[*] separately with LOAD did not change the result either. For a multi-value tag we got the right numbers by fetching the values with FT.TAGVALS and sending one count query with LIMIT 0 0 for each (the .NET code is below).

GROUPBY works as expected on single-value fields. Grouping by century with APPLY "floor(@year / 100) * 100" AS century, for example, gave 3 books with an average of 17,656 kuruş for the 1800s and 5 books with 23,390 kuruş for the 1900s. The APPLY, FILTER, SORTBY and REDUCE steps form a pipeline that runs in order; handy for report-style questions.

Catalog Search from .NET with NRedisStack

StackExchange.Redis has no typed methods for JSON and the FT.* commands. NRedisStack, Redis's .NET library, adds extension methods on the same IDatabase (db.JSON(), db.FT(), and db.BF() for the Bloom filter from post 07); the connection is still the single multiplexer from post 02. On startup Bookstore.Api creates the index if it does not exist:

// FT.CREATE idx:catalog ON JSON PREFIX 1 bookstore:catalog: LANGUAGE turkish SCHEMA ...
public static async Task EnsureIndexAsync(IDatabase db)
{
    ISearchCommandsAsync ft = db.FT();
    if ((await ft._ListAsync()).Any(name => name.ToString() == Index))
    {
        return;
    }
    await ft.CreateAsync(Index,
        new FTCreateParams().On(IndexDataType.JSON).Prefix(RedisKeys.CatalogPrefix).Language("turkish"),
        new Schema()
            .AddTextField(new FieldName("$.title", "title"), weight: 2.0)
            .AddTextField(new FieldName("$.author", "author"))
            .AddTextField(new FieldName("$.description", "description"))
            .AddTagField(new FieldName("$.categories[*]", "category"))
            .AddNumericField(new FieldName("$.priceKurus", "price"), sortable: true)
            .AddNumericField(new FieldName("$.year", "year"), sortable: true)
            .AddNumericField(new FieldName("$.stock", "stock")));
}

The search endpoint builds the query from parts and escapes every value that comes from the user; the category counts come from count queries sent together with Task.WhenAll:

Query query = new Query(parts.Count == 0 ? "*" : string.Join(' ', parts))
    .ReturnFields("title", "price")
    .Limit((pageNo - 1) * pageSize, pageSize);
if (string.IsNullOrWhiteSpace(q))
{
    query.SetSortBy("price", ascending: true);                                // without words, relevance means nothing: sort by price
}
SearchResult result = await redis.GetDatabase().FT().SearchAsync(Index, query);

// Facet counts: FT.TAGVALS + one FT.SEARCH ... LIMIT 0 0 per category, all sent together
RedisResult[] values = await ft.TagValsAsync(Index, "category");
string[] categories = [.. values.Select(v => v.ToString())];
SearchResult[] counts = await Task.WhenAll(categories.Select(c =>
    ft.SearchAsync(Index, new Query($"@category:{{{Escape(c)}}}").Limit(0, 0))));

// Query syntax characters (-, @, {, |, : ...) must be escaped, otherwise "science-fiction" is a syntax error
public static string Escape(string value)
{
    StringBuilder sb = new(value.Length);
    foreach (char c in value)
    {
        if (!char.IsLetterOrDigit(c) && c != '_')
        {
            sb.Append('\\');
        }
        sb.Append(c);
    }
    return sb.ToString();
}

curl "http://localhost:5080/catalog/search?category=science-fiction&maxPrice=25000"
{"query":"@category:{science\\-fiction} @price:[-inf 25000]","total":2,"page":1,"hits":[{"isbn":"9780060850524","title":"Brave New World","priceKurus":19990},{"isbn":"9780553293357","title":"Foundation","priceKurus":22990}]}
curl "http://localhost:5080/catalog/categories"
[{"category":"classic","books":6},{"category":"science-fiction","books":3},{"category":"dystopia","books":2},{"category":"romance","books":2},{"category":"fantasy","books":1}]
curl -X PATCH http://localhost:5080/catalog/books/9780441013593/price -H "Content-Type: application/json" -d '{"priceKurus":24990}'
{"isbn":"9780441013593","priceKurus":24990}
curl "http://localhost:5080/catalog/search?category=science-fiction&maxPrice=25000"
{"query":"@category:{science\\-fiction} @price:[-inf 25000]","total":3,"page":1,"hits":[{"isbn":"9780060850524","title":"Brave New World","priceKurus":19990},{"isbn":"9780553293357","title":"Foundation","priceKurus":22990},{"isbn":"9780441013593","title":"Dune","priceKurus":24990}]}

The PATCH endpoint changes only $.priceKurus (JSON.SET … XX; 404 if the document does not exist), and Dune showed up in the next search with its new price. I saw two details with MONITOR. First, NRedisStack adds DIALECT 2 to every FT.SEARCH, while the server default, according to CONFIG GET search-default-dialect, is 1. A query you try in redis-cli and the one the application sends may run under different dialects; write DIALECT 2 in redis-cli too to keep them the same. Second, the first time I fetched the category counts, the list contained a bestseller with 0 books: FT.TAGVALS kept returning a value that no document had any more, until the index cleanup (GC) ran. That is why the endpoint drops values with a count of 0.

Turkish Search: the Limits of Stemming and the Dotted Capital I

Redis's stemming documentation says it uses the Snowball library and lists turkish among the supported languages. In Guess First "evlilikler" matched "evlilik"; "kızlar"/"kızları" and "savaşlar"/"savaş" matched as well. But not every suffix gives the right result:

127.0.0.1:6379> FT.SEARCH idx:catalog "@description:gezegenlerde" RETURN 1 title
1) (integer) 0
127.0.0.1:6379> FT.EXPLAIN idx:catalog "@description:gezegenlerde"
"@description:UNION {\n  @description:gezegenlerde\n  @description:+gezegen(expanded)\n  @description:gezegen(expanded)\n}\n"
127.0.0.1:6379> FT.EXPLAIN idx:catalog "@description:gezegeni"
"@description:UNION {\n  @description:gezegeni\n  @description:+gezege(expanded)\n  @description:gezege(expanded)\n}\n"

The word "gezegeni", meaning its planet, in Dune's description was reduced to the stem "gezege", while "gezegenlerde", meaning on the planets, in the query was reduced to "gezegen"; the stems differ, so there is no match. The same happened between "hazinesi" and "hazinesine", its treasure and to its treasure. Snowball's Turkish stemmer is rule based and uses no dictionary; in an agglutinative language it cannot bring some word pairs to the same stem. FT.EXPLAIN shows which terms a query expands to, so it is the first place to look in a case like this.

The capital letter problem is more basic: lowercasing "İngiliz" under the Unicode rules gives "i" plus a combining dot, which is not the same term as "ingiliz". The index does no Turkish-specific lowercasing. If the application normalizes both the documents and the query the same way (for example by writing text lowercased with ToLower(new CultureInfo("tr-TR")) and with accents simplified into a separate field to be indexed), the problem goes away. When Turkish search quality is critical, PostgreSQL's Turkish configuration and unaccent (the "PostgreSQL Full-Text Search: tsvector, tsquery, Turkish Configuration, unaccent and pg_trgm" post in the PostgreSQL series) or Elasticsearch's Turkish analyzer (in the Elasticsearch series) give you more control.

Scale, Valkey and When to Use Another Search Engine

To see the cost of the index I wrote 100,000 generated books both as JSON and as hashes and indexed both:

dotnet run --project src/Bookstore.RedisLab -- catalog-scale

100,000 books: JSON 45.6 MB, hash 46.1 MB
MEMORY USAGE one book: json=464 B, hash=533 B
idx:bench-json: indexed 100,000 docs in 6,130 ms, index memory 41.4 MB
idx:bench-hash: indexed 100,000 docs in 5,700 ms, index memory 41.4 MB
@category:{science\-fiction} @price:[-inf 20000]: 7,947 matches, p50 1.69 ms, p99 3.19 ms
@description:(ejderha hazine): 20,362 matches, p50 2.25 ms, p99 5.28 ms

In this data set the memory difference between JSON and hash is negligible; the index, however, takes as much space as the data itself (41.4 MB). Search in Redis means fitting the catalog into memory twice. Queries take milliseconds, but these times come from a single client on an idle server; the text query with 20,000 matches is slower than the filtered one because it sorts more.

In Valkey, JSON and search are not part of the core; in the valkey/valkey image FT.SEARCH is an unknown command. The modules come in the valkey/valkey-bundle image (in the lab under the bundle profile, on port 6381). I tried the same index there:

127.0.0.1:6381> FT.CREATE idx:catalog ON JSON PREFIX 1 bookstore:catalog: LANGUAGE turkish SCHEMA $.title AS title TEXT
(error) Bad arguments for LANGUAGE: Unknown argument `turkish`
127.0.0.1:6381> FT.CREATE idx:catalog ON JSON PREFIX 1 bookstore:catalog: SCHEMA $.title AS title TEXT WEIGHT 2
(error) Invalid field type for field `$.title`: The `WEIGHT` clause with a value other than `1.0` is not supported.
127.0.0.1:6381> FT.CREATE idx:catalog ON JSON PREFIX 1 bookstore:catalog: SCHEMA $.title AS title TEXT $.categories[*] AS category TAG $.priceKurus AS price NUMERIC
OK
127.0.0.1:6381> FT.SEARCH idx:catalog "@title:dune" RETURN 1 title
1) (integer) 1
2) "bookstore:catalog:9780441013593"
3) 1) "title"
   2) "Dune"
127.0.0.1:6381> FT.SEARCH idx:catalog "@category:{classic}" LIMIT 0 0
1) (integer) 1

valkey-search (version number 66049 in the image's MODULE LIST) supports text search, but has no Turkish stemming and no field weights; with multi-element category arrays it found 1 classic instead of 6, matching only the single-element array. So even though the command names are the same, an index definition and the queries from Redis do not move to Valkey as they are; try every feature you use on the target version.

When is the Redis Query Engine the right choice? When the data is already in Redis, when the search need is filtering + sorting + simple text search, and when latency matters (such as "in-stock classics under 200 TL" next to the cart). If you need relevance tuning, synonyms, advanced language analysis, highlighting and large data, Elasticsearch; if the data is already in SQL Server or PostgreSQL and search is a secondary feature, the database's own full-text search means fewer moving parts. Vector and hybrid search (FT.HYBRID) are the topic of the "Hybrid Search and Reciprocal Rank Fusion: Combining Keyword and Vector Search" post in the RAG series.

In Production

The master record of the catalog should stay in SQL Server; the documents in Redis are a search copy. Keep the copy up to date with an event, as we did with order events (the stream in post 10), or with a periodic synchronization job; if Redis becomes the only source, the persistence and memory decisions (post 12) turn directly into a risk of data loss. If the index definition has to change, build a second index under a new name (idx:catalog-v2), wait until it is indexed in the background, then switch the alias the application uses to the new index with FT.ALIASUPDATE; Redis 8.10 added FT.ALIASLIST to list aliases. The query timeout is 500 ms by default (search-timeout), and the default for search-on-timeout is return: when time runs out you get partial results, not an error. To avoid mistaking partial results for complete ones, check NRedisStack's SearchResult.Warnings; according to its documentation it is populated only on RESP3 connections. When planning memory, count the documents and the index together: total_index_memory_sz_mb in FT.INFO was almost as large as the data itself in this post's measurement.

Common Pitfalls

Making Redis the only record of the catalog: it replaces the main database without the cost of memory, persistence and reindexing being worked out.

Putting characters such as a hyphen, a dot or @ into a query without escaping them: science-fiction was a syntax error.

Relying on GROUPBY over a multi-value field: 6 classic books were counted as 1.

Keeping prices in JSON as fractional numbers: 279.90 × 100 = 27,989.999999999996.

Assuming a query tried in redis-cli without a dialect behaves like the one in the application: the server default is 1, NRedisStack sends 2.

Assuming Turkish stemming handles every suffix and that a search for "ingiliz" will find "İngiliz".

Moving an index definition from Redis to Valkey as it is: LANGUAGE turkish and WEIGHT were rejected.

A Small Experiment

★☆☆ idx:catalog already has $.stock AS stock NUMERIC. Add an inStock=true parameter to the search endpoint and put @stock:[1 +inf] into the query. Does The Hobbit drop out of the list?

★★☆ Add a second field named titleSearch to the documents: the title and description lowercased with the tr-TR culture. Add this field to the index and verify that a search for "ingiliz" finds both books.

★★★ Build a new index named idx:catalog-v2 in which the author field is a TAG, and move the application to it without downtime with FT.ALIASADD/FT.ALIASUPDATE. Did the search endpoint return any errors during the switch?

In the next post we look at how Redis writes its data to disk and how memory is managed: RDB, AOF and fsync, the memory cost of fork, maxmemory and what eviction policies mean outside caching, and big and hot keys.