Redis Lists and Sets: Simple Queues, Reliable Queues, Recently Viewed Items and Tags

Version note: The examples in this post were run with Redis 8.10.3 (redis:8), Valkey 9.1.3, .NET 10 and StackExchange.Redis 3.4.35.

Guess First

At Bookstore, the online bookstore used as the shared example across this site's series, order confirmation e-mails go through a Redis list: the API puts the order number on the queue with LPUSH, and a worker takes it with RPOP and sends the e-mail. One day the SMTP server is unreachable and the worker crashes right after taking order-1002 off the queue:

RedisValue job = reliable
    ? await db.ListMoveAsync(Queue, Processing, ListSide.Right, ListSide.Left) // LMOVE
    : await db.ListRightPopAsync(Queue);                                      // RPOP
if (job.IsNull)
{
    break;
}
if (job == "order-1002")
{
    throw new InvalidOperationException("SMTP server unreachable, worker crashed");
}
await SendEmailAsync(job);
if (reliable)
{
    await db.ListRemoveAsync(Processing, job, count: 1); // ack: LREM
}

There were three jobs on the queue. When the worker restarts, where is order-1002? What changes if you run the same scenario with LMOVE instead of RPOP?

dotnet run --project src/Bookstore.RedisLab -- queue-crash

      sent e-mail for order-1001
RPOP  worker: SMTP server unreachable, worker crashed
      queue: [order-1003]  processing: []
      sent e-mail for order-1001
LMOVE worker: SMTP server unreachable, worker crashed
      queue: [order-1003]  processing: [order-1002]

With RPOP, order-1002 is nowhere: Redis removed it from the list the moment it handed it to the worker, and the worker died before finishing. The customer will never get the confirmation e-mail, and nobody will notice. With LMOVE, the job was moved to a processing list in the same atomic step that took it off the queue; after the crash it is still there and can be processed again. This post covers lists and sets through Bookstore's real needs: a simple and a reliable queue, where blocking commands stand in StackExchange.Redis, "the last 10 books viewed", book tags and the daily unique visitor count.

List Commands

A Redis list is an ordered sequence of strings that you can add to and remove from at both ends. Operations at the ends are O(1); reaching an element in the middle or scanning the list is O(N):

127.0.0.1:6379> LPUSH bookstore:queue:emails order-1001 order-1002 order-1003 order-1004
(integer) 4
127.0.0.1:6379> LRANGE bookstore:queue:emails 0 -1
1) "order-1004"
2) "order-1003"
3) "order-1002"
4) "order-1001"
127.0.0.1:6379> RPOP bookstore:queue:emails
"order-1001"
127.0.0.1:6379> LLEN bookstore:queue:emails
(integer) 3
127.0.0.1:6379> LMOVE bookstore:queue:emails bookstore:queue:emails:processing RIGHT LEFT
"order-1002"
127.0.0.1:6379> LREM bookstore:queue:emails:processing 1 order-1002
(integer) 1

Since LPUSH adds on the left, together with RPOP it forms a FIFO queue. LRANGE key 0 -1 returns the whole list; that is fine for small lists, but on large ones it carries the same risk as HGETALL in post 04. Small lists live in a single listpack; as they grow they move to a quicklist, a linked list of listpacks. The limit is set by list-max-listpack-size; the default -2 means each node stays under 8 KB.

A Reliable Queue: LMOVE, Ack and a Reaper

The LMOVE flow from Guess First is the well-known reliable queue pattern. It has three parts. Taking: the worker takes a job with LMOVE queue processing RIGHT LEFT; the job moves from one list to the other inside a single Redis command, with no moment in which it could be lost. Acknowledging: when the job is done, the worker removes it from processing with LREM processing 1 job. Reaping: a separate task moves jobs that have stayed in processing too long back to the queue, because the worker that took them has most likely died.

The limits of the pattern show here too. The list does not record when a job entered processing; for the reaper to decide that a job "stayed too long" you have to store the time separately, for example in a hash. It does not record how many times a job was attempted or which worker holds it either; a job that keeps failing can circle forever. Delivering the same message to more than one group of workers is not possible. Redis Streams provide all of this out of the box through consumer groups, the pending entry list and XAUTOCLAIM; we cover them in post 10. A list-based queue suits very simple workflows or cases where some loss is tolerable.

Redis 8.10 added bulk moves to this flow: LMOVEM moves several elements in one command. A worker can take ten jobs at a time:

127.0.0.1:6379> RPUSH a j1 j2 j3 j4 j5
(integer) 5
127.0.0.1:6379> LMOVEM a b RIGHT LEFT COUNT 3 BULK
1) "j3"
2) "j4"
3) "j5"
127.0.0.1:6379> LMOVEM a b RIGHT LEFT EXACTLY 5 OBO
(nil)
127.0.0.1:6379> LMOVEM a b RIGHT LEFT COUNT 5 OBO
1) "j1"
2) "j2"
127.0.0.1:6379> EXISTS a
(integer) 0

COUNT n moves at most n elements; EXACTLY n moves exactly n or none at all. BULK moves the elements as a block, preserving their order from the source; OBO (one by one) behaves as if each element were moved with its own LMOVE. In this release one of the two is mandatory; writing COUNT 10 without an ordering gave me ERR syntax error. In StackExchange.Redis, the ListMoveAsync overload with ListMoveCount and ListMoveOrder parameters sends this command. Valkey does not have LMOVEM.

Blocking Commands and StackExchange.Redis

What should a worker do when the queue is empty? Redis's answer is blocking commands: BLPOP, BRPOP, BLMOVE (and, since 8.10, BLMOVEM) hold the connection until an element arrives or a timeout expires. The trouble is that in StackExchange.Redis the whole application shares one connection (post 02). That is why the package documentation says it does not, and will not, offer blocking pops. You can still send one with ExecuteAsync; let's see what happens:

Task<RedisResult> blpop = db.ExecuteAsync("BLPOP", Queue, 3); // the queue is empty: waits up to 3 seconds
await Task.Delay(100);
TimeSpan ping = await db.PingAsync();                            // another caller, same multiplexer
Console.WriteLine($"PING from another caller took {ping.TotalMilliseconds:N0} ms");

PING from another caller took 2,989 ms
BLPOP returned nil after 3,095 ms

For 3 seconds the rest of the application could not get a single reply from Redis; even PING waited for the BLPOP to time out. In an API that means every request freezing at once. The documentation suggests pub/sub instead of blocking pops: the data stays in the list, and pub/sub only sends workers a "there is work on the queue" signal; when the signal arrives, workers take jobs with an ordinary LMOVE. A simpler route is to poll with increasing intervals while the queue is empty. But considering the limits above, the right tool for a queue with workers is usually Streams.

Bookstore: the Last 10 Books Viewed

Beyond queues, the most common use of a list is a bounded history: the books a customer looked at last. Whenever a book page opens we put the book at the head of the list; if the same book is already there we remove it first, and we cap the list at 10 elements:

IDatabase db = redis.GetDatabase();
RedisKey key = RedisKeys.RecentlyViewed(customerId);

// Remove the book if already there, push it to the head, keep only the newest 10 (pipelined, not atomic)
await Task.WhenAll(
    db.ListRemoveAsync(key, request.Isbn),               // LREM key 0 isbn
    db.ListLeftPushAsync(key, request.Isbn),             // LPUSH
    db.ListTrimAsync(key, 0, MaxItems - 1),              // LTRIM key 0 9
    db.KeyExpireAsync(key, Lifetime));

# viewed 9780441013593, 9780451524935, 9780141439518 and 9780441013593 again
curl http://localhost:5080/customers/7/recently-viewed
["9780441013593","9780141439518","9780451524935"]

# viewed 12 more books
curl http://localhost:5080/customers/7/recently-viewed
["9780000000012","9780000000011","9780000000010","9780000000009","9780000000008","9780000000007","9780000000006","9780000000005","9780000000004","9780000000003"]
docker compose -f lab/compose.yaml exec redis redis-cli LLEN bookstore:customer:7:recent
(integer) 10

The book viewed twice did not repeat in the list, it moved to the front; after 15 views the list stayed at 10 elements. LTRIM is the basic tool for a capped list: called right after LPUSH, it keeps the list from ever being longer than one element over the limit. The four commands run in order in the same pipeline, but they are not atomic: two requests arriving at the same time from two tabs of the same customer can interleave. Here the worst outcome is a book repeated for a moment; the next request fixes it. Where you need full atomicity there are MULTI/EXEC and Lua (post 08). LREM scans the list from start to end; for a 10-element list that is nothing, but on lists with thousands of elements remember that LREM and LPOS are O(N).

Set Commands and Book Tags

A set is an unordered collection of unique strings. Membership checks are O(1), and operations between sets (intersection, union, difference) run on the server. In Bookstore each tag is a set, such as bookstore:tag:classic, holding the ISBNs of the books that carry it:

127.0.0.1:6379> SADD bookstore:tag:classic 9780141439518 9780451524935 9780547928227 9780141439549
(integer) 4
127.0.0.1:6379> SADD bookstore:tag:science-fiction 9780441013593 9780451524935 9780553293357
(integer) 3
127.0.0.1:6379> SADD bookstore:tag:dystopia 9780451524935 9780060850524
(integer) 2
127.0.0.1:6379> SMISMEMBER bookstore:tag:classic 9780141439518 9780441013593
1) (integer) 1
2) (integer) 0
127.0.0.1:6379> SINTER bookstore:tag:classic bookstore:tag:science-fiction
1) "9780451524935"
127.0.0.1:6379> SINTERCARD 2 bookstore:tag:classic bookstore:tag:science-fiction
(integer) 1
127.0.0.1:6379> SUNIONCARD 2 bookstore:tag:science-fiction bookstore:tag:dystopia
(integer) 4
127.0.0.1:6379> SDIFF bookstore:tag:science-fiction bookstore:tag:classic
1) "9780441013593"
2) "9780553293357"
127.0.0.1:6379> SDIFFCARD 2 bookstore:tag:science-fiction bookstore:tag:classic
(integer) 2
127.0.0.1:6379> OBJECT ENCODING bookstore:tag:classic
"intset"

The *CARD commands return only the number of elements, without building and sending the result. SINTERCARD has existed since Redis 7.0, and with LIMIT it can answer questions such as "are there at least 3 books in common?" without finishing the count. Its counterparts for union and difference, SUNIONCARD and SDIFFCARD, arrived with Redis 8.10; on Valkey 9.1.3 both returned ERR unknown command. The intset encoding is a small surprise: ISBN-13s fit into 64-bit integers, so Redis keeps this small set as a sorted array of integers. The limits are set-max-intset-entries (512), set-max-listpack-entries (128) and set-max-listpack-value (64); a non-numeric member or a size over the limit turns the set into a listpack first and then a hashtable.

Two endpoints in Bookstore.Api use these sets: books carrying all of the given tags (SINTER, and SINTERCARD for the count) and random suggestions for the "you may also like" box (SRANDMEMBER):

// Books that carry every given tag: SINTER; the count alone: SINTERCARD
app.MapGet("/books/by-tags", async (string[] tag, IConnectionMultiplexer redis) =>
{
    IDatabase db = redis.GetDatabase();
    RedisKey[] keys = [.. tag.Select(RedisKeys.Tag)];
    Task<RedisValue[]> books = db.SetCombineAsync(SetOperation.Intersect, keys);
    Task<long> count = db.SetIntersectionLengthAsync(keys);
    await Task.WhenAll(books, count);
    return Results.Ok(new { tags = tag, count = count.Result, isbns = books.Result.Select(b => b.ToString()) });
});

// A few random books from a tag for the "you may also like" box: SRANDMEMBER
app.MapGet("/books/suggestions", async (string tag, int count, IConnectionMultiplexer redis) =>
{
    RedisValue[] isbns = await redis.GetDatabase().SetRandomMembersAsync(RedisKeys.Tag(tag), count);
    return Results.Ok(isbns.Select(i => i.ToString()));
});

curl "http://localhost:5080/books/by-tags?tag=classic&tag=science-fiction"
{"tags":["classic","science-fiction"],"count":1,"isbns":["9780451524935"]}
curl "http://localhost:5080/books/suggestions?tag=classic&count=2"
["9780141439518","9780451524935"]
curl "http://localhost:5080/books/suggestions?tag=classic&count=2"
["9780547928227","9780451524935"]

With a positive count, SRANDMEMBER returns distinct members; a negative count allows the same member to come back more than once. If you want to take a random member out of the set, for a prize draw for example, there is SPOP.

Daily Unique Visitors

One of the most natural uses of a set is unique counting: a customer who visits 20 times in a day is counted once. We keep one set per day, and the return value of SADD also answers "is this today's first visit?". The weekly unique visitor count needs the size of the union of seven days; SUNIONCARD gives it without building the union:

// Unique visitors over the last 7 days without building the union: SUNIONCARD (Redis 8.10+)
app.MapGet("/visits/weekly", async (IConnectionMultiplexer redis, TimeProvider clock) =>
{
    DateOnly today = DateOnly.FromDateTime(clock.GetUtcNow().UtcDateTime);
    RedisKey[] days = [.. Enumerable.Range(0, 7).Select(i => RedisKeys.DailyVisitors(today.AddDays(-i)))];
    long unique = await redis.GetDatabase().SetCombineLengthAsync(SetOperation.Union, days);
    return Results.Ok(new { from = today.AddDays(-6), to = today, uniqueVisitors = unique });
});

To see the scale, I generated 20,000 visits a day from a pool of 50,000 customers and filled seven daily sets:

dotnet run --project src/Bookstore.RedisLab -- visitors

bookstore:visitors:20261108: 16,429 unique visitors, 622 KB, encoding=hashtable
7 days, SUNIONCARD        : 46,925 unique visitors (11.9 ms)
7 days, SUNIONCARD APPROX : 47,131 unique visitors (4.3 ms)

A single day's set takes 622 KB; with 16,429 members it is far past the intset limit and lives in a hashtable. The exact weekly count is 46,925. The APPROX option counts with a HyperLogLog-based estimate: 47,131, an error of 0.44 percent, in a third of the time. In StackExchange.Redis the approximate: true parameter of SetCombineLengthAsync sends it. With millions of visitors a day the memory of these sets becomes a real cost; post 07 does the same count with 12 KB HyperLogLogs and measures the memory difference.

In Production

Bound the size of your lists and sets, or at least watch it. I measured a set with a million members:

SMEMBERS: 1,000,000 members in 287 ms
SSCAN:    1,000,000 members in 596 ms
slowlog: SMEMBERS 66.8 ms

SMEMBERS locked the server for 67 ms; SSCAN (SetScanAsync) never showed up in SLOWLOG. The same warning applies to SINTER, SUNION and SDIFF on large sets: their cost grows with the number of elements. If you only need a number, use the *CARD variants and LIMIT. For lists used as queues, track the queue length (LLEN) and the size of the processing list as metrics; a growing processing list is a sign of dying workers. In a Redis that holds queues or visitor sets, make sure the eviction policy will not delete these keys; post 12 covers that in detail.

Common Pitfalls

Treating a list as a durable job queue: a job taken with RPOP is lost if the worker crashes. A processing list with LMOVE is a step forward, but there is no retry count, no timeout and no consumer tracking; those are in Streams (post 10).

Sending BLPOP through the shared multiplexer: the whole application got no reply from Redis for 3 seconds.

SMEMBERS on a big set: a million members made the server wait for 67 ms. Use SSCAN or the *CARD commands.

Forgetting LTRIM on a capped list: the "recently viewed" list grows without limit for every customer and turns into a memory leak.

Assuming Redis 8.10's new commands (LMOVEM, SUNIONCARD, SDIFFCARD) on Valkey or on an older managed Redis.

A Small Experiment

★☆☆ Add a non-numeric member to the bookstore:tag:classic set (for example draft-42) and look at OBJECT ENCODING again. Then add 130 more members: at what point does the encoding change once more?

★★☆ Add a reaper to the reliable queue: when a job is moved to processing, write the time it was taken to a hash (HSET bookstore:queue:emails:taken order-1002 <unix-ms>), and have the reaper move jobs older than 60 seconds back to the queue with LMOVE. What happens if the reaper and a worker touch the same job at the same time?

★★★ Repeat the visitors experiment with 200,000 visits a day. How much memory does a single day's set take? How do the time and error difference between SUNIONCARD and SUNIONCARD … APPROX change?

In the next post we move on to sorted sets: the bestsellers of the week, paging by score, deterministic ordering for equal scores, and sliding windows that use timestamps as scores.