Running Redis in Production: ACL, TLS, Dangerous Commands, Latency Diagnosis, OpenTelemetry and Testcontainers

Version note: The examples in this post were run with Redis 8.10.3 (redis:8), Valkey 9.1.3, .NET 10, StackExchange.Redis 3.4.35, OpenTelemetry 1.21.0 (Redis instrumentation 1.20.1-beta.1), Testcontainers.Redis 4.16.0 and xUnit v3 4.0.3.

Guess First

So far Bookstore.Api, the API of Bookstore, the online bookstore used as the shared example across this site's series, has connected to Redis as the passwordless default user, which can do anything. Let's give the application its own user: only keys and channels starting with bookstore:, read-write, stream, pub/sub, JSON and search commands; no dangerous commands, no Lua, but the Functions from post 08 can be called:

ACL SETUSER bookstore-api on ><api-password> resetkeys ~bookstore:* resetchannels &bookstore:* -@all +@read +@write +@stream +@pubsub +@connection +@transaction +@json +@search -@dangerous +fcall +fcall_ro +function|list

Which of the following commands work for this user?

127.0.0.1:6379> FLUSHALL
(error) NOPERM User bookstore-api has no permissions to run the 'flushall' command
127.0.0.1:6379> KEYS *
(error) NOPERM User bookstore-api has no permissions to run the 'keys' command
127.0.0.1:6379> CONFIG GET maxmemory
(error) NOPERM User bookstore-api has no permissions to run the 'config|get' command
127.0.0.1:6379> EVAL 'return 1' 0
(error) NOPERM User bookstore-api has no permissions to run the 'eval' command
127.0.0.1:6379> GET session:1
(error) NOPERM No permissions to access a key
127.0.0.1:6379> GET bookstore:order:id-seq
(nil)
127.0.0.1:6379> FCALL reserve_stock 2 bookstore:stock:{9780441013593} bookstore:stock:{9780441013593}:reservations order-1 1
(integer) -1
127.0.0.1:6379> PUBLISH other-channel x
(error) NOPERM No permissions to access a channel

The two commands the application really needs worked; the rest were refused. An application vulnerability or a leaked connection string can no longer wipe the data with FLUSHALL, lock the server with KEYS * or run arbitrary Lua. This post covers the pieces of running Redis in production: not exposing it to the network, ACLs, moving the application to Functions and patch discipline, TLS, SLOWLOG/LATENCY DOCTOR and StackExchange.Redis's timeout message, OpenTelemetry, and integration testing with Testcontainers.

Not Exposing Redis to the Network

The lab's Redis writes this line on every start: WARNING: Redis does not require authentication and is not protected by network restrictions. In the official image protected-mode is off (CONFIG GET protected-mode → no) and Redis listens on every interface; without the 127.0.0.1:6379:6379 port mapping in the compose file, anyone on the machine's network could connect without a password. With protected-mode on and no password configured, Redis accepts connections only from localhost; it is turned off in the container because connections come from the container network. The rule is simple: never expose Redis to the internet, keep it in the private network where the application runs, allow only the application servers with a firewall or security group, and turn off the passwordless default user.

Per-User Permissions with ACLs

An ACL rule is read in order: -@all closes everything, categories such as +@read and +@write open things up, and -@dangerous (commands such as FLUSHALL, KEYS, CONFIG, DEBUG, RESTORE) closes them again. ~bookstore:* is the key pattern, &bookstore:* the channel pattern. On the application side the password is not written into code or appsettings.json; it comes from an environment variable:

$env:ConnectionStrings__Redis="localhost:6379,name=bookstore-api,user=bookstore-api,password=<api-password>,tiebreaker=,configChannel=,abortConnect=false"
dotnet run --project src/Bookstore.Api

I ran the API with this user, tried every endpoint and looked at ACL LOG. Two things came up. GET /catalog/categories returned 500: FT.TAGVALS, which we use for facet counts in post 11 (O(N), it returns every tag value), is in the @dangerous category according to COMMAND INFO, so the rule closed it too; I opened it as a single command with ACL SETUSER bookstore-api +ft.tagvals. The second was the commands StackExchange.Redis tries while connecting: INFO, CONFIG GET, CLUSTER NODES/SLOTS, the __Booksleeve_TieBreak key and the __Booksleeve_MasterChanged channel. The package tolerates these refusals and the application works; tiebreaker= and configChannel= in the connection string turned off the last two, the first four stay in ACL LOG. Watching ACL LOG shows both a wrongly configured permission and someone trying something forbidden.

Users created with ACL SETUSER live in memory; after restarting the container, ACL LIST showed only user default on nopass … +@all. To keep them, start Redis with aclfile and use ACL SAVE (the file holds a SHA-256 hash of each password, not the password itself), or recreate the users during deployment with values from a secret store. The last step is to turn off the default user (ACL SETUSER default off) and keep a separate administration user with +@all. Valkey 9.1.3 rejected the same rule because of +@json (Unknown command or category name in ACL): a server without the JSON and search modules has no such categories.

No EVAL for the Application User: Functions and Patch Discipline

In post 08 the stock reservation was a Lua script sent from the API with EVAL/EVALSHA. With EVAL closed in the ACL, I moved the reservation endpoint to the reserve_stock function we loaded in post 08; the application no longer sends Lua text, it only calls a function registered on the server:

// FCALL reserve_stock 2 <stock> <reservations> <orderId> <quantity>
RedisResult result = await redis.GetDatabase().ExecuteAsync("FCALL", "reserve_stock", 2,
    RedisKeys.Stock(isbn), RedisKeys.StockReservations(isbn), request.OrderId, request.Quantity);

// On startup: if the library is not loaded, reservations will fail; log it
RedisResult libraries = await db.ExecuteAsync("FUNCTION", "LIST", "LIBRARYNAME", "bookstore");

There is a security reason for this. Redis's security advisories on GitHub published four Lua vulnerabilities together in October 2025: CVE-2025-49844 (use-after-free, 9.9 under CVSS 3.1), CVE-2025-46817 (integer overflow, 7.0), CVE-2025-46818 (running Lua code with another user's permissions, 6.0) and CVE-2025-46819 (out-of-bounds read, 6.3). In May 2026 Redis 8.8 fixed CVE-2026-23631 (Lua, use-after-free on writable replicas), CVE-2026-23479 (use-after-free in the blocked client flow) and CVE-2026-25243 (invalid memory access through RESTORE); in November 2025 there was also CVE-2025-62507 for XACKDEL in Redis 8.2. What they have in common is that the attacker has to be an authenticated user, and the workaround the advisories suggest is to close EVAL/EVALSHA and the related commands with ACLs. Protection has two parts: following patch releases and applying them quickly (watch Redis's security advisories on GitHub; since its GA in July 2026 the 8.10 line has had three security updates already: 8.10.1, 8.10.2 and 8.10.3), and giving each application only the commands it needs.

TLS

Traffic between Redis and the application is unencrypted by default, including the ACL password. For the lab I wrote a small command that produces a CA and a localhost certificate; the files are written to lab/tls/, which is in .gitignore:

dotnet run --project src/Bookstore.RedisLab -- make-certs
docker compose -f lab/compose.yaml -f lab/compose.tls.yaml up -d redis
dotnet run --project src/Bookstore.RedisLab -- tls-connect

plain               : It was not possible to connect to the redis server(s). Error connecting right now. To allow this multiplexer to continue retrying until it's able to connect, use abortConnect=false in your connection string or AbortOnConnectFail=false; in your code.
tls, CA not trusted : It was not possible to connect to the redis server(s). There was an authentication failure; check that passwords (or client certificates) are configured correctly: (AuthenticationException) The remote certificate is invalid because of errors in the certificate chain: UntrustedRoot
tls, lab CA trusted : connected, PING 0.7 ms

compose.tls.yaml starts Redis with --port 0 --tls-port 6379; the plain port is closed. The plain connection attempt could not connect, the attempt with ssl=true but without trusting the CA got a certificate chain error, and the one that trusts the CA connected:

ConfigurationOptions options = ConfigurationOptions.Parse("localhost:6379,ssl=true");
options.TrustIssuer(caFile);                              // accept certificates signed by this CA only

Writing a callback that turns certificate validation off (always returning true) makes TLS pointless; someone in the middle can read the traffic with their own certificate. With managed services the certificate comes from a public CA, so ssl=true alone is enough.

Diagnosis: SLOWLOG, LATENCY DOCTOR and the Timeout Message

Imagine a report job accidentally running a 2-second Lua loop on the server, while the API, running with asyncTimeout=1000, reads a price at the same time:

dotnet run --project src/Bookstore.RedisLab -- timeout-message

after 1900 ms: Timeout awaiting response (outbound=0KiB, inbound=0KiB, 1895ms elapsed, timeout is 1000ms), command=GET, next: GET bookstore:prices:9780441013593, inst: 0, qu: 0, qs: 0, aw: False, bw: Inactive, rs: ReadAsync, ws: Idle, in: 0, in-pipe: 0, last-in: 361, cur-in: 361, lm: 2/2/0/0, sync-ops: 0, async-ops: 1, serverEndpoint: localhost:6379, conn-sec: 2.01, aoc: 1, mc: 1/1/0, clientName: bookstore-api, IOCP: (Busy=0,Free=1000,Min=1,Max=1000), WORKER: (Busy=1,Free=32766,Min=10,Max=32767), POOL: (Threads=5,QueuedItems=0,CompletedItems=67,Timers=2), v: 3.4.35.20751 (see https://seredis.dev/Timeouts for some common client-side issues that can cause timeouts)

When reading the message, rule out the client side first: WORKER: (Busy=1,…,Min=10) says the thread pool is relaxed, qu: 0 that no command is waiting to be sent, in: 0 that no bytes are waiting to be read, and ws: Idle that the writer is idle. The command was sent and no answer came: the problem is on the server or in the network. In the sync-over-async experiment in post 02 the same message showed WORKER Busy=501 Min=10; there the server was healthy and there was no thread to process the reply. One more detail: the timeout was 1,000 ms, yet the exception came at 1,895 ms; the package catches timeouts with a periodic check, so the actual wait can be longer than the setting. On the server side the culprit is easy to find:

127.0.0.1:6379> SLOWLOG GET 1
1) 1) (integer) 2
   2) (integer) 1794658505
   3) (integer) 2215073
   4) 1) "EVAL"
      2) "local n = 0 for i = 1, 1000000000 do n = n + 1 end return n"
      3) "0"
   5) "<client-ip:port>"
   6) "report-job"
   7) (integer) 3
127.0.0.1:6379> LATENCY LATEST
1) 1) "command"
   2) (integer) 1794658505
   3) (integer) 2215
   4) (integer) 2215

SLOWLOG shows the command, its duration (2,215,073 µs) and the client name (report-job, the benefit of giving the multiplexer a name=). LATENCY LATEST and LATENCY DOCTOR record only when latency-monitor-threshold is greater than zero; the default is 0, and I set it to 100 ms for this experiment. The LATENCY DOCTOR report had the line "1. command: 1 latency spikes (average 2215ms…)" and advice on slow commands and deleting big keys. Beyond these, the values to watch are connected_clients, blocked_clients, instantaneous_ops_per_sec, used_memory, evicted_keys and the persistence metrics from post 12 in INFO, and the per-client omem (the output buffer from post 09) in CLIENT LIST.

Redis Spans with OpenTelemetry

I added OpenTelemetry tracing to Bookstore.Api: every HTTP request is a span and every Redis command inside it a child span. The Redis instrumentation package is still in beta; its version number says so plainly:

// Post 14: a span per Redis command under the ASP.NET Core request span; console output only when asked for
builder.Services.AddOpenTelemetry().WithTracing(tracing =>
{
    tracing.AddAspNetCoreInstrumentation().AddRedisInstrumentation();
    if (builder.Configuration.GetValue<bool>("Telemetry:Console"))
    {
        tracing.AddConsoleExporter();
    }
});

Activity.TraceId:            4f50a461c2936a5ba6a705c90c118cc8
Activity.ParentSpanId:       5e9527db79e128b8
Activity.DisplayName:        FCALL
Activity.Kind:               Client
Activity.Duration:           00:00:00.0019405
Activity.Tags:
    db.system: redis
    db.redis.database_index: 0
    db.statement: FCALL
    server.address: localhost
    server.port: 6379
Activity.Events:
    Enqueued [14.11.2026 12:16:35 +00:00]
    Sent [14.11.2026 12:16:35 +00:00]
    ResponseReceived [14.11.2026 12:16:35 +00:00]

This is a shortened part of the console output for a reservation request sent to the API running with --Telemetry:Console true. The FCALL span is a child of the POST /inventory/{isbn}/reservations span (5e9527db79e128b8); the command took 1.9 ms. The Enqueued, Sent and ResponseReceived events separate time spent in the multiplexer's queue from time spent on the network and the server; exactly the information you are looking for when diagnosing timeouts. db.statement is only the command name; the arguments (customer data) are not written to the span. In production you would send to a collector with the OTLP exporter instead of the console; collectors, backends and dashboards are the topic of the upcoming Observability series. The hit ratio and memory metrics on the cache side are in the "Measuring and Testing Caches: Hit Ratio, Metrics, Memory Usage and Consistency Tests" post in the Caching series.

Integration Testing with Testcontainers

A Lua function and a stream consumer cannot be tested with mocks; their behaviour lives in Redis itself. tests/Bookstore.Integration.Tests starts a real redis:8.10.3 container on every test run, loads the same bookstore.lua the deployment loads, and runs the tests against it:

[assembly: AssemblyFixture(typeof(Bookstore.Integration.Tests.RedisFixture))]

// One real Redis 8 container for the whole test run, with the bookstore Functions library loaded
public sealed class RedisFixture : IAsyncLifetime
{
    private readonly RedisContainer container = new RedisBuilder("redis:8.10.3").Build();

    public IConnectionMultiplexer Redis { get; private set; } = null!;

    public async ValueTask InitializeAsync()
    {
        await container.StartAsync();
        Redis = await ConnectionMultiplexer.ConnectAsync(container.GetConnectionString());
        string library = await File.ReadAllTextAsync(Path.Combine(AppContext.BaseDirectory, "bookstore.lua"));
        await Redis.GetDatabase().ExecuteAsync("FUNCTION", "LOAD", "REPLACE", library);
    }

    public async ValueTask DisposeAsync()
    {
        await Redis.DisposeAsync();
        await container.DisposeAsync();
    }
}

There are three tests: exactly 100 reservations when 200 concurrent buyers go for 100 copies (a test of the race from post 08), the same order not being reserved twice, and the StockConsumer from post 10 moving a malformed event to the dead-letter stream after three deliveries. The last one starts the worker's own class:

StockConsumer consumer = new(fixture.Redis, config, NullLogger<StockConsumer>.Instance);
await consumer.StartAsync(TestContext.Current.CancellationToken);

RedisValue poisonId = await db.StreamAddAsync(OrderEventsConsumer.Stream,
    [new("type", "order-created"), new("orderId", 9999), new("isbn", "9780451524935"), new("quantity", "two")]);

dotnet test --project tests/Bookstore.Integration.Tests

Test run summary: Passed!
  total: 3
  failed: 0
  succeeded: 3
  skipped: 0
  duration: 2s 374ms

In the first version every test class started its own container, and since xUnit runs classes in parallel, two Redis instances were up at the same time; xUnit v3's assembly fixture shares one container across all tests. Because the tests share the same Redis, each test uses its own keys (different ISBNs). The Testcontainers life cycle and cleanup between tests are covered with SQL Server in the upcoming Integration and E2E Testing series.

In Production

A checklist: Redis only on a private network; the default user off; each application with its own ACL user (API, worker and report job separately) and a durable ACL source; passwords in a secret store with planned rotation (ACL allows several passwords per user, so you delete the old one after the new one is in use); TLS; patch releases of Redis and its modules tracked. For diagnosis, give the multiplexer a meaningful name, turn on latency-monitor-threshold, collect SLOWLOG periodically and keep the full timeout message in the logs. Raising the timeout usually hides the real problem (a slow command, thread pool starvation, a big value); first look at which side the message points to.

Common Pitfalls

Every application connecting as the passwordless default user: one leak exposes all the data.

Setting up ACLs without looking at ACL LOG: a missing permission (FT.TAGVALS) shows up in production as a 500.

Assuming users created with ACL SETUSER are durable: after a restart only default was left.

Giving the application user EVAL: every Lua vulnerability of 2025–2026 required a user able to run scripts.

Turning off certificate validation with TLS.

"Fixing" the problem by raising the timeout: the WORKER, qu and in values in the message tell the real cause.

Testing Redis-dependent code with mocks: atomicity and consumer group behaviour only show up on a real Redis.

A Small Experiment

★☆☆ Create a second ACL user named bookstore-worker: only the bookstore:orders:events* keys and the @stream, @transaction and @connection categories. Does the worker run with this user? What does ACL LOG say?

★★☆ Change the compose file so that Redis starts with --aclfile /data/users.acl (the file must exist beforehand), create the users, run ACL SAVE and restart the container. Are the users still there, and what is in the file?

★★★ Add a test for the catalog search from post 11 to the integration tests: build the index with EnsureIndexAsync, write three books and check the result of the @category:{science\-fiction} query. What happens when you run the same test with the valkey/valkey-bundle:9 image?

In the last post of the series we look at versions: what came from Redis 7 to 8.x, where it diverges from Valkey 9 with that compatibility table built from measurements, migrating StackExchange.Redis from 2 to 3, and a version upgrade plan.