Redis Persistence and Memory Management: RDB, AOF, fsync, Fork Cost, maxmemory and Big/Hot Keys

Version note: The examples in this post were run with Redis 8.10.3 (redis:8), Valkey 9.1.3, .NET 10 and StackExchange.Redis 3.4.35.

Guess First

Since post 10, Bookstore, the online bookstore used as the shared example across this site's series, keeps its order events in a stream and its order numbers in a counter. For this post lab/persistence.yaml has two throwaway Redis instances: rdb runs with the official image's default settings, aof with --appendonly yes --appendfsync everysec. We write 5,000 orders to both, kill the containers right away with SIGKILL and start them again:

docker compose -f lab/persistence.yaml up -d
$env:BOOKSTORE_REDIS="localhost:6390"; dotnet run --project src/Bookstore.RedisLab -- write-orders 5000
$env:BOOKSTORE_REDIS="localhost:6391"; dotnet run --project src/Bookstore.RedisLab -- write-orders 5000
docker compose -f lab/persistence.yaml kill rdb aof
docker compose -f lab/persistence.yaml start rdb aof
$env:BOOKSTORE_REDIS="localhost:6390"; dotnet run --project src/Bookstore.RedisLab -- check-orders
$env:BOOKSTORE_REDIS="localhost:6391"; dotnet run --project src/Bookstore.RedisLab -- check-orders

How many orders are left in the two restarted instances? The "fsync every second" in aof can lose up to one second of data; did it lose any in this measurement?

wrote 5,000 orders, bookstore:order:id-seq = 5000, XLEN = 5000
wrote 5,000 orders, bookstore:order:id-seq = 5000, XLEN = 5000
 Container bookstore-persistence-rdb-1 Killed
 Container bookstore-persistence-aof-1 Killed
 Container bookstore-persistence-rdb-1 Started
 Container bookstore-persistence-aof-1 Started
bookstore:order:id-seq = (nil), XLEN = 0
bookstore:order:id-seq = 5000, XLEN = 5000

The Redis with default settings lost every order: no counter, empty stream. In the one with AOF they are all there, including those written in the last second. This post explains why: RDB and AOF, the memory cost of fork, switching the lab to AOF without losing data, Redis 8.10's BACKUP command, what maxmemory and eviction policies mean outside caching, the container's memory limit, and big and hot keys.

RDB: Snapshots, Fork and Copy-on-Write

The official image starts without a config file and uses Redis's defaults: save 3600 1 300 100 60 10000 (take a snapshot if there was at least 1 change in an hour, at least 100 in 5 minutes or at least 10,000 in 1 minute), AOF off. The experiment took a few seconds; no rule fired, and the last snapshot was of the empty database. That is exactly what the documentation says about RDB: a Redis that does not shut down cleanly loses everything since the last snapshot, and snapshots are usually minutes apart.

"Shut down cleanly" is an important detail. When I repeated the experiment with docker compose stop instead of kill, the 5,000 orders were kept; the log showed Received SIGTERM scheduling shutdown followed by Saving the final RDB snapshot before exiting. Because planned restarts lose nothing, the risk of RDB can go unnoticed; a crash, an OOMKilled container or a host going down has the same result as SIGKILL.

The snapshot is not written by Redis itself but by a child process it creates with fork. The child shares the parent's memory without copying it; when the parent changes a page, the operating system makes a copy of that page (copy-on-write). So the more you write during a snapshot, the more extra memory you need. I measured it with 2 million session keys (547 MB):

$env:BOOKSTORE_REDIS="localhost:6390"; dotnet run --project src/Bookstore.RedisLab -- fork-cost

2,000,000 keys, used_memory 547.26M
BGSAVE with 0% of keys rewritten: latest_fork_usec 3213, rdb_last_cow_size 1 MB, took 836 ms
BGSAVE with 50% of keys rewritten: latest_fork_usec 3473, rdb_last_cow_size 163 MB, took 1,707 ms

The fork itself took 3.2 ms; during that time Redis answers no command, and it grows with the data. With no writes during the snapshot the extra memory was 1 MB; while rewriting half of the keys it was 163 MB. That is why you should not give Redis all of the machine's or container's memory: snapshots and AOF rewrites need extra memory depending on the write load.

AOF: appendfsync and What "At Most 1 Second" Means

AOF appends every write command to the end of a file and rebuilds the data at startup by running those commands again. appendfsync decides how often data is forced to disk: always on every write (safest, slowest), everysec once a second (the default), no leaves it to the operating system. According to the documentation everysec can lose the last second of data in a disaster. In Guess First, however, nothing was lost. There is no contradiction: Redis hands every command to the operating system right away with write; fsync only flushes the operating system's in-memory buffer to disk. If the Redis process dies but the operating system stays up, the buffered data still reaches the disk. The one-second loss happens when the machine itself crashes or loses power; you cannot reproduce it by killing a container.

Since Redis 7.0 the AOF is not a single file but a directory (multi-part AOF): a base file, one or more incr files and a manifest saying which ones are current. After the experiment the aof container had an incr file of 958,086 bytes; BGREWRITEAOF produced a new base from the current state of the data, and the directory looked like this:

-rw------- 1 redis redis 29983 Nov 14 11:53 appendonly.aof.2.base.rdb
-rw------- 1 redis redis     0 Nov 14 11:53 appendonly.aof.2.incr.aof
-rw------- 1 redis redis   107 Nov 14 11:53 appendonly.aof.manifest

The base has a .rdb extension: with the default aof-use-rdb-preamble yes the base is written in RDB format, while the incr files are a command log. 10,000 commands shrank to a 30 KB snapshot. At startup Redis loads the base quickly and replays only the commands after it. The Functions library we loaded in post 08 is stored with these files too; the script cache is not.

Switching the Lab to AOF: the Wrong Way and the Right Way

The stream from post 10 and the stock reservations from post 08 are data that must not be lost; let's run the lab's Redis with AOF as well. The first idea is to add a command line to lab/compose.yaml and recreate the container:

  redis:
    image: redis:8
    # post 12: AOF next to RDB, a memory limit, and noeviction (streams and queues must never be evicted)
    command: ["redis-server", "--appendonly", "yes", "--maxmemory", "512mb", "--maxmemory-policy", "noeviction"]

With the eight catalog documents from post 11 loaded, I tried it in two ways:

### wrong
DBSIZE before: 8
 Container bookstore-redis-lab-redis-1 Recreated
DBSIZE after: 0
### right
DBSIZE before: 8
OK
aof_rewrite_in_progress:0
aof_last_bgrewrite_status:ok
 Container bookstore-redis-lab-redis-1 Recreated
DBSIZE after: 8

The wrong way (only changing compose and running docker compose up -d redis) opened an empty database. With AOF on, Redis does not look at dump.rdb at startup; since there was no AOF directory it created an empty base (the log says Creating AOF base file appendonly.aof.1.base.rdb on server start). The old container had written dump.rdb while shutting down, but the new container's first clean shutdown will overwrite it with empty data. The right way is the order in the documentation: first CONFIG SET appendonly yes on the running server, see aof_rewrite_in_progress:0 and aof_last_bgrewrite_status:ok in INFO persistence, then change compose and recreate the container.

Backup and Restore: the BACKUP Command

An RDB file can be copied from a running server, because it is written under a temporary name and renamed atomically when complete. Copying the AOF directory, on the other hand, can give an inconsistent backup if it coincides with a rewrite; the documentation's fix is to turn rewrites off temporarily. Redis 8.10 added the BACKUP command family for this; it is a new feature, and I only tried it on a single node:

docker compose -f lab/compose.yaml exec redis redis-cli

127.0.0.1:6379> BACKUP START
OK
127.0.0.1:6379> BACKUP LIST
1) "/data/backupdir/appendonly.aof.2.base.rdb"
127.0.0.1:6379> XADD bookstore:orders:events * type order-created isbn 9780141439518 quantity 1
"1794657412351-0"
127.0.0.1:6379> BACKUP SEAL
OK
127.0.0.1:6379> BACKUP LIST
1) "/data/backupdir/appendonly.aof.2.base.rdb"
2) "/data/backupdir/appendonly.aof.2.incr.aof"
3) "/data/backupdir/appendonly.aof.manifest"

START produces a base snapshot while writes keep being appended to the incr file; SEAL freezes the backup at that point. The backup also contains the XADD sent after START. The files have to be copied between SEAL and CLEANUP. When I sent the commands to redis-cli back to back from a file, SEAL failed with ERR No backup ready to seal (must be in the incrementing state): START produces the base snapshot in the background, and the backup cannot be sealed until state in BACKUP STATUS is incrementing. Wait for that state when you automate it. To restore, I started a new Redis with the preload-file setting:

docker compose -f lab/compose.yaml cp redis:/data/backupdir/. ./backup/
docker compose -f lab/compose.yaml exec redis redis-cli BACKUP CLEANUP
docker run -d --name bookstore-restore -v ./backup:/restore redis:8 redis-server --preload-file aof:/restore/appendonly.aof.manifest

The log showed DB loaded from base file, DB loaded from incr file and DB pre-loaded from append only file, and every event in the stream was there. Until you have actually restored and checked a backup, you do not know you have one; make this test a periodic job. Valkey 9.1.3 has no BACKUP (ERR unknown command).

maxmemory and Eviction: noeviction Outside Caching

maxmemory is the memory limit Redis will use; maxmemory-policy decides what happens at the limit. volatile-* policies consider only keys with a TTL, allkeys-* policies all keys. Which policy to choose for a cache is covered in the "IDistributedCache and Running Redis as a Cache: Serialization, Key Design, TTL and Eviction Policy" post in the Caching series; here we look at what happens when cache keys and a stream live in the same Redis. I set the limit to 20 MB and wrote first a 1,000-event order stream, then 1 KB cache keys:

$env:BOOKSTORE_REDIS="localhost:6390"; dotnet run --project src/Bookstore.RedisLab -- eviction

allkeys-lru: cache keys written 50,000, evicted_keys 32166, stream exists False
noeviction: cache keys written 17,862, evicted_keys 0, stream exists True
  OOM command not allowed when used memory > 'maxmemory'.
  GET still works: True, DEL works: True

allkeys-lru never returned an error to the cache writes, but it achieved that by silently deleting the order stream: written once and never read again, the stream was the least recently used key. The application saw no error; the workers would simply have thought no new events were arriving. With noeviction, write commands got an OOM error at the limit, while reads and deletes kept working. In a Redis that holds data which must not be lost, such as streams, queues and stock counters, the right policy is noeviction; the error is visible, can be monitored and can raise an alert. Putting the cache in a separate Redis instance resolves the conflict between the two needs. In Redis 8.10.3 the default policy is already noeviction, but managed services may come with a different default; check yours with CONFIG GET maxmemory-policy. The allkeys-lrm/volatile-lrm policies added in Redis 8.6, which evict the key least recently modified, do not protect a stream either, since they prefer keys that are only read; Valkey 9.1.3 does not accept these two policies.

Container Limits and OOMKilled

Without maxmemory there is no limit; Redis grows up to the container's memory limit. The limited service in lab/persistence.yaml is limited to 128 MB:

docker compose -f lab/persistence.yaml --profile oom up -d limited
$env:BOOKSTORE_REDIS="localhost:6392"; dotnet run --project src/Bookstore.RedisLab -- fill 300
docker inspect -f "State={{.State.Status}} OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}" bookstore-persistence-limited-1

32 MB written, used_memory 42.89M
64 MB written, used_memory 82.89M
96 MB written, used_memory 122.90M
after 120 MB: RedisConnectionException: SocketClosed on localhost:6392/Interactive
State=exited OOMKilled=true ExitCode=137

Redis returned no error; the process was killed, the client only saw a dropped connection, and the restarted container came up empty. When I tried again on the same container with CONFIG SET maxmemory 96mb, an OOM command not allowed error came after 74 MB and the container stayed up. used_memory is about 1.25 times the data written; add fork's copy-on-write needs, client buffers and fragmentation, and maxmemory has to stay clearly below the container limit. The Kubernetes version of the same situation is in the "Resource Requests and Limits: QoS, OOMKilled, CPU Throttling and the .NET GC in Containers" post of the Kubernetes series.

Big and Hot Keys, UNLINK and Fragmentation

The size of a key determines how long a single command takes. redis-cli --bigkeys finds the largest keys by element count, --memkeys by memory; both use SCAN, so they can be run in production. On data with a 200,000-member visitor set next to 100,000 sessions, --memkeys reported the set at 9,634,725 bytes and the 50,000-event stream at 1,385,637 bytes. Deleting such a key is expensive too:

$env:BOOKSTORE_REDIS="localhost:6390"; dotnet run --project src/Bookstore.RedisLab -- big-key-delete

DEL    client   33.3 ms, slowlog 30.8 ms, max PING meanwhile 31.0 ms
UNLINK client    0.5 ms, slowlog -, max PING meanwhile 1.7 ms

DEL removed the 1-million-member set in 30.8 ms, and another client's PING waited all that time. UNLINK took the key out of the keyspace at once and freed the memory on a background thread. In Redis 8.10.3 lazyfree-lazy-user-del defaults to no, so DEL is still synchronous, and StackExchange.Redis's KeyDeleteAsync sends DEL. Use UNLINK for keys that may be big, or set this option to yes.

A hot key is one that is accessed very often: in a Cluster (post 13) it puts all the load on a single node. Redis 8.6 added HOTKEYS for this. When I started tracking with HOTKEYS START METRICS 2 CPU NET COUNT 3 and sent 200,000 HGETs to the price hash with redis-benchmark, HOTKEYS GET put bookstore:prices first by CPU time with 15,417 µs, ahead of a session key with 1,283 µs. Valkey 9.1.3 has no HOTKEYS; there you use redis-cli --hotkeys, which requires an LFU policy.

Fragmentation is the difference between the memory Redis uses (used_memory) and the memory the process got from the operating system (used_memory_rss). After writing 600,000 small keys and deleting half of them, mem_fragmentation_ratio was 2.27: 134 MB for 59 MB of data. After CONFIG SET activedefrag yes I waited 15 seconds and nothing changed. The reason is the default of active-defrag-ignore-bytes: defragmentation does not start until fragmentation exceeds 100 MB, and the difference here was 57 MB. With the threshold lowered to 10 MB, RSS dropped to 90 MB and the ratio to 1.53 within 15 seconds.

In Production

A starting point for a Redis that holds durable data: AOF on (everysec), RDB snapshots on as well (both together, as the documentation recommends), maxmemory clearly below the container's or machine's memory, and noeviction. Metrics to watch: rdb_last_bgsave_status and aof_last_write_status in INFO persistence (if they are not ok there is a disk problem; when the RDB cannot be written, Redis rejects writes by default), latest_fork_usec and evicted_keys in INFO stats (the latter should be 0 on a Redis with durable data), the used_memory/maxmemory ratio and mem_fragmentation_ratio in INFO memory. Move backups to another machine and test restores regularly. With managed services (Azure Managed Redis, ElastiCache) persistence and backup are service settings, but the noeviction and memory headroom decisions are still yours.

Common Pitfalls

Using the official image with default settings for durable data: after SIGKILL all 5,000 orders were gone.

Turning on AOF just by changing the config and restarting: 8 keys went down to 0.

Keeping a cache and a stream in the same Redis with allkeys-lru: the stream was deleted without any error.

Giving the container a memory limit but no maxmemory: instead of an error, Redis died with OOMKilled and came back empty.

Deleting big keys with DEL: every client waited for 30 ms.

Expecting activedefrag yes to start right away: it does not start before the default 100 MB threshold is exceeded.

A Small Experiment

★☆☆ Run write-orders 5000 on the aof service with appendfsync always and compare the time with everysec. How big is the difference?

★★☆ Change fork-cost so that it rewrites 100% of the keys. Does rdb_last_cow_size get close to the data size? Based on that, how much memory would you give a 547 MB Redis?

★★★ Write a small .NET console application that runs BACKUP START, SEAL and CLEANUP, copies the files, opens the copy in a separate container with preload-file and compares XLEN.

In the next post we go beyond a single Redis: replication and WAIT, writes that can be lost in a failover, Sentinel, and in Cluster hash slots, hash tags and CROSSSLOT. We will also see why the stock keys from post 08 have curly braces in them.