Symmetric Encryption and AEAD: AES-GCM, ChaCha20-Poly1305, Nonce Management and Envelope Encryption

Version note: The code in this post was built and run with .NET 10, Entity Framework Core 10.0.15 (SQLite provider) and xUnit v3 4.0; the outputs were taken on Ubuntu 24.04 (OpenSSL 3.0).

Guess First

Phone numbers sit in plaintext in Bookstore's customer table, and ahead of a data protection audit the decision came down: "let's encrypt this field". The first draft goes like this: Aes.Create(), the key and the IV come from configuration, the mode is left at its default, and the output is written to the column as Base64. The code works, and there are no readable numbers in the database any more.

Now two questions: what happens to the column values of two customers who share the same phone number? If someone with write access to the database copies Ada's encrypted phone into Alan's row, does the application notice? The answer to the first is "the same value", because with a fixed IV the same input always gives the same output; the encrypted column tells you which customers share a number. The answer to the second is "no", because that mode only provides confidentiality, not integrity. In this post we build the structure that closes both gaps: authenticated encryption with AES-GCM, a fresh nonce for every encryption, associated data that binds the encrypted value to its context, a format ready for rotation, envelope encryption, and an Entity Framework Core value converter that brings all of it to the Customer.Phone field.

A Short Look at Block Cipher Modes

AES on its own encrypts just one 16-byte block. Encrypting longer data needs a mode of operation that says how blocks are chained. Three classic modes and their problems:

ECB encrypts every block separately with the same key; the same input block always gives the same output block. CBC fixes that with a random IV and chaining, but provides no integrity: a change in the ciphertext can carry through to the decrypted text unnoticed, and different responses to padding errors opened the door to serious attacks over the years. CTR encrypts a counter and XORs the result with the data, turning the block cipher into a stream cipher; it's fast, but again has no integrity. Let's encrypt four identical blocks with three modes:

// Four identical 16-byte blocks
byte[] plaintext = Encoding.ASCII.GetBytes(string.Concat(Enumerable.Repeat("ISBN 9780441013", 4).Select(s => s + "|")));
byte[] key = RandomNumberGenerator.GetBytes(32);
using Aes aes = Aes.Create();
aes.Key = key;

Print("ECB", aes.EncryptEcb(plaintext, PaddingMode.None));
Print("CBC", aes.EncryptCbc(plaintext, RandomNumberGenerator.GetBytes(16), PaddingMode.None));

using AesGcm gcm = new(key, tagSizeInBytes: 16);
byte[] ciphertext = new byte[plaintext.Length];
gcm.Encrypt(RandomNumberGenerator.GetBytes(12), plaintext, ciphertext, new byte[16]);
Print("GCM", ciphertext);

$ dotnet run --project src/Bookstore.CryptoLab -- modes
ECB: 9d5d97e8 9d5d97e8 9d5d97e8 9d5d97e8
CBC: 7abf9aaf 00454848 1248ca26 c09bd38d
GCM: c5438ced ad92ff5d b1496a62 e193971f

With ECB the four blocks are identical; repeated values in a table or shapes in an image stay visible in encrypted form. In the CBC and GCM output the blocks differ from each other. But what sets GCM apart doesn't show in this output: GCM, which adds an authentication tag to CTR encryption, is an AEAD mode. It produces a tag when encrypting, checks the tag when decrypting, and if even a single bit changed it throws an exception without returning anything. That's the right default in new code today; you'd use CBC or CTR only when compatibility with an old format requires it, and then together with a separate HMAC.

AEAD: AesGcm and ChaCha20Poly1305

.NET has two AEAD classes and their APIs have the same shape: nonce, plaintext, output buffer, tag buffer and optional associated data. Since .NET 8 the AesGcm constructor asks for the tag size explicitly; the old constructor gives a compiler warning:

warning SYSLIB0053: 'AesGcm.AesGcm(byte[])' is obsolete: 'AesGcm should indicate the required tag size for encryption and decryption. Use a constructor that accepts the tag size.'

Use 16 bytes for the tag size. According to the platform table on Microsoft Learn, Windows and Linux accept tags between 12 and 16 bytes and Apple platforms only 16; a truncated tag has both a lower security margin and no portability.

ChaCha20Poly1305 does the same job with the ChaCha20 stream cipher and the Poly1305 MAC instead of AES. The key is always 32 bytes, the nonce 12 bytes and the tag 16 bytes; there's nothing to configure:

using ChaCha20Poly1305 chacha = new(key);
chacha.Encrypt(nonce, plaintext, ciphertext, tag, "Customer.Phone"u8);

byte[] roundTrip = new byte[ciphertext.Length];
chacha.Decrypt(nonce, ciphertext, tag, roundTrip, "Customer.Phone"u8);

$ dotnet run --project src/Bookstore.CryptoLab -- chacha
ChaCha20-Poly1305: 13 + 16 tag bytes, round trip: +905550100001

Which one to pick? Modern x64 servers have hardware instructions for AES, so AES-GCM is fast and available everywhere. ChaCha20-Poly1305 is faster on devices without AES hardware support and easy to run in constant time in software; but on Windows it isn't available before build 20142. The nonce rules are equally strict for both. Bookstore uses AES-GCM on the server side; that's also why AES-GCM is in the CryptoRequirements list from the first post of the series.

Nonces: Why Single-Use and How Many Messages

GCM's most critical rule: the same nonce is never used twice with the same key. If it is, two problems appear at once. Because GCM uses CTR inside, the same nonce produces the same keystream for two messages; someone who puts the two ciphertexts side by side gets the XOR of the two plaintexts. Worse, two tags produced with the same nonce leak information about the authentication key, and that opens the way to forging tags. So a single repeat breaks both confidentiality and integrity.

How do we generate the nonce? There are two ways. A counter: a value incremented on every encryption; it never repeats, but the counter has to be persistent and kept in one place, which is hard for a web application running on several instances. Random: 12 bytes from RandomNumberGenerator on every encryption. No state needed, but a random choice in a 96-bit space also has a chance of collision. In Special Publication 800-38D, which defines GCM, NIST limits the number of encryptions with one key to 2^32 when random nonces are used. That's about 4.3 billion encryptions; a distant limit for Bookstore's phone field, but reachable for high-volume log or event encryption. The fix is to rotate the key well before the limit; the keyId in our format is there for exactly that.

Associated Data and a Format Ready for Rotation

Associated data is extra bytes that aren't encrypted but are included in the tag. If you don't supply the same bytes when decrypting, the tag doesn't match. We use it to bind the encrypted value to its context: if we pass the customer id as associated data for Ada's phone, that value can't be decrypted once it's copied into Alan's row. Bookstore, the online bookstore used as the shared example across this site's series, keeps the encryption code in the Fields folder of the Bookstore.Crypto library:

// Layout: [version][keyId][nonce: 12][ciphertext: n][tag: 16]
public sealed class FieldEncryptor(FieldKeyRingProvider keys)
{
    public const byte FormatVersion = 1;
    private const int HeaderSize = 2;
    private const int NonceSize = 12;
    private const int TagSize = 16;

    public FieldEncryptor(FieldKeyRing keyRing) : this(new FieldKeyRingProvider(keyRing))
    {
    }

    public byte[] Encrypt(ReadOnlySpan<byte> plaintext, ReadOnlySpan<byte> associatedData)
    {
        FieldKeyRing keyRing = keys.Current;
        byte keyId = keyRing.ActiveKeyId;
        byte[] payload = new byte[HeaderSize + NonceSize + plaintext.Length + TagSize];
        payload[0] = FormatVersion;
        payload[1] = keyId;

        Span<byte> nonce = payload.AsSpan(HeaderSize, NonceSize);
        Span<byte> ciphertext = payload.AsSpan(HeaderSize + NonceSize, plaintext.Length);
        Span<byte> tag = payload.AsSpan(HeaderSize + NonceSize + plaintext.Length, TagSize);
        RandomNumberGenerator.Fill(nonce);

        using AesGcm aes = new(keyRing.Get(keyId), TagSize);
        aes.Encrypt(nonce, plaintext, ciphertext, tag, BuildAad(payload.AsSpan(0, HeaderSize), associatedData));
        return payload;
    }

    public byte[] Decrypt(ReadOnlySpan<byte> payload, ReadOnlySpan<byte> associatedData)
    {
        if (payload.Length < HeaderSize + NonceSize + TagSize || payload[0] != FormatVersion)
        {
            throw new CryptographicException("Unknown field encryption format.");
        }

        int length = payload.Length - HeaderSize - NonceSize - TagSize;
        byte[] plaintext = new byte[length];
        using AesGcm aes = new(keys.Current.Get(payload[1]), TagSize);
        aes.Decrypt(
            payload.Slice(HeaderSize, NonceSize),
            payload.Slice(HeaderSize + NonceSize, length),
            payload.Slice(HeaderSize + NonceSize + length, TagSize),
            plaintext,
            BuildAad(payload[..HeaderSize], associatedData));
        return plaintext;
    }

    // The header is authenticated too: changing version or keyId breaks the tag
    private static byte[] BuildAad(ReadOnlySpan<byte> header, ReadOnlySpan<byte> associatedData) => [.. header, .. associatedData];
}

Every byte of the format has a job. The version byte lets us recognize old records if we ever change the algorithm or the layout. The keyId says which key a record was encrypted with; after rotation, old records keep decrypting with the old key. The nonce is stored with the encrypted value because it isn't secret, it only has to be single-use. Because we add the header to the associated data too, if someone changes the keyId byte to get the record decrypted with another key, the tag doesn't match. The cost is small: a 13-byte phone number becomes 43 bytes.

$ dotnet run --project src/Bookstore.CryptoLab -- aead
plaintext 13 bytes -> payload 43 bytes (2 header + 12 nonce + 13 ciphertext + 16 tag)
same input again, different payload: True
decrypted for Ada: +905550100001
moved to Alan's row: AuthenticationTagMismatchException
one ciphertext bit flipped: AuthenticationTagMismatchException
keyId byte changed: KeyNotFoundException: Field key 2 is not in the key ring.

On the last line we stopped before verification even started, because key 2 isn't in the key ring. The case where both keys exist is in the tests: turning the keyId from 1 into 2, or changing the nonce, the ciphertext or the tag, all give AuthenticationTagMismatchException:

[Theory]
[InlineData(1)]   // keyId in the header
[InlineData(5)]   // nonce
[InlineData(20)]  // ciphertext
[InlineData(42)]  // tag
public void Any_changed_byte_is_rejected(int index)
{
    byte[] payload = Encryptor(1).Encrypt(Phone, Ada);
    payload[index] ^= index == 1 ? (byte)0x03 : (byte)0x01;   // keyId 1 -> 2, a key that exists
    Assert.Throws<AuthenticationTagMismatchException>(() => Encryptor(1).Decrypt(payload, Ada));
}

Envelope Encryption: DEK, KEK and AES Key Wrap

So far we derived the key from the master key with HKDF, as in the previous post. That works, but it has a limit: rotating the master key means re-encrypting every record encrypted with every key derived from it. On a table with millions of rows that's a job that takes hours. Envelope encryption solves this with two layers. A DEK, the data encryption key, encrypts the data. The DEK itself is wrapped with a KEK, the key encryption key, and stored in the database. The KEK lives inside Key Vault or an HSM and never leaves it; at startup the application sends the wrapped DEK and gets the unwrapped one back.

To wrap the DEK we use AES-KWP, AES Key Wrap with Padding, which arrived in .NET 10 and is defined in RFC 5649. Why not AES-GCM? Key wrap is a deterministic construction designed for wrapping a short random key; it needs no nonce, so it leaves no room for nonce management mistakes. Its output is only 8 bytes longer than its input:

// A data key (DEK) as stored: wrapped with a key encryption key (KEK)
public sealed record WrappedDataKey(byte KeyId, string KekId, byte[] WrappedKey);

public static class EnvelopeKeys
{
    public static WrappedDataKey CreateDataKey(byte keyId, string kekId, ReadOnlySpan<byte> kek)
    {
        byte[] dek = RandomNumberGenerator.GetBytes(32);
        try
        {
            return new WrappedDataKey(keyId, kekId, Wrap(dek, kek));
        }
        finally
        {
            CryptographicOperations.ZeroMemory(dek);
        }
    }

    public static byte[] Unwrap(WrappedDataKey dataKey, ReadOnlySpan<byte> kek)
    {
        using Aes aes = Aes.Create();
        aes.SetKey(kek);
        return aes.DecryptKeyWrapPadded(dataKey.WrappedKey);
    }

    // KEK rotation: the data stays as it is, only the 40-byte wrapped keys change
    public static WrappedDataKey Rewrap(WrappedDataKey dataKey, ReadOnlySpan<byte> oldKek, string newKekId, ReadOnlySpan<byte> newKek)
    {
        byte[] dek = Unwrap(dataKey, oldKek);
        try
        {
            return dataKey with { KekId = newKekId, WrappedKey = Wrap(dek, newKek) };
        }
        finally
        {
            CryptographicOperations.ZeroMemory(dek);
        }
    }

    private static byte[] Wrap(ReadOnlySpan<byte> dek, ReadOnlySpan<byte> kek)
    {
        using Aes aes = Aes.Create();
        aes.SetKey(kek);
        return aes.EncryptKeyWrapPadded(dek);
    }
}

$ dotnet run --project src/Bookstore.CryptoLab -- envelope
DEK 2 wrapped by kek-2026: 40 bytes 81432ce3107662e6...
DEK 2 wrapped by kek-2027: 40 bytes 490024644793d6ff...
same DEK after KEK rotation: True
unwrap with the retired KEK: CryptographicException

In the lab we derive the KEK from the master key as a stand-in for Key Vault; in production you never touch the KEK, you call the wrap/unwrap operations of Key Vault or the HSM and the KEK never leaves the service. Where the KEK lives, who can reach it and how it's rotated is the subject of the Configuration and Secret Management series. What matters here is the difference between the two rotations: KEK rotation only re-wraps the 40-byte wrapped keys, and the data doesn't change at all. DEK rotation makes a new keyId active; old records keep being read with the old DEK and are moved to the new DEK in the background.

Encrypting Customer.Phone with an EF Core Value Converter

Now we tie it all to Bookstore's customer model. The new Bookstore.Customers project has the Customer entity and BookstoreContext; the database in this post is SQLite. Phone is plaintext in memory and encrypted in the database; a value converter does the conversion:

// Associated data = column name: binds the ciphertext to the column, not to the row
public sealed class EncryptedStringConverter(FieldEncryptor encryptor, string column)
    : ValueConverter<string, string>(
        plaintext => encryptor.EncryptString(plaintext, column),
        payload => encryptor.DecryptString(payload, column));

protected override void OnModelCreating(ModelBuilder modelBuilder)
{
    modelBuilder.Entity<Customer>(customer =>
    {
        customer.Property(c => c.Name).HasMaxLength(200);
        customer.Property(c => c.Phone)
            .HasConversion(new EncryptedStringConverter(encryptor, "Customer.Phone"));
        customer.Property(c => c.PhoneIndex).HasMaxLength(24);
        customer.HasIndex(c => c.PhoneIndex);
    });
}

We have to be honest here: a value converter sees only the value, not the entity. So we can't pass the customer id as associated data; the best context we can give is the column name. That stops the encrypted value from being moved to another column, but not to another row in the same column. Rather than hide it, we documented it with a test:

[Fact]
public async Task Converter_binds_to_the_column_not_the_row()
{
    (Customer ada, Customer alan) = await SeedAsync();
    await using BookstoreContext db = NewContext();

    // Someone with write access to the database copies Alan's ciphertext into Ada's row
    await db.Database.ExecuteSqlAsync(
        $"UPDATE Customers SET Phone = (SELECT Phone FROM Customers WHERE Id = {alan.Id}) WHERE Id = {ada.Id}", Ct);

    Customer reloaded = await db.Customers.SingleAsync(c => c.Id == ada.Id, Ct);
    Assert.Equal("+905550100002", reloaded.Phone);   // decrypts without error: the known limit
}

Is that acceptable? For the phone field, yes: the attack needs write access to the database, and someone with that access can already do much more. For fields where row binding really matters, a payment instruction for example, you do the encryption in domain code instead of a converter and pass the entity id as associated data; the lab's aead command showed how that behaves. You're making a conscious choice between transparency and context.

The lab command adds two customers, reads the columns as stored in the database with SQL, searches by phone, then rotates the DEK and re-encrypts the old records:

$ dotnet run --project src/Bookstore.CryptoLab -- customers
Ada   Phone=AQHsFgs6FzT5BJffgpwL4L59... (60 chars) PhoneIndex=YFjLBVUw3HSlSwExbaQb+A==
Alan  Phone=AQFHf3OFJ4sWJoYZ7D5C8WVm... (60 chars) PhoneIndex=3uGlghWi8bXO3EiyJK2Bqg==
lookup by phone -> Ada +905550100001
after adding Grace: Ada=key 1, Alan=key 1, Grace=key 2
after re-encryption: Ada=key 2, Alan=key 2, Grace=key 2

It's no accident that every Base64 value starts with AQ: the first two bytes are the version (1) and the keyId (1). For rotation we don't create a new encryptor; we swap the same encryptor's key ring with FieldKeyRingProvider.Replace; I'll explain why with a real bug under Common Pitfalls. For re-encryption, marking the Phone property as modified and saving is enough; the converter writes the value again with the active key. On a real table you'd write a background job that does this in small batches filtered by keyId, not in one go.

Searchability: Blind Index and Always Encrypted

The price of encryption with random nonces is losing searchability: the same phone gives a different encrypted value every time, and WHERE Phone = @phone no longer works. Customer service still wants to find a calling customer by phone. The fix is to put a blind index next to the encrypted column: a keyed HMAC of the normalized value. We derive its key with KeyPurposes.CustomerPhoneBlindIndex from the previous post, so it's independent of the encryption key:

// Keyed, deterministic lookup value for an encrypted field (equality search only)
public sealed class BlindIndex(byte[] key, int lengthInBytes = 16)
{
    public string Compute(string normalizedValue)
    {
        byte[] mac = HMACSHA256.HashData(key, Encoding.UTF8.GetBytes(normalizedValue));
        return Convert.ToBase64String(mac, 0, lengthInBytes);
    }
}

This design has three consequences. First, normalization is critical: the first version of the lab turned a search in the 0090 555 010 0001 form into +00905550100001, and the search came back empty with "Sequence contains no elements". The same normalization function must be used when saving and when searching. Second, a blind index only gives equality search; a query like "starting with 555" isn't possible. Third, a blind index leaks something: customers with the same phone have the same index value. The difference from an unkeyed hash is that an attacker can't recover the value by trying every possible number; without the HMAC key that's not possible. Truncating the index to 16 bytes is deliberate too: filtering out a few false matches in the application leaves less information behind than storing a full-length value.

If you use SQL Server, the same need has a database-side solution: Always Encrypted. The client driver (Microsoft.Data.SqlClient) does the encryption and decryption; the column encryption key that encrypts the column lives in the database, and the column master key that wraps it lives in Key Vault or a certificate store. That's the same as our envelope structure. Deterministic mode allows equality search, but identical values give identical encrypted values; randomized mode allows no search. The version with secure enclaves supports richer queries. Always Encrypted's advantage is that application code barely changes; its drawbacks are being tied to SQL Server and SqlClient, limits on supported queries and data types, and the fact that, because encryption has to happen outside the database, encrypting an existing column or rotating a key means moving the data to the application and writing it back. Application-level encryption is independent of the database and its format is entirely under your control.

Production Notes

Track the number of keys and how they're used. Even if you're far from the 2^32 encryption limit per key with random nonces, rotating keys regularly limits how much data is affected if a key leaks. A simple query or metric showing how many records are on which keyId is the only way to see that a rotation has really finished; remove an old DEK from the key ring only when that number is zero.

Build and verify the key ring at application startup: check that every DEK unwraps and that the active keyId is in the ring. If Key Vault can't be reached, the application shouldn't start; an application that stays up while unable to read its encrypted columns means the error surfaces on the first customer request.

Performance: AES-GCM is very fast with AES hardware instructions; for small fields such as a phone number almost all the cost is creating the AesGcm instance. On a very hot path you can cache an instance per key, but AesGcm isn't thread safe; keep it per thread or in a pool. Don't optimize without measuring.

Watch logs and error messages: when logging an AuthenticationTagMismatchException, don't write the payload to the log. Which record, which keyId and which column is enough; a tag failure is a sign of either data corruption or tampering, and both call for an alert.

Common Pitfalls

The first is forgetting EF Core's model cache. The first version of the lab did rotation by creating a new FieldEncryptor with active key 2 and a new context. The output was:

after adding Grace: Ada=key 1, Alan=key 1, Grace=key 1
after re-encryption: Ada=key 1, Alan=key 1, Grace=key 1

Grace hadn't been encrypted with the new key. EF Core builds the model once per context type and caches it; the converter created in OnModelCreating holds the first context's encryptor for the life of the process. The encryptor you pass to later contexts means nothing to the converter. The fix is to make the encryptor a singleton and rotate inside it by swapping the key ring; that's what FieldKeyRingProvider is for.

The second is keeping the nonce fixed or deriving it from the key. The idea "let's derive the nonce from each record's id" means the same nonce is reused when the same record is updated. The nonce must be generated fresh on every encryption.

The third is using decrypted data before the tag is verified. .NET's AEAD APIs already prevent that; if the tag doesn't match, the output buffer is cleared and an exception is thrown. But if you build your own "encrypt, then HMAC" with CBC, verify the HMAC before decrypting.

The fourth is the wrong length for the encrypted column. Because of Base64 and the 30-byte overhead, a 13-character phone becomes 60 characters. Define the column by the plaintext length and on SQL Server you'll get truncated records or errors.

The fifth is making the blind index key the same as the encryption key. The rule from the previous post: a separate key for every purpose.

A Small Experiment

★☆☆ Reduce the BlindIndex length from 16 bytes to 4 bytes and generate the index for 100,000 random phone numbers. How many collisions did you get? Use that number to discuss the balance between length and leaked information.

★★☆ Write a row-bound version: add SetPhone(string phone, FieldEncryptor encryptor, BlindIndex index) to Customer and use Id as associated data; remove the converter. Rewrite the Converter_binds_to_the_column_not_the_row test for this version and show that it now expects AuthenticationTagMismatchException.

★★★ Write the re-encryption work as a background job: it should find records whose keyId isn't the active key in batches of 500, re-encrypt them and log progress for each batch. What do you need to add to filter by keyId: reading it from the payload, or a separate column?

In the next post we move to the question a symmetric key can't answer: how do you get a key to a party you've never talked to before? RSA with OAEP padding, elliptic curves, key agreement with ECDH, and encrypting the purchase order file Bookstore sends to a supplier with hybrid encryption so that only the supplier can open it.