Asymmetric Cryptography: RSA-OAEP, Elliptic Curves, ECDH Key Agreement and Hybrid Encryption

Version note: The code in this post was built and run with .NET 10 and xUnit v3 4.0; the outputs were taken on Ubuntu 24.04 (OpenSSL 3.0).

Guess First

Every week Bookstore sends purchase order files to its suppliers: how many copies of which ISBN, at what unit price. The file is dropped into an SFTP folder and the supplier picks it up from there. A new supplier contract added the clause "files must be encrypted end to end": the server in between, the backup system or anyone with access to the folder must not be able to read the file.

From the previous post we have AES-GCM. The problem: to open the file, the supplier needs the same key. How do you get the key to them? By e-mail, by phone, as another file next to the file? Whichever channel you choose, it has to be more secure than the file you're trying to protect. In this post we solve that problem, key distribution: public key cryptography, RSA and elliptic curves, key agreement with ECDH and hybrid encryption, which ties them together.

What a Public Key Solves

In asymmetric cryptography every party has a key pair. The private key stays with its owner only; the public key can be given to anyone. If you encrypt a message with the public key, only the holder of the private key can open it. The supplier sends us their public key once; we encrypt every file with it. Nothing secret passes through any channel.

But there are two constraints. First, asymmetric operations are much slower than symmetric ones and the amount of data they can encrypt directly is limited by the key size. Second, you need to be sure the public key really belongs to the supplier; if the key that reached us was swapped on the way, we encrypt the file for the attacker. We solve the first constraint in this post with hybrid encryption. The second, binding a public key to its owner, is the job of X.509 certificates and the subject of the sixth post in the series. Here we assume we got the supplier's public key through a trusted channel, for example during the contract process.

RSA: OAEP, PKCS#1 v1.5 and Key Size

RSA is used for encryption together with a padding scheme, and the choice matters. The old PKCS#1 v1.5 padding is open to padding oracle attacks: if the server reveals whether the padding of what it decrypted was valid, through an error message, the way it closes the connection or its response time, an attacker can use that signal to decrypt a ciphertext step by step. Daniel Bleichenbacher published this attack in 1998; the 2017 ROBOT study showed the same flaw was still present on 27 of the top 100 domains in the Alexa list. In new code the only right choice is OAEP, with SHA-256 as the hash: RSAEncryptionPadding.OaepSHA256.

Let's see how much data we can encrypt with RSA:

using RSA rsa = RSA.Create(2048);
foreach (int size in new[] { 190, 191 })
{
    try
    {
        byte[] ciphertext = rsa.Encrypt(new byte[size], RSAEncryptionPadding.OaepSHA256);
        Console.WriteLine($"{size} bytes -> {ciphertext.Length} bytes ciphertext");
    }
    catch (CryptographicException ex)
    {
        Console.WriteLine($"{size} bytes -> {ex.GetType().Name}: {ex.Message}");
    }
}

// Same input, two encryptions: OAEP is randomized
byte[] a = rsa.Encrypt("Dune"u8, RSAEncryptionPadding.OaepSHA256);
byte[] b = rsa.Encrypt("Dune"u8, RSAEncryptionPadding.OaepSHA256);
Console.WriteLine($"same plaintext, same ciphertext: {a.AsSpan().SequenceEqual(b)}");

$ dotnet run --project src/Bookstore.CryptoLab -- rsa-limit
190 bytes -> 256 bytes ciphertext
191 bytes -> OpenSslCryptographicException: error:0200006E:rsa routines::data too large for key size
same plaintext, same ciphertext: False

With a 2048-bit key, OAEP-SHA256 encrypts at most 190 bytes: two hash lengths and two bytes of padding come out of the 256-byte modulus. The error message comes from OpenSSL; on Windows you'll see CNG's own message in the same situation, but the type is still a CryptographicException. Our order file is 229 bytes; it already doesn't fit. Splitting the data into pieces and encrypting each with RSA isn't a solution either: it's slow, big, and doesn't protect the order or integrity of the pieces.

For key size, NIST's key management guideline SP 800-57 counts RSA-2048 at the 112-bit and RSA-3072 at the 128-bit security level; it accepts protecting new data with a 112-bit key through the end of 2030 and requires at least 128 bits from 2031. NIST IR 8547, published in October 2026, tied this timeline to the quantum transition: RSA of any size will be deprecated after 2030 and disallowed after 2035. If you're generating a new key, start at 3072 bits; how quantum computers affect RSA and what replaces it is the subject of the series' post-quantum post.

Elliptic Curves: Why Smaller Keys

Elliptic curve cryptography reaches the same security level with much smaller keys. The difference is clear when we export the keys:

foreach (int bits in new[] { 2048, 3072 })
{
    using RSA rsa = RSA.Create(bits);
    Console.WriteLine($"RSA-{bits,-5} public key {rsa.ExportSubjectPublicKeyInfo().Length,3} bytes, private key {rsa.ExportPkcs8PrivateKey().Length,4} bytes");
}

foreach (ECCurve curve in new[] { ECCurve.NamedCurves.nistP256, ECCurve.NamedCurves.nistP384 })
{
    using ECDiffieHellman ecdh = ECDiffieHellman.Create(curve);
    Console.WriteLine($"{curve.Oid.FriendlyName,-9} public key {ecdh.ExportSubjectPublicKeyInfo().Length,3} bytes, private key {ecdh.ExportPkcs8PrivateKey().Length,4} bytes");
}

$ dotnet run --project src/Bookstore.CryptoLab -- key-sizes
RSA-2048  public key 294 bytes, private key 1216 bytes
RSA-3072  public key 422 bytes, private key 1793 bytes
nistP256  public key  91 bytes, private key  138 bytes
nistP384  public key 120 bytes, private key  185 bytes

P-256 is at the same 128-bit security level as RSA-3072, but its public key is a fifth of the size. Key generation is much faster too: generating an RSA key means searching for large primes, while an EC key is just a random number. P-256 and P-384 are curves defined by NIST and supported by .NET on every platform; according to the platform table in the first post of the series, support for other curves such as Brainpool varies by platform. There's no class in .NET 10 for the X25519 curve you often see in modern protocols; we'll cover its status in .NET in the last post of the series. Bookstore uses P-256.

A Shared Secret with ECDH and Why It Goes Through HKDF

We don't "encrypt" with an elliptic curve directly; we use ECDH, where two parties compute the same shared secret, each from their own private key and the other side's public key. Bookstore generates a new, temporary key pair (an ephemeral key) for every message; it computes a shared secret from its ephemeral private key and the supplier's public key. The supplier computes the same secret from its own private key and Bookstore's ephemeral public key. The ephemeral public key travels with the file; the private key is destroyed once the message is encrypted.

Using the shared secret directly as an AES key is wrong. The raw output of ECDH is a coordinate of a curve point; it's not a uniformly random value. That's the case we mentioned when covering HKDF in the previous post: if the input isn't a strong, uniform key, the extract step and a salt are a must. DeriveRawSecretAgreement, which came with .NET 8, gives us the raw secret, and we run it through HKDF:

// The raw ECDH output is not uniformly random: always run it through a KDF, with a salt
private static byte[] DeriveKey(byte[] shared, byte[] ephemeralPublic, string keyId) =>
    HKDF.DeriveKey(HashAlgorithmName.SHA256, shared, 32, salt: ephemeralPublic,
        info: Encoding.UTF8.GetBytes($"bookstore/purchase-order/v1|{keyId}"));

We pass the ephemeral public key as the salt: it isn't secret, it differs for every message and both sides have it. info binds the key to its purpose and to the recipient's key identifier; the same shared secret produces a different key in another protocol. Shortcuts that skip the KDF step, such as "SHA-256 of the raw ECDH output", are both non-standard and lose the context information.

Hybrid Encryption: The Purchase Order File

Hybrid encryption combines the two worlds: asymmetric cryptography carries only a 32-byte symmetric key, and the data itself is encrypted by the previous post's AEAD, AES-GCM. Bookstore, the online bookstore used as the shared example across this site's series, keeps the code in the Hybrid folder of the Bookstore.Crypto library:

// What travels to the recipient. Binary fields are Base64Url; the whole record is serialized as JSON.
public sealed record HybridEnvelope(int Version, string Algorithm, string KeyId, string EncapsulatedKey, string Nonce, string Ciphertext, string Tag);

public static HybridEnvelope Encrypt(ECDiffieHellman recipientPublicKey, string keyId, ReadOnlySpan<byte> plaintext)
{
    // A fresh key pair for every message: the sender keeps no private key
    using ECDiffieHellman ephemeral = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256);
    byte[] ephemeralPublic = ephemeral.ExportSubjectPublicKeyInfo();
    byte[] shared = ephemeral.DeriveRawSecretAgreement(recipientPublicKey.PublicKey);
    byte[] key = DeriveKey(shared, ephemeralPublic, keyId);
    try
    {
        return Aead.Seal(Version, Algorithm, keyId, ephemeralPublic, key, plaintext);
    }
    finally
    {
        CryptographicOperations.ZeroMemory(shared);
        CryptographicOperations.ZeroMemory(key);
    }
}

Aead.Seal reuses the idea from the previous post: the version, the algorithm name, the keyId and the ephemeral public key aren't encrypted but are included in the tag as associated data. So if someone changes the algorithm name or the keyId in the file, opening it fails. On the decrypting side we also check that the ephemeral public key really is on P-256; don't use a key that came from outside without validating it against the curve you expect.

The lab command plays both sides: the supplier generates the key pair and hands over its public key as PEM, Bookstore encrypts the order, and the supplier opens its encrypted private key with a passphrase and reads the order:

$ dotnet run --project src/Bookstore.CryptoLab -- purchase-order
supplier-0042.key.pem: -----BEGIN ENCRYPTED PRIVATE KEY----- (8 lines)
supplier-0042.pub.pem: -----BEGIN PUBLIC KEY----- (4 lines)
PO-2026-11-0042.json.enc: 638 bytes for a 229-byte order
{
  "version": 1,
  "algorithm": "ECDH-P256.HKDF-SHA256.A256GCM",
  "keyId": "supplier-0042/2026",
  "encapsulatedKey": "MFkwEwYHKoZIzj0CAQYI...",
  "nonce": "A1HQBsW6CVgK9lJf",
  "ciphertext": "ASfmhuFdpGIn2Cjxepdb...",
  "tag": "9rbS3dE2Wcyv4o_Tu0kE4g"
}
supplier reads: {"purchaseOrderId":"PO-2026-11-0042","supplierId":"sup...
other supplier's key: AuthenticationTagMismatchException
keyId changed in transit: AuthenticationTagMismatchException
wrong passphrase: CryptographicException

The 229-byte order became a 638-byte file; most of the difference is the 91-byte ephemeral public key, Base64Url and JSON. Another supplier's key produces a different shared secret, hence a different AES key, and the tag doesn't match. We chose Base64Url for the binary fields: the + and / characters of standard Base64 can turn into escape sequences in JSON and cause trouble in URLs.

This construction isn't new. The ECDH-ES mode of JWE, which we saw in the JWT post of the Authentication and Authorization series, applies the same idea in JSON form; HPKE, standardized in RFC 9180, defines the same approach at the protocol level. If the other side speaks one of these standards, don't invent your own format, use theirs. Our format is a bilateral agreement with the supplier; the version and algorithm fields are there to make a later move to HPKE easier.

If the supplier has an RSA key, the structure is the same and only the key transport part changes: a random DEK is generated and encrypted with OAEP:

public static HybridEnvelope Encrypt(RSA recipientPublicKey, string keyId, ReadOnlySpan<byte> plaintext)
{
    byte[] dek = RandomNumberGenerator.GetBytes(32);
    try
    {
        byte[] wrapped = recipientPublicKey.Encrypt(dek, RSAEncryptionPadding.OaepSHA256);
        return Aead.Seal(Version, Algorithm, keyId, wrapped, dek, plaintext);
    }
    finally
    {
        CryptographicOperations.ZeroMemory(dek);
    }
}

Storing and Moving Keys: PEM, PKCS#8 and Certificates

Keys are usually written to files or configuration as PEM. The content is a standard structure: SubjectPublicKeyInfo for a public key (-----BEGIN PUBLIC KEY-----), PKCS#8 for a private key (-----BEGIN PRIVATE KEY-----). .NET has import and export methods for both and works out the algorithm from the PEM label:

// Supplier side: generate a key pair once, send only the public key to Bookstore
using (ECDiffieHellman supplierKey = ECDiffieHellman.Create(ECCurve.NamedCurves.nistP256))
{
    File.WriteAllText(Path.Combine(folder, "supplier-0042.pub.pem"), supplierKey.ExportSubjectPublicKeyInfoPem());
    PbeParameters pbe = new(PbeEncryptionAlgorithm.Aes256Cbc, HashAlgorithmName.SHA256, iterationCount: 600_000);
    File.WriteAllText(Path.Combine(folder, "supplier-0042.key.pem"), supplierKey.ExportEncryptedPkcs8PrivateKeyPem(passphrase, pbe));
}

// Supplier side: open the private key with the passphrase and decrypt
using ECDiffieHellman supplierPrivate = ECDiffieHellman.Create();
supplierPrivate.ImportFromEncryptedPem(File.ReadAllText(Path.Combine(folder, "supplier-0042.key.pem")), passphrase);

If you have to write a private key to disk, use encrypted PKCS#8 (-----BEGIN ENCRYPTED PRIVATE KEY-----). The key is encrypted with an AES key derived from the passphrase with PBKDF2; 600,000 iterations matches the PBKDF2-HMAC-SHA256 recommendation in the password storage post of the Web Security (OWASP Top 10) series. The lab's passphrase is dev-only-supplier-key-passphrase; in reality the value comes from a secret store or a person's memory, never from code.

Instead of a bare public key the supplier may send an X.509 certificate. A certificate carries the public key together with the owner's name and a signature; getting the key out of it is one line:

using X509Certificate2 received = X509Certificate2.CreateFromPem(pem);
using RSA publicKey = received.GetRSAPublicKey()!;
Console.WriteLine($"{received.Subject}, has private key: {received.HasPrivateKey}, RSA-{publicKey.KeySize}");

HybridEnvelope envelope = RsaHybrid.Encrypt(publicKey, "supplier-0042/2026-rsa", PurchaseOrder);
Console.WriteLine($"{envelope.Algorithm}: encapsulated key {Base64Url.DecodeFromChars(envelope.EncapsulatedKey).Length} bytes");

$ dotnet run --project src/Bookstore.CryptoLab -- rsa-certificate
CN=supplier-0042.example.com, has private key: False, RSA-3072
RSA-OAEP-SHA256.A256GCM: encapsulated key 384 bytes
round trip: True

The certificate in the lab is self-signed and generated in memory; in real life you shouldn't trust the key inside a certificate without validating its chain, validity period and intended usage. How to do that is the subject of the sixth post. The key carried with RSA-3072 is 384 bytes; compare that with the 91-byte ephemeral public key of ECDH.

Production Notes

Manage the recipient's key with a keyId. When the supplier renews its key, new files are encrypted with the new key; old files are marked with the old keyId, so the supplier knows which private key to use. Tie key changes to the contract and the calendar: record when you received the new public key and when you stopped using the old one.

This construction has no forward secrecy: even though the sender uses a new ephemeral key for every message, if the recipient's long-lived private key leaks, every past file encrypted to that key can be opened. That's unavoidable in asynchronous scenarios such as file transfer; the protection is to rotate the recipient key regularly and not keep old private keys longer than needed. We'll see how live connections get forward secrecy in the TLS post.

On performance, RSA private key operations (decrypting and signing) are much slower than public key operations, and EC operations usually sit in between. Unless you encrypt thousands of files per second the difference doesn't matter; but don't generate RSA keys on a request path.

Where the private key lives matters more than the code itself. In production the private key should live in an HSM or Key Vault, or at the very least as encrypted PKCS#8 in a secret store with restricted access.

Common Pitfalls

The first and most common pitfall is putting a private key in the repo or the log. The key.pem generated for a test ends up in a commit, and one day the repo becomes public. That's why this project's .gitignore has *.pem and *.key lines; the secret scanning covered in the DevSecOps series catches it in the pipeline too. On the logging side, never log the key object or the PEM text; the keyId is enough.

The second is using RSA with PKCS#1 v1.5 padding. Unless compatibility with an old system forces you, use OAEP; if it does, make sure decryption failures don't leak to the outside as different responses.

The third is using the ECDH shared secret directly as a key, or not giving HKDF any context. If the same key pair is used for two different purposes, the info parameter is the only thing separating those two uses.

The fourth is using an incoming public key without validating it: a different curve, an invalid point or the wrong algorithm. ImportSubjectPublicKeyInfo checks that the point is on the curve; checking that the curve is the one you expect is up to you.

The fifth is confusing "encrypted" with "signed". Since the supplier's public key is available to everyone, anyone can produce this file. Proving the file really came from Bookstore takes a digital signature, which is the subject of the next post.

A Small Experiment

★☆☆ In the purchase-order command, raise the PbeParameters iteration count from 600,000 to 6,000,000 and measure how long ImportFromEncryptedPem takes. Who pays that time, and how often? Why would it be wrong for a web server to do this on every request?

★★☆ Extend EcdhHybrid to work with P-384: add the curve to the algorithm name and do the curve check on the decrypting side based on the algorithm name. Test which exception you get when a file encrypted with P-256 is opened with a P-384 key.

★★★ Design supplier key rotation: Bookstore holds several supplier public keys mapped by keyId, new files are encrypted with the newest key, and on the supplier side write a SupplierKeyRing that picks the right private key by keyId. What should happen when an unknown keyId arrives?

In the next post we fix this post's last pitfall: proving who produced the file. Digital signatures with RSA-PSS and ECDSA, signature formats, the canonicalization problem when signing JSON, and a signed license file for Bookstore's institutional installations that run offline.