Compression and Archives: GZip, Brotli, ZipArchive's Async APIs and Zip Slip

Version note: The code in this post was built and run with .NET 10; one comparison was also run with .NET 11 (SDK 11.0.100).

Guess First

We compress the 20,000-order, 6.8 MB NDJSON export of Bookstore, the online bookstore used as the shared example across this site's series, with BrotliStream. With CompressionLevel.Optimal it takes 10.6 ms. How many milliseconds does compressing the same data with CompressionLevel.SmallestSize take: 20, 100, or more?

Second question: we put 1,000 invoice XML files (1.7 MB in total) into a zip once and into a tar.gz once. Which one is smaller, and by how much?

Third: to write the "all invoices" zip straight to the response in ASP.NET Core, you use .NET 10's new async APIs: ZipArchive.CreateAsync, ZipArchiveEntry.OpenAsync and await using everywhere. Does this code work on Kestrel?

In the previous posts we produced JSON, XML and CSV files; in this post we look at the layer that carries them. The examples are in the Bookstore.IO.Compression console app and in Bookstore.Api in the series' repo:

dotnet run --project src/Bookstore.IO.Compression -- formats
dotnet run --project src/Bookstore.Api --urls http://127.0.0.1:5188

Four Formats: Deflate, ZLib, GZip and Brotli

.NET has four compression streams, and the first three are really the same algorithm. Deflate (RFC 1951) is raw compressed data with no header. ZLib (RFC 1950) adds a 2-byte header and an Adler-32 checksum at the end of the same data. GZip (RFC 1952) adds a 10-byte header, a CRC32 and the original size; this is the format of .gz files and of Content-Encoding: gzip in HTTP. Brotli (RFC 7932) is a separate algorithm developed by Google and designed for the web. We compressed the same export with all four at three levels; the time is the shortest of five runs on a developer machine:

Input: 6,808,653 bytes NDJSON
Format   Level               Bytes  Ratio     ms  Header
Deflate  Fastest           798,052   8.5x    4.6  AA56F2
Deflate  Optimal           325,989  20.9x   14.0  ECBD4D
Deflate  SmallestSize      315,959  21.5x   31.1  ECBD4B
ZLib     Fastest           798,058   8.5x    6.0  7801AA
ZLib     Optimal           325,995  20.9x   15.3  789CEC
ZLib     SmallestSize      315,965  21.5x   32.6  78DAEC
GZip     Fastest           798,070   8.5x    4.9  1F8B08
GZip     Optimal           326,007  20.9x   14.1  1F8B08
GZip     SmallestSize      315,977  21.5x   31.4  1F8B08
Brotli   Fastest           602,912  11.3x    4.8  ABFFFF
Brotli   Optimal           411,196  16.6x   10.6  9B4CE4
Brotli   SmallestSize      242,577  28.1x 6921.1  9B4CE4

There are a few things to read from the table. The sizes of Deflate, ZLib and GZip show byte-for-byte the same data; the 6 and 18 bytes of difference are only the header and footer. ZLib's first byte is always 78, and the second byte encodes the level (01, 9C, DA). GZip always starts with 1F 8B 08; you tell whether an incoming file really is gzip by looking at these three bytes, not at the extension. Deflate and Brotli have no recognizable start, so you have to know the format from outside.

The answer to the first question is on the last line: 6,921 ms, 650 times Optimal. Brotli SmallestSize gives the smallest file (23 percent smaller than GZip), but this is not work to do during a request. It makes sense for compressing static files once at build time and storing them; on an endpoint that runs on every request it ties up the server. There is also an unexpected result: Brotli Optimal came out larger than GZip Optimal. In .NET, Optimal maps to a fast level for Brotli; we will see that in the next section.

A short rule for choosing a format: Brotli or GZip for HTTP responses (according to the client's Accept-Encoding; ASP.NET Core's response compression middleware does this for you), GZip for files and archives, ZLib or Deflate for a protocol between two of your own services. Note too that the deflate encoding in HTTP is actually the ZLib format; sending raw Deflate causes problems with some clients.

CompressionLevel and the .NET 9 Options

The CompressionLevel enum carries only four values: Optimal, Fastest, NoCompression and SmallestSize, added in .NET 6. .NET 9 brought detailed settings for two algorithms: ZLibCompressionOptions for the ZLib family (a level from 0 to 9 and a strategy) and BrotliCompressionOptions for Brotli (a quality from 0 to 11). With the same data:

var zlib = new ZLibCompressionOptions { CompressionLevel = 9, CompressionStrategy = ZLibCompressionStrategy.Default };
await using var zlibStream = new ZLibStream(output, zlib, leaveOpen: true);

var brotli = new BrotliCompressionOptions { Quality = 9 };
await using var brotliStream = new BrotliStream(output, brotli, leaveOpen: true);

ZLib   level  1:     798,058 bytes    6.2 ms
ZLib   level  6:     325,995 bytes   15.8 ms
ZLib   level  9:     315,965 bytes   33.2 ms
Brotli quality  1:   602,912 bytes    4.9 ms
Brotli quality  4:   411,196 bytes   10.8 ms
Brotli quality  9:   294,010 bytes   62.4 ms
Brotli quality 11:   242,577 bytes 7002.2 ms

Comparing the sizes with the previous table shows what the enum maps to: in the ZLib family Fastest is level 1, Optimal 6 and SmallestSize 9; for Brotli Fastest is quality 1, Optimal 4 and SmallestSize 11. Brotli's real sweet spot is not in the enum: quality 9 produces a file 7 percent smaller than GZip's best in 62 ms. If you use Brotli, choose the quality through the options instead of the enum and measure with your own data.

The second change in .NET 9 is invisible but important: DeflateStream, ZLibStream, GZipStream and ZipArchive now use zlib-ng instead of zlib; according to Microsoft's notes the goal is more consistent and efficient results across operating systems and hardware. This has a side effect: the compressed bytes for the same input and level can change between .NET versions. Tests that store a hash of compressed output as a fixture can break on an upgrade; compare decompressed data in tests.

ZipArchive: Create, Read and Update

GZip compresses a single stream; to carry several files in one package you need ZIP or tar. ZipArchive opens in three modes. Create only writes and does not require the target stream to be seekable: it writes each entry in turn and appends the central directory at the end. Read first reads the central directory at the end, so it needs a seekable stream. Update does both. The code that writes the invoices to a zip:

public static async Task WriteAsync(Stream destination, IEnumerable<Order> orders, CancellationToken ct = default)
{
    await using ZipArchive archive = await ZipArchive.CreateAsync(destination, ZipArchiveMode.Create, leaveOpen: true, entryNameEncoding: null, ct);
    foreach (Order order in orders)
    {
        ZipArchiveEntry entry = archive.CreateEntry($"invoices/{order.OrderNumber}.xml", CompressionLevel.Optimal);
        await using Stream entryStream = await entry.OpenAsync(ct);
        await InvoiceXml.Create(order, IssueDate).SaveAsync(entryStream, SaveOptions.None, ct);
    }
}

Wrote 1,000 invoices to out/invoices.zip: 652,184 bytes in 81 ms
  invoices/BS-0000001.xml: Length 2,250, CompressedLength 562
  invoices/BS-0000002.xml: Length 2,250, CompressedLength 551
  Sum of Length: 1,748,940, sum of CompressedLength: 530,162

InvoiceXml is the UBL-like invoice we wrote in the series' XML post. An entry has two sizes, and mixing them up is a common mistake: Length is the uncompressed size, CompressedLength the size in the archive. Both are numbers written inside the archive and nobody verifies them; we will get to the security side of that shortly. Entry names are always separated with /; building a name with Path.Combine on Windows and putting \ in it leaves strange file names for people who open the archive on other platforms.

In Create mode only one entry can be open at a time. If you try to open a second one while the first entry's stream is open:

IOException: Entries cannot be created while previously created entries are still open.

That is why you cannot write the invoices to the same archive with Parallel.ForEachAsync. If you want parallelism, produce the XML in parallel and let a single writer write to the archive in order; we built that shape with a Channel in the Pipelines post.

.NET 8 added overloads to ZipFile that take a stream instead of a file path (ZipFile.CreateFromDirectory(string, Stream), ZipFile.ExtractToDirectory(Stream, string)); zipping a folder straight into a stream without writing a file is one line. .NET 10 also improved Update mode: according to Microsoft's notes, all entries used to be loaded into memory and rewritten in this mode, which no longer happens, and extraction runs in parallel.

One more note: we said Read mode needs a seekable stream; if you pass a non-seekable stream, ZipArchive does not fail, it copies the whole stream into memory first. When we opened a 13.4 MB zip with 20,000 entries through a non-seekable wrapper, the heap grew by 35.7 MB; with the file itself, by 18.8 MB. When reading uploaded zips, check the stream's CanSeek.

.NET 10's Async ZIP APIs and the Hidden Synchronous Write

One of the most awaited I/O features of .NET 10 was the async ZIP APIs: ZipArchive.CreateAsync, ZipArchiveEntry.OpenAsync, ZipFile.OpenAsync, ZipFile.CreateFromDirectoryAsync, ZipFile.ExtractToDirectoryAsync and extensions such as ExtractToFileAsync. The code above uses them. To answer the third question, we first wrote a small stream that, like Kestrel's response body, is not seekable and counts synchronous and async writes separately, and had the same code write to it:

Runtime 10.0.13: 1,000 entries
  async writes:  5,001 (650,184 bytes)
  sync writes:   2,000 (18,000 bytes)

Even though we wrote everything asynchronously, there are two synchronous writes per entry, 18 bytes in total. The cause is in the runtime: in Create mode the inner stream that OpenAsync returns does not override DisposeAsync, the base Stream.DisposeAsync calls the synchronous Dispose, and DeflateStream writes its last block synchronously. The bug is recorded as issue 121624 in the dotnet/runtime repository and was fixed in the 11.0.0 milestone. We ran the same code with .NET 11:

Runtime 11.0.0: 1,000 entries
  async writes:  7,001 (668,184 bytes)
  sync writes:       0 (0 bytes)

When writing to a file or a MemoryStream, this difference is irrelevant. But Kestrel does not allow synchronous writes. When we used this code in an endpoint on .NET 10, the request crashed at the end of the first invoice and the client received a broken 720-byte zip:

fail: Microsoft.AspNetCore.Server.Kestrel[13]
      Connection id "0HNQ1U65K0RU4", Request id "0HNQ1U65K0RU4:00000001": An unhandled exception was thrown by the application.
      System.InvalidOperationException: Synchronous operations are disallowed. Call WriteAsync or set AllowSynchronousIO to true instead.
         at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponseStream.Write(Byte[] buffer, Int32 offset, Int32 count)
         at System.IO.Compression.DeflateStream.PurgeBuffers(Boolean disposing)
         at System.IO.Compression.DeflateStream.Dispose(Boolean disposing)
         ...
         at System.IO.Compression.WrappedStream.Dispose(Boolean disposing)
         at System.IO.Stream.Close()
         at System.IO.Stream.DisposeAsync()

The answer to the third question: no on .NET 10, yes on .NET 11. What matters is that you only see this bug in a real environment, with a stream that is not seekable and rejects synchronous writes. A unit test that writes to a file always passes.

Streaming the "All Invoices" Zip to the Response

The "download all invoices" button in the Angular UI is a simple download link; all the work happens on the server. The classic solution is to write the zip to a temporary file or a MemoryStream first and send it afterwards. For 20,000 invoices that means a 13 MB buffer on the server, and the user only gets the first byte once the zip is completely ready. Because ZipArchive's Create mode does not need seeking, we can write the entries straight to the response body. The smallest fix for the synchronous last block on .NET 10 is to allow synchronous writes for this endpoint only:

// All invoices in one zip: no disk, no MemoryStream; entries go to the response as they are produced.
app.MapGet("/api/invoices/zip", async (int count, HttpContext context) =>
{
    // On .NET 10 the last few bytes of each entry are written synchronously (dotnet/runtime#121624, fixed in .NET 11).
    context.Features.GetRequiredFeature<IHttpBodyControlFeature>().AllowSynchronousIO = true;
    context.Response.ContentType = "application/zip";
    context.Response.Headers.ContentDisposition = "attachment; filename=\"invoices.zip\"";
    await InvoiceZip.WriteAsync(context.Response.Body, SampleOrders.Generate(count), context.RequestAborted);
});

$ curl -s -D - -o invoices.zip -w "first byte %{time_starttransfer}s, total %{time_total}s, %{size_download} bytes\n" "http://127.0.0.1:5188/api/invoices/zip?count=20000"
HTTP/1.1 200 OK
Content-Type: application/zip
Date: Sat, 14 Nov 2026 01:50:57 GMT
Server: Kestrel
Transfer-Encoding: chunked
Content-Disposition: attachment; filename="invoices.zip"

first byte 0.000451s, total 0.990429s, 13390980 bytes

$ unzip -tq invoices.zip
No errors detected in compressed data of invoices.zip.

The response goes out without Content-Length, with Transfer-Encoding: chunked: the server does not know the final size of the zip and does not need to. The first byte arrived in under a millisecond, the 13.4 MB zip with 20,000 invoices finished in a second, and unzip verified its integrity. Allowing synchronous writes is normally something you avoid, because it blocks a thread while writing; here it only applies to the last 18 bytes of each entry, and its scope is a single request. Delete this line when you move to .NET 11. If the client aborts the download, the RequestAborted token stops the production. The general rules of file downloads (Content-Disposition, range requests, limits) are in the file upload and download post of the ASP.NET Core Web API series.

Tar and tar.gz: System.Formats.Tar

Since .NET 7, tar archives are supported directly through System.Formats.Tar. Tar does not compress, it lays the files out one after another; for compression you write the tar stream into a GZipStream. To answer the second question we wrote the same 1,000 invoices to a folder and packaged them both ways:

await using (var file = File.Create(tarGz))
await using (var gzip = new GZipStream(file, CompressionLevel.Optimal))
{
    await TarFile.CreateFromDirectoryAsync(dir, gzip, includeBaseDirectory: false);
}

await using (var file = File.Create(zip))
{
    // .NET 8: ZipFile's overload that takes a stream.
    ZipFile.CreateFromDirectory(dir, file, CompressionLevel.Optimal, includeBaseDirectory: false);
}

1,000 files, 1,748,940 bytes
zip:       627,796 bytes
tar.gz:     72,306 bytes
First tar entry: BS-0000206.xml, 1,266 bytes, format Pax

The tar.gz is almost one ninth of the zip. The reason is structural: ZIP compresses each entry separately, so across 1,000 invoices it compresses the repeated namespace declarations, element names and ISBNs 1,000 separate times. tar.gz compresses all the files as one stream and reuses a pattern it saw in one invoice for the following ones. In return, tar.gz has no random access: to read the 900th invoice you have to decompress the 899 before it; in ZIP you go there directly thanks to the central directory. For archives that go to users and are opened with a double click on Windows, ZIP; for backups and exports moved between Linux servers and extracted in full, tar.gz. The default tar format is Pax; it carries long file names and large files without trouble.

Safely Extracting an Uploaded Zip: Zip Slip and Zip Bombs

Institutional customers upload their bulk order files as zips. Two pieces of information inside an archive are under the attacker's control: the entry name and the entry's sizes. Zip Slip means using ../ in an entry name to get a file written outside the target folder. For testing we produced a zip with two entries: invoices/BS-0000001.xml and ../outside.txt.

ZipFile.ExtractToDirectory: IOException: Extracting Zip entry would have resulted in a file outside the specified destination directory.
Naive: invoices/BS-0000001.xml -> extract/naive/invoices/BS-0000001.xml
Naive: ../outside.txt -> extract/outside.txt
SafeZipExtractor: Entry '../outside.txt' is outside the target directory.

.NET's own ExtractToDirectory performs this check. The problem is code that walks the entries one by one and builds its own path with Path.Combine(target, entry.FullName): on the second line the file was written outside the naive folder. When you have to extract a zip by hand (filtering, limits, per-file validation), you have to do the check yourself. The general treatment of path traversal is in the OWASP Top 10 series' post on path traversal; the check here applies the same principle to zips:

string root = Path.GetFullPath(destination);
if (!Path.EndsInDirectorySeparator(root)) root += Path.DirectorySeparatorChar;
StringComparison comparison = OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;

await using ZipArchive archive = await ZipArchive.CreateAsync(zip, ZipArchiveMode.Read, leaveOpen: false, entryNameEncoding: null, ct);
if (archive.Entries.Count > limits.MaxEntries)
    throw new InvalidDataException($"Archive has {archive.Entries.Count} entries, limit is {limits.MaxEntries}.");

foreach (ZipArchiveEntry entry in archive.Entries)
{
    // Unlike Path.Combine, Path.Join does not switch the target for an entry name that starts at the root.
    string target = Path.GetFullPath(Path.Join(root, entry.FullName));
    if (!target.StartsWith(root, comparison))
        throw new InvalidDataException($"Entry '{entry.FullName}' is outside the target directory.");

    if (entry.FullName.EndsWith('/'))
    {
        Directory.CreateDirectory(target);
        continue;
    }
    if (entry.CompressedLength > 0 && (double)entry.Length / entry.CompressedLength > limits.MaxCompressionRatio)
        throw new InvalidDataException($"Entry '{entry.FullName}' has compression ratio {entry.Length / entry.CompressedLength:N0}:1.");

    Directory.CreateDirectory(Path.GetDirectoryName(target)!);
    await using Stream input = await entry.OpenAsync(ct);
    await using var output = new FileStream(target, FileMode.CreateNew, FileAccess.Write, FileShare.None, 4096, useAsync: true);
    int read;
    while ((read = await input.ReadAsync(buffer, ct)) > 0)
    {
        // Do not trust Length from the header: count the bytes actually extracted.
        total += read;
        if (total > limits.MaxTotalBytes)
            throw new InvalidDataException($"Extracted size exceeds {limits.MaxTotalBytes:N0} bytes.");
        await output.WriteAsync(buffer.AsMemory(0, read), ct);
    }
    files++;
}

Three details matter. Adding a separator to the end of the root path stops a root of /data/extract from matching a sibling folder such as /data/extract-evil. Using Path.Join stops Path.Combine from switching the target entirely to an entry name that starts at the root, such as /etc/x. FileMode.CreateNew refuses to overwrite an existing file.

A zip bomb is a small archive that fills the disk or memory when extracted. For testing we produced a single entry made of 1 GB of zeros:

1,043,756 byte zip, entry Length 1,073,741,824, CompressedLength 1,043,638
ratio check: Entry 'report.csv' has compression ratio 1,028:1.
size check: Extracted size exceeds 100,000,000 bytes.

A 1 MB zip expands to 1 GB. The first defense is the ratio check: the invoices in this post compressed 4 times and the NDJSON export 21 times; an entry that compresses a thousand times is suspicious. But the ratio is computed from numbers written inside the archive, and those numbers can lie. That is why the real defense is on the second line: even with the ratio check turned off, the loop that counts the extracted bytes stopped at 100 MB. The entry count limit is there against archives with millions of empty entries. All of these come on top of an upload size limit, not instead of it.

In Production

Choose the level by how often the work runs. For compression on every request, Fastest or Optimal; for a file produced once and downloaded many times, a higher level. Brotli quality 11 took 7 seconds for 6.8 MB; running that in a request means the server loses a CPU core for the duration of that request.

Do not compress already compressed data again. PDF, JPEG, PNG and zip files are already compressed; when putting them in a zip, CompressionLevel.NoCompression gives almost the same size without spending CPU.

Error handling is different in streaming endpoints. For an exception after the response has started, you cannot change the status code; the client gets half a zip. So finish the risky work (authorization, data validation) before the first byte and make the integrity of the download checkable on the client side.

Extract uploaded archives into a separate folder and with limits: entry count, total extracted size, ratio, allowed extensions. Validate the extracted files like any other user input too; coming from inside an archive does not make them trustworthy.

Common Pitfalls

Using the output without disposing the GZipStream. The last block and the footer are written on dispose; a MemoryStream read before that contains a broken gzip. We saw this in the series' post on the Stream model.

Reading the entry size from the wrong property. CompressedLength is the size in the archive, Length the uncompressed size; both are the archive's claims, not verified information. Trusting Length to allocate a buffer in memory lets the attacker choose the buffer size.

Writing to the same ZipArchive in parallel. In Create mode only one entry can be open at a time; parallel code either gets an exception or corrupts the archive. Parallelize the production, gather the writing in a single writer.

Assuming everything is async because you use the async APIs. On .NET 10, the stream that OpenAsync returns writes synchronously during dispose; test in a real environment with streams that reject synchronous writes (such as Kestrel's response body).

Turning an entry name into a path without validating it. Path.Combine(target, entry.FullName) is open to Zip Slip; do not write any entry to disk without GetFullPath plus a root check.

A Small Experiment

★☆☆ Add the ZLibCompressionStrategy.HuffmanOnly and RunLengthEncoding strategies to the formats command. How do size and time change for the NDJSON data? For what kind of data could these strategies make sense?

★★☆ Rewrite the /api/invoices/zip endpoint so that it works without AllowSynchronousIO. Hint: write a wrapper similar to WriteCountingStream; let it collect synchronous writes in its own buffer, and let a FlushPendingAsync method you call after each entry is closed send those bytes to the response asynchronously. How many bytes does the buffer grow to at most?

★★★ Add an allow list for entry names to SafeZipExtractor (only .xml and .csv, at most two folder levels deep) and catch the case where the same entry name appears twice in the archive. Why is that case a separate risk?

The largest file in this post was 13 MB. In the next post we scale up: extracting daily revenue from a 5 GB order log with constant memory, splitting the file into offset ranges and processing them in parallel, and seeing when memory-mapped files really speed things up, with real measurements of three approaches.