Reading HTTP/1.1 Off the Wire: Message Format, Method Semantics and Keep-Alive

Version note: The requests in this post were made with macOS's nc, OpenSSL 4.0.3 and .NET 10 (ASP.NET Core 10 Kestrel); tests were run with xUnit v3 4.0.2. The HTTP (Hypertext Transfer Protocol) description follows RFC (Request for Comments) 9110 and RFC 9112.

Guess First

You write a request to the Bookstore API (Application Programming Interface) by hand over a plain TCP (Transmission Control Protocol) connection, without any HTTP library, and you see this in the response:

Transfer-Encoding: chunked

3d
{"id":1,"isbn":"978-0000000001","title":"Title 1","price":11}
0

What's the 3d above the JSON (JavaScript Object Notation)? Why is there a 0 below it? And one more: if you sent the same request without the Host header, would the server still return the same JSON? In this post we read HTTP/1.1 as it is on the wire, byte by byte.

Semantics and Syntax: RFC 9110 and RFC 9112

In 2022 the HTTP specs were reorganized and split into two layers. RFC 9110 (HTTP Semantics) defines the meaning shared by all versions: methods, status codes, the meaning of header fields, content negotiation. RFC 9112 covers only HTTP/1.1's format on the wire: how a message is split into lines and how the length of the body is determined. HTTP/2 (RFC 9113) and HTTP/3 (RFC 9114) carry the same semantics in an entirely different wire format. In practice this means: a GET being idempotent, or what 304 means, is version-independent; but the answer to "where does the body end?" depends on the version.

Writing Requests by Hand: nc and openssl s_client

HTTP/1.1 is a text protocol; you can write it by hand. I open a TCP connection with nc (netcat) and send the request with printf. Line endings must be CRLF (Carriage Return + Line Feed, \r\n); cat -v shows them as ^M:

$ (printf 'GET /api/books/1 HTTP/1.1\r\nHost: localhost\r\n\r\n'; sleep 1) | nc 127.0.0.1 5491 | cat -v
HTTP/1.1 200 OK^M
Content-Type: application/json; charset=utf-8^M
Date: Sun, 08 Nov 2026 10:28:17 GMT^M
Server: Kestrel^M
Transfer-Encoding: chunked^M
^M
3d^M
{"id":1,"isbn":"978-0000000001","title":"Title 1","price":11}^M
0^M
^M

For HTTPS (HTTP Secure) you do the same with openssl s_client, which opens the TLS (Transport Layer Security) connection; -quiet hides the handshake details and -alpn http/1.1 prevents a switch to HTTP/2:

$ (printf 'GET /whoami HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n'; sleep 1) | openssl s_client -connect 127.0.0.1:5592 -servername localhost -CAfile certs/ca.crt -alpn http/1.1 -quiet
depth=1 CN=Bookstore Local Root CA
verify return:1
depth=0 CN=bookstore.localhost
verify return:1
HTTP/1.1 200 OK
Content-Length: 6
Connection: close
Content-Type: text/plain; charset=utf-8
Date: Sun, 08 Nov 2026 10:29:03 GMT
Server: Kestrel

green
80FFF8F501000000:error:0A000126:SSL routines::unexpected eof while reading:ssl/record/rec_layer_s3.c:708:

The error on the last line is harmless: the server honored the Connection: close request and closed the TCP connection but didn't send TLS's closing message (close_notify); OpenSSL 3 and later report that as an error. Since the body length was announced with Content-Length, we know the response arrived complete.

Anatomy of a Message: Start Line, Headers, Body

Every HTTP/1.1 message has three parts. The first line is the start line: in a request the method, request target and version (GET /api/books/1 HTTP/1.1), in a response the version, status code and reason phrase (HTTP/1.1 200 OK). Then come header lines, each in the form Name: value. An empty line (just CRLF) says the headers are done, and the body begins if there is one.

In HTTP/1.1 the Host header is mandatory. The reason is virtual hosting: a server hosting dozens of sites on the same IP (Internet Protocol) address and port learns which site a request is for only from Host. RFC 9112 requires a 400 for an HTTP/1.1 request that lacks this header. Kestrel does exactly that:

$ (printf 'GET /api/books/1 HTTP/1.1\r\n\r\n'; sleep 1) | nc 127.0.0.1 5491 | cat -v
HTTP/1.1 400 Bad Request^M
Content-Length: 0^M
Connection: close^M
Date: Sun, 08 Nov 2026 10:28:19 GMT^M
Server: Kestrel^M
^M

That's the answer to the second guessing question: a 400 and a closed connection instead of JSON. In ASP.NET Core the value of the Host header can additionally be restricted with the Host Filtering middleware; not leaving AllowedHosts at * in production protects against links generated from fake Host values (in password reset emails, for example).

Body Framing: Content-Length and Chunked

TCP is a byte stream; it doesn't know where messages end. In HTTP/1.1 there are two standard ways to find the end of the body. The first is Content-Length: the sender says up front how many bytes the body is. Kestrel uses it when it returns a string of known size:

$ (printf 'GET /whoami HTTP/1.1\r\nHost: localhost\r\n\r\n'; sleep 1) | nc 127.0.0.1 5491 | cat -v
HTTP/1.1 200 OK^M
Content-Length: 5^M
Content-Type: text/plain; charset=utf-8^M
Date: Sun, 08 Nov 2026 10:28:18 GMT^M
Server: Kestrel^M
^M
blue

blue plus a newline: 5 bytes. The second way is Transfer-Encoding: chunked: the sender sends the body in pieces, writing each piece's length in hexadecimal before it, and finishes with a zero-length piece. The 3d in the guessing section is hexadecimal 61: the JSON is exactly 61 bytes. The 0 at the end means "the body is done." Because the minimal API serializes JSON straight into the response stream, it doesn't know the total size before it starts writing and chooses chunked. It's more visible in an endpoint that flushes pieces separately:

// Writing without a Content-Length makes Kestrel use chunked encoding on HTTP/1.1
app.MapGet("/stream", async (HttpContext ctx) =>
{
    ctx.Response.ContentType = "text/plain";
    for (var i = 1; i <= 3; i++)
    {
        await ctx.Response.WriteAsync($"part {i}\n");
        await ctx.Response.Body.FlushAsync();
    }
});

Transfer-Encoding: chunked^M
^M
7^M
part 1
^M
7^M
part 2
^M
7^M
part 3
^M
0^M
^M

Each piece is seven bytes (part 1 plus \n). Chunked encoding lets the server send content whose end it doesn't know (a report, a log stream, an AI (Artificial Intelligence) response) as it's produced. The same rules apply when the request has a body; to create an order:

$ body='{"bookId":1,"quantity":2}'
$ (printf "POST /api/orders HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/json\r\nContent-Length: ${#body}\r\n\r\n$body"; sleep 1) | nc 127.0.0.1 5491
HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Date: Sun, 08 Nov 2026 10:28:30 GMT
Server: Kestrel
Location: /api/orders/1214
Transfer-Encoding: chunked

19
{"bookId":1,"quantity":2}
0

A 201 and a Location header with the new resource's address. Choosing status codes is covered in "Status Codes and Responses in ASP.NET Core: Ok, CreatedAtAction and ProblemDetails" in the ASP.NET Core Web API series.

Method Semantics: Safe and Idempotent

RFC 9110 assigns two properties to every method. A safe method doesn't ask for any change on the server from the client's point of view; an idempotent method means sending the same request once or ten times doesn't change the outcome.

This distinction matters most for retries. When the connection drops after a request was sent, the client can't know whether the request reached the server. Repeating an idempotent request is safe; repeating a POST means a double order. I wrote that rule as a DelegatingHandler:

// Retries a failed request once, but only if repeating it cannot cause a second side effect
public sealed class IdempotentRetryHandler(int maxRetries = 1) : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        var response = await base.SendAsync(request, cancellationToken);
        for (var attempt = 0;
             attempt < maxRetries && response.StatusCode == HttpStatusCode.ServiceUnavailable && HttpMethodSemantics.IsIdempotent(request.Method);
             attempt++)
        {
            response.Dispose();
            response = await base.SendAsync(request, cancellationToken);
        }
        return response;
    }
}

The server has a /flaky endpoint that returns 503 on every other call. With the same client, first a GET, then a POST:

$ dotnet Bookstore.Net.Lab.dll retry http://127.0.0.1:5491/flaky
GET  -> 200 OK ok after 2 calls
POST -> 503 ServiceUnavailable

The GET got a 503 on the first attempt, the handler retried, and the second call succeeded. The POST got a 503 and wasn't retried. An xUnit test verifies that HttpMethodSemantics matches the table in RFC 9110. In a real project the Polly-based Microsoft.Extensions.Http.Resilience package does this job. Its standard retry strategy retries regardless of method; DisableForUnsafeHttpMethods() turns retries off for POST, PATCH, PUT, DELETE and CONNECT, a more cautious rule that also leaves out the idempotent PUT and DELETE. The details are in the upcoming External Service Integration and Resilience series. The way to make a POST safely retryable is an idempotency key: the client gives every order an ID, and the server doesn't create a new order when it sees the same ID a second time.

Persistent Connections, Head-of-Line Blocking and 6 Connections

In HTTP/1.0 a new TCP connection was opened for every request. In HTTP/1.1 connections stay open by default (persistent connections, commonly called keep-alive); unless one side sends Connection: close, the same connection is used for later requests. I sent two requests back to back over the same nc connection, asking for the connection to be closed on the second:

$ (printf 'GET /conn HTTP/1.1\r\nHost: localhost\r\n\r\nGET /conn HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n'; sleep 1) | nc 127.0.0.1 5491
HTTP/1.1 200 OK
...
{"id":"0HNPTG51MT3T4","remotePort":64506,"protocol":"HTTP/1.1"}
0

HTTP/1.1 200 OK
Connection: close
...
{"id":"0HNPTG51MT3T4","remotePort":64506,"protocol":"HTTP/1.1"}
0

Both responses carry the same connection ID and the same client port. The Connection: close in the second response says the server will honor the request and close the connection.

But in HTTP/1.1 a connection can carry only one response at a time. Sending the second request without waiting for the first response (pipelining) is in the spec, but responses must come back in order: a slow first response holds up everything behind it. This is called head-of-line blocking, and because of buggy proxies browsers never turned pipelining on. The browsers' answer is parallelism: opening several connections to the same host. Chrome and Firefox limit this to 6 connections per host; a page with 30 small files queues up in batches over HTTP/1.1. A good part of the "Queueing" and "Stalled" times in DevTools comes from this. That's the main problem HTTP/2 solves; the topic of the eighth post.

A Defensive Note: Request Smuggling

Having two ways to determine the end of the body creates a risk. If a request carries both Content-Length and Transfer-Encoding, a proxy in front and the server behind it may see the end of the body in different places. Then bytes the proxy took as one request are interpreted as two requests by the server, and the second, "hidden" request can get mixed into another user's connection. This class of attack is called request smuggling. RFC 9112 says that when both headers arrive together, Transfer-Encoding wins, and recommends treating such a request as suspicious and closing the connection. I checked how my own server handles it:

$ (printf 'POST /api/orders HTTP/1.1\r\nHost: localhost\r\nContent-Type: application/json\r\nContent-Length: 4\r\nTransfer-Encoding: chunked\r\n\r\n0\r\n\r\n'; sleep 1) | nc 127.0.0.1 5491 | cat -v
HTTP/1.1 400 Bad Request^M
Content-Length: 0^M
Connection: close^M
Date: Sun, 08 Nov 2026 10:28:31 GMT^M
Server: Kestrel^M
^M

Kestrel rejected the request with a 400 and closed the connection; it doesn't allow ambiguous framing. That's the core of the defense: every component in the chain (CDN (Content Delivery Network), load balancer, reverse proxy, application server) should reject ambiguous framing, and preferably the proxy and server should talk over a protocol like HTTP/2 that determines length with frames. Keeping your components up to date is the most effective measure here.

In Production

Decide retries per method. A resilience setting that automatically repeats non-idempotent calls produces double orders and double payments during a network glitch. Use an idempotency key for POST or turn retries off.

Be ready for chunked responses. With no Content-Length you can't know the total size up front; client code that shows a progress bar, enforces a size limit or sizes a buffer must take that into account.

Don't trust headers, validate them. Host, X-Forwarded-* and Content-Length are under the client's control. In ASP.NET Core fill in AllowedHosts, configure the Forwarded Headers middleware to trust only known proxies, and set request body limits (MaxRequestBodySize) deliberately.

Common Pitfalls

Writing line endings as \n. HTTP/1.1 requires CRLF; some servers are tolerant, but use \r\n in test requests you write by hand.

Forgetting the Host header. It's mandatory in HTTP/1.1; without it you get a 400.

Taking chunked for an error. The hexadecimal numbers in the body aren't part of the content but framing; HTTP libraries strip them for you.

Mixing up PUT and POST. Using PUT for "an update that adds something" makes a retry that expects idempotency dangerous.

Assuming keep-alive lasts forever. The server, a proxy or the client closes idle connections; the idle timeouts we saw in the third post apply here too.

A Small Experiment

★☆☆ Send a HEAD request to your own API with nc. Is there a Content-Length in the response, is there a body? Why?

★★☆ Over the same nc connection, send /slow?ms=500 and /whoami back to back without waiting for the first response. In what order and when do the responses arrive?

★★★ Extend IdempotentRetryHandler to retry POST requests too when they carry an Idempotency-Key header, and write an endpoint on the server that ignores a second order arriving with the same key. Verify both sides together with an xUnit test.

The requests we wrote by hand went all the way to the server and back every time. Yet most responses are the same again and again. In the next post we move to HTTP caching: Cache-Control, ETag, conditional requests and 304 Not Modified.