HTTP Clients in Go: http.Client, Timeouts, Cancellation with context, Reusing the Transport and Retries

Version note: The code in this post was built and run with Go 1.27.4; the Bookstore module's go line is 1.26.0, and the code also compiles and passes its tests with Go 1.26.11. The connection reuse measurement shows a behaviour change in Go 1.27, so it was run with both versions. In this post, Bookstore.Api's stock endpoint is represented by a small stand-in server written in Go; the .NET service itself was not run.

Guess First

The Go service of Bookstore, the online bookstore used as the shared example across this site's series, gets stock data from Bookstore.Api on the .NET side. A lab sends 50 requests in a row with the same shared http.Client and counts on the server side how many new TCP connections were opened. In the second and third run calls, the response body is only Closed, never read:

same := func() *http.Client { return shared }
readAndClose := func(r *http.Response) { io.Copy(io.Discard, r.Body); r.Body.Close() }
closeOnly := func(r *http.Response) { r.Body.Close() }

run("read + Close, 1 KB", same, "small", readAndClose)
run("Close without reading, 1 KB", same, "small", closeOnly)
run("Close without reading, 1 MB", same, "big", closeOnly)

How many connections are opened with Go 1.26.11 and with Go 1.27.4?

$ GOTOOLCHAIN=go1.26.11 go run ./labs/19-reuse
read + Close, 1 KB                  1 connections,  +3 goroutines
Close without reading, 1 KB        50 connections,  +0 goroutines
Close without reading, 1 MB        50 connections,  +0 goroutines
...
$ go run ./labs/19-reuse      # Go 1.27.4
read + Close, 1 KB                  1 connections,  +3 goroutines
Close without reading, 1 KB         1 connections,  +3 goroutines
Close without reading, 1 MB        50 connections,  +0 goroutines
...

In Go 1.26, every body closed without being read closed the connection too, and every request started with a new TCP connection. With the change mentioned in post 14, Go 1.27 reads small bodies itself during Close and puts the connection back into the pool; with a 1 MB body it still closes the connection. This post covers writing HTTP clients in Go: http.Client and Transport, which timeout covers what, cancellation with context, bodies and connection reuse, retries and testing. At the end there is an inventory.Client that calls Bookstore's stock service.

http.Client, Transport and the Defaults

http.Client is a thin layer: it handles redirects, cookies and the total time. The real work is done by http.Transport; the connection pool, TLS, proxies and HTTP/2 live in it. If you think in .NET terms, Client ≈ HttpClient and Transport ≈ SocketsHttpHandler. The defaults are:

• http.DefaultClient (and http.Get) come with Timeout: 0, so there is no limit on the total time.
• http.DefaultTransport allows 30 s to connect and 10 s for the TLS handshake, and keeps idle connections for 90 s; it has no limit for waiting for the response headers.
• MaxIdleConnsPerHost defaults to 2: at most two idle connections are kept per server.

The Go counterpart of the old new HttpClient() trap in .NET is creating a new Transport for every request; the right way is to create the client once and share it between all goroutines. http.Client and Transport are designed for use from many goroutines at once. Compared with the handler lifetime covered in the IHttpClientFactory post of the External Service Integration and Resilience series, there is one difference: Go's Transport has no setting that limits the maximum lifetime of a connection, and IdleConnTimeout only closes connections that sit idle.

Timeouts: Who Counts Which Phase?

An HTTP request goes through several phases, and in Go each phase has its own setting:

I tried the settings in labs/19-timeouts against a stand-in stock service that answers slowly. slow-headers waits 2 seconds before it starts the response; slow-body sends the headers at once and writes the body 2 seconds later:

fmt.Println("http.DefaultClient.Timeout =", http.DefaultClient.Timeout)

fmt.Println("Client.Timeout = 1s:")
c := &http.Client{Timeout: time.Second}
get(c, context.Background(), base+"slow-headers")
get(c, context.Background(), base+"slow-body")

fmt.Println("Transport.ResponseHeaderTimeout = 1s:")
t := http.DefaultTransport.(*http.Transport).Clone()
t.ResponseHeaderTimeout = time.Second
c = &http.Client{Transport: t}
get(c, context.Background(), base+"slow-headers")
get(c, context.Background(), base+"slow-body") // not covered: the headers were fast

fmt.Println("context.WithTimeout 1s, no client timeout:")
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
get(http.DefaultClient, ctx, base+"slow-body")

$ go run ./labs/19-timeouts
http.DefaultClient.Timeout = 0s
Client.Timeout = 1s:
  no response                 after 1s: Get "http://127.0.0.1:52716/api/stock/slow-headers": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  200 OK                      after 1s: context deadline exceeded (Client.Timeout or context cancellation while reading body)
Transport.ResponseHeaderTimeout = 1s:
  no response                 after 1s: Get "http://127.0.0.1:52716/api/stock/slow-headers": net/http: timeout awaiting response headers
  200 OK                      after 2s: <nil>
context.WithTimeout 1s, no client timeout:
  200 OK                      after 1s: context deadline exceeded
a 500 is not an error:
  500 Internal Server Error   after 0s: <nil>
Transport.TLSHandshakeTimeout = 1s:
  no response                 after 1s: Get "https://127.0.0.1:52717/api/stock/9780441172719": net/http: TLS handshake timeout

Three things stand out. Client.Timeout also counts while the body is read: Do returned successfully, a 200 arrived, and the error came from io.ReadAll. ResponseHeaderTimeout, on the other hand, stops once the headers arrive; with the slow body the request took 2 seconds and no timeout fired. context.WithTimeout covers the same span as Client.Timeout; the difference is that it belongs to the caller: r.Context() in a handler is also cancelled when the client drops the connection, and that cancellation travels down to the call to the external service. For the TLS line I used a TCP server that accepts the connection and never answers. I did not try net.Dialer.Timeout; that needs an address on the local machine that does not answer.

As a rule, use three layers: a short Transport timeout for each phase (to notice a slow connection early), Client.Timeout to limit a single attempt, and a context as the budget for the whole job, retries included.

Closing the Body and Connection Reuse

The full output of the lab from Guess First:

$ go run ./labs/19-reuse
read + Close, 1 KB                  1 connections,  +3 goroutines
Close without reading, 1 KB         1 connections,  +3 goroutines
Close without reading, 1 MB        50 connections,  +0 goroutines
new Transport per request          50 connections, +150 goroutines
no Close at all, 1 KB              50 connections, +150 goroutines
100 requests, 20 at a time, MaxIdleConnsPerHost=0   92 connections
100 requests, 20 at a time, MaxIdleConnsPerHost=20  20 connections

"+3 goroutines" is one open connection: the client's read and write goroutines plus the server's connection goroutine. In the Go 1.27.4 source code, the limit for this draining is 256 KB and 50 ms; with a larger or slower body the connection is still closed. A new Transport per request and never calling Close are the two worst cases: 50 requests left 50 connections and 150 goroutines behind, and they stay until the program ends or the server closes the connection. In a long-running service that means running out of file descriptors. The last two lines show that a service sending 20 requests at once makes almost every request on a new connection with the default MaxIdleConnsPerHost=2; only two connections can go back to the pool. The rule has not changed: always defer resp.Body.Close(), and read the body to the end when you are done or limit its size with io.LimitReader.

Bookstore: inventory.Client

The new internal/inventory package puts the rules above in one place. The client is created once, and every phase's timeout is written out explicitly:

// NewClient returns a Client with explicit timeouts for every phase.
func NewClient(baseURL string) *Client {
    transport := &http.Transport{
        Proxy:                 http.ProxyFromEnvironment,
        DialContext:           (&net.Dialer{Timeout: 2 * time.Second, KeepAlive: 30 * time.Second}).DialContext,
        TLSHandshakeTimeout:   3 * time.Second,
        ResponseHeaderTimeout: 3 * time.Second, // from the end of the request to the response headers
        MaxIdleConns:          100,
        MaxIdleConnsPerHost:   32, // the default is 2
        IdleConnTimeout:       90 * time.Second,
        ForceAttemptHTTP2:     true,
    }
    return &Client{
        BaseURL: baseURL,
        HTTP:    &http.Client{Transport: transport, Timeout: 5 * time.Second}, // one attempt, body included
        Retries: 2,
        Backoff: 100 * time.Millisecond,
    }
}

func (c *Client) getJSON(ctx context.Context, path string, dst any) error {
    req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.BaseURL+path, nil)
    if err != nil {
        return err
    }
    req.Header.Set("Accept", "application/json")
    resp, err := c.HTTP.Do(req)
    if err != nil {
        return err // no answer: connection, timeout or cancellation
    }
    defer resp.Body.Close() // always, or the connection never goes back to the pool

    switch {
    case resp.StatusCode == http.StatusNotFound:
        return ErrNotFound
    case resp.StatusCode != http.StatusOK: // a 4xx or 5xx is not an error for http.Client
        body, _ := io.ReadAll(io.LimitReader(resp.Body, 256))
        return &StatusError{Code: resp.StatusCode, Body: string(body), RetryAfter: retryAfter(resp.Header)}
    }
    if err := json.NewDecoder(io.LimitReader(resp.Body, 1<<20)).Decode(dst); err != nil {
        return fmt.Errorf("decode stock level: %w", err)
    }
    return nil
}

For http.Client, a 404 or a 500 is not an error but a response; err is set only when no response arrived at all. Checking the status code is therefore your job. An unexpected response comes back as a *StatusError with the first 256 bytes of the body; the caller can look at the status code with errors.As, and a 404 becomes the ErrNotFound sentinel. url.PathEscape keeps the path intact even if the ISBN contains characters such as /; NormalizeISBN from post 15 already rejects those, but the client does not trust its caller.

Retries, Backoff and Idempotency

A retry is safe only if sending the request again gives the same result, that is, if the request is idempotent. GET is; a POST that creates an order is not. Stock retries errors that may be temporary two more times:

// retry runs call until it succeeds, fails for good, or ctx ends.
func (c *Client) retry(ctx context.Context, call func() error) error {
    wait := c.Backoff
    for attempt := 0; ; attempt++ {
        err := call()
        if err == nil || attempt == c.Retries || !temporary(err) || ctx.Err() != nil {
            return err
        }
        d := wait / 2
        if wait > 0 {
            d += rand.N(wait) // jitter: callers that failed together do not retry together
        }
        var se *StatusError
        if errors.As(err, &se) && se.RetryAfter > d {
            d = se.RetryAfter // the server knows best how long to wait
        }
        select {
        case <-time.After(d):
        case <-ctx.Done():
            return errors.Join(err, context.Cause(ctx))
        }
        wait *= 2
    }
}

// temporary reports whether another attempt may succeed.
func temporary(err error) bool {
    var se *StatusError
    if errors.As(err, &se) {
        switch se.Code {
        case http.StatusRequestTimeout, http.StatusTooManyRequests, http.StatusInternalServerError,
            http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:
            return true
        }
        return false
    }
    var ue *url.Error // what http.Client.Do returns for transport errors
    return errors.As(err, &ue)
}

The wait doubles with every attempt, and a random jitter is added, so a hundred clients that failed at the same moment do not retry in the same millisecond. If the server sent Retry-After, the client waits as long as it says. When the context ends, the wait is cut short. labs/19-retry connects to a server that returns a 503 first, then a 429 with Retry-After: 1:

$ go run ./labs/19-retry
  server: call 1 at 0s
  server: call 2 at 70ms
  server: call 3 at 1.07s
{ISBN:9780441172719 Available:12 Warehouse:IST-1} <nil> after 1.07s
  server: call 1 at 0s
  server: call 2 at 80ms
stock service: 429 Too Many Requests: "slow down\n"
context deadline exceeded after 500ms

The second call had a 500 ms budget; the client started to wait the one second Retry-After asked for, stopped when the context ended, and returned both errors together with errors.Join. Go's Transport also does a small retry of its own: if a connection taken from the pool was closed by the server and the request can be sent again, it sends the request once more on a new connection. In the source code this decision is in Request.isReplayable: GET, HEAD, OPTIONS and TRACE requests without a body or with a GetBody, plus requests that carry an Idempotency-Key header. The same idea is used to retry a POST safely: the client generates a key, and the server does not process a second request with the same key again. In .NET, Polly and Microsoft.Extensions.Http.Resilience do this work; pieces such as the circuit breaker and rate limiter from the Polly v8 post of the External Service Integration and Resilience series are not in Go's standard library. If you need them, you keep them small and write them yourself, or you pick a library.

Testing with httptest

In post 12 we tested handlers with httptest; a client test uses it the other way round: instead of a real server, a server started inside the test behaves like Bookstore.Api:

// stub answers like Bookstore.Api; fail lists the status codes of the
// first answers.
func stub(t *testing.T, fail ...int) (*Client, *atomic.Int32) {
    t.Helper()
    var calls atomic.Int32
    mux := http.NewServeMux()
    mux.HandleFunc("GET /api/stock/{isbn}", func(w http.ResponseWriter, r *http.Request) {
        if n := int(calls.Add(1)); n <= len(fail) {
            w.Header().Set("Retry-After", "0")
            http.Error(w, "try again", fail[n-1])
            return
        }
        if r.PathValue("isbn") != "9780441172719" {
            http.NotFound(w, r)
            return
        }
        w.Header().Set("Content-Type", "application/json")
        w.Write([]byte(`{"isbn":"9780441172719","available":12,"warehouse":"IST-1"}`))
    })
    srv := httptest.NewServer(mux)
    t.Cleanup(srv.Close)
    c := NewClient(srv.URL)
    c.Backoff = time.Millisecond
    return c, &calls
}

$ go test -v -run 'TestStock$' ./internal/inventory
=== RUN   TestStock
...
--- PASS: TestStock (0.01s)
    --- PASS: TestStock/ok (0.00s)
    --- PASS: TestStock/two_503_then_ok (0.01s)
    --- PASS: TestStock/429_then_ok (0.00s)
    --- PASS: TestStock/three_503:_retries_used_up (0.00s)
    --- PASS: TestStock/400_is_not_retried (0.00s)
PASS

The tests reduce the wait to 1 ms and check the number of retries with a counter on the server side. A third test calls a server that never answers with a 300 ms context and checks that the client stops within that time. If you want full control over time, Go 1.27's httptest.NewTestServer works together with synctest from post 12.

Production Notes

Write the duration and result of every external call to the log and to metrics. Adding status, attempt and duration_ms to the slog line from post 13 is the easiest way to tell an external service slowing down from your own service slowing down; the details of metrics are in the Observability series.

Do not multiply retries across layers. If the API gateway tries 3 times, the Go service 3 times, and the .NET service 3 times against the database, a single failure turns into 27 requests. Retry in one layer only and limit the total time with a context.

Size the connection pool for the load. If you send 20 concurrent requests to the same service, set MaxIdleConnsPerHost to at least that; otherwise every request does the TCP and TLS handshakes from scratch. If the server closes idle connections sooner than the client's IdleConnTimeout, connections in the pool die silently; Go fixes that by itself for requests that can be sent again, and you see an error for the others.

Common Pitfalls

Going to production with http.Get or http.DefaultClient. There is no timeout; a service that does not answer keeps goroutines waiting forever.

A new http.Client and Transport for every request. The connection pool is lost, every request opens a new connection, and the old ones stay idle.

Forgetting resp.Body.Close(). A connection and two goroutines leak per request; whenever err == nil, always defer resp.Body.Close().

Not treating 4xx and 5xx as errors. err == nil only says a response arrived; decoding the body without checking the status code means trying to read an error page as stock data.

Retrying a POST blindly. If the server processed the first request but could not send the response, the second attempt creates a second order; either do not retry, or use an Idempotency-Key.

A Small Experiment

★☆☆ Add a 300 KB endpoint to labs/19-reuse and see how many connections Go 1.27.4 opens for a body closed without being read. Explain the result with the 256 KB limit in the source code.

★★☆ Add a log/slog line to inventory.Client for every attempt (attempt, status, duration_ms), and make retryAfter also read the HTTP-date form with http.ParseTime. Add a row to the table test.

★★★ Connect the stock check of orders.Service to inventory.Client: if a BOOKSTORE_INVENTORY_URL setting exists, read the stock of every line from Bookstore.Api before placing the order; otherwise use today's catalog. When the stock service does not answer, should the order get a 503, or continue with local stock? Write down your decision and test it with the fakes from post 12.

With this post, the Backend with Go series comes to an end. In nineteen posts we started from Go's philosophy and toolchain and went through types, interfaces, error handling, generics, goroutines and channels, a framework-free API with net/http, JSON, SQL Server, testing, production and upgrading releases; the last five posts covered the standard library packages you use every day: strings, time, collections, files and the HTTP client. Bookstore's Go service is now a small, single-binary application that takes orders, reduces stock, turns its log into a report, and calls another service safely. If you want to go further, the gRPC and GraphQL series covers communication between services, the Observability series adds metrics and traces next to the logs, and the Docker and Kubernetes series cover running this binary in production in detail. The best way to try Go as a second backend language is to take the code from this series and write a small side service in your own team in Go.