Mapping External Service Integration: Failure Modes, Anti-Corruption Layer and Choosing a Client (Hand-Written, Refit, Kiota)

Version note: The code in this post was built and run with .NET 10, WireMock.Net 2.31, Refit 16.4 and Kiota 1.38.

Guess First

During checkout Bookstore calls two external services: it asks the cargo carrier for a price, then asks the payment provider to charge the card. One Friday evening the payment provider slows down and every request takes 5 seconds. No request fails. What happens?

Second question: the carrier quietly ships a new version and the price field moves from AMT to PRICE.AMOUNT. You use a client generated from their OpenAPI document. Do you get an exception?

Third question: you sent the payment request, waited 30 seconds and got no answer. Was the customer's card charged?

The first answer is "nothing blows up, but every checkout takes 5 seconds, threads and connections pile up, and then everything slows down". The second is "usually not, the field just becomes null", and we will see that with real output shortly. The third is "you don't know", and that is the topic of post 06 in this series. This post is the map of the series: what can happen to an external service, where Bookstore draws the line, and how we write the client.

What Can Happen to an External Service

The Reliability Patterns post in the System Design series covered timeout, retry and circuit breaker as concepts. Here we look at the same ground from the point of view of the person writing the integration: the other side is outside our control, and whatever the SLA says, some days will be bad. We start by listing the failures we can expect and what Bookstore should do for each. This table is also the table of contents for the rest of the series:

The last row matters. A declined card is a business outcome, not a technical fault. Code that carries it as an exception into a retry loop tries the declined card three more times, and with some providers that is enough to get the card blocked for fraud. Retrying a service whose TLS certificate has expired is pointless as well: you get the same error within seconds. Separating transient failures from permanent ones comes before any resilience setting.

Bookstore's Outside World

Bookstore, the online bookstore used as the shared example across this site's series, talks to four external systems in this series: a payment provider that takes card payments, a cargo carrier with a pricing and shipment API, an e-invoice integrator, and an SMTP server that sends e-mail. Each carries a different risk:

We never connect to a real provider. Throughout the series every external service is a fake written with WireMock.Net, a library that opens an HTTP server on a port and works with rules like "when this request arrives, return that response". The Bookstore.FakeProviders project in the repo starts two fakes and takes their behavior from the command line. Here is the carrier fake:

// Legacy carrier: upper-case field names, prices as strings
public static void Configure(WireMockServer server, FakeMode mode)
{
    server.Reset();
    var quotes = Request.Create().WithPath("/api/v2/quotes").UsingGet();

    var response = mode switch
    {
        FakeMode.Error => Response.Create().WithStatusCode(500).WithBody("Internal Server Error"),
        FakeMode.Throttle => Response.Create().WithStatusCode(429).WithHeader("Retry-After", "2"),
        FakeMode.Maintenance => Response.Create().WithStatusCode(503).WithHeader("Retry-After", "600")
            .WithHeader("Content-Type", "application/json").WithBodyAsJson(new { ERR_CODE = "MAINTENANCE", ERR_MSG = "Service unavailable until 03:00 UTC" }),
        FakeMode.SchemaChanged => Response.Create().WithStatusCode(200)
            .WithHeader("Content-Type", "application/json").WithBodyAsJson(new { QUOTE_ID = "Q-77812", PRICE = new { AMOUNT = 89.90m, CUR = "TRY" }, SVC = "STD", ETA_DAYS = 2 }),
        _ => Response.Create().WithStatusCode(200)
            .WithHeader("Content-Type", "application/json").WithBodyAsJson(new { QUOTE_ID = "Q-77812", AMT = "89.90", CUR = "TRY", SVC = "STD", ETA_DAYS = 2 })
    };

    if (mode == FakeMode.Slow)
    {
        response = response.WithDelay(TimeSpan.FromSeconds(5));
    }

    server.Given(quotes).RespondWith(response);
}

We start the fakes and the API in two terminals:

dotnet run --project src/Bookstore.FakeProviders -- --carrier Maintenance
dotnet run --project src/Bookstore.Api

Fake payment provider : http://localhost:9091 (Normal)
Fake legacy carrier   : http://localhost:9092 (Maintenance)
Press Ctrl+C to stop.

There is a reason the fake sets Content-Type explicitly. My first version didn't; the Refit client worked fine, while the Kiota-generated client returned an empty model without any error. Kiota picks its deserializer by looking at Content-Type. The closer a fake is to the real thing, the more a test against it means, and this detail comes back in the tests in post 05.

Port and Adapter: Keeping the Outside Model Out

In the Context Mapping post of the Domain-Driven Design series we wrote an anti-corruption layer for the same carrier's old SOAP tracking service. The idea is the same here: the outside system's vocabulary doesn't enter Bookstore. In this series we focus on one part of that layer in more depth: the error model. First the port, Bookstore's own language for payments:

namespace Bookstore.Payments;

// The port: Bookstore's own language for taking a payment.
public interface IPaymentGateway
{
    Task<PaymentResult> ChargeAsync(PaymentRequest request, CancellationToken cancellationToken);
}

public sealed record PaymentRequest(int OrderId, decimal Amount, string Currency, string CardToken);

public abstract record PaymentResult
{
    public sealed record Approved(string ProviderPaymentId) : PaymentResult;

    public sealed record Declined(DeclineReason Reason) : PaymentResult;
}

Nothing in this interface belongs to the provider. The amount comes in as a decimal; the provider wants a long in minor units. There are no status, source or card_error fields. A declined card is not an exception but a PaymentResult.Declined: the checkout code handles it with a switch and moves the order to PaymentDeclined.

Technical failures arrive as our own exception type. What matters is that the exception tells the caller what to do, not just what went wrong:

public enum PaymentFailureKind
{
    // Retrying later may help (5xx, 429, connection refused).
    Transient,
    // The provider said "not now" for a long time (maintenance, circuit open).
    Unavailable,
    // The request may or may not have been processed (timeout after sending).
    UnknownOutcome,
    // The provider answered with something we cannot understand.
    ContractViolation
}

public sealed class PaymentGatewayException(PaymentFailureKind kind, string message, Exception? innerException = null)
    : Exception(message, innerException)
{
    public PaymentFailureKind Kind { get; } = kind;

    public TimeSpan? RetryAfter { get; init; }
}

Keeping UnknownOutcome separate is the most valuable part of this enum. If the connection was never established (HttpRequestException), no money moved and it is safe to try again. If the request went out and then timed out, the state is unknown, and showing the customer "payment failed" is the shortest route to charging the card twice on the second attempt.

Translating the Error Model

The adapter is the only class that knows the provider's JSON, status codes and error codes. It is written as a plain typed client that takes HttpClient in its constructor:

public sealed class ProviderPaymentGateway(HttpClient httpClient, ILogger<ProviderPaymentGateway> logger) : IPaymentGateway
{
    private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web)
    {
        PropertyNamingPolicy = JsonNamingPolicy.SnakeCaseLower
    };

    public async Task<PaymentResult> ChargeAsync(PaymentRequest request, CancellationToken cancellationToken)
    {
        var body = new ProviderChargeRequest(
            Amount: ToMinorUnits(request.Amount),
            Currency: request.Currency,
            Source: request.CardToken,
            Description: $"Bookstore order {request.OrderId}");

        HttpResponseMessage response;
        try
        {
            response = await httpClient.PostAsJsonAsync("v1/charges", body, Json, cancellationToken);
        }
        catch (HttpRequestException ex)
        {
            // Nothing reached the provider (DNS, connection refused, TLS): safe to try again later.
            throw new PaymentGatewayException(PaymentFailureKind.Transient, "Payment provider is unreachable.", ex);
        }
        catch (TaskCanceledException ex) when (!cancellationToken.IsCancellationRequested)
        {
            // The request left our process; we do not know if the card was charged.
            throw new PaymentGatewayException(PaymentFailureKind.UnknownOutcome, "Payment provider did not answer in time.", ex);
        }

        using (response)
        {
            return response.StatusCode switch
            {
                HttpStatusCode.Created or HttpStatusCode.OK => await ReadApprovedAsync(response, cancellationToken),
                HttpStatusCode.PaymentRequired => await ReadDeclinedAsync(response, cancellationToken),
                HttpStatusCode.TooManyRequests => throw new PaymentGatewayException(
                    PaymentFailureKind.Transient, "Payment provider is throttling us.")
                    { RetryAfter = response.Headers.RetryAfter?.Delta },
                HttpStatusCode.ServiceUnavailable => throw new PaymentGatewayException(
                    PaymentFailureKind.Unavailable, "Payment provider is unavailable.")
                    { RetryAfter = response.Headers.RetryAfter?.Delta },
                >= HttpStatusCode.InternalServerError => throw new PaymentGatewayException(
                    PaymentFailureKind.Transient, $"Payment provider returned {(int)response.StatusCode}."),
                _ => throw new PaymentGatewayException(
                    PaymentFailureKind.ContractViolation, $"Unexpected status {(int)response.StatusCode} from payment provider.")
            };
        }
    }

    // Provider contract: kept internal so it never leaks into the domain.
    private sealed record ProviderChargeRequest(long Amount, string Currency, string Source, string Description);

    private sealed record ProviderCharge(string Id, string Status, long Amount, string Currency);
    // ...
}

Note the catch (TaskCanceledException) when (!cancellationToken.IsCancellationRequested) line: HttpClient also throws TaskCanceledException when its own timeout elapses. The filter separates "the user cancelled" from "the provider didn't answer". We will look at that relationship with real output in the next post.

Exceptions are turned into HTTP responses at the edge of the API, inside an IExceptionHandler. The mechanics are in the Global Error Handling post of the ASP.NET Core Web API series; what matters here is the mapping. The customer never sees the provider's error message:

(int status, string title)? mapped = exception switch
{
    PaymentGatewayException { Kind: PaymentFailureKind.UnknownOutcome } =>
        (StatusCodes.Status202Accepted, "Payment is being verified."),
    PaymentGatewayException { Kind: PaymentFailureKind.Transient or PaymentFailureKind.Unavailable } =>
        (StatusCodes.Status503ServiceUnavailable, "Payment service is temporarily unavailable."),
    ShippingCarrierException { Kind: ShippingFailureKind.Transient or ShippingFailureKind.Unavailable } =>
        (StatusCodes.Status503ServiceUnavailable, "Shipping service is temporarily unavailable."),
    PaymentGatewayException or ShippingCarrierException =>
        (StatusCodes.Status502BadGateway, "An external service returned an unexpected response."),
    _ => null
};

Sending the checkout request from Bookstore.Api.http in the repo with the fakes in different modes gave these responses. With the carrier in maintenance:

HTTP/1.1 503 Service Unavailable
Content-Type: application/problem+json

{"type":"https://tools.ietf.org/html/rfc9110#section-15.6.4","title":"Shipping service is temporarily unavailable.","status":503,"traceId":"00-a0139010768c527b6a2e245fa2a405c0-2864805a92cd9ba7-00"}

With the payment provider in maintenance (--payments Maintenance) the provider's Retry-After: 600 is passed on, so the client knows when to try again. When the card is declined the response is 200 and the body carries the business outcome:

HTTP/1.1 503 Service Unavailable
Retry-After: 600

HTTP/1.1 200 OK
{"orderId":1001,"status":"PaymentDeclined","total":548.90,"providerPaymentId":null,"declineReason":"CardDeclined"}

In the schema-change mode we got a 502, and the log said exactly what broke:

warn: Bookstore.Infrastructure.ExternalServiceExceptionHandler[0]
      External service failure mapped to 502
      Bookstore.Shipping.ShippingCarrierException: Carrier quote could not be mapped: QUOTE_ID=Q-77812, AMT=null.

That may look like a bad result, but it isn't. The alternative is treating the price as 0 or null and completing the order with free shipping. The decimal.TryParse check in the mapping catches the contract change on the first request and raises an alert.

Choosing a Client: Hand-Written, Refit, Kiota

There are three common ways to write the HTTP code inside an adapter. For the carrier we used a hand-written typed client like the one above. Let's call the same API with Refit and Kiota and see how they behave when the schema changes. With Refit you write an interface and the source generator writes the implementation at compile time:

// Refit: we write the interface, the source generator writes the implementation.
public interface ILegacyCarrierApi
{
    [Get("/api/v2/quotes")]
    Task<RefitQuote> GetQuoteAsync(string from, string to, int weight, CancellationToken cancellationToken = default);
}

services.AddRefitGeneratedClient<ILegacyCarrierApi>()
    .ConfigureHttpClient(c => c.BaseAddress = baseAddress);

I fell into a trap here. My first attempt used AddRefitClient, as the Microsoft Learn samples do, and with Refit 16.4 the app stopped on the first call with this error:

Unhandled exception. System.NotSupportedException: This interface needs the reflection request builder, which is not installed. Add a reference to the Refit.Reflection NuGet package to opt in to it, or change the interface so every method generates inline (the RF006 diagnostic reports the methods that cannot) and use a generated client via RestService.ForGenerated or AddRefitGeneratedClient.

In recent versions the reflection-based request builder lives in a separate package. Registering the source-generated client with AddRefitGeneratedClient makes that package unnecessary and keeps the client compatible with trimming and Native AOT.

Kiota works the other way around: you have an OpenAPI document and generate the client from it. We installed Kiota into the repo's local tool manifest, so no global install is needed:

dotnet tool restore
dotnet kiota generate -l CSharp -d specs/legacy-carrier.openapi.yaml -c LegacyCarrierApiClient -n Bookstore.ResilienceLab.Generated.LegacyCarrier -o samples/Bookstore.ResilienceLab/Generated/LegacyCarrier

Generation completed successfully
Client base url set to http://localhost:9092

The generated client is a chain of builders that follows the segments of the URL:

var quote = await kiota.Api.V2.Quotes.GetAsync(q =>
{
    q.QueryParameters.From = "34000";
    q.QueryParameters.To = "06100";
    q.QueryParameters.Weight = 850;
});

We call both, first in normal mode and then in schema-change mode:

dotnet run --project samples/Bookstore.ResilienceLab -- clients

--- carrier mode: Normal
Refit : QUOTE_ID=Q-77812, AMT=89.90
Kiota : QUOTE_ID=Q-77812, AMT=89.90, unknown fields=[]
--- carrier mode: SchemaChanged
Refit : QUOTE_ID=Q-77812, AMT=null
Kiota : QUOTE_ID=Q-77812, AMT=null, unknown fields=[PRICE]

That answers the opening question: neither client threw. Refit left the missing field null. Kiota left it null too, but kept the unknown PRICE field in its AdditionalData dictionary, which at least leaves a trace you can log and alert on. Whichever client you pick, the code that asks "is this response valid for me" is your mapping. A generated client doesn't replace the anti-corruption layer; it is a tool used inside it.

A short rule for choosing: if there are only a few endpoints and the provider has no OpenAPI document, or one you can't trust, write a typed client by hand. If you want to design the interface yourself and the number of endpoints is moderate, use Refit. If the provider publishes a proper OpenAPI document with dozens of endpoints (e-invoice integrator APIs usually look like this), generate with Kiota or NSwag. NSwag is an older tool that also generates C# clients from OpenAPI; its output is a single file that uses HttpClient directly.

Keeping Generated Code in the Repo and Versioning

We committed the files Kiota generated. kiota-lock.json records which OpenAPI document, which Kiota version and which settings produced them, and dotnet kiota update regenerates with the same settings. Generating at build time is possible too, but then the build depends on an external file and a tool version. Committed generated code shows up as a diff in a pull request when the provider changes its OpenAPI document, and it goes through code review. That is the cheapest place to catch a schema change: in a pull request, not in production.

Keep the provider's API version inside the adapter as well, not in configuration. Our carrier client uses the api/v2/quotes path; moving to v3 means a new adapter and new tests, not an appsettings value. Some providers ask for the version in a header instead of the path; send that header from inside the adapter too, as a constant.

Production Notes

Measure at least three numbers for every external service: request count, error rate by status code, and latency distribution (p50, p99). Average latency misleads; p99 is the first to show that a provider is slowing down. We will see how to collect these in post 05.

Keep the provider's error body in the log, but never show it to the user. The 502 response above has only a generic title; what broke is in the log. Card tokens, API keys and full request bodies must stay out of the log. We will look at what .NET's HttpClient logging does about this by default in the next post.

When converting money to the provider's unit, make the rounding explicit. ToMinorUnits uses MidpointRounding.AwayFromZero; for 548.90 TRY the provider receives 54890. A rounding mistake here shows up months later in the accounting reconciliation.

Common Pitfalls

Using the provider's DTOs in the domain. Code that keeps a ProviderCharge as a property on the order entity needs a database migration when the provider changes. DTOs stay private or internal inside the adapter.

Carrying every failure with the same exception. If HttpRequestException is used for both "could not connect" and "got a 404", the code above it cannot decide whether to retry. Design failure kinds around the decision the caller has to make.

Making a declined card an exception. Carrying a business outcome as an exception pushes it into generic retry and circuit breaker logic. Declined cards raise the breaker's failure ratio, and the breaker then closes the road to a perfectly healthy provider.

Trusting the generated client and skipping validation. As the output above shows, a schema change doesn't produce an exception. Check required fields in the mapping.

A Small Experiment

★☆☆ Add a rule to the fake payment provider that returns the insufficient_funds code (for the tok_insufficient_funds token) and verify with Bookstore.Api.http that checkout returns InsufficientFunds.

★★☆ For LegacyCarrierClient.Map, consider an AMT value of "89,90" (with a comma). How does the current code interpret it? Write a test, look at the real result, then decide whether the behavior is correct.

★★★ Write the Kiota client as a second implementation of IShippingCarrier (KiotaLegacyCarrierClient). Log a warning when AdditionalData isn't empty. Add a comparison command that runs both implementations against the same fake.

In this post we registered clients with AddHttpClient without looking at what sits behind it. Next we go inside IHttpClientFactory: named, typed and keyed clients, handler lifetime, adding tokens with a DelegatingHandler, and how HttpClient.Timeout relates to CancellationToken.