Concurrency, Idempotency and Long-Running Operations: ETag/If-Match, Idempotency-Key, 202 Accepted and Status Monitors

Version note: The code in this post was built and run with C# 14 and .NET 10 (ASP.NET Core 10.0.15); the HTTP examples are real responses from this application, and the OpenAPI 3.2 document was linted with Redocly CLI 2.78.0 and Spectral 6.17.1.

Guess First

In the back office screen of Bookstore, the online bookstore used as the shared example across this site's series, Ayşe opens the edit form for Dune and writes a description. Meanwhile the nightly price update job, which runs under the same user account, cuts Dune's price to 459 TRY. Ayşe saves the form. Here is what happens on the API as it was at the end of the previous post:

$ curl -s -i http://127.0.0.1:52318/books/9780441172719 | grep -E '^ETag|^\{'
ETag: "1"
{"isbn":"9780441172719","title":"Dune","author":"Frank Herbert","authors":["Frank Herbert"],"category":"science-fiction","price":{"amount":489.00,"currency":"TRY"},"inStock":true}

$ curl -s -X PATCH -H "Authorization: Bearer ayse" -H "Content-Type: application/merge-patch+json" -d '{"price":{"amount":459.00,"currency":"TRY"}}' http://127.0.0.1:52318/books/9780441172719 | jq -c '{title, price}'
{"title":"Dune","price":{"amount":459.00,"currency":"TRY"}}

$ curl -s -X PUT -H "Authorization: Bearer ayse" -H "Content-Type: application/json" -d '{"title":"Dune","author":"Frank Herbert","category":"science-fiction","price":{"amount":489.00,"currency":"TRY"},"description":"The first novel of the Dune saga."}' http://127.0.0.1:52318/books/9780441172719 | jq -c '{title, price, description}'
{"title":"Dune","price":{"amount":489.00,"currency":"TRY"},"description":"The first novel of the Dune saga."}

$ curl -s -i http://127.0.0.1:52318/books/9780441172719 | grep -E '^ETag'
ETag: "3"

No request failed, but the price cut is gone: the form wrote back the 489 TRY it had when it was opened. Since the third post the API has returned an ETag for every book and checked If-Match when it was sent. Where is the problem: in Ayşe's form, in the job, or in the contract? The same week, the "Place order" button in Ada's mobile app resends a request on its own when it times out. The first request had actually succeeded. What do these two problems have in common?

The Contract of Optimistic Concurrency

What the two problems have in common is a client writing without knowing the state on the server. In a lost update, where one write silently overwrites another write made in between, the client does not know whether the resource has changed since it read it. With the resent order, it does not know the outcome of the first request. HTTP's answer to the first one is the conditional request: the client sends back the ETag of the version it read in If-Match, and if the version has changed the server returns 412 Precondition Failed. The mechanism is defined in RFC 9110, HTTP Semantics, and Bookstore has had it since the third post. What was missing is that If-Match was optional. Neither the nightly job nor Ayşe's form sent the header, so the check never ran.

An optional protection does not protect the client you want to protect. RFC 6585 defines 428 Precondition Required for exactly this case: the server requires the request to be conditional, and preventing lost updates is the example given in the definition of this status code. Bookstore's new rule (BS-60): every PUT, PATCH and DELETE that changes an existing resource must send If-Match.

$ curl -s -X PATCH -H "Authorization: Bearer ayse" -H "Content-Type: application/merge-patch+json" -H 'If-Match: "1"' -d '{"price":{"amount":459.00,"currency":"TRY"}}' http://127.0.0.1:52318/books/9780441172719 | jq -c '{title, price}'
{"title":"Dune","price":{"amount":459.00,"currency":"TRY"}}

$ curl -s -X PUT -H "Authorization: Bearer ayse" -H "Content-Type: application/json" -d '{"title":"Dune","author":"Frank Herbert","category":"science-fiction","price":{"amount":489.00,"currency":"TRY"},"description":"The first novel of the Dune saga."}' http://127.0.0.1:52318/books/9780441172719 | jq -c '{status, title, detail}'
{"status":428,"title":"Precondition required","detail":"Send If-Match with the ETag from your last GET."}

$ curl -s -X PUT -H "Authorization: Bearer ayse" -H "Content-Type: application/json" -H 'If-Match: "1"' -d '{"title":"Dune","author":"Frank Herbert","category":"science-fiction","price":{"amount":489.00,"currency":"TRY"},"description":"The first novel of the Dune saga."}' http://127.0.0.1:52318/books/9780441172719 | jq -c '{status, title, detail}'
{"status":412,"title":"Precondition failed","detail":"If-Match \"1\" does not match the current ETag \"2\"."}

The job now sends the ETag it read and changes the price. If Ayşe's form sends no header it gets a 428, and if it sends the "1" it had when it was opened it gets a 412. What the Angular screen does on a 412 (a conflict screen that shows the current state and lets the user merge the two changes) and how the ETag is derived from the database's rowversion are covered in the Solution Patterns series post "Optimistic Concurrency End to End: From rowversion to ETag, If-Match, 412 and a Conflict UI". Three decisions remain on the contract side. First, creating a new book with PUT does not ask for If-Match, because there is no version to match. Second, adding 428 to the contract is a breaking change: every writing client that sends no header breaks. In Bookstore only the back office writes books, so we did it as a team in a single release without a new version; for a resource with external consumers this change would need a new api-version, as in post 06. Third, If-Match: * is still accepted; it means "write as long as the resource exists" and is the escape hatch for a deliberate overwrite.

Idempotency-Key: A Resent POST

Ada's order is a different problem. According to RFC 9110, PUT and DELETE are idempotent: sending the same request twice has the same effect as sending it once, so a client can safely resend it after a timeout. POST is not idempotent; a second POST /orders means a second order. And the client cannot know whether the first request reached the server and, if it did, whether it was processed. Idempotency-Key solves this: the client generates a unique key for each new operation and uses the same key when it resends the request. If the server sees the key for the first time it processes the request; if it has seen it before it returns the first result again.

There is no standard for the header yet. The draft of the IETF working group for HTTP APIs reached version -08 on 22 October 2026; on the datatracker its status is "Active Internet-Draft" and a working group document, so it is not an RFC and may still change. According to the draft the value is a Structured Field String, written in quotes, and the server decides and documents how long a key is kept. The draft also defines the error cases: 400 when a key is missing on an operation that requires one, 422 when the same key arrives with a different body, and 409 when the same key arrives while the first request is still being processed. Bookstore implements these as they are, with a problem type in the catalog for each:

$ KEY='"5f0c6e1a-8b2d-4c3e-9a71-2d4b8e6f1c90"'

$ curl -i -X POST -H "Authorization: Bearer ada" -H "Content-Type: application/json" -H "Idempotency-Key: $KEY" -d '{"items":[{"isbn":"9780441569595","quantity":1}]}' http://127.0.0.1:52318/orders
HTTP/1.1 201 Created
Content-Type: application/json; charset=utf-8
Date: Sat, 14 Nov 2026 23:22:24 GMT
Server: Kestrel
Content-Language: en
Location: /orders/01a1dc5e-871b-7891-a6d3-00a058028c9a
Transfer-Encoding: chunked
Api-Version: 2026-09-01
Deprecation: @1794700800
Sunset: Mon, 17 May 2027 00:00:00 GMT
Link: <https://api.example.com/docs/versions/2026-09-01>; rel="deprecation"; type="text/html"

{"id":"01a1dc5e-871b-7891-a6d3-00a058028c9a","status":"Pending","customerId":"ada","createdAt":"2026-11-14T23:22:24.923353Z","items":[{"isbn":"9780441569595","title":"Neuromancer","quantity":1,"unitPrice":{"amount":279.00,"currency":"TRY"}}],"total":{"amount":279.00,"currency":"TRY"}}

$ curl -s -X POST -H "Authorization: Bearer ada" -H "Content-Type: application/json" -H "Idempotency-Key: $KEY" -d '{"items": [{"isbn": "9780441569595", "quantity": 1}]}' http://127.0.0.1:52318/orders | jq -c '{id, status}'
{"id":"01a1dc5e-871b-7891-a6d3-00a058028c9a","status":"Pending"}

$ curl -s -X POST -H "Authorization: Bearer ada" -H "Content-Type: application/json" -H "Idempotency-Key: $KEY" -d '{"items":[{"isbn":"9780441569595","quantity":2}]}' http://127.0.0.1:52318/orders | jq -c '{status, title, detail}'
{"status":422,"title":"Idempotency-Key is already used","detail":"This Idempotency-Key was used with a different request body; use a new key for a new request."}

$ KEY2='"0d3f9b52-7a1e-4c86-b2f4-91e7c05a3d68"'

$ curl -s -o first.json -X POST -H "Authorization: Bearer ada" -H "Content-Type: application/json" -H "Idempotency-Key: $KEY2" -d '{"items":[{"isbn":"9780261102217","quantity":1}]}' http://127.0.0.1:52318/orders & sleep 0.3; curl -s -X POST -H "Authorization: Bearer ada" -H "Content-Type: application/json" -H "Idempotency-Key: $KEY2" -d '{"items":[{"isbn":"9780261102217","quantity":1}]}' http://127.0.0.1:52318/orders | jq -c '{status, title}'; wait; jq -c '{id, status}' first.json
{"status":409,"title":"A request with this Idempotency-Key is in progress"}
{"id":"01a1dc5e-8d2a-7717-bdb1-07294e620c02","status":"Pending"}

$ curl -s -X POST -H "Authorization: Bearer ada" -H "Content-Type: application/json" -d '{"items":[{"isbn":"9780441569595","quantity":1}]}' http://127.0.0.1:52318/orders | jq .
{
  "type": "https://api.example.com/problems/idempotency-key-missing",
  "title": "Idempotency-Key is missing",
  "status": 400,
  "detail": "This operation requires an Idempotency-Key header (a quoted string, e.g. a UUID); see the type URI.",
  "instance": "/orders",
  "traceId": "00-da58baa56ac4b046f7373f7c227114c9-9ea8c2fddddd9d14-00"
}

$ curl -s -H "Authorization: Bearer ada" "http://127.0.0.1:52318/orders?pageSize=3" | jq -c '.items[] | {id, status, total: .total.amount}'
{"id":"01a1dc5e-8d2a-7717-bdb1-07294e620c02","status":"Pending","total":349.00}
{"id":"01a1dc5e-871b-7891-a6d3-00a058028c9a","status":"Pending","total":279.00}
{"id":"01a1cfa1-0a00-7b74-ace2-8223a65ed389","status":"Confirmed","total":568.00}

The Deprecation and Sunset headers in the examples are the warning we give in post 06 to clients that send no version. The second request is the same JSON formatted with extra spaces, yet it returns the first order's id; the fingerprint we use to recognize a retry is computed from the bound request, not from raw bytes. The draft leaves the fingerprint algorithm to the server; a checksum of the whole payload works, and so does comparing selected fields. Had we chosen raw bytes, a client that formats the same order differently would get a 422. The parallel second request gets a 409, and when the first one finishes there is a single order; the list shows that too.

The rules we wrote into the contract (BS-61, BS-62): the key is required, scoped to the caller and the operation (Ada's key never collides with Grace's), and kept for 24 hours. Server errors are not stored: if the operation stopped half way with an exception, nothing was committed and the retry runs from the start. 4xx results are stored, though; a request that got a 422 for insufficient stock gets the same 422 when resent with the same key, even if stock has arrived in the meantime. This is a deliberate choice: the same key always gets the same answer. How the key table is kept in the database, the transaction boundary and its cleanup are covered in the Solution Patterns series post "Idempotency Keys: Not Creating the Same Order Twice on Double Clicks and Retries". The sending side, making idempotent calls to an external service and handling a request with an "unknown outcome", is covered in the External Service Integration and Resilience series. The security side of this header in sensitive business flows is in the API Security series post on sensitive business flows.

Long-Running Operations: 202 Accepted

Institutional customers want a PDF invoice for every order. Generating it means going to the accounting system for tax details, building the PDF and writing it to the archive; in Bookstore it takes a few seconds. Doing this inside a synchronous GET or POST means running into the 2-second timeout in the school's integration, into the connection time limits of gateways, and into the question "I got a timeout; was the work done?". 202 Accepted in RFC 9110 exists for this case: the request has been accepted but processing has not finished. The RFC says this status code is deliberately noncommittal and suggests that the response describe the request's status and point to a status monitor. HTTP does not say what a status monitor looks like; that is where the guidelines come in.

$ curl -s -i -X POST -H "Authorization: Bearer linus" http://127.0.0.1:52318/orders/01a1a612-9c80-7e46-8590-67e08b9d2434/invoices | tee accepted.txt
HTTP/1.1 202 Accepted
Content-Type: application/json; charset=utf-8
Date: Sat, 14 Nov 2026 23:22:40 GMT
Server: Kestrel
Content-Language: en
Retry-After: 1
Transfer-Encoding: chunked
Api-Version: 2026-09-01
Deprecation: @1794700800
Sunset: Mon, 17 May 2027 00:00:00 GMT
Link: <https://api.example.com/docs/versions/2026-09-01>; rel="deprecation"; type="text/html"
Operation-Location: /operations/01a1dc5e-c688-7d43-9846-be76e9816e1a

{"id":"01a1dc5e-c688-7d43-9846-be76e9816e1a","kind":"invoice-generation","status":"NotStarted","percentComplete":0,"createdAt":"2026-11-14T23:22:41.160705Z","lastUpdatedAt":"2026-11-14T23:22:41.160705Z"}

$ OP=$(tail -1 accepted.txt | jq -r .id); sleep 1

$ curl -s -i -H "Authorization: Bearer linus" http://127.0.0.1:52318/operations/$OP | grep -E '^(HTTP|Retry-After|\{)'
HTTP/1.1 200 OK
Retry-After: 1
{"id":"01a1dc5e-c688-7d43-9846-be76e9816e1a","kind":"invoice-generation","status":"Running","percentComplete":25,"createdAt":"2026-11-14T23:22:41.160705Z","lastUpdatedAt":"2026-11-14T23:22:41.913127Z"}

$ curl -s -X POST -H "Authorization: Bearer linus" http://127.0.0.1:52318/orders/01a1a612-9c80-7e46-8590-67e08b9d2434/invoices | jq -c '{status, type, operationLocation}'
{"status":409,"type":"https://api.example.com/problems/invoice-already-requested","operationLocation":"/operations/01a1dc5e-c688-7d43-9846-be76e9816e1a"}

$ sleep 3; curl -s -H "Authorization: Bearer linus" http://127.0.0.1:52318/operations/$OP | jq .
{
  "id": "01a1dc5e-c688-7d43-9846-be76e9816e1a",
  "kind": "invoice-generation",
  "status": "Succeeded",
  "percentComplete": 100,
  "createdAt": "2026-11-14T23:22:41.160705Z",
  "lastUpdatedAt": "2026-11-14T23:22:44.172468Z",
  "resourceLocation": "/orders/01a1a612-9c80-7e46-8590-67e08b9d2434/invoices/01a1dc5e-d249-7eba-be96-e00a01730945"
}

$ curl -s -D - -o invoice.pdf -H "Authorization: Bearer linus" "http://127.0.0.1:52318$(curl -s -H "Authorization: Bearer linus" http://127.0.0.1:52318/operations/$OP | jq -r .resourceLocation)" | head -3; file invoice.pdf
HTTP/1.1 200 OK
Content-Length: 650
Content-Type: application/pdf
invoice.pdf: PDF document, version 1.4, 1 pages

The design is based on the status monitor of the Azure guidelines: 202, an Operation-Location header with the monitor's address, Retry-After telling how many seconds to wait before the next poll, and the states NotStarted, Running, Succeeded, Failed and Canceled. Why a separate header rather than Location? RFC 9110 defines the meaning of Location as the created resource in a 201 and the redirect target in a 3xx; what it means alongside a 202 is undefined. The monitor gives the result of a finished operation in resourceLocation; while the operation is running every response carries Retry-After, and once it is done it does not. A second request for the same order does not generate a second PDF; it gets a 409 and the address of the running operation. This is a natural idempotency for invoicing: the rule "an order has one invoice" does the key's job.

Google's AIP-151 solves the same problem with a different shape: a long-running method returns a common Operation resource whose done field becomes true when it finishes, which carries the result in response or the failure in error, and progress in metadata, and every API uses the same Operations service. AIP-151's rule of thumb is 10 seconds: work that may take longer should be a long-running operation. The common idea of both approaches is to turn the state of the work, not the work itself, into a resource. Bookstore departs from Azure in one point: Azure wants a long-running create to start with PUT and a client-chosen id rather than with POST, which makes a resent request idempotent. Because we generate the invoice id on the server, we chose POST and got retry protection from the "one invoice per order" rule. We wrote this deviation into ADR-0005.

Failure, Cancellation and Retention

An operation can still fail after its 202, and that can no longer be reported with an HTTP status code; RFC 9110 also points out that HTTP has no way to send the status code of an asynchronous operation later. The failure lives inside the monitor as a Problem Details object. Failures known before the work starts (an invoice for a cancelled order, for example) get an ordinary error response instead of a 202; AIP-151 likewise wants errors that prevent an operation from starting to be returned as ordinary errors.

$ OP2=$(curl -s -X POST -H "Authorization: Bearer grace" http://127.0.0.1:52318/orders/01a18d45-6de0-707a-8dfe-2a2217fc695a/invoices | jq -r .id); sleep 3; curl -s -H "Authorization: Bearer grace" http://127.0.0.1:52318/operations/$OP2 | jq -c '{status, percentComplete, error}'
{"status":"Failed","percentComplete":25,"error":{"type":"https://api.example.com/problems/invoice-details-missing","title":"Invoice details missing","status":422,"detail":"Institution city-library has no tax number on file; ask your account manager to add it."}}

$ OP3=$(curl -s -X POST -H "Authorization: Bearer ada" http://127.0.0.1:52318/orders/01a06655-e3a0-7a3b-8199-c6b41c80317f/invoices | jq -r .id); sleep 1; curl -s -X POST -H "Authorization: Bearer ada" "http://127.0.0.1:52318/operations/${OP3}:cancel" | jq -c '{status, percentComplete}'
{"status":"Canceled","percentComplete":25}

$ curl -s -X POST -H "Authorization: Bearer ada" "http://127.0.0.1:52318/operations/${OP3}:cancel" | jq -c '{status, detail}'
{"status":409,"detail":"Operation 01a1dc5e-e26c-70e3-b95b-406f1f7f7779 has already ended with Canceled."}

$ curl -s -H "Authorization: Bearer ada" http://127.0.0.1:52318/operations/$OP | jq -c '{status, title}'
{"status":404,"title":"Not found"}

City Library has no tax number on file, and the job finds this out when it reaches the accounting system. The problem type inside error has the same shape as the others in the catalog; the client can show this failure with the same code the interceptor from post 04 uses. Cancellation follows the custom method rule from the second post, POST /operations/{operationId}:cancel: it stops running work and returns 409 on a finished one. In the last request Ada asks for the operation Linus started and gets a 404: the monitor is visible only to the person who started the operation, because it contains order and invoice details.

Retention is part of the contract too (BS-64). The Azure guidelines require the monitor to be kept for at least 24 hours after the work ends and this period to be documented; AIP-151 allows expired operations to be deleted and suggests 30 days as a rule of thumb. Bookstore deletes the monitor after 24 hours and returns 404. Having this period in writing matters: monitors kept forever turn into a storage problem, and an undocumented retention period turns into "it worked yesterday" complaints. Once a client has reached the result, it should keep resourceLocation, not the monitor's address.

In ASP.NET Core

Idempotency-Key is an endpoint filter added to POST /orders: it reads the key and a hash of the bound request, runs the handler on the first request and stores the returned IResult, and on a retry executes the same IResult again. Because endpoint filters run after model binding, you have to use the bound object in context.Arguments rather than reading the request body again; endpoint filters are covered in detail in the ASP.NET Core Web API series post "Minimal APIs in Depth Part 2: Route Groups, Endpoint Filters, the TypedResults Catalog and OpenAPI Metadata". Long-running work is written to a Channel<Operation> and a BackgroundService runs it in order; cancellation goes through a CancellationTokenSource tied to the work's CancellationToken. In the sample both the keys and the operations live in memory; they are lost when the application restarts or runs as several instances. In a real system the keys live in a database every instance sees, and the work in a durable queue.

Common Pitfalls

Keeping long work synchronous. Work left synchronous because "it usually takes 3 seconds" takes 30 seconds on a busy day; the client times out, resends, and the work is done twice. Work whose duration varies and depends on other systems should be designed with 202 from the start.

Keeping monitors forever or not documenting their lifetime. Every operation leaves a record behind; if the period is not documented, clients use the monitor's address as a permanent link.

Leaving If-Match optional. A concurrency check only works for clients that send the header, that is, the most careful ones. If the check is not required, the lost update comes from the script that sends no header.

Tying the idempotency key to its value only, not to the user. If the key is not scoped to the caller, two different users sending the same key may lead to one of them seeing the other's order.

Recognizing retries from the raw body. Different formatting of the same JSON, or a different property order, turns a real retry into a 422.

A Small Experiment

★☆☆ Write a small polling loop that respects Retry-After: it should start the invoice operation, poll the monitor until it reaches a terminal state and finally download the PDF. Where do you read the wait time from?

★★☆ Try what happens when the first request with a key stops half way with an exception: add a temporary throw to CreateOrder and send it twice. What does the second request get, and why?

★★★ Design the institutional bulk order approval (Linus approving 40 pending orders at once) as a long-running operation: which resource and method start it, what progress information does the monitor carry, and if it partly fails does it return an error, or does it finish successfully and say which orders were not approved?

The question in the last exercise is the topic of the next post: many operations in one request. We design schools sending a bulk order with a 500-line list, reporting partial success with a line-by-line report, delivering the result by polling or webhook, and writing rate limits down as a contract.