Blog
Mert Özen's blog posts on software development, backend, frontend, and DevOps.
Retry and Idempotency in Background Jobs: Living with At-Least-Once
A background job runs at least once and sometimes twice; retries make that even more likely. Hangfire retries 10 times at growing intervals by default and ExceptOn takes permanent errors out of retries; in Quartz 4 a retry policy is declared explicitly. "Check, then send" produces two emails from two concurrent runs; claiming first with a unique key brings it down to one, and an Idempotency-Key header does the same end to end with an external API.
Retry and Redelivery in Consumers: Immediate, Delayed and Exponential Backoff
Handle transient failures with a short in-memory retry and longer waits through the broker; never retry permanent failures, send them to the DLQ. Real output shows nack-requeue looping 8036 times in 5 seconds on RabbitMQ 4.4, TTL + DLX wait queues, the quorum queue's built-in delayed retry, scheduled retry in Service Bus, Kafka retry topics and MassTransit UseMessageRetry.
Reliability Patterns: Timeouts, Retries, Backoff with Jitter, Circuit Breakers and Load Shedding
Against a slow or failed dependency you need, in order, a timeout chosen from p99.9, jittered retries at a single layer and only for idempotent operations, a retry budget, a circuit breaker and bulkhead per dependency, and finally load shedding and graceful degradation. Real simulation output for retry amplification (27 and 243 calls), jitter, a 10% retry budget and a circuit breaker, failover, and .NET's standard resilience handler defaults.