Looking Back: Distributed Monolith Symptoms, Costs and What We Would Merge Again

Version note: The measurements in this post were taken with .NET 10, Aspire 13.6 and the code at the end of post 14 of this series; for comparison, the site's reference Bookstore monolith (a single Bookstore.Api) was run in the same SQL Server container.

Guess First

Over fourteen posts we split Bookstore, the online bookstore used as the shared example across this site's series, into six services, a BFF, four databases and a message broker. Compared with a monolith doing the same catalog, cart and order work, how many times more memory do the application processes use? How much longer does the order request take? And most important: is this system really made of services that can be deployed independently, or has it turned into the distributed monolith we described in the last post of the Microservices series? Every number in this post comes from a real measurement; the estimates are marked as estimates.

A Distributed Monolith Audit

In the "Monolith'ten Microservice'e" post of the Microservices series we listed the symptoms of a distributed monolith: services deployed together, shared code that keeps growing, requests travelling from service to service in synchronous chains, a shared database. We audited the system we built against that list, not by opinion but with the repository's history and the traces from post 12.

Changing together. We counted, with git diff-tree, which service folders each of the 16 commits in the repository touched. The posts that build a service's functionality naturally touch several, but the interesting ones are the contract changes: the commits of posts 06 (integration events), 07 (the saga) and 10 (CustomerId in events) each changed three to five services together. Adding a field to an event was not a breaking change (the rule from post 06), but the producer that fills the field and the consumers that use it changed together for the same feature. What keeps this from turning into a deployment lock is the order: the producer adds the field first, the consumers use it later. The schema test from post 13 is the tool that protects that order.

A finding we did not expect: Directory.Packages.props changed in 11 of the 16 commits. Central Package Management keeps every package version in one file, and every service workflow from post 14 watches that file in its paths filter; so adding a package to one service runs every service's pipeline. We share no code, but we share a build input. This is not a mistake but a choice: the same versions in every service, a security patch in one line. The price is six pipelines for every package change.

Shared code. The rule from post 01 limited shared code to two projects, and the architecture test checked it on every build. The measurement shows that the rule held:

Contracts consists only of records and hardly grew after post 07; the only additions to ServiceDefaults were the trace and metric lines from post 12 and the health endpoint change from post 14. There is no "shared business library".

Synchronous chains. In the checkout trace from post 12, the only synchronous step the user waits for is from the BFF to Ordering; stock, payment and notification move through messages. The gRPC call from Ordering to Inventory in post 05 was removed in post 07. The remaining synchronous calls feed screens: the book page calls Catalog and Inventory in parallel and still loads when Inventory does not answer (post 09). Payments' call to the payment provider is synchronous, but it goes outside and was designed for unknown results (post 08). There is no shared database: locally there are four separate databases in one SQL Server container, and no service knows another service's connection string.

The Cost: Local Resources

We measured with the whole system running under aspire run, after one checkout (docker stats for the Docker containers, resident memory for the .NET processes):

Starting the system took 53 seconds when the containers were persistent (a minute in the end-to-end test from post 13, where the containers were created from scratch). We saw the effect of these numbers throughout the series: when Docker's VM was shared with other containers, SQL Server dropped orders because it ran out of memory (post 11), and the SQL Server on kind restarted three times for lack of memory (OOMKilled, post 14). Keeping this system running on a developer's laptop takes a large share of a 16 GB machine.

The Cost: Latency and Code

For comparison we ran the site's reference Bookstore monolith (a single Bookstore.Api with Catalog, Inventory and Ordering modules) against its own database on the same SQL Server and placed 20 orders on both sides (excluding the first three warm-up requests; the measuring tool for the microservice side is tools/measure/checkout-latency.py in the repository, the commit analysis tools/measure/commit-areas.sh):

monolith POST /api/orders: n=20 p50=5.8 ms p95=15.1 ms status={'Placed'}
monolith RSS 146 MB

microservices POST /api/orders: n=20 p50=22.7 ms p95=431.0 ms
outcomes: {'Confirmed': 20}
microservices POST to Confirmed: p50=1212 ms p95=2139 ms

This is not a comparison of exactly the same work, and that has to be said plainly. The monolith reserves stock synchronously in one transaction and the order becomes Placed; there is no payment, authentication, e-mail or live notification. On the microservice side there is the BFF, a cookie session, token validation and the outbox, and the confirmation includes the 1-second processing time the fake provider adds on purpose; without it our own chain takes about 200 ms; in post 07, with the system idle, we measured 0.13–0.15 seconds. Still, the numbers point in a direction: the request the user waits for got about four times slower and its tail got much longer (15 ms against 431 ms at p95), and the memory of the application processes went from 146 MB to about 1.1 GB, 7.4 times as much.

The code side: the monolith's src is 813 lines of C# without migrations, in one project. The microservice version is 3,767 lines of C# in 62 files and 11 projects; on top of that come 610 lines for the AppHost, the Kubernetes manifests and the workflows, and 1,179 lines of tests. Most of the extra code is not functionality but distance: the outbox, the saga, contracts, token forwarding, retry rules, migrators.

CI Duration and Operational Load

A clean Release build of the whole solution takes 11 seconds (with the packages cached), the 56 tests 82.7 seconds; the end-to-end test almost alone sets the test duration (post 13). An image build takes 8–10 seconds per service. These are small numbers; the real load is in the number of parts that have to be operated:

• 6 deployed services, 4 migration Jobs and 6 images; one workflow per service and a shared reusable workflow.
• 4 databases, 4 exchanges, 2 command queues, 3 consumer queues and 4 dead letter queues.
• An identity provider, two clients and their secrets; a webhook secret; RabbitMQ, SQL Server and Redis passwords.
• New kinds of failure to watch: stuck sagas (post 07), payments with an unknown result (post 08), dropped WebSockets (post 11), payments whose webhook never came (post 12).

An estimate of the cloud bill, not run and without prices; prices vary by region and provider. According to the manifests from post 14, 100m CPU and 192 MiB of requests per service and two replicas mean 1.2 vCPU and about 2.3 GiB of requests in total for six services; the identity provider and the observability infrastructure come on top. Next to that come a managed SQL Server (four databases on one server or in an elastic pool), a managed RabbitMQ, a managed Redis, and storage for traces, logs and metrics. The same table for the monolith: one application, one database. Most of the bill is set not by the application pods but by the number of managed components and the volume of observability data. The calculation itself is in the Cloud Fundamentals series.

What Would We Merge Again?

Notifications into the BFF. Notifications has two jobs: pushing to the browser and sending e-mails. For the push, the browser is already connected to the BFF, and the BFF carries hub traffic to Notifications through YARP (post 11); that is one more network hop, deployable and pipeline, and one more proxy on the WebSocket's path. With the hub in the BFF, the BFF would have to connect to RabbitMQ; we would accept that. E-mail can stay a separate worker or become an e-mail provider's queue. If we started again today, we would not open Notifications as a separate service.

We would rethink Inventory's gRPC. In post 05 we set up gRPC for checkout's synchronous call; in post 07 checkout moved to messages, and today gRPC remains only for the book page's stock query. For that we have an HTTP/2-only port, a shared proto, call credentials without TLS and the token delay we saw in post 12. A single HTTP endpoint that shows stock on the book page, or a stock summary fed to Catalog through events, would be simpler.

The boundaries that proved right: Payments should stay separate. It talks to an external provider, has its own secret, webhook and reconciliation, and must keep the PCI scope small; we paid the price of separating it in post 08 and got the benefit. Ordering and the saga belonging together was also right: the owner of the workflow is the owner of the order. Catalog is read-heavy, and scaling it independently makes sense. In post 01 we deliberately left Shipping as a module, and not once in this series did we need to split it; the "not every module is a service" decision held.

The order from the "Module'ü Service'e Çıkarmak" post of the Modular Monolith series applies here too: to split a module, first look for a concrete reason such as independent scaling, a separate team or a separate security boundary. In Bookstore we found that reason most clearly in Payments.

Before Going to Production: What Is Missing

What we left out during the series, on purpose or for lack of time, and what has to be closed before Bookstore is opened to real customers:

• Token refresh and a server-side session: the access token gives 401 after 10 minutes (measured in post 11), the cookie comes in three chunks (post 10).
• An Idempotency-Key on Ordering's POST: a request repeated after a network error opens a second order (posts 04 and 11).
• Creating Wolverine's tables in the migration Job; creating the schema at startup caused problems in posts 07 and 14.
• Backup and recovery: backups of the four databases, a restore drill, and RPO/RTO targets for how much data loss and downtime we accept.
• SLOs and alerts: checkout duration (the histogram from post 12), stuck sagas, DLQ depth, payments with an unknown result.
• Runbooks: reprocessing a message from the DLQ, closing a stuck saga, rotating the webhook secret.
• NetworkPolicy, a RabbitMQ user per service, image signing and a secret store (posts 10 and 14).

How to write alerts and runbooks is covered in the SRE and Production Incidents series.

In Production

Measure, then decide. The numbers in this post come from a laptop; in production the latency distribution, the memory and the bill will be different. But the method is the same: measuring distributed monolith symptoms from the repository's history and from traces is more reliable than opinion.

Merging is a migration too. Moving Notifications into the BFF requires both the old and the new path to work between two deployments: first the BFF opens the hub and listens to the events, Angular switches to the new address, then Notifications is shut down. The strangler in reverse.

Common Pitfalls

Counting microservices as a measure of success. The value of six services lies in six teams deploying without waiting for each other; Bookstore's team is three developers.

Forgetting shared build inputs. Even without shared code, a single package file, a single pipeline template or a single base image can move every service together.

Comparing with unfair numbers. Saying "7 times the memory" without accounting for functionality the monolith lacks (payment, identity, notifications) is incomplete; you have to separate which difference comes from distribution and which from new functionality.

Closing the way back. Splitting a service is easy, merging it is hard; when you separate contracts and databases, keep in mind that you may merge one day.

A Small Experiment

★☆☆ In your own repository, count how many service folders each of the last 50 commits touched (git diff-tree --no-commit-id --name-only -r). What do the commits that touch more than three services have in common?

★★☆ Move Notifications' hub into the BFF: the BFF listens to OrderConfirmed and OrderCancelled from RabbitMQ and the /hubs route in YARP goes away. Repeat the browser run from post 11; how did the push latency and the system's memory change?

★★★ Add this series' payment flow to the monolith version of Bookstore as a module: a Payments module, the same fake provider, the same webhook and the same reconciliation, but a single transaction and a background job instead of the outbox and the saga. Compare lines of code, memory and order latency with the measurements in this post.

This is where the series ends. We split Bookstore by drawing its boundaries first, then built the services, messages, saga, payment flow, gateway, identity, interface, traces, tests and deployment, and finally measured the price. Before you set out on a similar path in your own project, go back to the decision list in the "Microservice'lerin Bedeli" post of the Microservices series: if none of the reasons on that list outweighs the price in this post for you, a monolith with well-separated modules is usually the better start.