Service Discovery: Client-Side, Server-Side, Kubernetes DNS and .NET Service Discovery

Version note: The code in this post was built and run with C# 14 and .NET 10; the service discovery example ran with Microsoft.Extensions.ServiceDiscovery 10.11.0.

Guess First

The Web BFF (Backend for Frontend) from the previous post went to Inventory at a fixed address such as http://localhost:5302. In production that address became http://10.0.4.17:8080 and was written into the BFF's configuration file. The back-to-school campaign is starting and Inventory is scaled out to three instances. One instance is restarted overnight after a memory problem and gets a new IP (Internet Protocol) address. The next day someone writes all three addresses into the BFF's configuration by hand; this time one of the instances crashes but stays in the list.

Three questions: after scaling out, how many Inventory instances does the BFF send requests to? What happens to the restarted instance? And with one instance down in the three-address list, roughly what share of the BFF's requests to Inventory fail?

Why a Fixed URL Isn't Enough

In the monolith, one module calling another was a method call; there was no address problem. With microservices every call goes to a network address, and that address doesn't stay fixed for three reasons. The first is scaling: the number of instances of the same service grows and shrinks with load, and every instance has its own address. The second is lifecycle: containers and Kubernetes pods die and are reborn, coming up on another machine with another IP; during a deployment, old and new instances live side by side for a while. The third is environments: what's localhost:5302 on a developer's machine is a different address in the test environment and another in production.

Service discovery is the answer to all three: the client uses a logical name like "Inventory", and a mechanism turns that name into the physical addresses that are healthy right now. Chris Richardson's pattern language on microservices.io splits this mechanism into two questions: where and how do instances record their addresses (registration), and how does a client find those addresses (discovery).

Service Registry and Registration

A service registry is a database holding the addresses of each service's currently running instances. The registry itself is a critical component: if it goes down, no new connections can be made, so it has to run with high availability. Consul, etcd and Netflix's Eureka are well-known examples of this role; in Kubernetes the job is inside the platform, in the API (Application Programming Interface) server and etcd.

There are two ways for an instance to get into the registry. With self registration the instance registers itself on startup, says "still alive" (a heartbeat) at intervals and removes its record when shutting down. Simple, but every service in every language has to carry the registry's client library. With third-party registration a registrar, not the instance, does the registering: the deployment platform registers the instance because it knows it started it, and deregisters it based on its health check. The service's code knows nothing about the registry. One of the examples Richardson gives for this pattern is Kubernetes itself; most .NET teams today get discovery this way.

Client-Side and Server-Side Discovery

Once the address is in the registry, there are two models depending on how the client uses it:

The gain of client-side discovery is control: the client decides which instance to go to, skips the hop in between and can pick a nearby instance by zone. The gain of server-side discovery is simplicity: all the client knows is a name and an address, whatever its language. In practice the two models often blend; below we'll see how .NET's client-side discovery library hands over to server-side discovery in Kubernetes.

Discovery in Kubernetes: Services and DNS

In Kubernetes you put a Service object in front of a Deployment's pods, and that Service gets a stable virtual IP (the cluster IP) and a stable DNS (Domain Name System) name: inventory.bookstore.svc.cluster.local (service name, namespace, svc and the cluster domain). For a pod in the same namespace, just inventory is enough. As pods die and are reborn, Kubernetes updates the endpoint list behind the Service itself; a pod whose readiness probe fails drops out of the list. All the client sees is a stable name; spreading requests to healthy pods is the job of kube-proxy's rules on the nodes (or of the network plugin in use). So out of the box, Kubernetes does third-party registration and server-side discovery.

Two more details are useful. A headless Service (clusterIP: None) gets no virtual IP; the DNS query returns the pods' IPs directly and leaves the choice to the client, opening the door to client-side discovery. Named ports also get an SRV (service record) entry: _http._tcp.inventory.bookstore.svc.cluster.local gives both the address and the port. The readiness endpoint we wrote in the ASP.NET Core Web API series post "Health Checks in ASP.NET Core Web API: Liveness, Readiness and Database Checks" is where Kubernetes looks when deciding whether to put a pod behind the Service; this is where discovery and health checks meet. Kubernetes itself is the topic of the upcoming Kubernetes series in the DevOps category.

Service Discovery in .NET: Microsoft.Extensions.ServiceDiscovery

.NET's own solution is the Microsoft.Extensions.ServiceDiscovery package, which ships with Aspire but doesn't depend on it. It adds a message handler to HttpClient: it treats the host name in the base address as a service name, gets the address list from endpoint providers and picks one from the list on each request. There are two default providers: one that reads from configuration and the pass-through provider we'll see below. The Web BFF's Inventory client:

HostApplicationBuilder builder = Host.CreateApplicationBuilder();
builder.Logging.ClearProviders();
builder.Configuration.AddJsonFile(file);
builder.Services.AddServiceDiscovery();
builder.Services.AddHttpClient("inventory", c => c.BaseAddress = new Uri("https+http://inventory"))
    .AddServiceDiscovery();

{
  "Services": {
    "inventory": {
      "http": [ "http://localhost:5402", "http://localhost:5403" ]
    }
  }
}

https+http://inventory is a priority list: first look for HTTPS (HTTP Secure) endpoints of the inventory service, and fall back to HTTP (Hypertext Transfer Protocol) if there are none. Microsoft's documentation gives the reason as the common practice of using HTTP in development and HTTPS in production; if an HTTPS endpoint is found, HTTP endpoints aren't used at all. If you want to allow only HTTPS in production, you can restrict schemes with ServiceDiscoveryOptions.AllowAllSchemes = false and AllowedSchemes = ["https"]. Each line in the Services:inventory:http section of the configuration is one instance. The two Inventory instances are stubs that return their own port; six requests:

services.two.json
  call 1: served by 5403
  call 2: served by 5402
  call 3: served by 5403
  call 4: served by 5402
  call 5: served by 5403
  call 6: served by 5402

The requests were spread across the two instances in turn; the package's default endpoint selector is RoundRobinServiceEndpointSelector. This is client-side discovery and client-side load balancing in their purest form.

Aspire's role here is to produce this configuration for you: when you reference a project with WithReference(inventory) in the AppHost, Aspire injects Inventory's endpoints as environment variables in the form services__inventory__http__0; since the .NET configuration system reads a double underscore as :, that's exactly the JSON (JavaScript Object Notation) above. Aspire itself is the topic of the upcoming "Distributed Applications with .NET Aspire" series in the Backend category.

When you go to Kubernetes in production, you don't have to change the same code. If there's no entry for inventory in configuration, the pass-through provider kicks in: it leaves the service name as it is, as a DNS name, and the request goes to the inventory host, that is, to the Kubernetes Service. Configuration or Aspire on the developer's machine, the platform's DNS in the cluster; no conditions in the code. One detail to watch: pass-through uses the first scheme in the priority list. Emptying the configuration completely and running the same client sent the request straight to the HTTPS port (and got a DNS error, since there's no host called inventory on a developer machine):

services.none.json
  call 1: HttpRequestException: nodename nor servname provided, or not known (inventory:443)

So in the cluster, https+http://inventory expects Inventory to speak TLS (Transport Layer Security) on 443. If in-cluster traffic is plain HTTP, make the base address http://inventory in the production configuration, or plan TLS between services from the start. If you want to reach individual pods in Kubernetes, the DNS SRV provider in the Microsoft.Extensions.ServiceDiscovery.Dns package can be used with headless Services; in most scenarios you don't need it.

Discovery Is Not a Health Check

Back to the last question from Guess First. I added a third instance to the same configuration, but nothing listens on 5404: an instance that crashed but wasn't removed from the list.

services.three.json
  call 1: served by 5403
  call 2: HttpRequestException: Connection refused (localhost:5404)
  call 3: served by 5402
  call 4: served by 5403
  call 5: HttpRequestException: Connection refused (localhost:5404)
  call 6: served by 5402

Two of six requests, a third, failed. Round robin doesn't know whether an address in the list is alive; whoever keeps the list is also responsible for health information. Since the configuration provider reads a static list, it never removes the dead instance. A Kubernetes Service, on the other hand, removes a pod whose readiness probe fails from its endpoint list, and Consul does the same based on its own health checks. That's why the discovery decision is made together with the load balancing and health check decisions: right behind "where do I find the address?" comes "who notices that the address is dead, and how quickly?" Even in Kubernetes there are a few seconds between the moment a pod crashes and the moment it leaves the list; requests in that window need client-side timeouts and retries. What that means at the service level is in this series' "Microservices in Production" post.

Consul and Service Mesh: When Do You Need Them?

Consul takes on the registry role outside Kubernetes or in environments spanning several platforms: services running on virtual machines, multiple Kubernetes clusters, setups mixing on-premises and cloud. It has its own health checks, a DNS interface and a key/value store. In a system like Bookstore that runs entirely in one Kubernetes cluster, adding Consul means managing something the platform already provides a second time.

A service mesh (Istio, Linkerd) takes discovery a step further: a sidecar proxy is placed next to every pod (or, in newer meshes, a proxy at the node level) and all traffic between services goes through these proxies. Application code goes to inventory over plain HTTP; the proxy takes care of discovery, load balancing, retry and timeout policies, mTLS (mutual TLS) encryption between services and percentage-based traffic splitting (canary deploys). The gain is that these behaviours arrive the same way in every language without changing application code. The cost is a proxy added to every pod, a few milliseconds added to every call, a new control plane to learn and operate, and the question "is the bug in my code or in the mesh?" whenever something goes wrong.

A mesh belongs in systems where services number in the dozens, several languages are in use and mTLS is a compliance requirement. For Bookstore, with four teams, all .NET and a dozen services, Kubernetes Services, Microsoft.Extensions.ServiceDiscovery and Microsoft.Extensions.Http.Resilience meet the same need with far fewer parts. Whether we'll need a mesh for mTLS is something we'll discuss again on the security side in the next post.

Common Pitfalls

Thinking of discovery separately from health checks and load balancing

The output above is exactly this: the address list is right, but one address in it is dead. When you choose discovery, you should also have chosen the answer to "when, and by whom, is a dead instance removed from the list?"

Writing endpoints without a scheme

While preparing this post's example I first wrote the addresses as "localhost:5402", without a scheme. Because the client tries HTTPS first with https+http://inventory, it tried to connect to these addresses over HTTPS, and all six requests failed with HttpRequestException: The SSL connection could not be established, see inner exception. Write every address in configuration with its scheme (http://localhost:5402); the values Aspire generates look like that too.

Putting a load balancer in front of a load balancer

Client-side round robin is pointless in a client that goes to a Kubernetes Service's cluster IP: there's a single address in the client's list and kube-proxy does the real spreading. Know which layer does load balancing and do it in only one.

Forgetting the DNS cache

DNS answers are cached, and long-lived connections can stay on an old address. In .NET, IHttpClientFactory's handlers are renewed every two minutes by default; if you hold your own HttpClient with a SocketsHttpHandler, set PooledConnectionLifetime, otherwise new pods don't get traffic.

Leaving the registry as a single point

If you run your own registry (Consul, Eureka), it's now the most critical component in the system; if it isn't clustered, it becomes a single point of failure shared by all services.

A Small Experiment

★☆☆ In the example, rename the http section in the configuration to https but leave the addresses with http://. Which addresses does https+http://inventory find now? Then set the base address to just http://inventory and try again.

★★☆ Set AllowAllSchemes = false and AllowedSchemes = ["https"] on ServiceDiscoveryOptions. What happens when you send a request with a configuration that has only HTTP endpoints? Write in one sentence why you'd turn this setting on in production.

★★★ Make a paper design for Bookstore's Inventory in Kubernetes: the Service type (normal or headless), the endpoint the readiness probe checks, the BFF's timeout and retry settings. What do requests arriving in the first few seconds after a pod crashes go through, and which layer tolerates it?

Services now find each other by name. But how do we know that a request reaching Inventory really comes from the BFF, and whether it comes on behalf of a user or on its own behalf? In the next post we'll look at security in microservices, from edge authentication to service-to-service trust.