Free lesson · Microservices Interview

Service Discovery - Client-Side vs Server-Side

Medium11 min readmicroservicessystem-designdistributed-systems

Service Discovery - Client-Side vs Server-Side

In a monolith, there is nothing to discover. The application is one process, and every function call resolves to a memory address. In microservices, every service call is a network call to another process running on another machine - and the caller needs to know where that machine is.

The address of a service instance is not fixed. Instances are created and destroyed by container orchestrators. They move across hosts during rescheduling. They scale up and down based on traffic. The IP address that was valid 5 minutes ago may belong to a different service now. Service discovery is the mechanism that provides callers with the current network location of service instances - dynamically, reliably, and in real time.

This is infrastructure that most engineers never think about until it breaks. But when discovery fails - when a caller sends traffic to a terminated instance, or when a new instance receives traffic before it is ready - the failure is indistinguishable from a service outage. Understanding how discovery works is essential for designing resilient microservices.

The Service Discovery Problem

Consider a Product Page Service that needs to call the Recommendation Service. The Recommendation Service has 5 instances running across 3 hosts. The Product Page Service needs to answer: which instances are available, what are their IP addresses and ports, and which of them are healthy?

In a static environment (a traditional data center with fixed servers), the addresses are configured in a file and rarely change. In a dynamic environment (Kubernetes, ECS, Lambda), instances are created and destroyed continuously. A discovery mechanism must:

1. Register new instances as they start
2. Deregister instances as they stop
3. Health check instances to detect failures
4. Serve the current set of healthy instances to callers
5. Handle the gap between an instance failing and the registry learning about it

Two architectural approaches solve this problem: client-side discovery and server-side discovery.

Client-Side Discovery

In client-side discovery, the calling service queries a service registry to get the list of available instances, then chooses one instance using a load balancing algorithm (round robin, random, least connections).

How It Works

1. The Recommendation Service instances register themselves with the service registry on startup
2. The Product Page Service queries the registry: "give me all healthy instances of the Recommendation Service"
3. The registry returns a list: [10.0.1.5:8080, 10.0.1.6:8080, 10.0.2.3:8080]
4. The Product Page Service applies its load balancing algorithm and selects one instance
5. The Product Page Service calls the selected instance directly

graph TD
    PPS["Product Page\nService"] -->|"1. Query registry"| SR["Service Registry\n(Consul / Eureka)"]
    SR -->|"2. Return instances\n[10.0.1.5, 10.0.1.6, 10.0.2.3]"| PPS
    PPS -->|"3. Direct call"| RS1["Recommendation\n10.0.1.5:8080"]
    
    RS1 -->|"Register"| SR
    RS2["Recommendation\n10.0.1.6:8080"] -->|"Register"| SR
    RS3["Recommendation\n10.0.2.3:8080"] -->|"Register"| SR

This diagram shows client-side service discovery. Three Recommendation Service instances register with the service registry. The Product Page Service queries the registry and receives the list of instances. The Product Page Service then calls one instance directly, using its own load balancing algorithm to choose which one.

Implementations

Netflix Eureka. The original microservices service registry, built by Netflix for their Java-based architecture. Services register via HTTP. Clients query the registry and cache the results locally. Eureka uses a peer-to-peer replication model for registry availability - if one Eureka server goes down, clients can use the cached results until another server is available.

HashiCorp Consul. A service registry with built-in health checking, key-value storage, and multi-datacenter support. Services register via HTTP or DNS. Consul provides a DNS interface - the caller resolves recommendation.service.consul and gets the IP addresses of healthy instances. This DNS-based discovery is transparent to the application; no client library is needed.

Pros

Flexibility. The caller controls the load balancing algorithm. Different callers can use different strategies - the Product Page Service uses round robin, while the Analytics Service uses least-connections because its requests are long-running.

No proxy overhead. The caller connects directly to the service instance. No intermediate proxy adds latency or becomes a bottleneck.

Local caching reduces registry dependency. The client caches the instance list. If the registry is temporarily unavailable, the client continues using the cached list. This makes the system resilient to registry failures - at the cost of potentially stale entries.

Cons

Client complexity. Every calling service must implement registry querying, caching, health-check-aware filtering, and load balancing. This logic must be implemented in every programming language used in the system - Java, Go, Python, Node.js each need their own discovery client.

Coupling to the registry. The caller is coupled to the specific registry technology (Eureka, Consul). Switching registries requires updating every caller.

Stale cache risk. The client's cached instance list can become stale if the registry is slow to deregister failed instances. The client may send traffic to a terminated instance and receive connection refused errors until the cache refreshes.

Server-Side Discovery

In server-side discovery, the calling service sends the request to a load balancer (or router), and the load balancer resolves the service name to an instance address using the service registry. The caller does not know about the registry - it sends traffic to a stable endpoint, and the load balancer handles instance selection.

How It Works

1. The Recommendation Service instances register with the registry
2. The Product Page Service sends the request to http://recommendation-service/api/recommendations - a stable hostname
3. The load balancer resolves recommendation-service to the list of healthy instances from the registry
4. The load balancer selects one instance and forwards the request
5. The response returns through the load balancer to the caller

Implementations

Kubernetes DNS + kube-proxy. Kubernetes is the dominant implementation of server-side discovery. Each Kubernetes Service gets a stable DNS name (recommendation-service.default.svc.cluster.local). The DNS name resolves to a virtual IP (ClusterIP). kube-proxy on each node routes traffic from the ClusterIP to a healthy pod. The caller uses the DNS name - it never knows the individual pod IPs.

AWS Elastic Load Balancer + ECS/EKS. ECS and EKS register tasks and pods with an ALB (Application Load Balancer). The ALB health-checks instances and routes traffic to healthy ones. The caller uses the ALB's DNS name.

Envoy proxy (sidecar). A sidecar proxy (Envoy) runs alongside each service instance. The proxy queries the registry and handles load balancing, health checking, and retries. The service calls localhost:port and the proxy routes to the correct instance. This is the sidecar pattern used by service meshes (covered in Topic 6).

Pros

Simple callers. The calling service uses a DNS name or a stable hostname. No registry client, no caching, no load balancing logic. A GET http://recommendation-service/api/recommendations is all the caller needs.

Language-agnostic. Since the caller just makes an HTTP call to a hostname, any language can call any service without a discovery client library. The load balancer handles the complexity.

Centralised load balancing policy. The load balancing algorithm is configured at the load balancer, not at each caller. Changing from round robin to least connections is a load balancer configuration change, not a code change in every calling service.

Cons

Additional network hop. Every request goes through the load balancer (or proxy), adding latency. For a Kubernetes ClusterIP, the overhead is minimal (iptables/IPVS rules on the same node). For an external ALB, the overhead is 1-5ms per request.

Load balancer as a single point of failure. If the load balancer fails, all traffic to the service fails. Kubernetes mitigates this by distributing kube-proxy across all nodes. External load balancers mitigate it through redundancy (multi-AZ ALBs).

Less flexibility for callers. The caller cannot customize the load balancing algorithm per call. The load balancer applies one algorithm to all callers.

Kubernetes Service Discovery in Detail

Since Kubernetes is the dominant platform for microservices, its service discovery deserves specific attention.

How Kubernetes DNS Works

When you create a Kubernetes Service (e.g., recommendation-service in the default namespace), Kubernetes DNS creates a DNS entry: recommendation-service.default.svc.cluster.local. Any pod in the cluster can resolve this hostname.

The DNS resolution returns the Service's ClusterIP - a virtual IP that does not correspond to any physical interface. When traffic is sent to the ClusterIP, kube-proxy (running on every node) intercepts it and forwards it to one of the Service's backend pods using iptables or IPVS rules.

The backend pods are identified by a label selector on the Service. Pods that match the selector and pass their readiness probe are included in the Service's endpoint list. Pods that fail their readiness probe are removed - traffic is not sent to pods that are not ready.

This readiness-based discovery is one of Kubernetes' strongest features. A pod that is starting up (loading data, warming caches, establishing database connections) reports not-ready through its readiness probe. Kubernetes does not add it to the Service's endpoints until the probe passes. Conversely, a pod that is overloaded can fail its readiness probe to temporarily remove itself from the discovery pool - a form of self-protective load shedding.

The readiness probe can be an HTTP endpoint (/health/ready returns 200 when the pod is ready), a TCP check (the port is open), or an exec command (a script returns exit code 0). For most microservices, an HTTP readiness probe that checks database connectivity and cache warmth is the right choice - it verifies that the service can actually serve traffic, not just that the process is running.

The distinction between readiness and liveness probes is critical for service discovery: the readiness probe controls whether the pod receives traffic (service discovery). The liveness probe controls whether the pod is restarted (crash recovery). A pod that fails its readiness probe is removed from the Service but keeps running - it has time to recover. A pod that fails its liveness probe is killed and restarted. Confusing the two (using a liveness probe where a readiness probe is needed) causes unnecessary restarts when the pod is merely overloaded, not dead.

Headless Services for Client-Side Discovery in Kubernetes

A headless Service (ClusterIP: None) returns the individual pod IPs in the DNS response instead of a virtual IP. This gives the caller direct access to pod addresses - enabling client-side load balancing within Kubernetes.

gRPC clients commonly use headless services because gRPC maintains long-lived connections that bypass the kube-proxy load balancing. With a regular ClusterIP Service, all gRPC requests from one client go to the same pod (because the connection is reused). With a headless Service, the gRPC client resolves all pod IPs and load-balances across them at the application level.

Interview: Mentioning headless services for gRPC shows Kubernetes operational depth. "For gRPC services, I would use a headless Kubernetes Service because gRPC uses long-lived HTTP/2 connections. A regular ClusterIP Service would route all requests from one client to the same pod. The headless Service returns individual pod IPs, and the gRPC client load-balances across them using round robin."

Choosing Client-Side vs Server-Side Discovery

The decision between client-side and server-side discovery depends on the platform, the team's operational maturity, and the specific requirements of the service interactions.

If you are on Kubernetes: Use Kubernetes Services (server-side discovery) as the default. It is built in, requires no additional infrastructure, and handles registration, health checking, and deregistration automatically. Use headless services for gRPC or when you need client-side load balancing.

If you are not on Kubernetes: Use Consul or Eureka for the registry, with either client-side discovery (Eureka clients in Java services) or server-side discovery (Consul + Envoy sidecar). The sidecar approach gives server-side simplicity without a central load balancer bottleneck.

For cross-cluster or multi-cloud: Service mesh discovery (Istio's service registry, Consul Connect) provides discovery across Kubernetes clusters, cloud providers, and hybrid environments. This is the most complex setup but necessary for architectures that span multiple deployment environments.

Interview: The discovery answer in an interview should name the specific implementation. "For service discovery, I would use Kubernetes Services with ClusterIP for most services - server-side discovery with no additional infrastructure. For gRPC services, I would use headless services so the gRPC client can load-balance across pod IPs directly. For external service registration (services outside the Kubernetes cluster), I would use Consul with DNS-based discovery."

Health Checking and Stale Entries

The most dangerous moment in service discovery is the gap between an instance failing and the registry reflecting that failure. During this gap, callers receive the failed instance's address from the registry and send traffic to it - receiving connection refused errors or timeouts.

Reducing the Gap

Frequent health checks. The registry pings each instance every 5-10 seconds. If an instance fails to respond to 3 consecutive health checks (configurable), it is removed from the registry. The gap is at most 15-30 seconds - the time for 3 missed health checks.

Self-deregistration on shutdown. When a service instance shuts down gracefully (SIGTERM), it deregisters itself from the registry before stopping. This eliminates the gap for planned shutdowns (deployments, scaling down). Kubernetes handles this automatically - when a pod enters the Terminating state, it is removed from the Service's endpoint list immediately.

Client-side health checking. The caller performs its own health checks against discovered instances, independent of the registry. If an instance fails the caller's health check, the caller removes it from its local list - even if the registry has not deregistered it yet. This reduces the effective gap to the caller's health check interval (typically 1-5 seconds).

Warning: The stale entry problem is the most common service discovery failure mode. During a deployment, old instances are terminated and new instances start. If the registry is slow to deregister the old instances, callers send traffic to terminated pods. The callers see connection refused errors until the registry catches up. Mitigation: configure aggressive deregistration (immediate on SIGTERM), short health check intervals (5 seconds), and client-side retries with instance exclusion.

Graceful Shutdown and Service Discovery

When a service instance shuts down, it must stop receiving new traffic before stopping its processing. The sequence matters:

1. Receive SIGTERM. The orchestrator sends SIGTERM to the process.
2. Deregister from discovery. The instance removes itself from the registry (or the orchestrator removes it from the endpoint list).
3. Stop accepting new connections. The instance's HTTP server stops accepting new requests.
4. Drain in-flight requests. The instance continues processing requests that are already in progress until they complete (or a drain timeout expires).
5. Shut down. The process exits.

If step 2 and step 5 happen simultaneously (the instance is removed from the registry and immediately killed), in-flight requests are dropped. The drain period - typically 10-30 seconds - gives active requests time to complete.

In Kubernetes, the graceful shutdown sequence involves the pod's terminationGracePeriodSeconds (default: 30 seconds) and a preStop hook that adds a short delay (5-10 seconds) to allow the endpoint removal to propagate before the container stops accepting connections. Without the preStop delay, traffic can arrive at the pod after it has stopped accepting connections but before the endpoint removal has propagated to all kube-proxy instances.

What Comes Next

The next lesson is an exercise: design a service discovery system with health checks that handles graceful shutdown during rolling deployments. The exercise tests your understanding of the registration lifecycle - from instance startup through health checking to graceful shutdown - and the edge cases that cause traffic errors during deployments.

Back to Microservices Interview