Free lesson · System Design Fundamentals

Service Discovery

Hard10 min readsystem-designhlddistributed-systems

Lesson 25.1 - Service Discovery

The first lesson of Module 25.

In a monolith (24.2), all code runs in the same process. The Order module calls the Payment module by invoking a function - a direct, in-process call that resolves at compile time. There is no "finding" the Payment module - it's right there, in the same binary, at a known memory address. In a microservices architecture (24.3), the Order Service needs to call the Payment Service - but the Payment Service is running in a different process, on a different machine, at a network address that might change at any moment. The Payment Service might be running 5 instances today and 50 tomorrow (auto-scaling). The instances might be on different IP addresses (containers are ephemeral - each deployment gets new IPs). Some instances might be healthy and others might be failing. The Order Service must answer a fundamental question before it can make a single network call: where is the Payment Service right now?

Service discovery is the mechanism that answers this question. It maintains a registry of all running service instances - their network addresses, health status, and metadata - and provides a way for services to look up other services' locations at runtime. Without service discovery, microservices would need hardcoded IP addresses (brittle - IPs change when containers restart), manual configuration files (doesn't scale - updating 50 services' config files when one service moves), or DNS with long TTLs (stale - DNS caches don't reflect real-time instance changes).

The Problem in Detail

In a traditional deployment (VMs with static IPs), you might configure the Order Service with:

# Static configuration - works with 3 fixed servers
payment-service:
  hosts:
    - 10.0.1.10:8080
    - 10.0.1.11:8080
    - 10.0.1.12:8080

This works until:

• Auto-scaling adds instances. The Payment Service scales from 3 to 10 instances during Black Friday. The 7 new instances have IPs that aren't in the Order Service's config file. Those instances receive no traffic - they're invisible.
• A container restarts. The Payment Service instance at 10.0.1.10 crashes and Kubernetes restarts it - but the new container gets IP 10.0.2.47. The Order Service still sends requests to 10.0.1.10 - which no longer exists. Requests fail.
• A deployment rolls out. The Payment Service deploys a new version. Old instances are terminated, new ones start with new IPs. For 30 seconds during the rollout, the Order Service's config points to a mix of dead and alive instances.
• A health check fails. One Payment Service instance is running but unhealthy (database connection lost, out of memory, deadlocked). The Order Service sends requests to it - they timeout. The Order Service doesn't know this instance is unhealthy.

Service discovery solves all four problems by maintaining a real-time registry of healthy service instances that updates automatically as instances start, stop, scale, and fail.

Two Approaches: Client-Side vs Server-Side Discovery

Client-Side Discovery

The client service (Order Service) queries the service registry directly to get the list of available instances, then chooses one (using a load-balancing algorithm - round-robin, random, least connections) and makes the request directly to that instance.

flowchart TD
    Order["Order Service\n(client)"]
    
    Registry["SERVICE REGISTRY\n(Consul / eureka / etcd)\n\nPayment Service instances:\n• 10.0.2.47:8080 ✅ healthy\n• 10.0.3.12:8080 ✅ healthy\n• 10.0.1.99:8080 ❌ unhealthy\n• 10.0.4.33:8080 ✅ healthy"]
    
    P1["Payment Instance 1\n10.0.2.47:8080"]
    P2["Payment Instance 2\n10.0.3.12:8080"]
    P3["Payment Instance 3\n10.0.4.33:8080"]
    
    Order -->|"1. Query: where is\nPayment Service?"| Registry
    Registry -->|"2. Response:\n[10.0.2.47, 10.0.3.12, 10.0.4.33]\n(healthy instances only)"| Order
    Order -->|"3. Direct request\n(client picks instance\nvia load balancing)"| P1

    style Registry fill:#fbbf24,stroke:#f59e0b,color:#000

A client-side service discovery diagram. The Order Service (the client) needs to call the Payment Service. Step 1: the Order Service queries the service registry (Consul, Eureka, or etcd) asking "where is the Payment Service?" Step 2: the registry responds with a list of healthy Payment Service instances - their IP addresses and ports. The registry excludes unhealthy instances (10.0.1.99 failed its health check). Step 3: the Order Service selects one instance from the list using a client-side load balancing algorithm (round-robin, random, weighted, or least-connections) and sends the request directly to that instance. The key characteristic: the CLIENT makes the load-balancing decision. The client has the full list of instances and chooses which one to call. This requires a service discovery client library in every service (the library handles querying the registry, caching results, and load balancing). Netflix's Eureka and Ribbon libraries pioneered this pattern - every Netflix microservice uses client-side discovery with Eureka as the registry and Ribbon as the client-side load balancer.

How instances register: each service instance registers itself with the registry on startup - sending its address, port, service name, and metadata (version, environment, datacenter). The registry periodically health-checks each instance (HTTP health endpoint, TCP probe, or gRPC health check). If an instance fails the health check, the registry marks it as unhealthy - it's excluded from query responses. When an instance shuts down gracefully, it deregisters itself. If it crashes (ungraceful shutdown), the health check eventually fails and the registry removes it.

Implementations:

• Netflix Eureka: the pioneer of client-side discovery. Each service registers with the Eureka server and periodically sends heartbeats. The Eureka client library (built into each service) caches the registry locally - so even if the Eureka server is temporarily unavailable, services can still discover each other from their local cache. Used by Netflix's entire microservice fleet (hundreds of services, thousands of instances).
• Consul (HashiCorp): a multi-purpose service discovery and configuration tool. Services register via HTTP API or DNS. Health checks are configurable (HTTP, TCP, gRPC, script). Consul also provides key-value storage and distributed locking. Used by many organizations as a general-purpose service mesh control plane.
• etcd + custom client: etcd (the key-value store behind Kubernetes) can serve as a service registry - services register as key-value entries (key = service name, value = instance address), with leases (TTL) for automatic deregistration on failure. Requires building a custom client library (unlike Eureka or Consul, which provide official client libraries).

Pros of client-side discovery: no intermediate proxy (lower latency - the client connects directly to the service instance). The client has full control over load balancing (can implement sophisticated algorithms - weighted round-robin based on instance health, latency-based routing, canary routing). The registry can be temporarily unavailable without disrupting existing connections (clients use cached instance lists).

Cons of client-side discovery: every service needs a discovery client library (coupling to a specific discovery technology - Eureka, Consul, or custom). Each programming language needs its own client library (Java services use a Java Eureka client; Go services need a Go client; Python services need a Python client). Client-side load balancing logic is duplicated across every service.

Server-Side Discovery

The client service sends requests to a load balancer (or router), which queries the service registry, selects an instance, and forwards the request. The client doesn't know about the registry or individual instances - it sends all requests to the load balancer's address.

flowchart TD
    Order["Order Service\n(client)"]
    
    LB["LOAD BALANCER / ROUTER\n(AWS ALB / Kubernetes Service /\nNginx / Envoy)\n\nQueries registry.\nSelects healthy instance.\nForwards request."]
    
    Registry["SERVICE REGISTRY\n(Consul / k8s etcd /\nAWS Cloud Map)"]
    
    P1["Payment Instance 1"]
    P2["Payment Instance 2"]
    P3["Payment Instance 3"]
    
    Order -->|"1. Request to\npayment-service:8080\n(logical name)"| LB
    LB -->|"queries"| Registry
    LB -->|"2. Forward to\nhealthy instance"| P2

    style LB fill:#fbbf24,stroke:#f59e0b,color:#000

A server-side service discovery diagram. The Order Service sends a request to a logical service name (payment-service:8080) - not to a specific instance IP. The request goes to a load balancer or router (AWS ALB, Kubernetes Service, Nginx, or Envoy proxy). The load balancer queries the service registry to get the list of healthy Payment Service instances, selects one using its load-balancing algorithm, and forwards the request to that instance. The Order Service is completely unaware of individual instances, the registry, or the load-balancing algorithm. It only knows the logical service name. This is the approach Kubernetes uses: a Kubernetes Service is a server-side discovery mechanism. When the Order Service calls payment-service.default.svc.cluster.local, Kubernetes' kube-proxy (or the CNI's iptables/eBPF rules) routes the request to one of the Payment Service's pods - selected by round-robin from the set of pods matching the Service's selector. The pods register automatically when they start (via the kubelet) and are deregistered when they stop. Health checks (readiness probes) exclude unhealthy pods from the rotation.

Implementations:

• Kubernetes Services: the dominant server-side discovery mechanism. A Kubernetes Service is a stable DNS name (e.g., payment-service.default.svc.cluster.local) backed by a set of pods. kube-proxy or iptables rules route requests to healthy pods. Registration is automatic (pods matching the Service's label selector are included). Health checks (readiness probes, liveness probes) exclude unhealthy pods. This is built into Kubernetes - no additional infrastructure needed.
• AWS ALB/NLB + Cloud Map: AWS Application Load Balancer or Network Load Balancer routes requests to registered targets (EC2 instances, ECS tasks, Lambda functions). AWS Cloud Map provides DNS-based service discovery. ECS tasks register automatically with Cloud Map on startup.
• Envoy + control plane: Envoy proxy (used by Istio, Consul Connect, and AWS App Mesh) acts as a server-side discovery proxy. The control plane (Istio's Pilot, Consul's server) pushes service instance information to Envoy. Envoy routes requests to healthy instances. This is the service mesh approach - covered in 25.8.

Pros of server-side discovery: the client is simple - just send requests to a logical name (no discovery library, no registry queries, no client-side load balancing). Language-agnostic - any language can call http://payment-service:8080. Load balancing logic is centralized in the load balancer - one implementation, not duplicated across every service.

Cons of server-side discovery: the load balancer is an additional network hop (adds 0.1-1ms latency per request). The load balancer is a potential single point of failure (mitigated by running multiple load balancer instances - but still a component to manage). The load balancer must be scaled to handle all inter-service traffic (in a high-throughput system, the load balancer can become a bottleneck).

Client-Side vs Server-Side: The Decision

The industry has moved toward server-side discovery via Kubernetes. In 2015, client-side discovery (Netflix OSS stack - Eureka, Ribbon, Hystrix) was the standard. In 2025, Kubernetes Services provide server-side discovery out of the box - no additional libraries, no per-language clients, no explicit registry management. Service meshes (Istio, Linkerd - 25.8) further abstract discovery into the infrastructure layer, making it invisible to application code.

DNS-Based Discovery: The Simplest Approach

The simplest form of service discovery is DNS. Each service has a DNS name that resolves to the IP addresses of its healthy instances. Kubernetes does this by default: payment-service.default.svc.cluster.local resolves to the ClusterIP (a virtual IP that kube-proxy routes to pods) or directly to pod IPs (headless Service).

Standard DNS limitations for service discovery:

• TTL caching: DNS responses are cached by clients for the duration of the TTL (Time-To-Live). If a service instance fails and a new one starts with a different IP, clients with a cached DNS response continue sending requests to the dead instance - until the cache expires. Short TTLs (5-10 seconds) mitigate this but increase DNS query traffic.
• No health checking: standard DNS doesn't health-check instances. A DNS record for a crashed instance persists until manually removed or TTL-expired. Consul and Kubernetes solve this by integrating health checks with DNS - unhealthy instances are removed from DNS responses immediately.
• No load balancing metadata: DNS returns a list of IPs but no information about instance load, response time, or capacity. The client picks randomly or round-robin - no intelligent routing.

Consul DNS extends standard DNS with health-aware resolution: payment-service.service.consul resolves only to healthy instances (Consul's health checks filter unhealthy ones before the DNS response). This combines the simplicity of DNS with the health-awareness of a service registry.

Health Checks: The Critical Complement

Service discovery without health checking is dangerous - it routes traffic to dead or unhealthy instances. Three types of health checks:

Liveness checks: "is the process alive?" A TCP connection to the service port, or an HTTP GET to /healthz that returns 200 if the process is running. If the liveness check fails: the instance is dead - remove it from the registry, restart it (Kubernetes liveness probe restarts the container).

Readiness checks: "can the process handle requests?" An HTTP GET to /ready that returns 200 only when the service has completed initialization (database connections established, caches warmed, config loaded). If the readiness check fails: the instance is alive but not ready - remove it from the load balancer's rotation (don't send it traffic) but don't kill it. Kubernetes readiness probes do exactly this: a pod that fails its readiness probe is removed from the Service's endpoints (no traffic routed) but the pod continues running (it might become ready after warmup).

Deep health checks: "are all dependencies healthy?" An HTTP GET to /health that checks the database connection, the cache connection, downstream service availability, and disk space. If any dependency is unhealthy, the check fails - the instance is removed from rotation. Caution: deep health checks can cause cascading failures - if the database is slow, ALL instances fail their health checks simultaneously, and the entire service is removed from discovery. Use deep checks judiciously - fail the health check for truly critical dependencies (database), not for non-critical ones (analytics service).

Six Takeaways from This Lesson

1. Service discovery maintains a real-time registry of healthy service instances - their addresses, ports, and health status. Services query the registry to find other services at runtime.

2. Client-side discovery (Eureka, Consul client): the client queries the registry and selects an instance. Lower latency but requires per-language client libraries. Used by Netflix's microservice fleet.

3. Server-side discovery (Kubernetes Services, ALB): the client sends requests to a load balancer, which queries the registry and forwards. Simpler clients, language-agnostic. The dominant modern approach via Kubernetes.

4. DNS-based discovery is the simplest approach - but limited by TTL caching, no built-in health checking, and no load metadata. Consul DNS adds health-aware resolution.

5. Health checks are mandatory: liveness (is it alive?), readiness (can it handle traffic?), and deep checks (are dependencies healthy?). Discovery without health checking routes traffic to dead instances.

6. The industry trend is server-side discovery via Kubernetes, with service meshes (25.8) further abstracting discovery into the platform layer - invisible to application code.

What's Next

Lesson 25.2 covers the API Gateway Pattern - the single entry point for all client requests, handling routing, authentication, rate limiting, and cross-cutting concerns before requests reach individual services.

Back to System Design Fundamentals