AlgoInsight

Free lesson · Node.js Deep Dive

Service Discovery and Inter-Service Communication

Hard10 min readnodejsjavascriptbackend

Lesson 21.2 - Service Discovery and Inter-Service Communication

Once you have services, they need to find each other. Service A needs to call service B; it needs to know where B lives and how to reach it. In a static world, you'd hard-code an IP address; in any real production environment, you can't - IPs change, services scale up and down, instances are replaced.

This lesson covers the mechanics. How services discover each other (mostly DNS in modern environments). How they communicate (synchronous via HTTP/gRPC; asynchronous via queues from 21.3). What a service mesh adds. The trade-offs between sync and async communication.

I've worked with service-to-service communication at multiple scales at Microsoft - VS Code Remote backends, Azure DevTools, dev container infrastructure. The patterns are well-established; the trade-offs are clear once you've seen them in production. By the end of this lesson you should know how to wire services together in a way that's debuggable, resilient, and operationally sustainable.

The Discovery Problem

Service A wants to call service B. What URL does it use?

flowchart TB
    subgraph "Static (Wrong)"
        A1[Service A]
        B1["B at 10.0.5.123:3000"]
        A1 -->|hardcoded| B1
        B1 -.IP changes,<br/>scaling, restarts.-> X[Breaks]
    end

    subgraph "DNS-Based (Common)"
        A2[Service A]
        DNS[DNS<br/>service-b.namespace.svc]
        B2[Service B<br/>multiple instances]
        A2 --> DNS --> B2
    end

    subgraph "Service Mesh (Sidecar)"
        A3[Service A]
        SA[Sidecar A]
        SB[Sidecar B]
        B3[Service B]
        A3 -->|localhost| SA
        SA -->|mTLS| SB
        SB -->|localhost| B3
    end

    style X fill:#f87171,stroke:#ef4444,color:#000
    style DNS fill:#34d399,stroke:#10b981,color:#000
    style SA fill:#a78bfa,stroke:#8b5cf6,color:#000
    style SB fill:#a78bfa,stroke:#8b5cf6,color:#000

A diagram of three service discovery approaches. The static approach hardcodes an IP address in service A's code - this breaks the moment service B's IP changes, which it will on every scale event, restart, or deploy. The DNS-based approach has service A look up service B by name through DNS, which resolves to the current set of healthy instances; this is how Kubernetes does it natively. The service mesh approach adds sidecar proxies to both services; A talks to its local sidecar over localhost, the sidecars handle the network communication including mutual TLS and routing, and B receives the call from its local sidecar.

The three approaches reflect a progression in operational sophistication.

DNS-Based Discovery (Kubernetes)

In Kubernetes, every Service object gets a stable DNS name:

apiVersion: v1
kind: Service
metadata:
  name: user-service
  namespace: production
spec:
  selector:
    app: user-service
  ports:
    - port: 3000
      targetPort: 3000

From inside the cluster, this service is reachable at:

http://user-service.production.svc.cluster.local:3000
http://user-service.production:3000        # short form
http://user-service:3000                   # within same namespace

DNS resolves to the set of currently healthy pod IPs. Calls round-robin across them. When pods are added or removed, DNS resolution reflects the change.

For most modern Kubernetes-based deployments, this is all the service discovery you need. The platform handles registration; consumers use DNS names; the system stays correct as services scale.

The mechanics:

  1. Pods register with the Kubernetes API when ready.
  2. The Service object's selector matches pods by label.
  3. CoreDNS (inside the cluster) exposes Service names as DNS records.
  4. Clients resolve via DNS; get an IP; connect.

In Node code, this looks like normal HTTP requests:

async function getUser(userId: string) {
  const res = await fetch(`http://user-service:3000/users/${userId}`);
  return await res.json();
}

No service-discovery library; no client-side load balancing logic. The platform handles everything.

Headless Services for Direct Pod Access

Sometimes you want the list of all pod IPs, not just one (e.g., for stateful services like databases, or for client-side load balancing in gRPC). Headless services (clusterIP: None) return all pod IPs:

apiVersion: v1
kind: Service
metadata:
  name: user-service-headless
spec:
  clusterIP: None   # headless
  selector:
    app: user-service
  ports:
    - port: 3000

DNS now returns multiple A records (one per pod). The client picks one; this enables client-side load balancing.

For most use cases, regular services with kube-proxy's round-robin is sufficient. Headless services are for specific patterns.

Service Mesh

A service mesh (Istio, Linkerd, Consul Connect) adds a layer above DNS. Every service gets a sidecar proxy; the proxies handle:

The pattern: your service makes HTTP calls to http://user-service:3000 (DNS-based). The local sidecar intercepts the call, applies policies, and forwards to the destination service's sidecar, which then forwards to the actual service.

# Istio: traffic splitting for canary deploy
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: user-service
spec:
  hosts:
    - user-service
  http:
    - route:
        - destination:
            host: user-service
            subset: v1
          weight: 90
        - destination:
            host: user-service
            subset: v2
          weight: 10   # 10% canary

90% of requests go to v1; 10% to v2. The application code doesn't change; the mesh handles the routing.

When service mesh fits:

When service mesh doesn't:

For most teams with <20 services, DNS-based discovery without a mesh is the right level of complexity. Add a mesh when the manual workarounds for what it provides become substantial.

REST vs gRPC for Service-to-Service

When services do talk to each other synchronously, two main protocols:

REST/HTTP with JSON. The default. Familiar; debuggable; works with browsers and standard tools.

gRPC (covered in 17.4). Binary protocol over HTTP/2. Better latency; better throughput; strongly typed contracts via protobuf.

For internal service-to-service communication:

Use REST when:

Use gRPC when:

For most teams, start with REST. The familiarity and tooling win unless gRPC's specific benefits are needed. Migrate specific high-traffic paths to gRPC if performance becomes the bottleneck.

For service-to-service calls in Node:

// REST
async function getUser(userId: string) {
  const res = await fetch(`http://user-service/users/${userId}`, {
    headers: { 'Authorization': `Bearer ${serviceToken}` },
    signal: AbortSignal.timeout(5000),
  });
  if (!res.ok) throw new ServiceError(res.status, await res.text());
  return await res.json();
}

// gRPC (with generated client from .proto)
async function getUser(userId: string) {
  const client = new UserServiceClient('user-service:50051', credentials.createInsecure());
  return await new Promise((resolve, reject) => {
    client.getUser({ id: userId }, (err, response) => {
      if (err) reject(err);
      else resolve(response);
    });
  });
}

The REST version is more familiar; the gRPC version has better performance and types.

Authentication Between Services

Internal services still need authentication. "It's internal, who cares" is wrong - internal services can be reached by compromised pods, by buggy clients, by anyone with cluster access. Treat service-to-service communication as needing auth.

Three common approaches:

1. Service Tokens

Each service has a long-lived token; presents it as a Bearer header. The receiver verifies.

// Service A
const TOKEN = process.env.SERVICE_TOKEN;
const res = await fetch('http://user-service/users/42', {
  headers: { 'Authorization': `Bearer ${TOKEN}` },
});

// Service B middleware
function authenticateService(req, res, next) {
  const token = req.headers.authorization?.replace('Bearer ', '');
  if (!VALID_SERVICE_TOKENS.has(token)) {
    return res.status(401).send('Unauthorized');
  }
  next();
}

Simple; works. Downsides: token rotation requires coordinated deploys; revocation is manual.

2. JWT with Short-Lived Tokens

Each service signs a JWT identifying itself; signed with a key the recipient can verify. Short expiration (e.g., 5 minutes) bounds the damage if a token leaks.

Often combined with a token-issuing service that each service calls to get fresh tokens.

3. Mutual TLS (mTLS)

Each service has a certificate; both ends verify each other's certificates during the TLS handshake. No tokens; identity comes from the cert.

This is what service meshes provide automatically. Without a mesh, you'd configure mTLS yourself - workable but complex.

For most setups:

Load Balancing

When service B has 10 instances, traffic to it should distribute. Two main models:

Server-side load balancing. A load balancer (LB) sits between A and B's instances. A calls the LB; the LB picks an instance.

In Kubernetes, this is what kube-proxy does via iptables/IPVS rules. Service IP forwards to one of the matching pod IPs. Round-robin by default.

Client-side load balancing. A knows the list of B's instances; picks one for each call. Smarter algorithms (least connections, latency-aware) possible.

For most setups, server-side via Kubernetes Services is sufficient. Client-side is needed for:

Sync vs Async Communication

A fundamental choice. Service A needs something from service B.

Synchronous (request/response):

Asynchronous (message-based):

flowchart LR
    subgraph "Synchronous"
        A1[Service A]
        B1[Service B]
        A1 -->|"1. HTTP request"| B1
        B1 -->|"2. HTTP response"| A1
        Note1["A waits for B<br/>A blocked on B's latency<br/>If B is down, A fails"]
    end

    subgraph "Asynchronous"
        A2[Service A]
        Q[Queue]
        B2[Service B]
        A2 -->|"1. Send message"| Q
        A2 -.completes immediately.-> A2
        Q -->|"2. Deliver later"| B2
        Note2["A doesn't wait<br/>A's success doesn't depend on B<br/>B can be down briefly without impact"]
    end

    style A1 fill:#a78bfa,stroke:#8b5cf6,color:#000
    style A2 fill:#a78bfa,stroke:#8b5cf6,color:#000
    style Q fill:#fbbf24,stroke:#f59e0b,color:#000

A diagram comparing synchronous and asynchronous service communication. Synchronously, service A makes an HTTP request to service B and waits for the response - A is blocked on B's latency, and if B is down, A's request fails. Asynchronously, service A sends a message to a queue and completes immediately; service B processes the message later, possibly with delay; A's success doesn't depend on B's availability. Both have legitimate uses: synchronous for queries that need immediate answers; asynchronous for fire-and-forget work and decoupling availability.

When to use synchronous:

When to use asynchronous:

Most production systems mix both. Synchronous for queries; asynchronous for events and side effects. The split is one of the most important architecture decisions.

Async messaging is covered in 21.3.

Patterns to Avoid

A few specific antipatterns:

Synchronous Chains

Service A → Service B → Service C → Service D

Each step adds latency. Each step adds failure risk. If any service in the chain is slow, the whole request is slow. If any fails, the whole request fails.

Fix: break the chain. Either:

Service A Calling Itself via the Network

Service A → DNS → Service A

Common bug. Service A calls a function that's actually in the same service via HTTP (because of how the code is organized). Pure overhead.

Fix: detect this in code review. Local function calls instead of network calls when in the same service.

Sync When Async Would Do

POST /orders
  ↓ calls
POST /notifications  (sync)
  ↓ calls
POST /email-service  (sync)

The order creation depends on email delivery. If email is slow, order creation is slow. If email is down, order creation fails.

Fix: make notifications async. Order creation succeeds; notifications happen eventually.

Distributed Monolith

Multiple services that must deploy together because of tight coupling. All the costs of microservices; none of the benefits.

Fix: either consolidate (probably right) or actually decouple (rare but possible).

A Realistic Pattern

For a typical Node microservices system:

Synchronous calls (HTTP/REST):

Asynchronous messages (covered in 21.3):

Service discovery:

Authentication:

Communication libraries:

This is the boring, well-established pattern that most production Node systems converge on. Variations exist; the basics are the same.

What I Want You to Take From This Lesson

Six things.

One: In Kubernetes, DNS-based discovery is usually sufficient. No service-discovery library needed.

Two: Service mesh adds value for many-service systems; overkill for small ones.

Three: REST is the right default for service-to-service. gRPC for specific performance needs.

Four: Internal services need authentication. Short-lived JWTs or mTLS.

Five: Mix sync and async. Sync for queries; async for events.

Six: Avoid synchronous chains. Each hop adds latency and failure risk.

Try It

Exercise: Warmup (20 minutes). In a Kubernetes cluster (minikube or kind), create two simple services. Make one call the other via DNS. Verify it works.

Exercise: Core (30 minutes). Add authentication between the services using short-lived JWTs. Verify unauthenticated calls are rejected.

Exercise: Core (30 minutes). Implement timeouts and basic error handling for inter-service calls. Simulate the target service being down; verify the caller handles it gracefully.

Exercise: Stretch (60 minutes). Set up Linkerd (lighter than Istio) on a Kubernetes cluster. Compare the experience to plain DNS. Note what mTLS and traffic metrics it provides automatically.

What's Next

Lesson 21.3 covers async messaging - the alternative to synchronous calls that often makes systems more robust.

Back to Node.js Deep Dive