Free lesson · System Design Advanced

Kubernetes Origin and Philosophy - The Borg Lineage

Hard10 min readsystem-designadvanced

Before Kubernetes existed as open-source software, the ideas behind it ran every Google service you have ever used. Search results, Gmail messages, YouTube videos, Google Maps tiles - all of them scheduled, managed, and kept alive by an internal system called Borg. I work at Google today, and Borg is still running. It manages billions of containers across millions of machines in Google's data centres. Kubernetes is not a copy of Borg - it is a reimagining of Borg's core ideas for the rest of the world, built by engineers who learned from Borg's mistakes as much as its successes.

Understanding the philosophy Kubernetes inherited from Borg is essential because it explains design decisions that otherwise seem arbitrary or unnecessarily complex. Why is everything declarative instead of imperative? Why do controllers run in infinite reconciliation loops? Why does the system use labels instead of a rigid hierarchy? Why does every pod get its own IP address instead of sharing the host's network? The answers to all of these are rooted in what Google learned from operating Borg at planet scale for over a decade. If you skip the philosophy, Kubernetes feels like a collection of arbitrary conventions. If you understand the philosophy, the entire system clicks into place.

What Borg Taught Google

graph TD
    subgraph "Borg's Core Lessons (2003-2014)"
        L1["Lesson 1: Declarative > Imperative<br/>──────────────────<br/>Operators should declare the<br/>desired state ('run 5 replicas<br/>of this service') - NOT issue<br/>step-by-step commands ('start<br/>a container on machine A, then<br/>start one on machine B...').<br/><br/>Why: imperative commands break<br/>when machines fail mid-execution.<br/>Nobody re-issues the command.<br/>Declarative state survives<br/>failures because the system<br/>continuously reconciles actual<br/>state toward desired state —<br/>automatically, indefinitely,<br/>without human intervention."]
        
        L2["Lesson 2: Reconciliation Loops<br/>──────────────────<br/>Every component continuously<br/>observes actual state, compares<br/>it to desired state, and takes<br/>action to close the gap.<br/><br/>observe → diff → act → repeat<br/><br/>If a container crashes, no human<br/>needs to intervene. The loop<br/>detects the gap (actual=2,<br/>desired=3) and creates a new<br/>container. This makes the<br/>entire system self-healing<br/>without central coordination."]
        
        L3["Lesson 3: Labels Over Hierarchy<br/>──────────────────<br/>Borg originally used a rigid<br/>hierarchy: cluster → cell →<br/>job → task. This hierarchy<br/>broke down for cross-cutting<br/>concerns. 'Show me all tasks<br/>owned by team X across all<br/>cells' required traversing<br/>every branch of the tree.<br/><br/>Kubernetes replaced hierarchy<br/>with labels: arbitrary key-value<br/>pairs on any object. Flexible,<br/>composable, queryable."]
        
        L4["Lesson 4: IP-per-Pod<br/>──────────────────<br/>Borg gave each task a unique<br/>port on the host, not a unique<br/>IP. This caused massive port<br/>conflict management overhead<br/>and required a port broker.<br/><br/>Kubernetes fixed this: every<br/>pod gets its own IP address.<br/>No port conflicts. Any service<br/>can bind to port 80 without<br/>worrying about what else is<br/>running on the node."]
    end

This diagram shows the four core lessons Google learned from operating Borg that directly shaped Kubernetes. Lesson 1: declarative state management beats imperative commands because declarations survive machine failures - the system continuously reconciles toward the desired state without anyone re-issuing failed commands. Lesson 2: reconciliation loops where components observe actual state, compare to desired state, and act to close the gap make the system self-healing without central coordination or human intervention. Lesson 3: labels replaced Borg's rigid cluster-cell-job-task hierarchy with flexible key-value pairs, allowing cross-cutting queries that a fixed tree structure couldn't express. Lesson 4: IP-per-pod networking eliminated Borg's port conflict management overhead by giving every pod its own unique IP address.

The Declarative Model: Tell the System What, Not How

This is the single most important concept in Kubernetes. You do not tell Kubernetes to "start a container on node-3." You tell Kubernetes "I want 3 replicas of this container running at all times." Kubernetes figures out which nodes to use, starts the containers, and - critically - keeps monitoring. If a node dies and takes one replica down, Kubernetes notices that actual state (2 replicas) differs from desired state (3 replicas) and starts a new replica on a healthy node. No human intervention. No pager alert. No manual restart.

The declarative model is expressed through YAML manifests - files that describe the desired state of every resource. A Deployment manifest says "I want 3 replicas of nginx:1.25, each with 256 MB memory limit, exposed on port 80." The Kubernetes control plane reads this manifest, compares it to the current cluster state, and takes whatever actions are needed to make reality match the declaration - now and continuously into the future.

Important: The YAML is not ceremony - it is the desired state stored in version control. Many engineers complain about "YAML engineering," but the YAML manifest serves a critical purpose. When you kubectl apply -f deployment.yaml, you are not issuing a one-time command. You are submitting a desired-state document that the system will continuously enforce, even after you close your laptop. If the cluster crashes and recovers, the desired state persists in etcd, and the reconciliation loops resume enforcement automatically. This is fundamentally different from docker run, which is an imperative command - if the container crashes, it stays dead unless something else explicitly restarts it.

Reconciliation Loops: The Self-Healing Pattern

graph TD
    subgraph "The Reconciliation Loop"
        OBSERVE["OBSERVE<br/>──────────────────<br/>Controller watches the API server<br/>for changes to resources it manages.<br/><br/>Example: ReplicaSet controller<br/>watches all ReplicaSets and Pods.<br/>It knows the current state of<br/>every Pod matching a ReplicaSet's<br/>label selector.<br/><br/>Uses the Watch API - event-driven,<br/>not polling. Changes are pushed<br/>to watchers in near-real-time."]
        
        OBSERVE --> DIFF["DIFF<br/>──────────────────<br/>Compare observed state to desired:<br/><br/>Desired: 3 replicas<br/>Observed: 2 running pods<br/>→ Diff: 1 pod short<br/><br/>Desired: 3 replicas<br/>Observed: 4 running pods<br/>→ Diff: 1 pod excess<br/><br/>Desired: 3 replicas<br/>Observed: 3 running pods<br/>→ Diff: none → do nothing"]
        
        DIFF --> ACT["ACT<br/>──────────────────<br/>Take the MINIMUM action to<br/>close the gap:<br/><br/>1 pod short → create 1 pod<br/>(not 3 - the other 2 exist)<br/><br/>1 pod excess → delete 1 pod<br/><br/>0 diff → no action, sleep<br/>until next Watch event<br/><br/>The controller never rebuilds<br/>from scratch - it always<br/>incrementally converges."]
        
        ACT --> OBSERVE
    end

This diagram shows the three-phase reconciliation loop at the heart of every Kubernetes controller. In the observe phase, the controller watches the API server via the Watch API for changes to resources it manages - this is event-driven, not polling, so changes arrive in near-real-time. In the diff phase, observed state is compared to desired state: if 3 replicas are desired and only 2 exist, the diff is 1 pod short; if 4 exist, the diff is 1 excess; if 3 exist, the diff is zero. In the act phase, the controller takes the minimum action to close the gap - creating exactly 1 pod if short, deleting 1 if excess, or doing nothing if states match. The loop then returns to observe and repeats indefinitely.

This pattern runs everywhere in Kubernetes simultaneously. The ReplicaSet controller ensures the right number of pods. The Deployment controller ensures rollouts progress correctly. The Node controller marks nodes unhealthy when heartbeats stop. The Endpoint controller updates Service endpoints when pods join or leave. The StatefulSet controller ensures ordered creation and deletion. Each controller runs its own independent loop. Together, they create a system that continuously self-heals without any central coordinator telling them what to do.

Tip: The reconciliation loop is also the pattern for building custom controllers. If you write a Kubernetes operator (a custom controller for application-specific resources), you implement the same observe-diff-act loop. Your operator watches your Custom Resource Definition (CRD), compares current application state to the declared desired state, and takes corrective action. The Prometheus Operator, the Cert-Manager, and every database operator work this way. Kubernetes is extensible because the same self-healing pattern that powers built-in controllers is available to anyone.

Labels and Selectors: Flexible Grouping

graph TD
    subgraph "Labels: How Kubernetes Objects Connect"
        POD["Pod with Labels<br/>──────────────────<br/>app: payment-service<br/>version: v2<br/>environment: production<br/>team: checkout<br/>region: us-east<br/><br/>These key-value pairs are<br/>not metadata in the passive<br/>sense - they are the PRIMARY<br/>mechanism for grouping,<br/>selecting, and routing."]
        
        POD --> SVC["Service Selector<br/>──────────────────<br/>selector:<br/>  app: payment-service<br/>  version: v2<br/><br/>Routes traffic to all pods<br/>matching BOTH labels.<br/>Pod joins/leaves the Service<br/>automatically when labels<br/>match/unmatch."]
        
        POD --> NP["Network Policy Selector<br/>──────────────────<br/>podSelector:<br/>  team: checkout<br/><br/>Restricts ingress to pods<br/>owned by the checkout team.<br/>Any pod with this label gets<br/>the firewall rule applied."]
        
        POD --> MON["Prometheus Scrape Target<br/>──────────────────<br/>serviceMonitor:<br/>  selector:<br/>    environment: production<br/><br/>Discovers all production<br/>services for metrics scraping.<br/>New services with the label<br/>are auto-discovered."]
    end

This diagram shows how Kubernetes labels connect objects across the system. A pod carries labels like app: payment-service, version: v2, environment: production, and team: checkout. These aren't passive metadata - they're the primary mechanism for grouping and selection. A Service with selector app: payment-service and version: v2 routes traffic to all pods matching both labels. A Network Policy with podSelector team: checkout applies firewall rules to pods owned by that team. A Prometheus ServiceMonitor with selector environment: production auto-discovers all production services for metrics scraping. The same pod participates in all three selections simultaneously.

Warning: Label design is an architectural decision, not a cosmetic one. At Razorpay, before I joined Google, our early Kubernetes deployment used inconsistent label keys across teams. One team labelled pods with app, another used application, a third used service-name - all meaning the same thing. When we tried to build platform-wide service mesh routing rules, the inconsistent labels meant we could not write uniform selectors. We spent two weeks relabelling 200+ Deployments across 8 teams. Establish a label taxonomy before your first production deployment, document it in a shared standard, and enforce it with admission controllers that reject resources missing required labels.

The Production War Story

The moment that taught me why declarative infrastructure matters happened during my time at Razorpay, before I joined Google. We ran our payment processing platform on Kubernetes - 40+ microservices handling real monetary transactions across Indian payment networks. On a Friday evening at 7:43 PM (it is always Friday evening), one of our three Kubernetes worker nodes experienced a hardware failure. The node went offline without warning - motherboard failure, no graceful shutdown, no drain, no advance indication.

In a pre-Kubernetes world, this would have been a serious incident. Someone would need to be paged. They would need to SSH into the remaining servers, determine which services were affected, find capacity, manually restart each service, update load balancer configurations, verify health, and monitor for cascading failures. On a Friday evening. For a payment system processing real money.

With Kubernetes, here is what actually happened: the node controller detected that node-2 stopped sending heartbeats. After the configured timeout (40 seconds in our cluster), the node was marked NotReady. The pods on that node were marked for rescheduling. The scheduler evaluated the two remaining healthy nodes for capacity - both had sufficient CPU and memory headroom because we had configured resource requests conservatively. New pods were created on healthy nodes. The ReplicaSet controllers verified that desired replica counts were restored. The Endpoint controllers updated Service endpoints to reflect new pod IPs. Total time from hardware failure to full service recovery: under 90 seconds. No human intervention. No pager. No escalation.

We discovered the node failure on Monday morning when we reviewed cluster metrics and saw that node-2 had been offline since Friday. Payment processing had been uninterrupted the entire weekend. That 90-second automatic recovery - from unexpected hardware failure to full service restoration with zero downtime - is what the declarative model and reconciliation loops buy you in production.

Interview: "What is the declarative model in Kubernetes and why does it matter?" is a foundational interview question. The strong answer: "Kubernetes uses a declarative model where operators submit desired-state documents (YAML manifests) rather than issuing imperative commands. The system stores desired state in etcd and runs reconciliation loops that continuously observe actual state, compare it to desired state, and take the minimum action to close any gap. This matters because declarations survive failures: if a node crashes, the desired state persists in etcd, and controllers automatically restore the correct number of replicas without human intervention. An imperative system requires someone to notice the failure, assess the impact, and re-issue the failed commands. The declarative model replaces human reaction time with automated convergence toward the declared state - which is why a node can fail on Friday night and services recover in 90 seconds without anyone being paged."

Six Takeaways

Kubernetes descends from Google's Borg. It inherits Borg's core insights - declarative state, reconciliation loops, label-based grouping - while fixing Borg's mistakes (rigid hierarchy, port-based networking that caused conflict management overhead).
Declarative state management means you describe what you want, not how to achieve it. YAML manifests define desired state. The system continuously reconciles actual state toward that declaration, surviving failures automatically.
Reconciliation loops are the self-healing mechanism. Every controller runs an observe-diff-act loop via the Watch API. When actual state diverges from desired state, controllers take the minimum corrective action.
Labels replace rigid hierarchies with flexible grouping. Services, Network Policies, and monitoring systems all use label selectors to discover and group objects - the same pod can participate in multiple selections simultaneously.
The declarative model survives failures that imperative models cannot. An imperative command is fire-and-forget. A declaration persists in etcd and is continuously enforced by controllers, even after node failures or cluster restarts.
Label taxonomy is an architectural decision. Inconsistent labels across teams compound into routing, monitoring, and policy enforcement problems. Define the taxonomy early and enforce it with admission controllers.

Exercises

Exercise: Starter: A team deploys their application with docker run on 3 servers, manually starting containers on each and configuring a load balancer by hand. (a) What happens when Server 2 crashes at 3 AM? How does the team recover? Estimate the time. (b) Rewrite this as a Kubernetes Deployment with 3 replicas. What happens when a node fails? (c) Explain the fundamental difference between the imperative approach (docker run) and the declarative approach (kubectl apply -f deployment.yaml). (d) Why does Kubernetes persist desired state in etcd rather than just executing commands?

Exercise: Intermediate: Trace the reconciliation loop for a Deployment with replicas: 3 when the following events occur in sequence: (a) A pod is OOMKilled. Which controller detects this? What action does it take? (b) A node goes offline. How long before Kubernetes reacts? What happens to the pods on that node? (c) An engineer manually deletes a pod with kubectl delete pod. What happens and why? (d) The Deployment spec is changed to replicas: 5. Trace the reconciliation from desired=5, actual=3 through to desired=5, actual=5, naming each controller involved.

Exercise: Advanced: Design a label taxonomy for a platform with 50 microservices, 3 environments (dev, staging, production), 8 teams, and 3 release tracks (stable, canary, experimental). (a) Define the required label keys and their allowed values. (b) Write an OPA/Gatekeeper constraint that rejects any Deployment missing required labels. (c) Write the label selectors for: "all production pods owned by the checkout team," "all canary pods in staging," "all pods running version v2.3 of the payment service." (d) A service mesh (Istio), monitoring (Prometheus), and cost attribution tool all depend on labels. How do you keep them consistent across all three systems?

Back to System Design Advanced