Free lesson · System Design Fundamentals

Vertical vs Horizontal Scaling

Medium11 min readsystem-designhlddistributed-systems

Lesson 18.1 - Vertical vs Horizontal Scaling

The first lesson of Module 18.

"Just scale horizontally" is the default advice in system design - add more machines, distribute the load, grow linearly. It sounds universally correct. But horizontal scaling introduces distributed systems complexity that vertical scaling avoids entirely: data partitioning, cross-node coordination, network failure handling, consistency protocols, and operational overhead for a cluster of machines instead of one. Vertical scaling - making one machine bigger - is simpler, often cheaper, and handles far more load than most engineers assume. A single modern server with 128 cores, 2 TB of RAM, and NVMe SSDs handles workloads that engineers reflexively distribute across 20 machines.

The trade-off isn't "vertical = bad, horizontal = good." It's: vertical scaling is simpler and sufficient for longer than most teams realize, but it has a ceiling. Horizontal scaling is more complex but has no ceiling. The art is knowing where the ceiling is for your workload and not introducing horizontal complexity before you've reached it.

Vertical Scaling: More Machine

Vertical scaling (scaling up) means adding more resources to a single machine - more CPU cores, more RAM, faster disks, faster network. The same application, on the same code path, just runs on more powerful hardware.

flowchart LR
    subgraph Vertical["VERTICAL SCALING"]
        V1["Phase 1\n4 cores, 16 GB RAM\ndb.t3.xlarge\n
40/month\n\nHandles: 2,000 req/s"]
        V2["Phase 2\n16 cores, 64 GB RAM\ndb.r6g.2xlarge\n$580/month\n\nHandles: 12,000 req/s"]
        V3["Phase 3\n64 cores, 512 GB RAM\ndb.r6g.16xlarge\n$4,600/month\n\nHandles: 60,000 req/s"]
        V4["Phase 4\n128 cores, 2 TB RAM\ndb.x2idn.32xlarge\n$20,000/month\n\n🚫 Ceiling reached.\nNo larger instance exists."]
        
        V1 -->|"upgrade"| V2 -->|"upgrade"| V3 -->|"upgrade"| V4
    end

    style V4 fill:#fb7185,stroke:#f43f5e,color:#000

A diagram showing vertical scaling progression through four phases. Phase 1: a small instance with 4 cores and 16 GB RAM, handling 2,000 requests per second at 
40 per month. Phase 2: upgrade to 16 cores and 64 GB RAM - 12,000 requests per second at $580 per month. A 6x throughput increase for a 4x cost increase. Phase 3: upgrade to 64 cores and 512 GB RAM - 60,000 requests per second at $4,600 per month. Phase 4: the largest available instance - 128 cores and 2 TB RAM at $20,000 per month. This is the ceiling. No larger instance exists from the cloud provider. If the workload exceeds this machine's capacity, vertical scaling cannot help. At this point, horizontal scaling becomes necessary. The key insight: 60,000 requests per second on a single machine is MORE than most applications need. A startup at Series B with 500,000 daily active users generates roughly 5,000-10,000 requests per second at peak - well within Phase 2 or Phase 3 territory. The "we need to scale horizontally" conversation happens far later than most teams assume.

What vertical scaling handles (more than you think):

A PostgreSQL database on a db.r6g.16xlarge (64 vCPUs, 512 GB RAM) handles 60,000+ queries per second with proper indexing. That's enough for a product with millions of daily active users. Most startups shard their database years before they need to - introducing distributed systems complexity (cross-shard queries, distributed transactions, shard management) for a workload that a bigger machine handles trivially.

The advantages of vertical scaling:

1. Zero distributed complexity. One machine means no network partitions between nodes, no data partitioning, no cross-node coordination, no distributed transactions, no consensus protocols. The entire application runs in one address space. JOINs work. Transactions work. EXPLAIN works. Everything you know about single-machine debugging, profiling, and optimization applies.

2. Simpler operations. One server to monitor, one server to back up, one server to patch, one set of logs, one connection pool. A 10-person engineering team that operates 20 horizontally-scaled database shards spends significant engineering time on shard management, rebalancing, and cross-shard query issues. The same team with one large PostgreSQL instance spends that time on product features.

3. Lower total cost at moderate scale. A db.r6g.16xlarge ($4,600/month) handles what might require 8 smaller machines ($8,000-12,000/month) in a horizontally scaled setup - plus the engineering time to manage the cluster, the load balancer, the connection routing, and the operational runbooks.

4. Instant scaling. Upgrading from a db.r6g.4xlarge to a db.r6g.8xlarge takes minutes (RDS: modify instance, schedule during maintenance window, ~5 minutes of downtime for a restart). Adding 4 more shards to a horizontally scaled database takes weeks (data migration, routing updates, testing, dual-write verification - see M16.1).

Horizontal Scaling: More Machines

Horizontal scaling (scaling out) means adding more machines - distributing the workload across multiple servers. Each server handles a fraction of the total load. Adding servers increases total capacity linearly (in theory).

flowchart TD
    LB["Load Balancer"]
    
    S1["Server 1\n4 cores, 16 GB\n
40/month"]
    S2["Server 2\n4 cores, 16 GB\n
40/month"]
    S3["Server 3\n4 cores, 16 GB\n
40/month"]
    S4["Server 4\n4 cores, 16 GB\n
40/month"]
    SN["Server N\n..."]
    
    LB --> S1 & S2 & S3 & S4 & SN
    
    Note["Total: N × 2,000 req/s\nCost: N × 
40/month\nNo ceiling (add more servers)\nBut: distributed complexity"]

    style LB fill:#fbbf24,stroke:#f59e0b,color:#000

A horizontal scaling diagram. A load balancer distributes traffic across N servers, each with 4 cores and 16 GB RAM, handling 2,000 requests per second each. Total capacity is N times 2,000 requests per second - need more? Add another server. There's no ceiling: 10 servers = 20,000 req/s, 100 servers = 200,000 req/s. The cost is N times 
40 per month - linear. But unlike vertical scaling, horizontal scaling introduces distributed systems complexity: the load balancer must route traffic, session state must be shared or externalized, databases must be sharded or replicated, and the application must handle partial failures (Server 3 is down but Servers 1, 2, 4 are fine). Each added server is simple individually but the coordination between servers is complex.

The advantages of horizontal scaling:

1. No ceiling. Need 10x more capacity? Add 10x more servers. Need 100x? Add 100x. Cloud auto-scaling groups add and remove servers automatically based on load - scaling up in minutes during traffic spikes and scaling down during off-peak to save cost. There's no "biggest instance" limit.

2. Fault tolerance. If one server in a pool of 10 dies, the other 9 continue serving traffic (at reduced capacity - 90%). With vertical scaling, if the single server dies, everything is down (SPOF - M2.4). Horizontal scaling provides inherent redundancy: the system survives individual machine failures.

3. Geographic distribution. Deploy servers in multiple regions - US East, EU West, Asia Pacific - and route users to the nearest one. Vertical scaling can't put one machine in three locations simultaneously.

4. Cost-efficient at very large scale. Above a certain throughput, the largest available machine is either insufficient or prohibitively expensive. 100 small machines (

The costs of horizontal scaling:

1. Distributed systems complexity. Network partitions between machines. Data consistency across replicas. Distributed transactions (slow, complex). Cross-node communication latency. Partial failure handling (some nodes up, some down). Each of these adds engineering effort, operational burden, and failure modes that don't exist on a single machine.

2. State management. On a single server, user session state lives in the process's memory. With multiple servers, the load balancer might route the same user to a different server on each request - and that server doesn't have the user's session. Solutions: sticky sessions (fragile - defeats load balancing), externalized state (Redis - adds a dependency), or stateless architecture (18.4 - the best solution, but requires designing for it).

3. Data partitioning. If the database is the bottleneck (not the application servers), horizontal scaling means sharding the database (M16.1) - the most complex scaling operation. Cross-shard JOINs break. Distributed transactions are slow. Resharding is a multi-week migration.

4. Operational overhead. 20 servers × monitoring, logging, patching, deployment, health checks. CI/CD must deploy to all instances (rolling deployments, blue-green). Log aggregation (Datadog, ELK) is essential because logs are distributed across machines. A single server's logs are in one place - tail -f /var/log/app.log.

The Decision Framework

The question isn't "which is better?" but "at what scale does horizontal become necessary?"

The honest framing: most applications should scale vertically until they can't, then scale horizontally for the specific bottleneck (reads → replicas, writes → sharding, compute → more app servers). Don't scale horizontally preemptively - it's borrowing complexity against future capacity you might never need.

The Hybrid Reality

In practice, most production systems use both:

• Application tier: horizontal. Stateless app servers behind a load balancer. Adding servers increases request handling capacity. This is the easiest layer to scale horizontally because application servers are stateless (or can be made stateless - 18.4).
• Database tier: vertical first, then horizontal. Start with one PostgreSQL instance. Upgrade to a larger instance as needed. Add read replicas (15.4) when read volume exceeds one machine. Shard (16.1) only when write volume exceeds the largest instance - which most applications never reach.
• Cache tier: horizontal. Redis Cluster or Memcached distribute cached data across nodes (M11.2). Cache workloads benefit from horizontal scaling because cache access is inherently partitionable by key.

flowchart TD
    LB["Load Balancer"]
    
    App1["App Server 1"] 
    App2["App Server 2"]
    App3["App Server 3"]
    
    Cache["Redis Cluster\n(horizontal - 3 shards)"]
    
    DB_Primary["PostgreSQL Primary\n(vertical - largest instance)\n64 cores, 512 GB RAM"]
    DB_R1["Read Replica 1"]
    DB_R2["Read Replica 2"]
    
    LB --> App1 & App2 & App3
    App1 & App2 & App3 --> Cache
    App1 & App2 & App3 -->|"writes"| DB_Primary
    App1 & App2 & App3 -->|"reads"| DB_R1 & DB_R2
    DB_Primary -->|"WAL stream"| DB_R1 & DB_R2

    style DB_Primary fill:#fbbf24,stroke:#f59e0b,color:#000
    style LB fill:#a78bfa,stroke:#8b5cf6,color:#000

A hybrid scaling architecture showing different scaling strategies for each tier. The application tier is horizontally scaled: 3 stateless app servers behind a load balancer. Adding app servers increases request handling capacity - easy, no state coordination needed. The cache tier is horizontally scaled: a Redis Cluster with 3 shards distributes cached data by key hash. The database tier is primarily vertically scaled: one large PostgreSQL primary (64 cores, 512 GB RAM) handles all writes. Read replicas (a form of horizontal scaling for reads only) distribute read traffic. The primary is the largest instance the workload can justify - scaling UP to handle more writes. Only when write volume exceeds this machine's capacity would sharding be considered. This is the standard production architecture: horizontal for stateless compute and cache, vertical-first for the database. Most applications run this architecture through Series C and beyond.

A Brandon Story: The Premature Sharding

A team I'd advised at a Series A startup (50,000 DAU, ~3,000 requests per second at peak) decided to shard their PostgreSQL database. Their reasoning: "we're growing fast - by next year we'll need it." They spent 8 weeks implementing application-level sharding by user_id across 4 database shards.

The result: their database had been running at 15% CPU utilization on a db.r6g.xlarge (4 vCPUs, 32 GB RAM) - a $290/month instance. The sharding introduced: a shard routing layer (custom middleware), cross-shard query handling (for admin dashboards that aggregated across all users), a shard management service (health checks, failover), distributed migrations (running ALTER TABLE across 4 instances), and a new deployment process (deploy to 4 databases instead of 1). Total engineering investment: 8 weeks × 3 engineers = 24 engineer-weeks.

What they should have done: upgrade to a db.r6g.2xlarge (8 vCPUs, 64 GB RAM, $580/month). At 15% CPU on 4 vCPUs, they had 20x headroom before the larger instance would even be stressed. With the 2xlarge, they had approximately 80x headroom over their current load - enough for 240,000 DAU at their per-user query rate. They could have reached Series C before even considering sharding.

The 24 engineer-weeks were wasted. The sharding infrastructure was never needed during the 2 years I tracked them - their user base grew to 300,000 DAU and the database handled it comfortably on a single db.r6g.4xlarge (

The cost comparison:

Same infrastructure cost.

Lessons:

• Measure your actual utilization before scaling. 15% CPU on the smallest instance means you have 20x+ headroom - sharding is years away.
• Vertical scaling from a small to a large instance is a 5-minute operation. Sharding is an 8-week project. The asymmetry is enormous.
• Every shard adds operational overhead: monitoring, migrations, backups, failover, connection management. 4 shards ≈ 4x the operational burden.
• "We'll need it next year" is speculative. "The database is at 80% CPU on the largest available instance" is evidence. Scale based on evidence, not projection.
• A db.r6g.16xlarge (64 vCPUs, 512 GB RAM) handles more concurrent load than most startups will ever need. Exhaust vertical scaling before even considering horizontal for the database tier.

Six Takeaways from This Lesson

1. Vertical scaling is simpler - zero distributed complexity, one machine to operate, instant upgrades. It handles far more load than most engineers assume (50,000+ PostgreSQL queries/sec on a 64-core machine).

2. Horizontal scaling has no ceiling - add more machines indefinitely. But it introduces distributed systems complexity: partitioning, coordination, partial failures, state management.

3. Scale vertically until you can't, then horizontally for the specific bottleneck. Reads → replicas. Writes → sharding. Compute → more app servers. Don't scale everything horizontally.

4. The hybrid architecture is the standard: horizontal for stateless app servers and cache, vertical-first for the database. Most applications run this through Series C+.

5. Premature horizontal scaling is expensive. An 8-week sharding project on a database at 15% utilization wastes engineering time and adds permanent operational complexity.

6. Scale based on evidence (measured utilization), not projection ("we'll need it next year"). A 5-minute instance upgrade beats an 8-week sharding migration.

Try It

Exercise: Warmup (15 minutes). Check your production database's CPU and memory utilization (CloudWatch, top, htop). What percentage is currently used? How large is the next available instance size? How many times could you vertically scale before hitting the ceiling?

Exercise: Core (25 minutes). A startup with 100,000 DAU generates 8,000 requests/sec at peak. Their PostgreSQL database runs on a db.r6g.xlarge (4 vCPUs) at 40% CPU. A colleague proposes sharding. Calculate: how many more users could the current instance handle? What instance size would be needed for 1 million DAU? At what DAU would the largest instance (db.r6g.16xlarge, 64 vCPUs) reach 80% utilization?

Exercise: Stretch (30 minutes). Design a scaling roadmap for a social media application. Phase 1 (0-100K DAU), Phase 2 (100K-1M), Phase 3 (1M-10M), Phase 4 (10M+). For each phase: what scales vertically? What scales horizontally? What's the trigger for moving to the next phase?

What's Next

Lesson 18.2 covers concurrency vs parallelism - two related but distinct concepts that engineers frequently conflate, with different implications for system design.

Back to System Design Fundamentals