Free lesson · System Design Fundamentals

Scalability

Easy8 min readsystem-designhlddistributed-systems

Lesson 2.1 - Scalability

The first lesson of Module 2.

Scalability is the most overused word in system design - and the most misunderstood. It doesn't mean "fast." It doesn't mean "big." It means: can this system handle increased load by adding resources, without redesigning it? A system that serves 100 users well but collapses at 10,000 isn't scalable. A system that serves 100 users well and serves 10,000 users well by adding more machines - without rewriting the application - that's scalable. The distinction between "works at current load" and "works at 10x or 100x load" is the difference between a prototype and a production system.

Two Kinds of Scaling

Interactive animation: Scaling up makes one machine bigger. Scaling out adds machines.

There are exactly two ways to handle more load: make the machine bigger, or add more machines.

flowchart LR
    subgraph Vertical["Vertical Scaling (Scale Up)"]
        V1["Small server\n4 CPU, 16 GB RAM\n500 req/s capacity"]
        V2["Bigger server\n32 CPU, 128 GB RAM\n4,000 req/s capacity"]
        V1 -->|"upgrade hardware"| V2
    end
    
    subgraph Horizontal["Horizontal Scaling (Scale Out)"]
        LB["Load Balancer"]
        H1["Server 1\n500 req/s"]
        H2["Server 2\n500 req/s"]
        H3["Server 3\n500 req/s"]
        H4["Server 4\n500 req/s"]
        LB --> H1 & H2 & H3 & H4
    end

    style Vertical fill:#fbbf24,stroke:#f59e0b,color:#000
    style Horizontal fill:#34d399,stroke:#10b981,color:#000

A diagram comparing vertical and horizontal scaling side by side. On the left, vertical scaling: you replace a small server (4 CPUs, 16 GB RAM, handling 500 requests per second) with a bigger server (32 CPUs, 128 GB RAM, handling 4,000 requests per second). The application doesn't change - it just runs on better hardware. On the right, horizontal scaling: four identical servers, each handling 500 requests per second, sit behind a load balancer. Together they handle 2,000 requests per second. To handle more, add more servers. The load balancer distributes incoming requests across all servers, so no single server is overwhelmed. The fundamental difference: vertical scaling makes one machine more powerful; horizontal scaling adds more machines. Vertical is simpler (one server, no coordination) but has a ceiling - the biggest machine money can buy is still finite. Horizontal has no theoretical ceiling (keep adding servers) but introduces complexity - the application must handle being distributed across multiple machines, which introduces all the problems this course teaches: replication, partitioning, consistency, coordination.

Vertical scaling (scale up) means getting a bigger machine. More CPU cores, more RAM, faster SSDs, better network cards. The application doesn't change - it runs the same code on better hardware. This is the simplest kind of scaling because nothing about the architecture changes. A single PostgreSQL instance can handle remarkably high load on modern hardware - 50,000+ queries per second on a well-tuned server with NVMe SSDs and 256 GB of RAM. Many production systems run comfortably on a single powerful machine far longer than engineers expect.

But vertical scaling has a ceiling. The largest EC2 instance AWS sells (u-24tb1.metal: 448 vCPUs, 24 TB RAM) is finite. And more practically, cost scales non-linearly - a machine with 2x the CPU often costs 2.5-3x the price. A machine with 4x the RAM might cost 5x. You hit diminishing returns before you hit the physical limit. There's also a single-machine availability problem: one machine means one point of failure. If it dies, everything is down until it's replaced.

Horizontal scaling (scale out) means adding more machines. Each handles a fraction of the load. A load balancer (M4) distributes requests across them. To handle more traffic, add more machines. There's no theoretical ceiling - you can keep adding servers as long as the application supports it.

The catch: horizontal scaling requires the application to work as a distributed system. Data must be replicated or partitioned across machines. Requests must be routed to the right server. State must be coordinated. Sessions must be handled without assuming the same server handles every request from the same user. Every hard problem in this curriculum - replication (M28), partitioning (M29), consistency (M30), coordination (M31), distributed transactions (M32) - exists because of horizontal scaling. The complexity isn't accidental; it's the price of scale.

Scaling What, Exactly?

Interactive animation: Compute, storage, network, and the database each scale on their own terms.

"We need to scale" is the beginning of a conversation, not a conclusion. Scale what?

Scale for more users. A social network going from 1,000 to 10 million users. The database grows (more rows), the request rate grows (more concurrent users), and the data model may need to change (sharding user data across databases). This is the most common scaling challenge.

Scale for more data. A log aggregation system growing from 100 GB to 100 TB. The storage and query engine must handle the volume. The indexing strategy might need to change entirely (from a single database to a distributed search engine like Elasticsearch - M20). Data volume scaling often requires fundamentally different architectures, not just more servers.

Scale for more requests per second. An API going from 100 req/s to 100,000 req/s. The compute and network layers must absorb the traffic. Caching (M11-M12) often provides the first 10x; horizontal scaling provides the next 10x; architectural changes (async processing - M9, CDNs - M11) provide the rest.

Scale for more geographic distribution. A service expanding from one US data center to serving users globally. Latency becomes the constraint - a user in Tokyo hitting a server in Virginia experiences 150ms of network latency per round trip. Multi-region deployment, CDNs, and edge computing address this.

Scale for more features / teams. An application growing from 3 services maintained by one team to 300 services maintained by 50 teams. The organizational complexity (deployment coordination, API contracts, monitoring, debugging) scales with the system. Microservices architecture (M36) addresses this - but adds its own complexity.

Each type of growth stresses different parts of the system. "We need to scale" requires the follow-up: "scale WHAT, from WHAT to WHAT, by WHEN?"

Concrete Examples

Vertical scaling success: Stack Overflow. One of the world's most-visited websites serves hundreds of millions of page views per month on a remarkably small server fleet - under 20 servers for the main application. They scale vertically (powerful hardware with 512 GB RAM, aggressively optimized code with heavy caching) rather than horizontally. Their database is a single SQL Server instance - not sharded, not distributed. It works because their workload is overwhelmingly read-heavy, their data fits in memory, and their code is meticulously optimized. The lesson: vertical scaling carries you further than most engineers assume, especially with aggressive caching and query optimization.

Horizontal scaling success: Netflix. Serves 200+ million subscribers streaming video simultaneously across every continent. No single machine could handle this - not the video delivery (terabits per second of bandwidth), not the metadata (billions of device connections), not the recommendations (massive ML inference). Netflix horizontally scales everything: video delivery across thousands of CDN edge servers (Open Connect Appliances), metadata across distributed databases (Cassandra), recommendations across distributed compute clusters, and API traffic across hundreds of microservice instances. The entire architecture assumes horizontal distribution from the ground up.

Scaling failure: early Twitter (2007-2010). Twitter's original architecture was a Ruby on Rails monolith backed by a single MySQL database. As usage grew to millions of users, the database became the write bottleneck - every tweet was a write, and the fan-out (delivering a tweet to every follower's timeline) required reads proportional to follower count. The architecture couldn't scale horizontally because the data model wasn't designed for it - sharding the "following" graph across databases required fundamental redesign. The result: the notorious "Fail Whale" - frequent outages during peak traffic while the engineering team rebuilt the entire backend architecture. Twitter eventually moved to a distributed system with Cassandra, Manhattan (their custom key-value store), and a complete redesign of the tweet delivery pipeline. The rebuild took years and cost hundreds of millions of dollars.

The Twitter lesson is the most instructive: scalability isn't something you bolt on later. It's a property of the architecture. An architecture that works at 1x but can't work at 100x isn't just "not yet scaled" - it needs to be redesigned. The cost of that redesign grows with every user, every data record, and every feature built on the unscalable foundation.

The Premature Scaling Trap

Interactive animation: The signals that say it is time to scale - and the ones that only feel like it.

The opposite mistake is equally dangerous: scaling before you need to. Every horizontal scaling decision adds complexity - distributed coordination, eventual consistency, network partitions, operational overhead. If your system serves 500 users and you're building for 50 million, you're spending engineering time on problems you don't have instead of building the product people are waiting for.

There's a startup graveyard full of teams that built sophisticated distributed architectures for 50 million users and never got past 500 - not because the architecture was wrong, but because they spent 6 months building infrastructure instead of finding product-market fit.

The discipline: scale for the load you have and the load you'll have in 6 months. Not the load you dream about. When you hit a scaling bottleneck, you'll know - latency increases, error rates climb, the database connection pool is exhausted. At that point, measure, identify the bottleneck, and address it specifically. Don't pre-solve every possible scaling problem before it manifests.

A Brandon Story: The
2,000 Architecture for 500 Users

A startup I'd advised had 500 users. Their architecture: Kubernetes cluster with 12 nodes, DynamoDB with reserved capacity for 10,000 writes per second, Kafka event streaming cluster, ElastiCache Redis cluster, and a CDN in front. Monthly AWS bill:

2,000. The system could serve 10 million users. It currently served 500.

I asked them to run their application on a single EC2 instance with PostgreSQL. They resisted - "what about availability? What about scaling? What about event-driven architecture?" I pointed out: at 500 users, their PostgreSQL instance would be at 0.1% utilization. A single t3.medium ($30/month) could handle their load with 99x headroom.

They tried it. Monthly cost dropped to $400 (EC2 + RDS + basic setup). The application was faster - no network hops between services, no serialization overhead between Kafka and consumers, no Kubernetes networking complexity. Product development velocity doubled because engineers stopped debugging distributed systems issues and started building features.

Two years later, at 50,000 users, they hit their first real scaling bottleneck - read latency on a specific query. They added a read replica and a targeted cache. Cost: $200/month more. At 200,000 users, they introduced horizontal scaling for the API layer. At 500,000, they sharded the database. Each scaling step was a response to a measured bottleneck, not a preemptive architecture decision.

Lessons:

A single PostgreSQL instance handles more than most engineers believe - 50,000+ queries per second on modern hardware
Premature horizontal scaling adds distributed-systems complexity before you have distributed-systems traffic
2,000/month for 500 users is not "being prepared" - it's burning money on imaginary problems
Scale when you have a measured bottleneck, not an imagined one
Each scaling step should address a specific, measured constraint - not a general fear of "what if we grow"

Six Takeaways from This Lesson

Scalability = handling more load by adding resources without redesigning the system. Not "fast" or "big."
Vertical scaling (bigger machine): simple, no distribution complexity. Dismissed too quickly. A single PostgreSQL instance handles more than you think.
Horizontal scaling (more machines): no ceiling, but introduces every distributed-systems problem in this curriculum.
Specify what's scaling: users, data, requests, geography, or team size. Each stresses different components.
Premature scaling is as dangerous as no scaling. Build for 6 months ahead, not 6 years.
Twitter's Fail Whale teaches: scalability is an architectural property, not a bolt-on. Redesigning an unscalable system is orders of magnitude more expensive than designing for scale from the start - but only when scale is actually needed.

Try It

Exercise: Warmup (15 minutes). For a system you use daily (Gmail, Spotify, your company's app), estimate: how many users? How many requests per second? What's the data size? Could a single powerful server ($500/month, 32 CPUs, 128 GB RAM, NVMe SSDs) handle it?

Exercise: Core (25 minutes). A startup has 10,000 users and expects 1 million in a year. Their app is a CRUD API backed by PostgreSQL. At what point would they need to scale horizontally? List the bottlenecks in order: which component hits its limit first? (Hint: usually reads before writes, and the database before the API servers.)

Exercise: Stretch (45 minutes). Read the Stack Overflow architecture overview (link below). Note: how many servers they use, their caching strategy, their database approach. Then read about Netflix's architecture. Compare: why does Stack Overflow use vertical scaling while Netflix uses horizontal? What's different about their workloads?

What's Next

Lesson 2.2 covers availability - uptime as a number, the "nines," and what 99.9% actually means in practice.

Back to System Design Fundamentals