AlgoInsight

Free lesson · System Design Crash Course

Reading Existing Systems

Easy11 min readsystem-designhlddistributed-systems

Lesson 1.4 - Reading Existing Systems

The "how to actually learn HLD" lesson.

Here's something most courses don't tell you: you don't really learn system design from courses. You learn the vocabulary and the frameworks from courses (like this one). But the actual depth comes from reading the architectures that real engineers at real companies have built. Once you can read those, dissect them, and understand why they made the choices they did, your sense of system design gets sharper than any course alone could teach.

The good news: those architectures are mostly public. Major tech companies maintain engineering blogs where they describe in detail how their systems work. Netflix has a blog. Stripe has one. Discord has one. GitHub, Cloudflare, Slack, Uber, Airbnb, LinkedIn - they all publish architecture posts regularly. The information is sitting there, free, waiting for anyone willing to read it carefully.

This lesson is about doing that reading well. We'll cover where to find these posts, what questions to ask while reading, and how to extract real learning from them rather than just skimming.

Why This Matters More Than You'd Think

Let me make the case before we get to the how-to.

Junior engineers learn from courses and tutorials. Senior engineers learn from production. But there's a gap in between - you're past the tutorial stage but you don't yet have production experience at scale. Reading other people's architectures is how you bridge that gap.

When a Netflix engineer writes a blog post about how they redesigned their content recommendation system, they're giving you years of their team's learnings for free. They tell you what they tried first. What didn't work. What surprised them. What they'd do differently. This is information you literally cannot get from a textbook because it's the kind of knowledge that only comes from operating systems at scale.

If you read one architecture write-up per week, deeply, for a year, your system design intuition will exceed what most engineers ever develop. That's not an exaggeration. The compounding of "what I learned from reading how the experts solved X" is enormous.

The skill is in reading well, not just reading. Let me show you how.

Where to Find Good Architecture Writing

The major engineering blogs you should bookmark right now. These are organized roughly by what they're best at:

Streaming and global scale:

Payments and financial infrastructure:

Real-time messaging:

Developer platforms:

Edge computing and infrastructure:

Ride-sharing and marketplaces:

Travel and lodging:

Team collaboration:

Aggregation sites:

The "system design primer" types of resources:

You don't need to follow all of these. Pick three or four that align with your interests, and check them periodically. Different blogs have different posting frequencies - Netflix posts a lot; Discord posts less often but their posts are usually excellent.

The Questions to Ask While Reading

Interactive animation: Five questions for every architecture post: the problem, the constraints, the alternatives, the surprises, and what they would change.

Reading well is different from just reading. When you encounter an architecture post, here are the five questions you should be asking. Train yourself to actively think through each one.

Question 1: What problem were they solving?

Engineers don't redesign systems for fun. Every architecture decision is in response to a specific problem. The first thing to identify in any architecture post is: what triggered this work?

Common triggers:

Once you know the problem, the choices in the post become legible. They're not arbitrary preferences - they're responses to constraints. A design that seems weird in isolation often makes perfect sense once you know what problem it was solving.

For example: "Netflix moved from a monolith to microservices." Why? Not because microservices are cool - because they needed to deploy independently across hundreds of teams. The problem (independent deployment) explains the choice (microservices).

Question 2: What constraints shaped the design?

After understanding the problem, look for the constraints - the things the team had to work within. Constraints come in many forms:

Scale constraints. "We need to handle 1 million requests per second." That's a constraint.

Latency constraints. "End-to-end response time must be under 200ms." Constraint.

Reliability constraints. "99.99% uptime required." Constraint.

Cost constraints. "We can't spend more than $X per user." Constraint.

Compatibility constraints. "The existing API must keep working." Constraint.

Team / organizational constraints. "We have 10 engineers, and three of them are about to go on parental leave." Constraint.

Technology constraints. "We must use AWS because of an existing contract." Constraint.

These constraints often aren't stated explicitly in the post, but they're hidden behind every decision. Trying to infer them is a useful exercise. "Why did they shard by user ID instead of by region? Probably because the scale constraint demanded uniform distribution, and regional sharding would create hot regions."

Question 3: What were the alternatives?

The best architecture posts mention what the team considered and rejected. The shape of "we considered X but chose Y because Z" is golden - it tells you the trade-offs.

When alternatives aren't mentioned explicitly, try to guess them. "They went with Kafka. What other options exist? RabbitMQ, AWS SQS, Google Pub/Sub, Redis Streams. Why did they pick Kafka over these?" Sometimes you can figure it out from context. Sometimes you can't, and that's a question worth asking.

Question 4: What surprised them in production?

Look for sentences like:

These are gold. They're the production-only knowledge that you can only get from operating systems at scale. When an engineer tells you "we thought X would work fine, but in production we found Y," they're sharing knowledge that took them months and thousands of dollars of incidents to learn.

Make notes on these. Build a personal collection of "things experienced engineers told me they learned in production." Over time, this collection is incredibly valuable.

Question 5: What would they do differently?

Some posts include retrospective reflections. "If we were starting over today, we'd..." These are honest assessments from teams that have lived with their decisions. They tell you what the long-term consequences of design choices look like.

Even when posts don't include this explicitly, you can sometimes infer it. "They wrote this in 2018, and we know that by 2021 they migrated off this design. Why?" The follow-up is often the more interesting story.

A Worked Example: Reading One Real Post

Let me walk you through how I'd read one famous architecture post, asking these five questions as I go.

The post: "How Discord Stores Trillions of Messages" from Discord's engineering blog. It's a real post from 2023. (If you haven't read it, pause and read it now - it's only about 15 minutes long and will make this section more useful.)

Here's what I'd extract using the five questions:

Question 1: What problem were they solving? They needed to store an enormous and growing number of messages - trillions of them. The previous database (Cassandra) was hitting operational limits at scale. So the problem was: "Our existing message storage doesn't scale gracefully to where we're heading."

Question 2: What constraints shaped the design?

Question 3: What were the alternatives? Discord considered staying on Cassandra and tuning further. They could have moved to a different distributed database. They could have built something custom. They ultimately chose ScyllaDB - which is Cassandra-compatible but built differently. The reasoning: ScyllaDB's architecture handles the operational issues they had with Cassandra better, while preserving their existing data model.

Question 4: What surprised them in production? They learned that hot partitions (one channel getting massively more traffic than others) were harder to handle than expected. They also learned that backup/restore operations at trillion-scale had unique challenges that didn't show up at smaller scale.

Question 5: What would they do differently? They suggest some future improvements - things they're still working on. Production at this scale is an ongoing journey, not a finished destination.

If you read the post this way - actively, asking these five questions - you'll absorb the lessons. If you just skim it for vibes, you'll forget most of it within a week.

Interactive animation: The five questions applied to Discord's post on storing trillions of messages, from Cassandra's limits to ScyllaDB.

A Reading Routine That Works

Here's a routine I'd suggest for getting good at this:

Once a week, pick one architecture post from one of the blogs above. Read it twice. The first time, just read for understanding. The second time, take notes using the five questions.

Keep a personal knowledge base. Note the post title, the company, the problem, the key decisions, the surprises, and your takeaways. After a few months, you'll have your own library of architecture learnings.

Discuss what you read with other engineers. Even if it's just "did you see this Netflix post?" - talking through the architecture forces you to organize your thinking.

Try to apply the patterns. When you encounter a design challenge at work, ask yourself: "Have I read about something similar?" Often, you'll have. The patterns from these blogs show up in your own work eventually.

Read across companies, not just one. Netflix's solutions work for Netflix's problems. Discord's solutions work for Discord's problems. Reading widely gives you a sense of which patterns are common and which are specific.

A Suggested Starter Reading List

If you don't know where to start, here are five posts to read first. Each one is famous in the system design community for good reason:

  1. How Discord Stores Trillions of Messages - Scale of message storage. Already discussed above.

  2. Designing Instagram's Feed by Meta or any of Meta's posts about how Instagram works. Massive social feed, real-time scaling.

  3. Stripe's Engineering Blog on Idempotency Keys - Not a full architecture post, but the kind of detailed problem-solving that's worth studying.

  4. How Slack handles 10K+ messages per second - Slack's posts on real-time messaging are excellent. Browse for relevant titles.

  5. Uber's Marketplace Architecture - Browse Uber's blog for posts about how they match riders with drivers in real-time. Several relevant posts.

These are starting points. Once you read these and want more, the blogs above will keep you in material for years.

What to Do When Reading Feels Confusing

Some architecture posts will feel impenetrable at first. That's normal. Here's how to handle it:

Skim first, then read carefully. A first read just to get the vibe. A second read to actually understand. Posts about novel infrastructure (like Stripe's idempotency keys or Discord's message storage) often need multiple reads.

Look up terms you don't know. If a post mentions "consistent hashing" and you've never heard of it, pause and Google it. (You'll see it discussed in our Module 2 of this course.) Building vocabulary is the prerequisite for reading these posts.

Don't try to retain everything. No one remembers every detail of every post. Capture the headline lesson - "Discord uses ScyllaDB to handle trillions of messages" - and the one or two specific decisions that stood out. That's enough.

It's OK to find a post you can't understand yet. Some posts assume background you don't have yet. Bookmark them and come back in six months. You'll be amazed how much more accessible they feel after more learning.

Read with other people. A study group or book club, focused on architecture posts, is one of the highest-leverage learning activities you can do. Even an informal weekly discussion with one other engineer.

Beyond Blogs: Other Sources

Engineering blogs are the most accessible, but there are other sources worth knowing:

Conference talks. QCon, StrangeLoop, KubeCon, AWS re:Invent, Google Cloud Next - all have talks where engineers present their architectures. YouTube is full of these. Look for talks by recognizable companies on topics that interest you.

Papers. Some companies (Google, Meta, Amazon historically) publish formal research papers on their systems. The Dynamo paper (Amazon, 2007) is the foundational paper for NoSQL databases. The MapReduce paper (Google, 2004) launched the modern big data era. Papers are dense but for foundational ideas, they're worth reading.

Books. Some are written by people who built famous systems. "Designing Data-Intensive Applications" by Martin Kleppmann is the canonical reference. "Site Reliability Engineering" by Google is free online. "The Datacenter as a Computer" by Barroso and Hölzle is Google's perspective on hyperscale.

Podcasts. Software Engineering Daily, The Cloudcast, Software Engineering Radio. These often interview engineers about systems they've built.

You don't need to use all of these. Pick a primary source (probably engineering blogs) and supplement.

A Warning About Cargo Culting

One last thing worth saying: don't blindly copy patterns from these posts. A common mistake is to read a Netflix architecture post and think, "I should build my app the way Netflix built theirs." This is almost always wrong.

Netflix's architecture is designed for Netflix's problems: 230+ million users, video streaming, global presence, hundreds of engineering teams. If you're building a side project or an early-stage startup, your problems are nothing like Netflix's. Their solutions will overcomplicate your system without providing benefit.

The way to read these posts is to learn how to think about problems, not to copy specific solutions. The patterns underlying Netflix's architecture (eventual consistency, replication, caching, async work) are widely applicable. But the specific implementations are tuned to their scale and shouldn't be copied without your own analysis.

When in doubt, ask: "What is the simplest design that meets my actual requirements?" That's a Netflix engineer would tell you to start from.

Six Takeaways from This Lesson

  1. Reading published architectures is the highest-leverage way to learn HLD beyond what courses can teach you.

  2. The major tech companies have public engineering blogs. Netflix, Stripe, Discord, GitHub, Cloudflare, Uber, Airbnb, Slack - all of them. The information is free.

  3. Five questions sharpen your reading: What problem? What constraints? What alternatives? What surprised them? What would they change?

  4. A weekly reading routine compounds. One post per week, deeply read, for a year, will transform your system design intuition.

  5. Don't read in isolation. Discussing what you read with other engineers makes it stick.

  6. Don't cargo cult. Learn how the engineers thought; don't blindly copy their solutions. Most of these designs are for problems much bigger than yours.

Try It

Exercise: Warmup (15 minutes). Open the Netflix Tech Blog. Browse the post titles. Pick one that looks interesting to you. Just look at the headline and intro - don't read the whole thing yet.

Exercise: Core (45 minutes). Read the Discord trillions of messages post. After reading, write a one-paragraph summary using the five questions framework.

Exercise: Stretch (90 minutes). Pick three different blogs from the list above. Read one post from each. Compare and contrast - how do the different companies approach similar problems? What patterns recur across them?

What's Next

Lesson 1.5 covers something most beginners miss: failure isn't a bug; it's a design dimension. The shift in mindset from "what works" to "what fails" is one of the biggest leaps from junior to senior engineering.

Back to System Design Crash Course