Free lesson · System Design Interviews

Step 8: Bottlenecks and Trade-offs

Hard13 min readsystem-designinterview-prep

This is the step where you prove you can think like someone who's been paged at 2 AM because the system is down. Not "here's the architecture I drew on a whiteboard," but "here's what would actually break, here's the alert I'd set up, and here's the runbook I'd write." When I ask this question as an interviewer, I'm evaluating production maturity. Can you anticipate failure modes before they happen? Can you articulate what you traded away when you made each design decision?

I've seen plenty of candidates design beautiful architectures and then freeze when I ask "what breaks first?" The ones who impress me are the ones who've already been thinking about this throughout the interview - they point at a component on their diagram and say "this is my biggest risk" without hesitation. That confidence comes from understanding the trade-offs you made at every step. Let me walk through them.

Bottleneck Analysis

Bottleneck 1: S3 as a Single Content Store

Every paste's content lives in a single S3 bucket. S3 is extraordinarily reliable (11 nines of durability), but it's still a single dependency. If S3 experiences a regional outage - which has happened - every paste view returns an error.

flowchart TD
    READ["Read request<br/>(cache miss)"] --> S3{"S3 Available?"}
    
    S3 -->|"Yes"| SERVE["Serve content<br/>Normal operation"]
    S3 -->|"No (regional outage)"| FAIL["All cache-miss reads fail<br/>CDN-cached pastes still served<br/>New paste creation fails"]
    
    MITIGATION["Mitigations:<br/>1. S3 Cross-Region Replication<br/>2. Cache extension during outage<br/>3. CDN with long TTL absorbs impact"]

Bottleneck 1: S3 as the single content store. All paste content lives in one S3 bucket. If S3 experiences a regional outage, every cache-miss read fails. CDN-cached pastes continue to be served because the CDN doesn't need to contact origin. New paste creation also fails because it can't write to S3. Mitigations include S3 Cross-Region Replication for automatic failover to another region, extending cache TTLs during outages so cached content stays available longer, and relying on CDN's long TTL to absorb the impact for popular pastes.

Impact assessment: During a 1-hour S3 outage:

• CDN-cached pastes (popular content): Still served - CDN has its own cache
• Redis-cached small pastes: Still served - Redis doesn't depend on S3 for reads
• Cache-miss reads (uncommon pastes): Failed - S3 is the origin of truth
• New paste creation: Failed - can't write content to S3

Estimated user impact: If CDN + Redis cover 80% of reads, only 20% of reads fail during an S3 outage. That's still significant - about 50 failed requests per second.

Mitigation: Circuit breaker with graceful degradation.

def view_paste_with_circuit_breaker(paste_id, metadata):
    if s3_circuit_breaker.is_open():
        # S3 is known to be down - don't even try
        if metadata['content_size'] <= CONTENT_SIZE_THRESHOLD:
            # Try Redis content cache (may still have it)
            content = redis.get(f"paste:content:{paste_id}")
            if content:
                return {"metadata": metadata, "content": decompress(content)}
        
        # Can't serve content - return metadata with error
        return {
            "metadata": metadata,
            "error": "Content temporarily unavailable. Please try again in a few minutes.",
            "cached_until": metadata.get('cached_until')
        }
    
    try:
        content = s3.get_object(Bucket="paste-content", Key=metadata['s3_key'])
        s3_circuit_breaker.record_success()
        return {"metadata": metadata, "content": content}
    except S3Error:
        s3_circuit_breaker.record_failure()
        raise

Interview: What to say: "My biggest infrastructure risk is S3 dependency. If S3 has a regional outage, cache-miss reads and all writes fail. CDN and Redis absorb the impact for popular content - about 80% of reads still work. For the remaining 20%, I'd implement a circuit breaker that returns metadata with a 'content temporarily unavailable' message instead of a hard error. And for production, I'd enable S3 Cross-Region Replication with automatic failover."

Bottleneck 2: CDN Cache Invalidation Lag

When a paste is deleted, the CDN continues serving it from edge caches for 5-10 minutes. This is an inherent limitation of CDN architecture - propagating invalidations to 200+ edge nodes takes time.

Impact: A user deletes a paste containing sensitive information (a password, an API key). For 5-10 minutes, anyone with the URL can still access the content from CDN cache.

Why this matters: For a public Pastebin, 5-10 minutes of stale content after deletion is usually acceptable. But if someone pastes credentials by mistake and then deletes the paste, those credentials are exposed for 5-10 minutes. This is a security-adjacent concern.

Bottleneck 3: Large Paste Upload Processing

A 10 MB paste upload consumes server memory and CPU for compression:

Memory per 10 MB upload:
- Raw content in memory: 10 MB
- Gzip compression buffer: ~10 MB
- JSON parsing overhead: ~5 MB
Total: ~25 MB per concurrent large upload

At 150 peak write QPS, if 2% are 10 MB:
3 concurrent large uploads × 25 MB = 75 MB
Manageable on a server with 4+ GB RAM

At 10x (1,500 peak write QPS, 2% = 30 concurrent large uploads):
30 × 25 MB = 750 MB concurrent memory for large uploads

750 MB of memory for upload processing is significant but manageable. The risk is that multiple large uploads arrive simultaneously and cause memory pressure.

Mitigation: Streaming upload to S3.

Instead of buffering the entire upload in memory, stream the content directly to S3 as it arrives:

async def create_paste_streaming(request):
    """Stream large uploads directly to S3 without buffering entire content."""
    paste_id = generate_paste_id()
    
    # Create a streaming multipart upload to S3
    upload = s3.create_multipart_upload(
        Bucket="paste-content",
        Key=f"pastes/{paste_id}"
    )
    
    content_hash = hashlib.sha256()
    total_size = 0
    compressed_stream = gzip.GzipFile(fileobj=upload_buffer, mode='wb')
    
    async for chunk in request.stream():
        total_size += len(chunk)
        if total_size > 10 * 1024 * 1024:
            # Abort upload - exceeds size limit
            s3.abort_multipart_upload(...)
            return error(413, "Content exceeds 10 MB limit")
        
        content_hash.update(chunk)
        compressed_stream.write(chunk)
    
    compressed_stream.close()
    # ... complete multipart upload, store metadata ...

Streaming reduces memory per upload from 25 MB to ~1 MB (the chunk size), allowing the server to handle many more concurrent large uploads.

Bottleneck 4: Cleanup Worker Falling Behind

If the cleanup worker can't keep up with the rate of paste expiration, expired content accumulates in S3, increasing storage costs.

At 1x: 3M expired pastes/day → cleanup worker handles easily
At 10x: 30M expired pastes/day → 347 pastes/sec to clean up
At 100x: 300M expired pastes/day → 3,472 pastes/sec to clean up

At 100x, the cleanup worker needs to delete 3,472 S3 objects per second. Using batch deletes (1,000 per API call), that's 3.5 API calls per second - still within S3's limits. But the metadata cleanup (soft-delete + hard-delete) at this rate generates heavy database write load.

Mitigation: Partition-based cleanup.

With monthly table partitioning, cleaning up very old pastes is instant - just drop the partition. Only the "current month" partition needs row-by-row cleanup for pastes with short expiration times:

-- Instead of: DELETE FROM paste_metadata WHERE expires_at < NOW() - INTERVAL '6 months'
-- (scanning hundreds of millions of rows)

-- Drop the entire old partition:
DROP TABLE paste_metadata_2025_06;
-- Instant. Zero row scanning. Zero transaction log.

Tip: This is why partitioning was introduced in the scaling section. The cleanup bottleneck is the operational justification for table partitioning. When someone asks "why partition?", the answer is "because the alternative is scanning 500 million rows to find the 3 million that expired today."

Trade-off Analysis

Trade-off 1: S3 + CDN vs Database BLOBs

flowchart LR
    subgraph CHOSE["What We Chose: S3 + CDN"]
        S3_PRO1["✓ Unlimited storage scaling"]
        S3_PRO2["✓ Native CDN integration"]
        S3_PRO3["✓ Independent content scaling"]
        S3_CON1["✗ Two storage systems to manage"]
        S3_CON2["✗ Eventual consistency between S3 and DB"]
    end
    
    subgraph ALTERNATIVE["Alternative: PostgreSQL BLOBs"]
        DB_PRO1["✓ Single storage system"]
        DB_PRO2["✓ Transactional consistency"]
        DB_CON1["✗ Database bloat from large objects"]
        DB_CON2["✗ No native CDN integration"]
        DB_CON3["✗ Backup includes all content"]
    end

Trade-off 1: we chose S3 plus CDN over storing content as database BLOBs. The benefit is unlimited storage scaling, native CDN integration with CloudFront, and independent scaling of content and metadata. The cost is managing two storage systems and dealing with eventual consistency between S3 and the database. A database-only approach would give transactional consistency and a single system to manage, but would cause database bloat, slow backups, and no CDN integration. For a system storing 10 kilobyte to 10 megabyte objects, S3 is the clear winner.

What we gave up: Transactional consistency between content and metadata. When a paste is created, the S3 upload and the PostgreSQL insert are not in a single transaction. If the S3 upload succeeds but the DB insert fails, we have an orphaned S3 object. If the DB insert succeeds but the S3 upload fails, we have a metadata record pointing to non-existent content.

How we handle the inconsistency:

def create_paste_safely(content, **kwargs):
    paste_id = generate_paste_id()
    
    # Step 1: Upload to S3 first
    try:
        s3.put_object(Bucket="paste-content", Key=f"pastes/{paste_id}", Body=compressed)
    except S3Error:
        return error(500, "Failed to store content")
    
    # Step 2: Insert metadata
    try:
        db.execute("INSERT INTO paste_metadata ...", ...)
    except DBError:
        # DB failed after S3 succeeded - clean up the orphaned S3 object
        s3.delete_object(Bucket="paste-content", Key=f"pastes/{paste_id}")
        return error(500, "Failed to create paste")
    
    # Both succeeded
    return paste_id

The "upload to S3 first, then insert metadata" pattern ensures the common failure mode (DB fails after S3 succeeds) is handled by cleaning up the S3 object. The reverse failure (S3 fails) is caught before any metadata is written.

Trade-off 2: Tiered Caching vs Uniform Caching

What we gave up: Occasionally, a large paste that's being viewed frequently would benefit from Redis caching. With our tiered approach, it goes through CDN/S3 at higher latency. But the CDN effectively serves this role - popular large pastes are cached at edge nodes with <15ms latency.

Trade-off 3: Immutable Content vs Editable Content

What we gave up: Users can't edit a paste after creation. If they paste code with a typo, they must delete and create a new paste (with a new URL). This is a user experience trade-off for architectural simplicity. In practice, Pastebin's real-world usage is primarily "fire-and-forget" - people paste something, share the URL, and move on. Editing is rare enough that the simplicity trade-off is justified.

Trade-off 4: Delete Token vs User Accounts

What we gave up: Users who lose their delete token can't manage their paste. There's no "forgot delete token" flow (unlike "forgot password"). This is acceptable for an anonymous paste service, but would be a problem for a service where users create content they want to manage long-term.

Trade-off 5: Lazy Expiration + Background Cleanup vs Real-Time Expiration

What we gave up: For up to 1 hour after expiration, a paste's content still exists in S3 (consuming storage) and may still be served from CDN cache. The lazy check at the API layer returns 410, but CDN-cached content bypasses the API. The storage cost of this delay is negligible - a few extra hours of S3 storage for expired pastes - but the CDN serving expired content for up to 1 hour is a correctness concern for pastes with very short expiration times.

What Would Break First?

Here's my ordered prediction of the first production incidents:

1. A paste with credentials goes viral. Someone accidentally pastes an AWS secret key. It gets shared on social media. By the time they delete it, the CDN has cached it at 200+ edge nodes. The delete goes through, but the content is accessible from CDN cache for 5-10 minutes. During those 5-10 minutes, automated scrapers have already harvested the key.

2. Cleanup worker lag causes storage cost spike. The cleanup worker fails silently (network issue, crashed container). Expired pastes accumulate in S3 for 3 days. Storage costs spike from

   84/month to $600/month. Nobody notices until the monthly AWS bill arrives. Fix: monitoring on cleanup worker health and S3 storage growth rate.

3. A large paste (10 MB) uploaded repeatedly by a bot. An automated system creates 10 MB pastes every second - 10 MB/sec of write bandwidth. It hits the rate limit on request count (10 pastes/min) but the bandwidth-based rate limit wasn't configured. In 1 hour, it uploads 36 GB of content. Fix: enforce both request-count and bandwidth-based rate limits from day one.

4. DNS propagation during CDN configuration change. A CDN configuration update (changing cache TTL, adding a new behaviour) causes CloudFront to return 502 errors for 30 seconds during propagation. All paste views fail for 30 seconds. Users who bookmarked raw paste URLs get errors. Fix: use blue-green CloudFront distributions for configuration changes.

5. S3 and database become inconsistent. The application crashes between the S3 upload (step 1) and the database insert (step 2). An orphaned S3 object exists with no metadata record. Nobody notices because there's no monitoring for orphaned objects. Over time, orphans accumulate - wasting storage. Fix: a reconciliation job that lists S3 objects and verifies each one has a metadata record.

Interview: How to present this section: "If I'm being honest, the first production incident will probably be a security issue, not a scaling issue. Someone will paste credentials, delete the paste, but the CDN will serve the cached content for 5-10 minutes. My second concern is the cleanup worker failing silently - leading to a storage cost surprise. And my third concern is abuse - someone scripting 10 MB uploads to blow up our storage. I'd prioritise monitoring for these three scenarios on day one."

The Complete Design Summary

flowchart TB
    subgraph GLOBAL["Global"]
        CDN["CDN (CloudFront)<br/>200+ edge nodes<br/>Caches paste content"]
    end
    
    subgraph API_LAYER["Application"]
        LB["Load Balancer"]
        WRITE["Write API (2)"]
        READ["Read API (3)"]
    end
    
    subgraph DATA["Data"]
        REDIS[("Redis<br/>Metadata + small content")]
        DB[("PostgreSQL<br/>Metadata only")]
        S3[("S3<br/>All paste content<br/>gzip compressed")]
    end
    
    subgraph OPS["Operations"]
        CLEANUP["Cleanup Worker<br/>Hourly expiration"]
    end
    
    CDN -->|"Cache miss"| LB
    LB --> WRITE
    LB --> READ
    READ --> REDIS
    READ -.->|"Large pastes"| S3
    WRITE --> S3
    WRITE --> DB
    WRITE --> REDIS
    CLEANUP --> S3
    CLEANUP --> DB
    CLEANUP --> REDIS

Complete Pastebin architecture summary. The global layer is a CloudFront CDN with 200-plus edge nodes caching paste content. The application layer has a load balancer distributing traffic to 2 write API servers and 3 read API servers. The data layer has Redis caching metadata and small paste content, PostgreSQL storing metadata only, and S3 storing all paste content gzip-compressed. A cleanup worker runs hourly to delete expired pastes from S3, PostgreSQL, and Redis. The key design principles: content separated from metadata, tiered caching by content size, CDN as the primary content delivery mechanism, and immutable content for aggressive caching.

Key Takeaways

1. Know your failure modes before the interviewer asks. The three most likely incidents are: credentials exposed via CDN cache after deletion, cleanup worker failure causing storage cost spikes, and abuse from automated large-paste uploads. Name them proactively.

2. The S3-to-database consistency gap is the most important trade-off. Separating content storage from metadata storage means they can become inconsistent. Handle this with "S3 first, then DB" ordering and cleanup of orphaned objects.

3. CDN invalidation lag is a security-adjacent concern. Deleted content can be served from CDN cache for 5-10 minutes. For a public Pastebin, this is usually acceptable but should be documented and monitored.

4. Every design decision is a trade-off - articulate both sides. S3 vs DB BLOBs, tiered vs uniform caching, immutable vs editable, delete tokens vs accounts. The interviewer doesn't care which you chose - they care that you know what you gave up.

5. The cleanup worker is more operationally critical than the API servers. If an API server goes down, the load balancer routes around it. If the cleanup worker fails silently, storage costs grow unchecked for days. Monitor the cleanup worker more aggressively than the API servers.

6. End with practical monitoring, not theoretical architecture. "I'd set up CloudWatch alarms for S3 storage growth rate, cleanup worker execution time, CDN hit rate, and Redis memory usage" is a stronger closing statement than describing another architecture diagram.

Exercise: Exercise 1: Write a post-mortem for the following incident: "At 03:00 UTC, a user created a paste containing 47 AWS access keys harvested from public GitHub repos. The paste URL was shared on a hacking forum. By the time our abuse detection flagged the paste at 03:45 UTC, it had been viewed 12,000 times from 3,000 unique IPs. The paste was deleted, but CDN-cached versions were served for another 8 minutes. All 47 access keys were rotated by their owners within 24 hours." Include: timeline, impact, root cause, contributing factors, corrective actions.

Exercise 2: Design a reconciliation job that identifies orphaned S3 objects (objects with no matching metadata record) and orphaned metadata records (records pointing to non-existent S3 objects). How often should it run? How do you handle race conditions with concurrent paste creation? What do you do with the orphans - delete them immediately or quarantine them?

Exercise 3: The legal team receives a DMCA takedown request for a paste containing copyrighted code. Design the takedown workflow: How does the request arrive? Who reviews it? How is the paste removed? How do you ensure the content is removed from all cache tiers (CDN, Redis, local memory)? What response do viewers get after takedown - 404, 410, or a special "removed for legal reasons" page?

Back to System Design Interviews