Free lesson · System Design Interviews

Step 4: High-Level Architecture

Hard9 min readsystem-designinterview-prep

Here's where Pastebin's architecture diverges fundamentally from URL Shortener. If you draw the same boxes - load balancer, API server, cache, database - you've missed the point. Pastebin stores content, not mappings. The content lives in object storage, the metadata lives in a database, and a CDN sits in front of everything to handle bandwidth spikes. The architecture is a content delivery system with an API layer on top.

When I evaluate candidates on this question, I'm checking whether they separate content storage from metadata storage. That's the single biggest architecture decision in a Pastebin design, and it reveals whether you've internalised the estimation from step 2 - that this is a bandwidth-bound system storing variable-size content, not a QPS-bound system storing fixed-size records.

Let me draw the architecture and walk through each component.

The Architecture

flowchart TB
    CLIENT["Client<br/>(Browser / CLI / API)"] --> CDN["CDN<br/>(CloudFront)"]
    
    CDN -->|"Cache HIT<br/>(static content)"| CLIENT
    CDN -->|"Cache MISS"| LB["Load Balancer<br/>(ALB)"]
    
    LB -->|"POST /api/v1/pastes"| WRITE_API["Write API<br/>Servers"]
    LB -->|"GET /api/v1/pastes/:id"| READ_API["Read API<br/>Servers"]
    
    READ_API -->|"1. Check metadata cache"| CACHE[("Redis Cache<br/>Metadata + small pastes")]
    CACHE -->|"Metadata hit"| READ_API
    READ_API -->|"2. Fetch content"| S3
    
    WRITE_API -->|"3. Store content<br/>(gzip compressed)"| S3[("S3<br/>Object Storage<br/>Paste content")]
    WRITE_API -->|"4. Store metadata"| DB[("PostgreSQL<br/>Paste metadata")]
    WRITE_API -->|"5. Populate cache"| CACHE
    
    CLEANUP["Cleanup Worker<br/>(scheduled)"] -->|"Delete expired"| S3
    CLEANUP -->|"Delete expired"| DB
    CLEANUP -->|"Invalidate cache"| CACHE

High-level architecture for Pastebin. Clients connect through a CDN - CloudFront - which caches paste content at edge nodes. CDN cache misses go through a Load Balancer to API servers. Write API servers store paste content as gzip-compressed objects in S3, store metadata in PostgreSQL, and populate the Redis cache. Read API servers check the Redis cache for metadata and small paste content, then fetch large content directly from S3 or CDN. A scheduled Cleanup Worker deletes expired pastes from S3, PostgreSQL, and the Redis cache. The key architectural difference from URL Shortener is the separation of content storage in S3 from metadata storage in PostgreSQL.

Let me explain each component and why it exists.

Component-by-Component Breakdown

CDN (CloudFront) - The First Line of Defence

For URL Shortener, the CDN was optional - 302 redirects are tiny. For Pastebin, the CDN is essential. Here's why:

Paste content is static and cacheable. Once created, a paste's content never changes (immutability assumption from requirements). This is the ideal use case for CDN caching - static content served from edge nodes close to the user.

Bandwidth spikes are absorbed at the edge. A viral paste viewed 10,000 times per hour generates 10,000 × 10 KB = 100 MB of bandwidth per hour - manageable. But a viral 1 MB paste viewed 10,000 times generates 10 GB/hour. The CDN handles this at the edge without the origin server feeling the load.

Global latency reduction. A user in Tokyo fetching a paste hosted on a US server experiences 150-200ms of network latency. The CDN serves the cached content from a Tokyo edge node in <10ms.

CDN configuration for Pastebin:

Cache Policy:
- Cache raw paste content (GET /api/v1/pastes/:id/raw) for 1 hour
- Cache HTML paste view for 5 minutes (shorter because it may include dynamic elements like view count)
- Don't cache POST/DELETE requests
- Cache key: paste ID + encoding (gzip/br/identity)

Headers:
Cache-Control: public, max-age=3600    # 1 hour for raw content
Vary: Accept-Encoding                  # Different cached versions for gzip vs plaintext
ETag: "sha256-of-content"              # For conditional requests

Load Balancer (ALB)

Same function as URL Shortener - distributes traffic, health checks, SSL termination. The only difference is that Pastebin's load balancer needs to handle larger request bodies (up to 10 MB for paste creation).

ALB configuration note: AWS ALB has a default request body size limit of 1 MB. For Pastebin, you need to increase this to 10 MB in the target group settings. This is a production detail that shows you've deployed real applications.

Separate Read and Write API Servers

The separation logic is similar to URL Shortener but for different reasons:

flowchart LR
    subgraph WRITE["Write Servers (2 instances)"]
        W1["Upload processing<br/>S3 write<br/>DB metadata insert<br/>Cache population"]
    end
    
    subgraph READ["Read Servers (3 instances)"]
        R1["Metadata lookup<br/>Content URL resolution<br/>Small paste serving"]
    end
    
    NOTE["Write servers need more memory<br/>(buffering 10 MB uploads).<br/>Read servers need more instances<br/>(5:1 read-write ratio)."]

Separation of read and write API servers for Pastebin. Write servers handle paste creation - they buffer large uploads in memory, compress and store content in S3, write metadata to PostgreSQL, and populate the Redis cache. They need more memory per instance to handle 10 megabyte uploads. Read servers handle paste viewing - they look up metadata in Redis, resolve content URLs, and serve small pastes directly or redirect to CDN for large pastes. Read servers are lighter weight but need more instances because of the 5-to-1 read-write ratio.

S3 (Object Storage) - The Content Store

This is the biggest architectural difference from URL Shortener. Paste content lives in S3, not in the database.

Why S3, not PostgreSQL?

The S3 object structure:

Bucket: paste-content
├── pastes/
│   ├── x7Kf2mBp          # Paste content (gzip compressed)
│   ├── a3Xb9kLm          # Each paste is one S3 object
│   ├── pQ4r5sT2
│   └── ...

Each paste is stored as a single S3 object, gzip-compressed, with the paste ID as the key. The key structure is flat (no directories) because S3 doesn't have real directories - prefixes are just naming conventions.

S3 storage classes for cost optimisation:

flowchart LR
    NEW["New paste<br/>created"] -->|"Store in"| STANDARD["S3 Standard<br/>First 30 days"]
    STANDARD -->|"Lifecycle rule<br/>after 30 days"| IA["S3 Infrequent Access<br/>30 days - 1 year<br/>(40% cheaper)"]
    IA -->|"Lifecycle rule<br/>after 1 year"| GLACIER["S3 Glacier<br/>Instant Retrieval<br/>(68% cheaper)"]
    
    EXPIRED["Paste expires"] -->|"Lifecycle rule or<br/>cleanup worker"| DELETED["Deleted from S3"]

S3 storage class lifecycle for cost optimisation. New pastes are stored in S3 Standard for the first 30 days - this is the most frequently accessed period. After 30 days, a lifecycle rule moves the object to S3 Infrequent Access, which is 40% cheaper. After one year, another rule moves it to S3 Glacier Instant Retrieval, which is 68% cheaper but still provides millisecond retrieval. When a paste expires, it's deleted by either a lifecycle rule or the cleanup worker. This tiered approach reduces storage costs significantly for the 15% of pastes that never expire.

Interview: What to say: "I'd store paste content in S3, not in the database. S3 is purpose-built for storing variable-size objects - it handles 1 KB and 10 MB equally well, costs $0.023/GB/month, and integrates natively with CloudFront for CDN delivery. Putting 10 MB blobs in PostgreSQL would bloat the database, slow down backups, and couple content scaling with metadata scaling. Keeping them separate means I can scale content storage independently."

PostgreSQL - The Metadata Store

PostgreSQL stores the paste metadata - everything about the paste except the content itself. At 45 GB of active metadata, a single PostgreSQL instance is more than sufficient.

The metadata store handles:

• Paste ID lookups (for the view path)
• Expiration queries (for the cleanup worker)
• Delete token verification
• View count tracking (if stored here rather than in Redis)

Redis Cache - Metadata and Small Content

Redis caches two things:

1. Paste metadata - the small JSON response with paste_id, title, language, created_at, expires_at. Every view request checks this first.
2. Small paste content - for pastes under 50 KB, the content itself is cached in Redis. This avoids an S3 round-trip for the 70% of pastes that are small enough to cache economically.

# Cache strategy: two tiers based on content size
def view_paste(paste_id):
    # Tier 1: Check metadata cache
    metadata = redis.get(f"paste:meta:{paste_id}")
    if not metadata:
        metadata = db.query("SELECT * FROM paste_metadata WHERE paste_id = %s", paste_id)
        if not metadata:
            return 404
        redis.setex(f"paste:meta:{paste_id}", ttl=3600, value=json.dumps(metadata))
    
    # Tier 2: Content retrieval (size-dependent)
    if metadata.content_size <= 50_000:  # 50 KB threshold
        # Small paste - check Redis content cache
        content = redis.get(f"paste:content:{paste_id}")
        if not content:
            content = s3.get_object(Bucket="paste-content", Key=f"pastes/{paste_id}")
            redis.setex(f"paste:content:{paste_id}", ttl=3600, value=content)
        return {"metadata": metadata, "content": content}
    else:
        # Large paste - return CDN URL, let client fetch directly
        content_url = f"https://cdn.paste.ly/pastes/{paste_id}"
        return {"metadata": metadata, "content_url": content_url}

This two-tier approach is the key cache design insight for Pastebin. Small pastes (70% of traffic) are served entirely from cache - metadata + content in Redis. Large pastes (30% of traffic) get metadata from cache but content from CDN/S3. The cache memory budget stays reasonable because you're not caching 1 MB objects.

Cleanup Worker

A scheduled job (running hourly or daily) that removes expired pastes:

def cleanup_expired_pastes():
    # 1. Find expired pastes in batches
    expired = db.query(
        "SELECT paste_id FROM paste_metadata "
        "WHERE expires_at < NOW() AND deleted_at IS NULL "
        "LIMIT 10000"
    )
    
    for paste_id in expired:
        # 2. Delete content from S3
        s3.delete_object(Bucket="paste-content", Key=f"pastes/{paste_id}")
        
        # 3. Soft-delete metadata
        db.execute(
            "UPDATE paste_metadata SET deleted_at = NOW() WHERE paste_id = %s",
            paste_id
        )
        
        # 4. Remove from cache
        redis.delete(f"paste:meta:{paste_id}")
        redis.delete(f"paste:content:{paste_id}")
    
    # 5. Hard-delete old metadata records (older than 30 days after deletion)
    db.execute(
        "DELETE FROM paste_metadata "
        "WHERE deleted_at < NOW() - INTERVAL '30 days'"
    )

The Data Flow Summary

How This Architecture Differs from URL Shortener

Key Takeaways

1. Separate content storage (S3) from metadata storage (PostgreSQL). This is the defining architecture decision. Content and metadata have different sizes, different access patterns, and different scaling properties. Treating them as one system couples two things that should scale independently.

2. CDN is essential, not optional. Paste content is static and cacheable. A CDN absorbs bandwidth spikes, reduces global latency, and offloads the origin servers. For URL Shortener, CDN was a nice-to-have. For Pastebin, it's a core architecture component.

3. Two-tier caching: metadata + small content in Redis, large content from CDN/S3. Caching 1 MB pastes in Redis wastes memory. Caching metadata-only for large pastes and embedding content for small pastes gives the best balance of performance and memory efficiency.

4. S3 lifecycle policies reduce storage costs automatically. Move infrequently accessed pastes to cheaper storage tiers. Delete expired pastes via lifecycle rules. This is operational efficiency that shows production experience.

5. The cleanup worker is a first-class component, not an afterthought. Without cleanup, expired content accumulates in S3 and metadata in PostgreSQL. The cleanup worker's correctness - deleting S3 content, soft-deleting metadata, invalidating cache - is critical for storage cost control.

6. The architecture has ~8 components - clean and explainable. CDN, Load Balancer, Write API, Read API, Redis Cache, PostgreSQL, S3, Cleanup Worker. Each has a clear reason to exist. You should be able to draw this in 5 minutes and explain every data flow.

Exercise: Exercise 1: Draw this architecture on a whiteboard in 5 minutes, narrating each component and data flow as you draw. Time yourself. Can you explain the difference from URL Shortener in one sentence? ("Pastebin separates content storage in S3 from metadata in PostgreSQL because content is 1,000-10,000x larger than metadata.")

Exercise 2: The interviewer asks: "What if users want to upload images alongside their text pastes - like a paste with inline screenshots?" How does the architecture change? Does S3 still work? What content type handling is needed? How does CDN caching change for mixed-content pastes?

Exercise 3: Redesign the architecture for a Pastebin that supports real-time collaborative editing (like Google Docs for code). What components need to be added? (Hint: WebSocket servers for real-time sync, an operational transformation or CRDT engine for conflict resolution, and a completely different caching strategy because content is no longer immutable.)

Back to System Design Interviews