AlgoInsight

Worked system design solution

Design Pastebin

A web service for sharing text snippets via short URLs. Users paste text, get a short link, and anyone with the link can read it back. We design for moderate write traffic, high read traffic via CDN, and an expiry-driven storage model with hot/warm tiering.

HLDEasy22 min readstoragecdnurl-shortenerexpiryblob-storage

Requirements

Functional + non-functional scope and what we explicitly cut

Functional requirements (must-haves)

Non-functional requirements

Explicitly out of scope (v1)

Assumptions we're making (state these out loud)

Interview: Strong candidates make scope explicit before sketching boxes. The "no auth on reads" decision drives the entire CDN strategy below — surface it now or it'll bite you in step 6.

Capacity Estimation

Traffic, storage, bandwidth, and server count from first principles

Baseline (modeled on Pastebin.com at peak — public stats: ~1 M new pastes/day, ~10 M views/day):

Traffic (QPS)

Metric Average Peak (3× avg)
New pastes 12 / sec 36 / sec
Reads 120 / sec 360 / sec
Deletes ~1 / sec ~3 / sec

Even at peak, the absolute QPS is tiny. The interesting work is in storage growth, CDN bandwidth, and expiry sweeping.

Storage growth

Bandwidth

Server-count sanity check

A single modern API server can comfortably handle 5,000 QPS of small JSON requests. We need <100 QPS of API traffic at peak after the CDN absorbs reads → 2 API instances (one for redundancy) cover this baseline.

Storage:

Complexity: The numbers tell you what the bottleneck is not. At this scale it's not compute, not bandwidth, not storage. The hard parts are (a) the expiry/sweeping pipeline staying healthy and (b) the CDN hit-ratio staying high.

API Design

Full REST contract with status codes, headers, idempotency, errors, and rate limits

Three endpoints cover everything. All requests and responses are JSON. All timestamps are ISO-8601 UTC.

Common headers

Header Direction Purpose
Content-Type: application/json request required on POST/PUT/PATCH
Authorization: Bearer <jwt> request optional; required for delete + owner-protected reads
Idempotency-Key: <uuid> request recommended on POST to prevent dup-create on retry
X-Request-ID: <uuid> request optional client trace id; echoed in response
X-RateLimit-Limit response requests/minute allowed for caller
X-RateLimit-Remaining response requests left in the current window
X-RateLimit-Reset response epoch second when the window resets

1. Create a paste

POST /v1/pastes
Content-Type: application/json
Authorization: Bearer <optional>
Idempotency-Key: 7c9e6679-7425-40de-944b-e07fc1f90ae7

{
  "content":       "console.log('hello world');",
  "expiry":        "1d",
  "isPrivate":     false,
  "customAlias":   null,
  "language":      "javascript"
}

Field constraints:

Success — 201 Created:

{
  "id":            "aB3xY9",
  "url":           "https://paste.io/aB3xY9",
  "rawUrl":        "https://paste.io/raw/aB3xY9",
  "createdAt":     "2026-05-24T12:00:00Z",
  "expiresAt":     "2026-05-25T12:00:00Z",
  "ownerToken":    "0d7f...c91e",
  "language":      "javascript"
}

The ownerToken is returned only to the original creator and is the only credential that authorises delete. Lose it and the paste can only be cleaned up at expiry.

Error responses:

Status code field Returned when
400 content_too_large Body exceeds 10 MB
400 invalid_expiry expiry not in allowed enum
409 alias_taken customAlias already exists
422 alias_reserved Alias hits the reserved-word blocklist
429 rate_limited > 10 creates/minute from this IP
500 server_error Anything else

Every error follows the same envelope:

{
  "error": {
    "code":    "alias_taken",
    "message": "Custom alias 'meeting-notes' is already in use.",
    "requestId": "req_01HXYZ..."
  }
}

2. Read a paste

GET /v1/pastes/{id}
Accept: application/json

{
  "id":         "aB3xY9",
  "content":    "console.log('hello world');",
  "language":   "javascript",
  "createdAt":  "2026-05-24T12:00:00Z",
  "expiresAt":  "2026-05-25T12:00:00Z",
  "viewCount":  42
}

For browser readers, we also expose GET /raw/{id} returning text/plain — used by the CDN's edge function to serve the bare paste content with maximum cacheability.

Status codes:

3. Delete a paste

DELETE /v1/pastes/{id}
Authorization: Bearer <ownerToken>

204 No Content on success. 403 if the token doesn't match. Soft-delete: metadata row is marked deleted=true and blob is purged async (CDN purge fires within ~30 s, blob delete within ~5 min).

Rate limiting

Endpoint Limit / IP Limit / authed user
POST /pastes 10 / min 60 / min
GET /pastes/{id} 600 / min unrestricted
DELETE /pastes/{id} 30 / min 30 / min

Implemented as a token-bucket in Redis (bucket:{ip}) at the API gateway. Sliding-window with 1-minute buckets.

Idempotency

POST /v1/pastes accepts Idempotency-Key: if the same key is seen within 24 hours, we return the original response (same paste id, same URL) instead of creating a duplicate. Useful for clients that retry on network failure.

sequenceDiagram
  autonumber
  participant C as Client
  participant API as API service
  participant R as Redis (idempotency store)
  participant KV as DynamoDB
  C->>API: POST /pastes (Idempotency-Key: K)
  API->>R: GET idem:K
  alt key seen
    R-->>API: existing response blob
    API-->>C: 201 (cached response)
  else first time
    R-->>API: nil
    API->>KV: create paste
    API->>R: SETEX idem:K 86400 <response>
    API-->>C: 201 (fresh response)
  end

High-Level Design

Full system architecture diagram with every component, and the request lifecycle

Architecture

flowchart TB
  subgraph Client[Client tier]
    Browser["Web / Mobile / CLI client"]
  end

  subgraph Edge["Edge tier — globally distributed"]
    DNS["Route 53<br/>Anycast DNS"]
    WAF["WAF + DDoS<br/>AWS Shield / Cloudflare"]
    CDN["CDN edges<br/>~250 PoPs"]
    Shield["Origin Shield<br/>1 per region"]
  end

  subgraph API["API tier — regional"]
    ALB[Application Load Balancer]
    APISvc["Paste API service<br/>auto-scaling 4–40 pods"]
    RateLim[Rate-limit middleware]
  end

  subgraph SubSvc[Sub-services]
    IDGen["ID Generator<br/>Snowflake"]
    Auth["Auth / Token Service"]
  end

  subgraph Data[Data tier]
    Redis["(Redis cluster<br/>hot pastes + idempotency<br/>+ rate-limit tokens)"]
    KV["(DynamoDB<br/>metadata + TTL)"]
    Blob["(S3<br/>paste content)"]
  end

  subgraph Async[Async pipeline]
    Stream[DynamoDB Streams]
    Sweeper["TTL Sweeper<br/>Lambda"]
    Audit["Audit log → Kinesis"]
    Purger[CDN Purge worker]
  end

  Browser --> DNS
  DNS --> WAF
  WAF --> CDN
  CDN --> Shield
  Shield --> ALB
  ALB --> APISvc
  APISvc --> RateLim
  RateLim --> Redis
  APISvc --> Auth
  APISvc --> IDGen
  APISvc --> KV
  APISvc --> Redis
  APISvc --> Blob
  APISvc --> Audit
  KV --> Stream
  Stream --> Sweeper
  Sweeper --> Blob
  Sweeper --> Purger

Component responsibilities

Edge tier

API tier

Sub-services

Data tier

Async pipeline

Request lifecycle — write path

sequenceDiagram
  autonumber
  participant C as Client
  participant CDN
  participant API as API svc
  participant RL as Redis
  participant ID as ID Gen
  participant KV as DynamoDB
  participant S3
  C->>CDN: POST /v1/pastes
  CDN->>API: forward (no caching on POST)
  API->>RL: check rate limit (token bucket)
  RL-->>API: ok
  API->>RL: check idempotency-key
  RL-->>API: not seen
  API->>ID: nextId()
  ID-->>API: "aB3xY9"
  API->>S3: PUT /pastes/aB/3x/Y9.txt
  S3-->>API: ok
  API->>KV: PutItem (id, blobKey, expiresAt, ownerTokenHash)
  KV-->>API: ok
  API->>RL: SETEX idem:K 24h <response>
  API-->>C: 201 { id, url, ownerToken, expiresAt }

Request lifecycle — read path (the hot path)

sequenceDiagram
  autonumber
  participant C as Browser
  participant CDN
  participant API as API svc
  participant R as Redis hot cache
  participant KV as DynamoDB
  participant S3
  C->>CDN: GET /raw/aB3xY9
  alt CDN hit (≥95% of traffic)
    CDN-->>C: 200 cached content
  else CDN miss
    CDN->>API: forward
    API->>R: GET paste:aB3xY9
    alt Redis hit
      R-->>API: blob bytes
    else Redis miss
      API->>KV: GetItem aB3xY9
      KV-->>API: { blobKey, expiresAt }
      API->>S3: GET <blobKey>
      S3-->>API: blob bytes
      API->>R: SETEX paste:aB3xY9 1h
    end
    API-->>CDN: 200 + Cache-Control: max-age=86400
    CDN-->>C: 200 + caches for next time
  end

Database Design

Hybrid blob + KV model, schema, indexes, partitioning, and consistency choices

We split data into two stores because metadata and blob content have very different access patterns and cost profiles.

Metadata KV (DynamoDB)

field type notes
id string Partition key — 7-char base62
blobKey string s3://pastes/{aa}/{bb}/{id}.txt
expiresAt number Unix epoch seconds — DynamoDB TTL attribute
createdAt number epoch seconds
ownerTokenHash string bcrypt of owner token; required for DELETE
isPrivate boolean controls read-access policy at CDN edge
customAlias string? nullable; GSI for alias → id lookup
size number byte count, for analytics
language string? client hint
viewCount number incremented async via stream
deleted boolean soft-delete tombstone (cleaned by sweeper)

Capacity mode: on-demand. The traffic shape doesn't justify reserved capacity.

Indexes:

Partitioning: DynamoDB hash-partitions on the PK. With 7-char base62 ids, the distribution is uniform — no hot partitions even with a viral paste.

Object store (S3)

One object per paste, keyed by a 3-level prefix for partition friendliness:

s3://my-pastes-bucket/aB/3x/aB3xY9.txt

The two leading 2-char prefixes (aB then 3x) come from the first 4 chars of the id. S3 best-practice was historically "spread your prefixes to avoid hot partitions" — modern S3 doesn't strictly need it but the convention costs nothing and makes lifecycle policies easier.

S3 features we lean on:

Consistency choices

Operation Consistency Why
Read paste Eventually consistent Tolerable; 1 s lag at most
Create paste Strong write We commit S3 first, then KV. Order matters.
Custom alias claim Conditional write Compare-and-set on the alias-index GSI

The custom-alias claim is the only place we need strong consistency. We use DynamoDB's conditional write (attribute_not_exists) so two concurrent claims for the same alias resolve to exactly one winner.

await ddb.put({
  Item: { ...metadata },
  ConditionExpression: 'attribute_not_exists(customAlias)',
}).promise();
// throws ConditionalCheckFailedException → return 409 to client

Short-ID scheme

7 base62 chars ([0-9a-zA-Z]) → 62^7 = 3.5 trillion keyspace. At 1 M pastes/day, collision-free for ~9,500 years with a per-server counter, or forever with a Snowflake-style timestamped ID.

const ALPHABET = '0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
function encode(num: bigint): string {
  let s = '';
  while (num > 0n) {
    s = ALPHABET[Number(num % 62n)] + s;
    num /= 62n;
  }
  return s.padStart(7, '0');
}

Deep Dive: Caching

Four-layer caching strategy with concrete TTLs and the immutability win

Pastes are immutable once written (we never edit a paste; you can only delete it). That single property is the most important fact in this design — it means we can cache aggressively at every layer without ever worrying about stale reads.

Cache topology

flowchart LR
  Browser["Browser cache<br/>Cache-Control: max-age=300"] -->|miss| Edge["CDN Edge<br/>~250 PoPs<br/>TTL = min(expiry, 1d)"]
  Edge -->|miss| Shield["Origin Shield<br/>1 per region<br/>TTL = 1d"]
  Shield -->|miss| API[API service]
  API -->|hot path| Redis["(Redis<br/>top 1% pastes<br/>LRU, 10 GB cap)"]
  API -->|cold path| KV["(DynamoDB)"]
  API -->|blob fetch| S3["(S3)"]

  classDef cache fill:#fef3c7,stroke:#d97706,color:#000000
  class Browser,Edge,Shield,Redis cache

Layer 1 — Browser cache

Set Cache-Control: public, max-age=300 on /raw responses. A user refreshing the same paste 10 times in a minute hits zero servers. TTL is short (5 min) so deletions become visible quickly without forcing a server round-trip.

Layer 2 — CDN edge

Cache key: full URL including query params. TTL: min(expiryRemaining, 86400). Headers emitted by the API:

Cache-Control: public, max-age=86400, immutable
ETag: "v1-aB3xY9"
Vary: Accept-Encoding

immutable tells modern browsers + the CDN that the resource never changes within its lifetime — saves revalidation round-trips.

For private pastes: Cache-Control: private, max-age=300. Edges don't cache. Reads still hit the origin but the access-control check happens there.

Layer 3 — Origin shield

A single shield cache per region between the edges and the origin. Critical because:

sequenceDiagram
  participant E1 as Edge PoP-1
  participant E2 as Edge PoP-2
  participant E3 as Edge PoP-N
  participant S as Origin Shield
  participant O as Origin API
  par fan-in
    E1->>S: GET aB3xY9 (miss)
    E2->>S: GET aB3xY9 (miss)
    E3->>S: GET aB3xY9 (miss)
  end
  S->>O: GET aB3xY9 (one request)
  O-->>S: 200 + body
  par fan-out
    S-->>E1: 200 + body
    S-->>E2: 200 + body
    S-->>E3: 200 + body
  end

Layer 4 — Redis (origin-side hot cache)

Pastes that miss the CDN but are accessed > 5 times in an hour get promoted into Redis. Eviction: allkeys-lru with 10 GB cap.

Why Redis here even with the CDN? Because the CDN hit-ratio is great for popular pastes but bad for the long tail. A senior engineer's blog might link to a moderately-popular paste — getting 100 reads/hour. CDN TTL = 1 day means it'll be cached eventually, but the first read after a TTL-expiry hits the origin. Redis catches these.

Invalidation

Only happens on DELETE. Pipeline:

  1. API sets deleted=true on the metadata row.
  2. API DEL paste:aB3xY9 in Redis.
  3. API enqueues a CDN purge request via the async pipeline (Lambda → CloudFront API).
  4. Sweeper deletes the S3 object.
  5. After all this, any cached copy at any layer will return 404 within ~30 s.
async function deletePaste(id: string) {
  await ddb.update({
    Key: { id },
    UpdateExpression: 'SET deleted = :t',
    ExpressionAttributeValues: { ':t': true },
  });
  await redis.del(`paste:${id}`);
  await sqs.send({ MessageBody: JSON.stringify({ id, action: 'purge' }) });
}

Hit ratios in practice

Layer Hit ratio Notes
Browser 30% Same user, multiple views
CDN edge 90%+ Across all users for a given paste
Origin shield 5% Catches what edges miss
Redis 4% Long-tail popular pastes
Origin DB+S3 < 1% Cold reads, sweeper cleanup, audit

Complexity: A 90% CDN hit ratio means the origin sees 10% of total read traffic. At 99% hit ratio (achievable with origin-shield + good TTLs) the origin sees 1% — meaning your origin can be 10× smaller for the same total throughput. This is the single most important cost lever in the design.

Deep Dive: Scaling

What changes at 10×, 100×, and 1000× — and the multi-region topology

Baseline (1×) — single region

What we've described so far. Single AWS region, 4 API pods, on-demand DynamoDB, single Redis cluster, one S3 bucket. Handles ~360 QPS peak without breaking a sweat.

10× — same architecture, more replicas

flowchart LR
  LB[ALB] --> A1[API pod 1]
  LB --> A2[API pod 2]
  LB --> A3[API pod 3]
  LB --> An["…<br/>up to 40"]
  A1 & A2 & A3 & An --> R["(Redis cluster<br/>3 primaries +<br/>3 replicas)"]
  A1 & A2 & A3 & An --> KV["(DynamoDB<br/>on-demand)"]

What changes:

100× — multi-region active-passive

flowchart TB
  subgraph DNS[Route 53 latency-based routing]
    DNSnode[Geographic routing]
  end

  subgraph Primary["us-east-1 — write primary"]
    LBus[ALB] --> APIus[API fleet 60 pods]
    APIus --> Rus["(Redis cluster)"]
    APIus --> DDBus["(DynamoDB primary)"]
    APIus --> S3us["(S3 bucket us)"]
  end

  subgraph Replica["eu-west-1 — reads only"]
    LBeu[ALB] --> APIeu[API fleet 30 pods]
    APIeu --> Reu["(Redis cluster)"]
    APIeu --> DDBeu["(DynamoDB replica<br/>via Global Tables)"]
    APIeu --> S3eu["(S3 replica<br/>via CRR)"]
  end

  DNS --> Primary
  DNS --> Replica
  DDBus -.->|"async replication<br/>~1s lag"| DDBeu
  S3us -.->|"Cross-Region Replication<br/>~15 min lag"| S3eu

What changes:

1000× — multi-region active-active + sharded everything

flowchart TB
  subgraph US[us-east + us-west]
    USTraffic[Writes + reads]
  end
  subgraph EU[eu-west + eu-central]
    EUTraffic[Writes + reads]
  end
  subgraph AP[ap-south + ap-southeast]
    APTraffic[Writes + reads]
  end

  subgraph Conflict[Conflict resolution]
    Vector["CRDT / last-writer-wins<br/>on customAlias"]
  end

  US <--> Vector
  EU <--> Vector
  AP <--> Vector

  subgraph DataPlane[Per-region]
    Shard1["(KV shard 1<br/>id prefix aa-mm)"]
    Shard2["(KV shard 2<br/>id prefix nn-zz)"]
    Shard3["(KV shard 3<br/>id prefix 00-99)"]
  end

What changes:

Scaling specific subsystems

Expiry sweeping

Custom-alias contention

Rate limit storage

Audit log throughput

What we'd watch as we scale

Bottlenecks & Trade-offs

Failure modes, what we accept, monitoring, and what we revisit at scale

Top failure modes

1. CDN cache poisoning by a viral paste

A paste that gets linked from Hacker News goes from 0 to 100 K req/min in 10 seconds. If our CDN cache is cold for it, every edge cache-misses to the origin shield, which cache-misses to the origin.

Mitigation: origin shield + Redis hot cache + auto-scaling on the API tier. Tested by chaos engineering: we artificially purge the CDN for a paste each week and watch the latency / error budgets stay within SLA.

2. Hot-key DynamoDB partition

A single mega-viral paste's id is one partition key — DynamoDB caps single-partition reads at ~3,000 RPS. If 10 K users want the same paste simultaneously and the CDN is cold, the origin queues up.

Mitigation: detect viral pastes (Redis access counter) and promote to a per-region S3-only fast path — the API reads directly from S3 (which has no per-key limit) and bypasses DynamoDB.

3. S3 deletion not synced with CDN purge

Sweeper deletes the S3 object before the CDN purge fires → users hit the CDN cache and get a stale paste that's gone in S3. Functionally harmless but confusing.

Mitigation: purge first, delete second. The sweeper Lambda issues the CDN purge, waits for the 30-second propagation window, then deletes the S3 object. Costs an extra 30 s of storage per paste — irrelevant at our scale.

4. Idempotency-key collision

Client retries a create with the same Idempotency-Key but different content (bug on the client). We return the original response, content gets silently dropped on the floor.

Mitigation: we hash the request body and store (idempotencyKey, bodyHash) → response. If a later request shares the key but has a different hash, we reject with 422 Unprocessable — the client gets a clear "your idempotency key was reused with different content" error.

Accepted trade-offs

Trade-off What we got What we gave up
Eventual consistency on reads 1-s replication lag is fine Strictly fresh reads (we don't need them)
Soft-delete for 24 h Recovery from accidental deletes Storage cost for deleted pastes
No syntax highlighting server-side CPU savings + cacheable responses UX feature lives in the client
Single-language responses (UTF-8 only) Simple content-type story Multi-byte file uploads need explicit handling
Custom-alias reservation is FCFS Simple atomic claim model Premium-tier "I want this alias" is unhandled
10 MB hard cap on paste size Memory bounds on the API process Users can't paste huge files (use a file-share service)

Monitoring

API tier — Prometheus scrape:

CDN — CloudWatch dashboards:

DynamoDB — CloudWatch:

S3 — CloudWatch + access logs:

Business metrics — Athena over the audit log:

What we'd revisit at 10× growth

Interview: Strong candidates close with this kind of slide: failure modes named, mitigations spelled out, monitoring concrete, what-we'd-change-at-10× explicit. That's the maturity signal — it shows you've not just designed a system, you've operated one in your head.

Practice this question