Free lesson · System Design Advanced

TLS Internals at Depth

Hard14 min readsystem-designadvanced

Let me tell you about a production incident that cost us four hours and a very uncomfortable conversation with a VP. A certificate renewal that should have been routine caused a 12% error rate across a major service. The root cause? The renewed certificate used a different intermediate CA, and a subset of older clients didn't have that intermediate CA in their trust store. The certificate was technically valid. The chain was technically correct. And 12% of our traffic was getting TLS handshake failures.

The engineer who debugged it knew TLS at the level this lesson teaches. The engineers who couldn't debug it knew TLS as "the green padlock thing." There's a meaningful career gap between those two levels of understanding, and in a system design interview at the staff level, that gap shows.

The TLS Handshake: What Actually Happens

When your browser connects to a server over HTTPS, there's a negotiation that happens before a single byte of application data is exchanged. This negotiation - the TLS handshake - establishes the encryption parameters, authenticates the server, and creates the session keys that protect the conversation.

In TLS 1.2, this handshake takes two round trips before application data can flow. In TLS 1.3, it takes one round trip, and sometimes zero. That difference matters enormously at scale.

TLS 1.2 Full Handshake

Let me walk through exactly what happens in each message:

sequenceDiagram
    participant C as Client
    participant S as Server
    
    Note over C,S: Round Trip 1
    C->>S: ClientHello<br/>Supported cipher suites<br/>Client random<br/>Supported TLS versions
    S->>C: ServerHello<br/>Chosen cipher suite<br/>Server random<br/>Session ID
    S->>C: Certificate<br/>(Server's cert + chain)
    S->>C: ServerKeyExchange<br/>(DH parameters, if needed)
    S->>C: ServerHelloDone
    
    Note over C,S: Round Trip 2
    C->>C: Verify certificate chain
    C->>S: ClientKeyExchange<br/>(Client's DH public key)
    C->>S: ChangeCipherSpec
    C->>S: Finished (encrypted)
    S->>C: ChangeCipherSpec
    S->>C: Finished (encrypted)
    
    Note over C,S: Application data can now flow
    C->>S: HTTP Request (encrypted)
    S->>C: HTTP Response (encrypted)

This sequence diagram shows the TLS 1.2 full handshake between a client and server. In the first round trip, the client sends a ClientHello with supported cipher suites and a random value. The server responds with its ServerHello, choosing a cipher suite, then sends its certificate chain, key exchange parameters, and a ServerHelloDone message. In the second round trip, the client verifies the certificate chain, sends its own key exchange value, changes to encrypted mode, and sends a Finished message. The server also changes to encrypted mode and sends Finished. Only after these two full round trips can application data begin flowing.

ClientHello. The client announces what it supports: which TLS versions, which cipher suites, which elliptic curves. It also sends a random 32-byte value (the "client random") that will be used to derive session keys.

ServerHello. The server picks from the client's menu: one TLS version, one cipher suite, one curve. It sends its own 32-byte random value. The choice here matters - a misconfigured server that picks a weak cipher suite has just undermined the entire connection's security.

Certificate. The server sends its certificate plus the chain of intermediate certificates leading up to a root CA. This is where our production incident happened - the chain was valid but some clients didn't trust the intermediate.

Key Exchange. Using Diffie-Hellman (or its elliptic curve variant, ECDHE), the client and server agree on a shared secret without ever transmitting it. This is the mathematical magic of public-key cryptography - both sides compute the same secret from public values that an eavesdropper can see but cannot use.

Finished. Both sides derive session keys from the shared secret, the client random, and the server random. Then they each send a Finished message encrypted with those session keys. If both sides can decrypt each other's Finished message, the handshake succeeded.

TLS 1.3: One Round Trip

TLS 1.3 collapses the handshake to a single round trip by making a fundamental change: the client sends its key exchange parameters in the ClientHello itself, instead of waiting for the server to choose parameters first.

sequenceDiagram
    participant C as Client
    participant S as Server
    
    Note over C,S: Single Round Trip
    C->>S: ClientHello<br/>Supported cipher suites<br/>Client random<br/>Key share (ECDHE public key)<br/>Supported groups
    S->>C: ServerHello<br/>Chosen cipher suite<br/>Server random<br/>Key share (ECDHE public key)
    S->>C: EncryptedExtensions
    S->>C: Certificate (encrypted!)
    S->>C: CertificateVerify (encrypted!)
    S->>C: Finished (encrypted)
    
    Note over C,S: Client can send data immediately
    C->>S: Finished (encrypted)
    C->>S: HTTP Request (encrypted)
    S->>C: HTTP Response (encrypted)

This sequence diagram shows the TLS 1.3 handshake, which completes in a single round trip. The client sends its ClientHello including key share material upfront. The server responds with ServerHello, its key share, and then immediately sends the certificate, certificate verification, and finished message, all encrypted. The client can send its Finished and the first HTTP request together, meaning application data flows after just one round trip instead of two.

The key differences from 1.2:

Key share in ClientHello. The client guesses which key exchange group the server will want (usually X25519 or P-256) and sends its public key immediately. If the server supports that group, the handshake completes in one RTT. If not, the server sends a HelloRetryRequest and they fall back to two RTTs.

Everything after ServerHello is encrypted. In TLS 1.2, the certificate is sent in plaintext - an eavesdropper can see which site you're connecting to from the certificate. In TLS 1.3, the certificate is encrypted. This is a meaningful privacy improvement.

No more RSA key exchange. TLS 1.3 only supports ephemeral Diffie-Hellman. This means every connection gets unique session keys even if the server's private key is later compromised. This property is called forward secrecy, and TLS 1.3 makes it mandatory rather than optional.

Fewer cipher suites. TLS 1.2 supported dozens of cipher suites, many of them weak or broken. TLS 1.3 supports exactly five, all of which are considered strong. This simplification eliminates an entire class of downgrade attacks.

Important: The 1-RTT vs 2-RTT difference at scale. If your service handles 100 million new TLS connections per day and the average network RTT to your clients is 50ms, the TLS 1.3 upgrade saves 50ms × 100M = 5 billion milliseconds = roughly 58 days of cumulative user wait time per day. That's why major CDNs pushed TLS 1.3 adoption so aggressively.

TLS 1.3 Zero-RTT (0-RTT)

TLS 1.3 has one more trick: if a client has connected to a server before, it can send application data in the very first packet of the next connection, with zero round-trip wait. This is called 0-RTT or "early data."

How it works: during the first connection, the server issues a Pre-Shared Key (PSK) to the client. On subsequent connections, the client includes the PSK and immediately sends encrypted application data alongside its ClientHello. The server can process that data before the handshake is complete.

The catch: 0-RTT data is vulnerable to replay attacks. An attacker who captures the first packet can replay it, causing the server to process the same request twice. This means 0-RTT should only be used for idempotent requests - GET requests that fetch data, not POST requests that modify state.

graph TD
    A["First Connection"] --> B["Full 1-RTT Handshake"]
    B --> C["Server issues PSK ticket"]
    C --> D["Connection closes"]
    
    D --> E["Subsequent Connection"]
    E --> F["Client sends ClientHello + PSK + Early Data"]
    F --> G["Server processes early data<br/>BEFORE handshake completes"]
    G --> H["0-RTT: Application data<br/>flows immediately"]
    
    I["⚠️ Risk: Replay attacks<br/>Only use for idempotent requests"]
    H --> I

This diagram shows TLS 1.3 zero-RTT resumption. During the first connection, a full 1-RTT handshake occurs and the server issues a PSK ticket. On a subsequent connection, the client sends its ClientHello along with the PSK and early application data. The server can process this early data before the handshake completes, achieving zero round-trip latency. However, there's a risk of replay attacks, so this should only be used for idempotent requests like GETs.

Warning: 0-RTT replay is a real concern. In 2018, a major CDN provider enabled 0-RTT broadly and discovered that retry mechanisms in HTTP clients were causing duplicate POST requests. The lesson: enable 0-RTT only for safe methods (GET, HEAD, OPTIONS), and implement server-side replay detection for anything else.

Certificate Chains: Trust All the Way Up

When a server presents its certificate, the client doesn't just check the certificate itself - it validates the entire chain of trust from the server's certificate up to a root CA that the client already trusts.

The Chain Structure

graph TD
    A["Root CA Certificate<br/>Self-signed<br/>In client's trust store<br/>(e.g., DigiCert Global Root G2)"] --> B["Intermediate CA Certificate<br/>Signed by Root CA<br/>(e.g., DigiCert TLS RSA SHA256 2020)"]
    B --> C["Server Certificate<br/>Signed by Intermediate CA<br/>(e.g., *.example.com)"]
    
    D["Client validates:<br/>1. Server cert signature → Intermediate CA<br/>2. Intermediate cert signature → Root CA<br/>3. Root CA → in trust store?<br/>4. Not expired? Not revoked?"]
    
    C --> D

This diagram shows a three-level certificate chain. At the top is a Root CA certificate that is self-signed and present in the client's trust store. The Root CA signs an Intermediate CA certificate. The Intermediate CA signs the server's certificate. The client validates by checking each signature up the chain, verifying the root CA is in its trust store, and confirming no certificates are expired or revoked.

Root CA certificates are self-signed and baked into operating systems, browsers, and programming language trust stores. There are about 150 root CAs trusted by most systems. You never send the root certificate - the client already has it.

Intermediate CA certificates are signed by root CAs. They're the certificates that actually sign server certificates. CAs use intermediates rather than the root directly because if an intermediate is compromised, the root can revoke it without losing trust for all other intermediates. It's a blast radius containment strategy.

Server certificates (also called leaf certificates or end-entity certificates) are what the server actually presents. They're signed by an intermediate CA and contain the server's public key, the domain name, and the validity period.

The Chain Delivery Problem

Here's where production issues live. The server must send its own certificate plus all intermediate certificates in the correct order. If it omits an intermediate, clients that don't have that intermediate cached will fail to validate the chain, even though the certificate is perfectly valid.

This is exactly what happened in our incident. The renewed certificate used a different intermediate CA. Most clients had the old intermediate cached from previous connections. New clients - or clients that had cleared their cache - didn't have the new intermediate and couldn't build the chain.

Tip: Always test certificate changes with a clean client. Don't test with your browser that has been visiting the site for months and has intermediates cached. Use openssl s_client -connect your.domain:443 -showcerts to see exactly which certificates the server sends and whether the chain is complete. If you see "unable to verify the first certificate" on a clean client, your chain is broken.

Certificate Revocation: OCSP and OCSP Stapling

Certificates can be compromised or need to be revoked before they expire. The question is: how does a client know if a certificate has been revoked?

The Old Way: CRL (Certificate Revocation Lists)

CAs publish lists of revoked certificate serial numbers. Clients download these lists and check if the server's certificate is on them. The problem: CRLs can be megabytes in size, they need to be downloaded before the connection can proceed, and they add latency to every new TLS connection.

In practice, most clients stopped checking CRLs years ago because the latency was too painful and the lists were too large.

The Better Way: OCSP (Online Certificate Status Protocol)

Instead of downloading the entire revocation list, the client asks the CA "is this specific certificate revoked?" via an OCSP responder - a lightweight HTTP endpoint that the CA operates.

The problem with OCSP: the client needs to make a separate HTTP request to the CA's OCSP responder during the TLS handshake. This adds latency (one more RTT to an external server), adds a privacy concern (the CA sees which sites the client is visiting), and adds a reliability concern (if the OCSP responder is down, does the client fail-open or fail-closed?).

Most browsers fail-open - if they can't reach the OCSP responder, they proceed anyway. This is called "soft-fail" and it means an attacker who can block OCSP traffic can use a revoked certificate without detection.

The Right Way: OCSP Stapling

OCSP stapling flips the model. Instead of the client asking the CA about the certificate, the server periodically fetches a signed OCSP response from the CA and "staples" it to the TLS handshake. The client gets the revocation status as part of the handshake itself - no extra RTT, no privacy leak, no dependency on the CA's OCSP responder during connection time.

sequenceDiagram
    participant CA as CA OCSP Responder
    participant S as Server
    participant C as Client
    
    Note over CA,S: Background (every few hours)
    S->>CA: OCSP Request<br/>"Is my cert still valid?"
    CA->>S: Signed OCSP Response<br/>"Yes, valid until [timestamp]"
    S->>S: Cache OCSP response
    
    Note over S,C: During TLS Handshake
    C->>S: ClientHello
    S->>C: ServerHello + Certificate<br/>+ Stapled OCSP Response
    C->>C: Verify cert AND<br/>verify OCSP response signature
    
    Note over C: No extra RTT to CA!

This sequence diagram shows OCSP stapling. In the background, the server periodically requests an OCSP response from the CA's responder, asking if its certificate is still valid. The CA returns a signed, time-limited response. During TLS handshakes with clients, the server includes this cached OCSP response alongside its certificate. The client verifies both the certificate chain and the OCSP response signature without needing to contact the CA itself. This eliminates the extra round trip and privacy concern of traditional OCSP.

Interview: If an interviewer asks about TLS optimization, OCSP stapling is a strong answer. It shows you understand not just the handshake itself but the revocation verification layer. Mention that stapling eliminates the OCSP round trip, solves the privacy issue (no client-to-CA request), and removes the soft-fail vulnerability. Then connect it to latency: "At our scale, eliminating that OCSP round trip saves one RTT per new connection, which across 100M connections per day is significant."

Certificate Transparency

One more piece of the certificate ecosystem worth knowing at depth: Certificate Transparency (CT). CT is a public, append-only log of all certificates issued by participating CAs. It exists because CAs have been compromised and have issued fraudulent certificates - the Comodo breach in 2011, the DigiNotar hack in 2011, the Symantec trust issues in 2015-2017.

CT logs let domain owners monitor whether anyone has issued a certificate for their domain without authorization. Google requires all certificates to be CT-logged to be trusted by Chrome. If you see "SCT" (Signed Certificate Timestamp) in a TLS certificate, that's the proof that the certificate was submitted to a CT log.

For system design, CT matters because it's part of the trust infrastructure. If you're designing a system that issues its own certificates (internal mTLS, service mesh certificates), you should understand that public CAs are monitored via CT, and internal CAs are not - which is one reason why internal certificate compromise can go undetected longer.

The Production War Story

I mentioned the incident at the top, but let me give you the full picture because the debugging process is itself instructive.

The symptom: 12% of requests to a major service were failing with TLS handshake errors. Not timing out - failing immediately. The error rate appeared suddenly, exactly 3 hours after a certificate renewal.

The first instinct was wrong. The on-call engineer looked at the certificate, saw it was valid, saw the chain was complete, and concluded the certificate wasn't the problem. They spent 90 minutes investigating load balancer configurations.

The second instinct was closer but still wrong. Another engineer noticed the intermediate CA had changed and checked whether the server was sending the full chain. It was. They concluded the chain was fine.

The actual problem: the new intermediate CA (let's call it "DigiCert TLS RSA SHA256 2020 CA1") was newer than what a subset of clients had in their trust stores. These were clients running older Android versions and some IoT devices that hadn't updated their CA bundles in two years. They trusted the old intermediate but not the new one. From the server's perspective, the chain was correct. From those clients' perspective, the chain ended at an untrusted intermediate.

The fix was two-part: (1) immediately switch back to the old certificate while the new one was reissued with the original intermediate, and (2) implement cross-signing, where the new intermediate was also signed by an older, more widely trusted root.

The capacity lesson: your certificate chain's validity depends on your clients' trust stores, not just the CA's PKI hierarchy. If you serve a diverse client population (mobile apps, IoT devices, older browsers), you need to test certificate changes against the oldest client you support, not just against a modern browser.

TLS Performance Tuning in Production

A few production-relevant tuning knobs that show up in system design conversations:

Session Resumption

Establishing a new TLS session is expensive (even at 1 RTT in TLS 1.3). If a client reconnects to the same server, TLS supports session resumption where both sides reuse the previously negotiated session keys.

Two mechanisms:

Session Tickets (TLS 1.2 and 1.3): The server encrypts the session state into a ticket and sends it to the client. The client presents the ticket on reconnection, and the server decrypts it to restore the session. No server-side state required.
PSK (Pre-Shared Key) (TLS 1.3): Similar to session tickets but integrated into the 1.3 handshake and supports 0-RTT.

At scale, session resumption rates of 70-85% are typical for services with returning users. That means 70-85% of your connections skip the full handshake entirely.

Certificate Size

Certificate chains are sent during the handshake. Larger chains mean more bytes before application data can flow. At scale, the difference between a 2KB chain and a 6KB chain matters - those 4KB extra bytes across millions of connections add up to terabytes of bandwidth and, more importantly, extra TCP packets that add latency on slow connections.

Optimization	Impact
Use ECDSA certificates instead of RSA	3-5x smaller certificate size
Minimize intermediate chain length	1-2KB saved per extra intermediate
Enable OCSP stapling	Avoids separate OCSP request RTT
Enable session resumption	Skips handshake for returning connections
Upgrade to TLS 1.3	1 RTT instead of 2 for new connections

Connection Coalescing

HTTP/2 (covered in the next lesson) allows connection coalescing - reusing a single TLS connection for multiple domains if they share the same certificate and IP address. Using a wildcard certificate (*.example.com) or a Subject Alternative Name (SAN) certificate that covers multiple domains lets you serve api.example.com, cdn.example.com, and www.example.com over a single TLS connection, saving the handshake cost for each additional domain.

Six Takeaways

TLS 1.3 cuts the handshake to 1 RTT, and 0-RTT for repeat connections. At scale, shaving one RTT per new connection saves millions of cumulative seconds of user wait time. If your infrastructure still uses TLS 1.2, upgrading is one of the highest-leverage latency improvements available.
Forward secrecy is now mandatory. TLS 1.3 only supports ephemeral key exchange (ECDHE), which means every connection gets unique keys. Even if your server's private key is compromised later, past sessions remain protected. This is a meaningful security improvement over TLS 1.2's optional forward secrecy.
Certificate chain completeness is a production concern. The server must send all intermediate certificates. Missing intermediates cause handshake failures on clients that don't have the intermediate cached. Always test with a clean client after certificate changes.
OCSP stapling eliminates a round trip and a privacy leak. The server fetches the revocation status from the CA in the background and includes it in the handshake. This is strictly better than having every client query the OCSP responder independently.
0-RTT is powerful but dangerous. Zero round-trip resumption sends application data in the first packet, but that data is replayable. Only enable 0-RTT for idempotent requests and implement server-side replay detection.
Your certificate's trust depends on your clients. A chain that validates in Chrome on macOS might fail on Android 8 or an IoT device with a two-year-old CA bundle. Know your client population and test against the oldest supported platform.

Exercises

Exercise: Starter: A service handles 50 million new TLS connections per day. The average client RTT is 80ms. Calculate the cumulative latency savings of upgrading from TLS 1.2 (2 RTT handshake) to TLS 1.3 (1 RTT handshake). Then calculate the additional savings if 75% of connections use session resumption and skip the handshake entirely.

Exercise: Intermediate: You're debugging a TLS handshake failure that affects 5% of your mobile app users but zero desktop users. The certificate was recently renewed. Describe the diagnostic steps you would take, in order, to identify the root cause. What tools would you use? What would you check first?

Exercise: Advanced: Design a TLS configuration strategy for a service with three use cases: (1) public-facing web traffic from diverse clients including older mobile devices, (2) internal service-to-service communication within a Kubernetes cluster, and (3) a financial API that must never accept replayed requests. For each, specify the TLS version, certificate type (RSA vs ECDSA), key exchange, session resumption strategy, and whether 0-RTT should be enabled.

Back to System Design Advanced