Free lesson · Computer Networking

Rtp And Real Time Streaming

Easy10 min readnetworkingcomputer-networkstcp-ip

Delivering Media in Real Time

When you make a video call on Zoom, attend a virtual meeting on Google Meet, or watch a live broadcast on Twitch, audio and video data must travel from the sender to the receiver with minimal delay. The human perception threshold for conversational audio latency is approximately 150 milliseconds - beyond that, the conversation feels unnatural and participants begin talking over each other. Video can tolerate slightly more delay, but anything beyond 200-300ms creates a noticeable disconnect between audio and lip movement.

TCP cannot meet these requirements. Its retransmission mechanism adds at least one RTT of delay when a packet is lost, and its ordered delivery guarantee blocks all subsequent data until the lost packet is recovered. For a video call on a 100ms RTT connection, a single lost packet could cause a 200ms+ freeze - exactly the kind of interruption that makes real-time communication feel broken.

RTP (Real-Time Transport Protocol) is the solution. Defined in RFC 3550, RTP is an application-layer protocol designed specifically for delivering real-time audio and video over UDP. RTP does not provide reliability (lost packets are not retransmitted) or ordering enforcement (the application decides what to do with out-of-order packets). Instead, it provides the tools the application needs to manage real-time media: sequence numbers for reordering detection, timestamps for synchronization, and payload type identification so the receiver knows how to decode the data.

RTP is used by virtually every real-time communication system: WebRTC (the standard for browser-based video calling), VoIP (Voice over IP), IPTV, video conferencing platforms, and live broadcast systems.

RTP Header Structure

The RTP header is 12 bytes (minimum) and contains fields specifically designed for real-time media:

Sequence Numbers - Detecting Loss, Not Enforcing Order

RTP sequence numbers increment by one for each packet sent. Unlike TCP sequence numbers (which enforce strict ordering), RTP sequence numbers are informational - the application uses them to detect gaps (a missing number means a lost packet) and to reorder packets that arrive out of sequence. The application decides how to handle the gap:

Audio: Replace the missing audio sample with silence or a comfort noise generator. Modern audio codecs like Opus include packet loss concealment (PLC) that interpolates the missing audio from surrounding samples, making small gaps nearly imperceptible.

Video: If the lost packet contained part of a P-frame (predicted frame), the decoder may show visual artifacts (blocky squares, color smearing) until the next I-frame (keyframe) arrives and resets the decoder state. The application may request an immediate keyframe from the sender (a PLI - Picture Loss Indication) to speed up visual recovery.

Timestamps - Synchronization Across Streams

RTP timestamps represent the sampling instant of the media, not wall-clock time. For audio sampled at 48 kHz, the timestamp increments by 48,000 per second. For video at 90 kHz (the conventional RTP timestamp clock for video), the timestamp increments by 90,000 per second.

Timestamps serve two critical purposes:

Playback timing. The receiver uses timestamps to determine when to play each sample relative to others. Even if packets arrive with variable network delay (jitter), the timestamps tell the receiver the correct playback order and spacing.

Audio-video synchronization (lip sync). A video call generates separate RTP streams for audio and video, each with their own sequence numbers and timestamps. The receiver uses the RTCP protocol (described below) to correlate the RTP timestamps with wall-clock time, enabling synchronization between the audio and video streams. Without this synchronization, the viewer sees lips moving out of sync with the spoken words.

sequenceDiagram
    participant S as Sender (Camera + Mic)
    participant N as Network (UDP)
    participant R as Receiver (Display + Speaker)

    S->>N: RTP: Seq=100, TS=0<br/>Video I-frame (keyframe)
    S->>N: RTP: Seq=101, TS=3000<br/>Audio chunk
    S->>N: RTP: Seq=102, TS=3600<br/>Video P-frame
    Note over N: Seq=103 LOST!<br/>(Audio chunk)
    S->>N: RTP: Seq=104, TS=7200<br/>Video P-frame

    N->>R: Seq=100 ✅
    N->>R: Seq=101 ✅
    N->>R: Seq=102 ✅
    N->>R: Seq=104 ✅ (gap detected)

    Note over R: Seq=103 missing<br/>Audio: apply PLC<br/>(interpolate missing sample)<br/>Video: continues normally<br/>(loss was audio, not video)

This sequence diagram shows RTP handling packet loss. The sender transmits video and audio as separate RTP packets with incrementing sequence numbers. Packet 103, containing an audio chunk, is lost in the network. The receiver detects the gap when it receives packet 104 after 102 without seeing 103. Because the lost packet was audio, the receiver applies Packet Loss Concealment to interpolate the missing audio sample while video continues playing normally. The missing audio is a brief, nearly imperceptible gap rather than a freeze.

RTCP - Quality Feedback

RTP is always paired with RTCP (RTP Control Protocol), which provides feedback about the quality of the RTP stream. RTCP does not carry media data - it carries statistics and control information.

RTCP serves several functions:

Sender Reports (SR): The sender periodically sends reports containing the number of packets sent, the number of bytes sent, and the mapping between RTP timestamps and NTP (wall-clock) timestamps. This mapping is what enables audio-video synchronization - the receiver can correlate the RTP timestamp in each stream with absolute time.

Receiver Reports (RR): The receiver periodically sends reports containing packet loss statistics (fraction of packets lost since last report and cumulative loss), jitter measurements (variation in packet arrival times), and the highest sequence number received. The sender uses these reports to adapt its behavior - for example, reducing the video bitrate when the receiver reports high packet loss.

Feedback Messages: Extensions like RTCP-FB (RFC 4585) enable immediate feedback:

• PLI (Picture Loss Indication): "I lost video data - send a keyframe immediately"
• NACK: "I am missing specific packets - retransmit them" (selective retransmission for critical data)
• REMB (Receiver Estimated Maximum Bitrate): "My available bandwidth is approximately X bps - adjust your sending rate"

These feedback mechanisms enable adaptive bitrate in real-time communication. The sender continuously adjusts the video resolution, framerate, and compression quality based on the receiver's network conditions. If the receiver reports increasing packet loss or jitter, the sender reduces quality to maintain smooth playback. If conditions improve, quality increases. This adaptation happens continuously, typically multiple times per second.

SRTP - Securing Real-Time Media

SRTP (Secure RTP), defined in RFC 3711, adds encryption and authentication to RTP. SRTP encrypts the RTP payload (media data) while leaving the RTP header in cleartext (so routers and middleboxes can forward based on SSRC and other header fields without decrypting). SRTP also adds an authentication tag that prevents tampering with the packet.

WebRTC mandates SRTP - all browser-based video calls are encrypted end-to-end at the media transport level. The encryption keys are negotiated during the WebRTC signaling process using DTLS-SRTP (DTLS provides the key exchange, SRTP provides the per-packet encryption).

WebRTC - RTP in the Browser

WebRTC (Web Real-Time Communication) is the open standard that brings RTP-based real-time media to web browsers. Before WebRTC, browser-based video calling required plugins (Flash, Java applets). WebRTC provides native browser APIs for capturing audio/video, establishing peer-to-peer connections, and exchanging media - all using RTP/SRTP over UDP under the hood.

A WebRTC connection involves several protocols working together:

Signaling (application-defined, typically WebSocket over TCP): The two peers exchange connection metadata - what codecs they support, their network addresses and port candidates, and cryptographic fingerprints. This is done through SDP (Session Description Protocol) messages exchanged via the application's signaling server. SDP is not an RTP protocol - it is a text-based format that describes the media session parameters.

ICE (Interactive Connectivity Establishment): Because both peers are typically behind NAT, they need to discover their public addresses and find a network path between them. ICE coordinates this using:

• STUN (Session Traversal Utilities for NAT): A lightweight protocol where the peer asks a STUN server "what is my public IP and port?" The STUN server, being on the public internet, sees the peer's NAT-translated address and reports it back.
• TURN (Traversal Using Relays around NAT): A relay server that mediates traffic when direct peer-to-peer connection is impossible (certain NAT types block it). TURN adds latency because all media traffic passes through the relay, but it ensures connectivity when nothing else works.

ICE tries multiple connection paths simultaneously and selects the best one: direct peer-to-peer (fastest), peer-to-peer via STUN (slightly slower due to NAT traversal), or relayed via TURN (slowest, used as fallback). This multi-path probing is why WebRTC calls typically establish quickly even across complex network topologies.

DTLS (Datagram TLS): Provides the key exchange for SRTP encryption. DTLS runs over UDP (unlike TLS which runs over TCP) and establishes the cryptographic session that protects the media data.

SRTP: Encrypts and authenticates the RTP media packets using keys established by DTLS.

WebRTC Data Channels

WebRTC also supports data channels - arbitrary data streams between peers that run over SCTP (Stream Control Transmission Protocol) encapsulated in DTLS over UDP. Data channels can be configured as reliable (like TCP) or unreliable (like UDP), ordered or unordered. They are commonly used for in-call chat, file sharing, screen sharing metadata, and game state in browser-based multiplayer games.

Jitter Buffers

Network delay is not constant - packets arrive with varying delays, a phenomenon called jitter. If the receiver played each packet immediately upon arrival, the audio would sound choppy (packets arriving early are played too soon, packets arriving late create gaps).

A jitter buffer (also called a playout buffer) smooths out jitter by holding received packets for a short time before playing them. The buffer introduces a fixed delay (typically 20-100ms for voice, 100-500ms for video) but produces smooth, continuous playback.

The jitter buffer size is a trade-off between smoothness and latency. A larger buffer absorbs more jitter but adds more end-to-end delay. A smaller buffer keeps latency low but is more sensitive to network variations. Modern jitter buffer implementations are adaptive - they automatically adjust the buffer size based on observed jitter, increasing during periods of high network variability and decreasing when the network is stable.

graph TD
    subgraph RTPStack["RTP/RTCP Protocol Stack"]
        direction TB
        MEDIA["Audio/Video Codec<br/>(Opus, H.264, VP8)"]
        RTP_LAYER["RTP<br/>Sequencing, timestamping,<br/>payload identification"]
        RTCP_LAYER["RTCP<br/>Quality feedback,<br/>synchronization,<br/>bitrate adaptation"]
        SRTP_LAYER["SRTP<br/>Encryption +<br/>authentication"]
        UDP_LAYER["UDP<br/>Fast, unreliable<br/>delivery"]
        MEDIA --> RTP_LAYER
        MEDIA -.->|"feedback"| RTCP_LAYER
        RTP_LAYER --> SRTP_LAYER
        RTCP_LAYER --> SRTP_LAYER
        SRTP_LAYER --> UDP_LAYER
    end

    style MEDIA fill:#4f46e5,stroke:#4338ca,color:#fff
    style RTP_LAYER fill:#7c3aed,stroke:#6d28d9,color:#fff
    style RTCP_LAYER fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style SRTP_LAYER fill:#a78bfa,stroke:#8b5cf6,color:#fff
    style UDP_LAYER fill:#c4b5fd,stroke:#a78bfa,color:#000

This diagram shows the RTP protocol stack. Audio and video codecs produce media data at the top. RTP adds sequencing, timestamping, and payload identification. RTCP provides quality feedback, synchronization information, and bitrate adaptation signals back to the codec. Both RTP and RTCP are protected by SRTP, which adds encryption and authentication. The entire stack runs over UDP for fast, unreliable delivery.

Beyond RTP - Alternative Real-Time Protocols

While RTP dominates real-time communication, other protocols serve specific niches:

SRT (Secure Reliable Transport) is an open-source protocol developed by Haivision for live video contribution (sending video from cameras and encoders to production servers). SRT runs over UDP and adds selective retransmission with configurable latency - it retransmits lost packets only if they can arrive before the playout deadline. This makes SRT more reliable than raw RTP for contribution links (where the link is between two known endpoints with predictable latency) while maintaining low enough latency for live production.

RIST (Reliable Internet Stream Transport) is another UDP-based protocol for broadcast-quality video transport over the internet. Like SRT, RIST adds selective retransmission and FEC to UDP for professional video workflows.

Media over QUIC (MoQ) is an emerging IETF standard that uses QUIC as the transport for live media. MoQ leverages QUIC's stream multiplexing to deliver audio, video, and data as independent streams with per-stream reliability. This is a potential successor to RTP for some use cases, particularly where the media passes through relay infrastructure (CDN-style distribution) rather than direct peer-to-peer delivery.

Interview: RTP knowledge demonstrates understanding of real-time system design:

"Design a video conferencing system. How does the video data reach participants?" - Video is encoded (H.264 or VP9), packetized into RTP packets, encrypted with SRTP, and sent over UDP. Each participant sends their media to a Selective Forwarding Unit (SFU) server, which relays the appropriate streams to other participants without transcoding. RTCP provides feedback for adaptive bitrate - the SFU selects the appropriate quality layer based on each receiver's network conditions.

"How does the system handle packet loss during a video call?" - Audio codecs (Opus) use Packet Loss Concealment to interpolate missing samples - loss under 5% is usually imperceptible. For video, lost P-frame data causes artifacts until the next I-frame. The receiver sends a PLI (Picture Loss Indication) via RTCP-FB to request an immediate keyframe. The sender can also perform selective retransmission (NACK) for critical packets if the RTT is low enough for retransmission to arrive before the playout deadline.

"Why not just use TCP for video calls?" - TCP's retransmission delays cause freezing that is worse than the visual artifacts from lost packets. A 200ms retransmission delay on a 100ms RTT connection creates a freeze that disrupts the conversation. With RTP over UDP, the codec handles loss gracefully - a brief glitch is better than a freeze. TCP's HOL blocking would also block audio delivery when video packets are lost, desynchronizing the streams.

What Comes Next

The next topic covers multicast and broadcast - the addressing modes that enable one-to-many data delivery, used in live streaming, financial data feeds, and network management.

Exercise: Starter: Make a WebRTC video call (Google Meet, Jitter.me, or any browser-based call). During the call, open chrome://webrtc-internals in Chrome. You will see detailed RTP statistics: packets sent/received, packet loss percentage, jitter, round-trip time, and codec information. Watch how these metrics change as the call progresses.

Intermediate: Research the difference between an SFU (Selective Forwarding Unit) and an MCU (Multipoint Control Unit) for video conferencing. How does each architecture handle a 10-person video call? What are the bandwidth requirements for each? Why have modern platforms (Zoom, Google Meet, Teams) converged on the SFU model? How does Simulcast (sending multiple quality levels) enable the SFU to serve heterogeneous receivers?

Advanced: Research FEC (Forward Error Correction) in WebRTC. Instead of retransmitting lost packets, FEC sends redundant data that allows the receiver to reconstruct lost packets without a round trip. How does WebRTC's FEC (using Flexible FEC or RED redundancy) trade bandwidth for latency? Under what packet loss rates does FEC outperform NACK-based retransmission? How does the sender decide how much FEC redundancy to add?

Back to Computer Networking