Free lesson · Computer Networking

Bgp How The Internet Routes Between Organizations

Easy10 min readnetworkingcomputer-networkstcp-ip

The Protocol That Holds the Internet Together

OSPF and IS-IS handle routing within a single organization's network - your company, your university, your ISP. But the internet is not one network. It is tens of thousands of independently operated networks, each managed by different organizations with different policies, different business relationships, and different ideas about where traffic should flow. The protocol that connects these networks together - that literally makes the internet work as a unified system - is BGP (Border Gateway Protocol).

BGP is a path vector routing protocol that exchanges routing information between Autonomous Systems (ASes) - independently operated networks identified by a unique AS number. Every ISP, cloud provider, major enterprise, and content delivery network operates one or more autonomous systems. The global internet routing table - the collective knowledge of how to reach every publicly advertised network on the planet - is built entirely from BGP route advertisements exchanged between these autonomous systems. As of 2025, the global BGP routing table contains approximately 1 million routes.

BGP is standardized in RFC 4271 and is currently in version 4 (BGP-4). It is the only routing protocol used for inter-domain routing (routing between organizations). There is no alternative protocol and no competing standard. If BGP stopped working, the internet would fragment into disconnected islands within minutes.

Autonomous Systems

An Autonomous System (AS) is a collection of IP networks under the control of a single organization that presents a common routing policy to the internet. Each AS is assigned a unique AS Number (ASN) by a Regional Internet Registry (RIR).

Some well-known ASNs:

ASN	Organization
AS15169	Google
AS32934	Meta (Facebook)
AS16509	Amazon (AWS)
AS8075	Microsoft
AS13335	Cloudflare
AS7018	AT&T
AS3356	Lumen (Level 3)

When you send a request to google.com, your packet traverses multiple autonomous systems - your ISP's AS, possibly one or more transit provider ASes, and finally Google's AS15169. Each AS-to-AS transition is governed by BGP, which determines which path your traffic takes through the internet.

eBGP vs iBGP

BGP runs in two modes:

eBGP (External BGP) runs between routers in different autonomous systems. This is the "internet-facing" BGP that exchanges routes between organizations. eBGP sessions are typically configured between directly connected routers at the boundary between two networks - for example, your organization's border router and your ISP's border router.

iBGP (Internal BGP) runs between routers within the same autonomous system. iBGP distributes the BGP routes learned from external peers to all internal routers that need them. The key rule of iBGP: a route learned from one iBGP peer cannot be advertised to another iBGP peer (to prevent loops within the AS). This means every iBGP router must peer with every other iBGP router - a full mesh - or use route reflectors to reduce the number of required peerings.

graph LR
    subgraph AS1["AS 64501 (Your Company)"]
        R1["Border<br/>Router 1"]
        R2["Internal<br/>Router"]
        R3["Border<br/>Router 2"]
        R1 ---|"iBGP"| R2
        R2 ---|"iBGP"| R3
        R1 ---|"iBGP"| R3
    end

    subgraph AS2["AS 64502 (ISP A)"]
        ISP1["ISP A<br/>Router"]
    end

    subgraph AS3["AS 64503 (ISP B)"]
        ISP2["ISP B<br/>Router"]
    end

    R1 ===|"eBGP"| ISP1
    R3 ===|"eBGP"| ISP2

    style R1 fill:#4f46e5,stroke:#4338ca,color:#fff
    style R2 fill:#7c3aed,stroke:#6d28d9,color:#fff
    style R3 fill:#4f46e5,stroke:#4338ca,color:#fff
    style ISP1 fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style ISP2 fill:#a78bfa,stroke:#8b5cf6,color:#fff

This diagram shows the difference between eBGP and iBGP. AS 64501 (your company) has three routers connected by iBGP in a full mesh. Two of these are border routers. Border Router 1 connects to ISP A in AS 64502 via an eBGP session. Border Router 2 connects to ISP B in AS 64503 via a separate eBGP session. The eBGP sessions exchange routes between different organizations. The iBGP sessions distribute those externally learned routes to all internal routers within the same AS.

How BGP Sessions Work

Unlike OSPF (which runs directly over IP using protocol number 89), BGP runs over TCP on port 179. This is a deliberate design choice - TCP provides reliable, ordered delivery of BGP messages, which is critical because BGP updates can be large and must not be lost or reordered. When two BGP routers establish a session, they first complete the TCP three-way handshake, then exchange BGP OPEN messages to negotiate session parameters (AS numbers, hold time, capabilities), and finally exchange their full routing tables.

Once established, BGP sessions are maintained through periodic KEEPALIVE messages (default every 60 seconds). If a router misses three consecutive keepalives (the hold time, default 180 seconds), it declares the peer down and withdraws all routes learned from that peer. This is slower than OSPF's failure detection, which is why BGP convergence is typically measured in seconds to minutes rather than milliseconds.

BGP uses four message types:

OPEN: Sent once to establish the session. Contains the sender's AS number, hold time, and BGP identifier (router ID).

UPDATE: Contains new route advertisements (NLRI - Network Layer Reachability Information) and/or withdrawn routes. Each UPDATE carries the path attributes (AS path, next hop, local preference, MED, communities) for the advertised routes.

KEEPALIVE: Sent periodically to confirm the session is still active. A simple heartbeat with no routing data.

NOTIFICATION: Sent when an error occurs. Closes the BGP session. Common causes include hold timer expiration, malformed UPDATE messages, and capability mismatches.

BGP Route Selection Works

BGP uses a complex, multi-step decision process to select the best path when multiple paths to the same destination exist. Unlike OSPF (which simply picks the lowest-cost path), BGP considers policy - business relationships, economic incentives, and traffic engineering goals all influence routing decisions. The selection process evaluates attributes in this order:

Highest Local Preference. A locally configured value that expresses how much the AS prefers a particular path. Used to implement policies like "prefer traffic through ISP A over ISP B."
Shortest AS Path. The path that traverses the fewest autonomous systems. A route that goes through 3 ASes is preferred over one that goes through 5 ASes. AS path length is BGP's default tiebreaker and the most commonly discussed BGP metric.
Lowest Origin Type. IGP origin (route was learned from an interior protocol) is preferred over EGP, which is preferred over Incomplete.
Lowest MED (Multi-Exit Discriminator). A value set by a neighboring AS to indicate which entry point it prefers for traffic. Used when two ASes connect at multiple points - the neighbor can signal which connection point to prefer.
eBGP over iBGP. Prefer routes learned from external peers over internal peers.
Lowest IGP Cost to Next Hop. Among remaining equal paths, prefer the one where the internal cost to reach the BGP next-hop router is lowest.
Additional tiebreakers. Oldest route, lowest router ID, and other vendor-specific tiebreakers.

The first two steps - local preference and AS path length - are the most important in practice. Local preference is how network engineers implement routing policy. AS path length is the natural "shortest path" metric, though it can be manipulated by AS path prepending - artificially lengthening the AS path on certain advertisements to make a path less preferred.

BGP Communities

Communities are tags attached to BGP route advertisements that carry policy information. They allow network engineers to apply routing policies without modifying the route itself. Common uses include:

Tagging routes by customer type (enterprise, residential, wholesale)
Signaling traffic engineering preferences to upstream providers (e.g., "prefer this route for European traffic")
Implementing blackhole routing (tagging a route for immediate discard to stop a DDoS attack at the network edge)

The blackhole community is particularly important for security. When a network is under DDoS attack, the operator can advertise the targeted IP prefix with the blackhole community to upstream providers. The providers then drop all traffic to that prefix at their edge routers, stopping the attack traffic before it reaches the target network. This sacrifices reachability for the attacked IP but protects the rest of the network from collateral damage.

BGP and Internet Peering

BGP relationships between autonomous systems fall into three categories, defined by business relationships:

Transit. One AS pays another for internet connectivity. The transit provider carries the customer's traffic to and from the rest of the internet. The customer advertises its own routes to the provider, and the provider advertises the full internet routing table (or a default route) to the customer.

Peering. Two ASes agree to exchange traffic between their respective customers for free (settlement-free peering). Neither AS carries the other's traffic to third parties - they only exchange traffic destined for each other's networks. Peering reduces transit costs and latency for traffic between the two networks.

Customer. The reverse of transit - the AS that pays for connectivity. From the provider's perspective, customer routes are the most preferred because they generate revenue.

These relationships create a hierarchy: traffic flows "downhill" from providers to customers, and "sideways" between peers. BGP routing policies enforce this hierarchy - a transit provider advertises all routes to its customers but only its own and customer routes to its peers. Violating this convention (accidentally advertising transit routes to peers) causes route leaks that can disrupt internet routing.

Famous BGP Incidents

BGP's power comes with significant risk. Because BGP is trust-based - routers generally believe the route advertisements they receive - mistakes and attacks can have catastrophic consequences:

Pakistan YouTube Hijack (2008). The Pakistani government ordered ISPs to block YouTube. Pakistan Telecom accomplished this by announcing a more specific BGP route for YouTube's IP prefix. Due to a configuration error, this announcement leaked to the global internet, causing YouTube traffic worldwide to be routed to Pakistan and dropped. YouTube was unreachable for approximately two hours.

Google Cloud Outage (2017). Google accidentally leaked internal routes to external BGP peers, causing traffic destined for Google services to be misrouted through Japanese ISPs that could not handle the volume. The outage affected Google Cloud, YouTube, and other Google services for over an hour.

Facebook Outage (2021). A configuration change to Facebook's backbone routers caused BGP routes to Facebook's DNS servers to be withdrawn from the internet. Without DNS, nobody could resolve facebook.com, instagram.com, or whatsapp.com. The outage lasted approximately six hours - one of the longest major internet outages in history.

Cloudflare Outage (2022). A BGP route leak by a network in Romania caused Cloudflare's traffic in certain regions to be misrouted, leading to connectivity issues for thousands of websites that depend on Cloudflare's CDN and DDoS protection.

These incidents illustrate a fundamental tension: BGP is the protocol that holds the internet together, but it has minimal built-in security. Efforts like RPKI (Resource Public Key Infrastructure) and BGPsec aim to add cryptographic verification to BGP route announcements, allowing routers to verify that an AS is authorized to advertise a particular IP prefix. RPKI adoption is growing but not yet universal.

BGP in Cloud Architecture

BGP appears in cloud environments in several important ways:

AWS Direct Connect uses BGP to exchange routes between your on-premises network and your AWS VPC. When you establish a Direct Connect connection, you configure a BGP session between your border router and the AWS Direct Connect router. Your on-premises routes are advertised to AWS, and your VPC routes are advertised back. If you have both Direct Connect and a VPN backup, BGP handles automatic failover - if Direct Connect goes down, traffic shifts to the VPN path.

Transit Gateway in AWS supports BGP for dynamic route propagation between connected VPCs and on-premises networks. This eliminates the need to manually manage static routes as networks change.

GKE/EKS networking in some configurations uses BGP (via Calico or MetalLB) for pod network routing, advertising pod CIDR ranges to the physical network infrastructure.

Anycast routing relies on BGP. CDN providers like Cloudflare announce the same IP prefix from data centers worldwide. BGP naturally routes each client to the nearest announcing data center. This is why you reach a nearby Cloudflare edge server regardless of where you are in the world.

graph TD
    subgraph AWSHybrid["AWS Hybrid Architecture with BGP"]
        direction TB
        subgraph OnPrem["On-Premises (AS 65000)"]
            ONPREM_RTR["Border Router"]
        end

        subgraph AWS["AWS (AS 7224)"]
            DX["Direct Connect<br/>Router"]
            TGW["Transit<br/>Gateway"]
            VPC1["VPC A<br/>10.1.0.0/16"]
            VPC2["VPC B<br/>10.2.0.0/16"]
            DX --- TGW
            TGW --- VPC1
            TGW --- VPC2
        end

        ONPREM_RTR ===|"eBGP over<br/>Direct Connect"| DX
    end

    style ONPREM_RTR fill:#4f46e5,stroke:#4338ca,color:#fff
    style DX fill:#7c3aed,stroke:#6d28d9,color:#fff
    style TGW fill:#8b5cf6,stroke:#7c3aed,color:#fff
    style VPC1 fill:#a78bfa,stroke:#8b5cf6,color:#fff
    style VPC2 fill:#c4b5fd,stroke:#a78bfa,color:#000

This diagram shows a hybrid cloud architecture using BGP. An on-premises network with AS number 65000 connects to AWS with AS number 7224 through an eBGP session over Direct Connect. The AWS side has a Direct Connect router connected to a Transit Gateway, which in turn connects to two VPCs: VPC A at 10.1.0.0/16 and VPC B at 10.2.0.0/16. BGP dynamically exchanges routes between the on-premises network and AWS, enabling automatic routing and failover.

Interview: BGP appears in senior-level System Design discussions. Per the curriculum guidance: you will not be asked to implement BGP. But if you are designing a multi-region system and the interviewer asks "how does traffic reach the nearest data center?" - the answer involves BGP and anycast routing.

"How does your CDN route users to the nearest edge?" - Anycast. The same IP prefix is announced via BGP from every edge location. BGP's natural shortest-path behavior routes each user to the nearest edge. If an edge fails, its BGP announcement is withdrawn and traffic automatically shifts to the next nearest edge.

"How do you connect your on-premises data center to AWS?" - Direct Connect with BGP for dynamic route exchange. VPN as a backup path. BGP handles failover: if Direct Connect goes down, the VPN route (with longer AS path or lower local preference) becomes the active path automatically.

"What caused the Facebook outage in 2021?" - A BGP misconfiguration withdrew routes to Facebook's DNS servers from the internet. Without DNS, no one could resolve Facebook's domain names. The outage lasted six hours because the BGP withdrawal also affected internal tools Facebook engineers needed to fix the problem.

What Comes Next

The next topic covers IP Fragmentation and MTU - what happens when a packet is too large for a network link, how Path MTU Discovery prevents fragmentation, and why modern systems actively avoid fragmenting packets.

Exercise: Starter: Look up your ISP's AS number at bgp.tools or peeringdb.com. Then use the BGP looking glass at lg.he.net to trace the BGP path from Hurricane Electric to your ISP. How many autonomous systems does the path traverse?

Intermediate: Research the Facebook outage of October 2021 in detail. How did a BGP configuration change cause a six-hour outage? Why couldn't Facebook engineers fix it remotely? What was the role of DNS in amplifying the BGP problem? What safeguards could have prevented it?

Advanced: Research RPKI (Resource Public Key Infrastructure) - the internet's emerging defense against BGP hijacking. How does RPKI allow ASes to cryptographically sign their route announcements? What is a ROA (Route Origin Authorization)? Check the RPKI adoption statistics at NIST RPKI Monitor - what percentage of routes are now covered? Why is full adoption difficult, and what are the risks of partial adoption?

Back to Computer Networking