Free lesson · DevOps and Platform Engineering on AWS

Network architecture - the VPC design you will not regret

Easy9 min readawsdevopsplatform-engineering

The Decisions That Are Expensive To Reverse

Most infrastructure decisions can be changed. Three network decisions effectively cannot, and they are made in the first week:

Decision	Why it is irreversible
CIDR allocation	Renumbering means touching every resource, route, rule, and peer
Topology	Rearchitecting connectivity across live accounts
DNS architecture	Every name resolution path depends on it

This topic is about getting those three right, because everything else in networking is adjustable and these are not.

IP Address Planning

Allocate before you create the first VPC, and allocate for far more than you think you need.

Warning: Overlapping CIDR ranges are the most common hybrid and multi-account networking blocker, and there is no configuration that resolves them. Two networks using 10.0.0.0/16 cannot be routed together - not with a Transit Gateway, not with peering, not with anything.

The options once it has happened are all expensive: renumber one side (a project touching every resource), NAT between them (breaks anything that cares about source addresses, and complicates troubleshooting permanently), or use PrivateLink to expose specific services without routing (works, and only for service access rather than general connectivity).

It happens because VPCs get created with default or convenient ranges before anyone owns the allocation. By the time it is noticed there are thirty VPCs and an on-premises network in the way.

The allocation scheme:

Reserve a large block for AWS - a /8 from RFC 1918 if you can, or a substantial /12. Then subdivide by Region, then by environment, then by account.

10.0.0.0/8                    AWS overall
  10.0.0.0/12                 eu-west-1
    10.0.0.0/14               eu-west-1 production
      10.0.0.0/16             account A prod VPC
      10.1.0.0/16             account B prod VPC
    10.4.0.0/14               eu-west-1 non-production
  10.16.0.0/12                us-east-1

Leave gaps deliberately. An allocation with no headroom means the next Region or environment breaks the scheme.

Use AWS IPAM to manage it rather than a spreadsheet. It enforces allocation, prevents overlap, tracks utilisation, and integrates with VPC creation so an account cannot self-allocate a colliding range.

Know your on-premises and partner ranges before allocating. The overlap that matters most is usually with a corporate network that predates the cloud estate.

Sizing VPCs And Subnets

Important: Undersizing subnets is the mistake, and EKS is where it bites hardest. The VPC CNI assigns a VPC IP address to every pod, so a cluster running a few thousand pods consumes a few thousand addresses - plus the addresses held in the warm pool on each node.

A /24 subnet has 251 usable addresses. Three of those across three AZs is 753 addresses, which a moderately sized EKS cluster exhausts, and the failure is pods stuck in ContainerCreating with no obvious cause.

Size subnets a tier larger than the arithmetic suggests. A /20 per subnet costs nothing - addresses are free - and the alternative is a renumbering exercise. Also account for deployment headroom: a rolling deployment runs old and new tasks or pods simultaneously, so peak consumption is above steady state.

A workable default: a /16 per VPC, subdivided into /20 subnets - three private, three public, and reserved space for additional tiers.

Prefix delegation on the VPC CNI assigns blocks of addresses per node rather than individual ones, substantially improving density. Enable it on EKS clusters - Module 03 Topic 2.

Topology

graph TD
    subgraph SPOKE[Workload accounts]
        V1[VPC A]
        V2[VPC B]
        V3[VPC C]
    end
    subgraph NET[Network account]
        TGW[Transit Gateway]
        SS[Shared services VPC<br/>**centralised endpoints**<br/>**Route 53 Resolver**]
        EG[Egress VPC<br/>**centralised NAT**]
        INS[Inspection VPC<br/>optional]
    end
    V1 --> TGW
    V2 --> TGW
    V3 --> TGW
    TGW --> SS
    TGW --> EG
    TGW --> INS
    EG --> IGW[Internet]
    TGW --> DX[Direct Connect /<br/>VPN to on-premises]

This diagram shows a hub and spoke network topology. Workload VPCs in separate accounts attach to a Transit Gateway in a central network account. The Transit Gateway connects to a shared services VPC containing centralised VPC endpoints and Route 53 Resolver endpoints, an egress VPC providing centralised NAT to the internet, an optional inspection VPC, and Direct Connect or VPN connectivity to on-premises networks.

Transit Gateway hub-and-spoke is the default at any meaningful scale. VPC peering does not scale - it is a full mesh requiring n(n-1)/2 connections, no transitive routing, and a route table entry per peer in every VPC.

Peering remains correct for two or three VPCs, and specifically for a high-volume pair where the Transit Gateway's per-GB processing charge matters - a direct peering alongside the TGW for that pair is a legitimate optimisation.

Transit Gateway route tables are the segmentation mechanism, and this is the underused part: separate route tables for production and non-production, with attachments associated to the appropriate one, means production and development cannot route to each other at all. That is a stronger boundary than security groups, because it is not per-resource.

Centralised Egress And Endpoints

The two decisions that determine whether many accounts are affordable.

Important: NAT Gateways and interface VPC endpoints are per-AZ hourly charges, and they multiply by account. Twenty accounts each running three NAT Gateways is sixty NAT Gateways billing continuously before any traffic. The same arithmetic applies to interface endpoints - eight endpoints across three AZs in twenty accounts is 480 endpoint-hours per hour.

Centralising both is the design that makes a large account estate viable, and it must be designed early because retrofitting means renumbering routing across every account.

Centralised egress: NAT Gateways in a dedicated egress VPC, with spoke VPCs routing 0.0.0.0/0 to the Transit Gateway. One set of NAT Gateways serves the estate. The trade is a Transit Gateway processing charge on that traffic and a shared failure domain.

Centralised endpoints: interface endpoints in a shared services VPC, exposed to spokes via PrivateLink and resolved through Route 53 Resolver rules shared by RAM. One set of endpoints serves every account.

Gateway endpoints for S3 and DynamoDB are free and cannot be centralised - create them in every VPC without exception. There is no trade-off to evaluate.

DNS Architecture

The component that breaks most often and is diagnosed last.

Private hosted zones for internal names, shared across accounts via association rather than duplicated. A zone associated with many VPCs is one authoritative source; a zone per account drifts.

Route 53 Resolver rules, shared via RAM, are what make hybrid and cross-account resolution work centrally:

Outbound endpoint with forwarding rules - queries for corporate domains go to on-premises DNS
Inbound endpoint - on-premises DNS forwards AWS domains here
Rules shared by RAM - configured once in the network account, associated with every VPC

Warning: Configure and test resolution in both directions. The characteristic failure is configuring only the direction the first workload needed - AWS resolving on-premises names - and discovering weeks later that on-premises systems cannot resolve the names of services that have moved.

The symptom is confusing: connectivity tests pass, the network is fine, and an application fails in a way that looks like a service outage rather than a DNS problem.

Make bidirectional resolution part of the connectivity acceptance test, before any workload depends on it.

Sharing resolver rules via RAM is the difference between a central design and forty teams each configuring DNS differently. Do it from the first account.

Hybrid Connectivity

A single Direct Connect is a single point of failure, and it is a common design error because the connection itself is reliable - the failures are cut fibre, provider maintenance, and router problems at your end.

The minimum for production: Direct Connect with a VPN backup - cheap, and a degraded path rather than none.

Better: two Direct Connects at different locations, on different routers, ideally different providers. Two connections through the same conduit fail together.

Test the failover. An untested backup path has untested routing, untested MTU behaviour, and untested capacity.

Attach Direct Connect to the Transit Gateway via a transit VIF, so one connection reaches every VPC rather than one.

What To Build Now Versus Later

Day one, because retrofitting is expensive:

IP allocation scheme and IPAM
Transit Gateway and the account attachment pattern
Route table segmentation between environments
Shared services VPC with centralised endpoints
Centralised egress
Route 53 Resolver rules shared via RAM
Subnet sizing decisions

Defer until there is a requirement:

Network Firewall and the inspection VPC - real cost, and security groups plus DNS Firewall cover a great deal
Direct Connect, unless hybrid is immediate - weeks of lead time, so decide early even if you defer
IPv6 - plan the addressing so it is possible, implement when needed
Global Accelerator, multi-Region topology

The distinguishing question: does adding this later require renumbering or re-routing? If yes, decide now.

Making It Self-Service

A network design nobody can use without the platform team becomes the platform team's full-time job.

Tip: The network account should expose a small number of consumable things, and everything else should be invisible to workload teams.

Concretely, a team creating a new account should receive: a VPC with subnets already sized and tagged, a Transit Gateway attachment already associated with the right route table, resolver rules already applied, and routes to shared egress and shared endpoints already present. They should never open a networking ticket for a standard workload.

That is achieved by making the network part of the account baseline - vended as infrastructure as code alongside everything else in Topic 1's baseline, with the network account sharing the Transit Gateway, the resolver rules, and any shared subnets via AWS Resource Access Manager.

The signal that this is working: a new account is network-ready without a human in the network team doing anything. The signal that it is not: a backlog of connectivity requests, and teams building their own NAT because the ticket was slow.

Share, do not delegate. RAM sharing means the network account retains ownership of the Transit Gateway, the resolver rules, and the allocation, while workload accounts consume them. That keeps the design coherent without making the network team a bottleneck - teams cannot create a colliding VPC or attach to the wrong route table, because they are not the ones doing it.

Document the exception path. Some workloads genuinely need something non-standard - a dedicated egress for a compliance requirement, a direct peering for a high-volume path, an additional CIDR. A request process that exists and is used is better than a standard that gets worked around, and the requests themselves tell you what the next version of the standard should include.

Common Failure Modes

Warning: No IP allocation plan. Overlapping CIDR with no resolution.

Undersized subnets. EKS pods stuck pending.

Peering at scale. Full mesh, no transitivity.

No route table segmentation. Production and development mutually routable.

NAT per account. Multiplied hourly charges.

Interface endpoints per account. The same, worse.

No S3 and DynamoDB gateway endpoints. Free traffic through NAT.

DNS configured in one direction. Fails weeks later, diagnosed as a service outage.

Resolver rules per account. Inconsistent and duplicated.

Single Direct Connect. Single point of failure.

Failover untested.

Appliance mode not enabled on an inspection VPC attachment - Course 1 Module 10 Topic 9.

Why This Matters In An Interview

Interview: Network architecture is where a platform engineer's seniority is most visible, because the expensive mistakes are irreversible and predictable.

What interviewers are listening for:

CIDR planning before VPC creation, with IPAM.

Centralised endpoints and egress as the thing that makes many accounts affordable.

Transit Gateway route tables as a segmentation boundary.

Bidirectional DNS, tested.

Common follow-ups:

"Design the network for a 50-account organisation." → IPAM-managed allocation with headroom, Transit Gateway hub in a network account, route tables segmenting production from non-production, shared services VPC with centralised interface endpoints via PrivateLink, centralised egress, resolver rules shared by RAM, gateway endpoints in every VPC.

"Your on-premises network and a VPC both use 10.0.0.0/16." → Nothing routes them. Renumber, NAT, or expose specific services via PrivateLink. All are expensive, which is why allocation precedes VPC creation.

"Your NAT costs more than your compute." → NAT per account across many accounts, plus AWS service traffic routed through it. Centralise egress, and add gateway endpoints for S3 and DynamoDB - those are free and there is no reason not to.

"EKS pods are stuck in ContainerCreating." → IP exhaustion. Check subnet free addresses. Enable prefix delegation, and size subnets a tier larger next time.

Practice

Exercise: Starter. Document the CIDR ranges currently in use across every VPC and every on-premises network. Identify any overlap or any risk of it.

Intermediate. Calculate the cost of interface endpoints and NAT Gateways at your current and projected account count, then model the centralised alternative.

Advanced. Design the full topology for a stated organisation: allocation scheme, Transit Gateway route table segmentation, shared services, egress, and DNS. Identify which decisions would require renumbering if changed later.