Skip to content
About Me
----

Amazon VPC: Enterprise Subnetting, Routing & Production Triage

Architect Key Takeaways

Master Amazon Virtual Private Cloud (VPC): RFC 1918 CIDR engineering, multi-AZ high availability subnet design, NAT Gateways, and network troubleshooting runbooks.

Amazon VPC: Enterprise Subnetting, Routing & Production Triage

Section titled β€œAmazon VPC: Enterprise Subnetting, Routing & Production Triage”

Amazon Virtual Private Cloud (Amazon VPC) is the foundational networking boundary of your AWS cloud architecture. It provides complete control over your virtual networking environment, including resource placement, IP address allocation, subnets, route tables, network gateways, and security perimeters.

In modern enterprise architectures, a poorly designed VPC with overlapping CIDR blocks or single points of failure (like a lone NAT Gateway) can paralyze cross-account connectivity and cause catastrophic multi-region outages.


Designing a scalable corporate cloud network requires adhering to RFC 1918 private IPv4 address spaces (10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16) while preventing IP address collisions with on-premises corporate data centers or partner VPCs.

VPC CIDR: 10.100.0.0/16 (65,536 Total IP Addresses)
β”œβ”€β”€ Availability Zone A (us-east-1a)
β”‚ β”œβ”€β”€ Public Ingress Subnet [10.100.1.0/24] (251 usable IPs) ──> ALB, NAT-GW-A
β”‚ β”œβ”€β”€ Private Application Subnet [10.100.11.0/24] (251 usable IPs) ──> EC2, EKS Nodes
β”‚ └── Isolated Database Subnet [10.100.21.0/24] (251 usable IPs) ──> RDS Primary
β”œβ”€β”€ Availability Zone B (us-east-1b)
β”‚ β”œβ”€β”€ Public Ingress Subnet [10.100.2.0/24] (251 usable IPs) ──> ALB, NAT-GW-B
β”‚ β”œβ”€β”€ Private Application Subnet [10.100.12.0/24] (251 usable IPs) ──> EC2, EKS Nodes
β”‚ └── Isolated Database Subnet [10.100.22.0/24] (251 usable IPs) ──> RDS Standby
└── Availability Zone C (us-east-1c)
β”œβ”€β”€ Public Ingress Subnet [10.100.3.0/24] (251 usable IPs) ──> ALB, NAT-GW-C
β”œβ”€β”€ Private Application Subnet [10.100.13.0/24] (251 usable IPs) ──> EC2, EKS Nodes
└── Isolated Database Subnet [10.100.23.0/24] (251 usable IPs) ──> RDS Read Replica

[!IMPORTANT] The 5 AWS Reserved IP Addresses in Every Subnet: In a /24 subnet (which mathematically contains $2^8 = 256$ addresses), only 251 IP addresses are available for your resources. AWS reserves 5 addresses across all subnets:

  • 10.100.1.0: Network address (Standard IP requirement).
  • 10.100.1.1: AWS VPC Router.
  • 10.100.1.2: Amazon Route 53 DNS resolver (VPC CIDR base + 2).
  • 10.100.1.3: Reserved by AWS for future network expansion.
  • 10.100.1.255: Network broadcast address (VPC does not support standard broadcast, but retains reservation).

Traffic isolation in a VPC is enforced strictly through dedicated Route Tables associated with each subnet:

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Internet Gateway (IGW) β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚ Bidirectional Public Traffic (0.0.0.0/0)
β–Ό
[ Public Subnet: 10.100.1.0/24 ]
β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β–Ό β–Ό
[ Application Load Balancer ] [ NAT Gateway (AZ-A) ]
β”‚ (Private Traffic) β”‚ (Outbound Only 0.0.0.0/0)
β–Ό β–Ό
[ Private Subnet: 10.100.11.0/24 (EC2 / EKS Compute Workloads) ]
β”‚
β–Ό (Internal Port 5432 / 3306 Only)
[ Isolated DB Subnet: 10.100.21.0/24 (Zero Internet Access) ]
  1. Public Route Table (Points to Internet Gateway):
    10.100.0.0/16 ──> local
    0.0.0.0/0 ──> igw-0a1b2c3d4e5f67890
  2. Private App Route Table (Points to NAT Gateway in same AZ):
    10.100.0.0/16 ──> local
    0.0.0.0/0 ──> nat-0123456789abcdef0 (AZ-A NAT Gateway)
  3. Isolated Database Route Table (Strictly Local):
    10.100.0.0/16 ──> local
    # NO 0.0.0.0/0 route! Completely air-gapped from both ingress and egress.

3. Production Terraform Pattern: Highly Available Multi-AZ VPC

Section titled β€œ3. Production Terraform Pattern: Highly Available Multi-AZ VPC”
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.5"
name = "corp-prod-vpc"
cidr = "10.100.0.0/16"
azs = ["us-east-1a", "us-east-1b", "us-east-1c"]
public_subnets = ["10.100.1.0/24", "10.100.2.0/24", "10.100.3.0/24"]
private_subnets = ["10.100.11.0/24", "10.100.12.0/24", "10.100.13.0/24"]
database_subnets = ["10.100.21.0/24", "10.100.22.0/24", "10.100.23.0/24"]
# NAT Gateway Architecture: High Availability
enable_nat_gateway = true
single_nat_gateway = false # True HA: Provisions one NAT per AZ
one_nat_gateway_per_az = true
# DNS Configuration
enable_dns_hostnames = true
enable_dns_support = true
# Database Subnet Isolation
create_database_subnet_route_table = true
create_database_internet_gateway_route = false # Guarantees zero internet route
tags = {
Environment = "Production"
ManagedBy = "Terraform"
}
}

Incident 1: Private Subnet EC2 Cannot Download Patches or Reach Internet

Section titled β€œIncident 1: Private Subnet EC2 Cannot Download Patches or Reach Internet”

Symptom: Running apt update or yum update hangs and fails with Connection timed out.

Triage Protocol:

  1. Inspect Subnet Route Table: Verify that the private subnet is associated with a route table containing 0.0.0.0/0 targeted to a nat-xxxxxx (NAT Gateway).
  2. Verify NAT Gateway Subnet Placement: Ensure the NAT Gateway is physically deployed inside a Public Subnet (a subnet whose route table points to igw-xxxxxx). Placing a NAT Gateway in a private subnet breaks routing.
  3. Verify Elastic IP (EIP): Ensure an active Elastic IP is allocated and bound to the NAT Gateway.
  4. Inspect Security Groups: Ensure the EC2 Security Group outbound rules permit egress on port 80 (HTTP) and port 443 (HTTPS) to 0.0.0.0/0.
  5. Inspect Network ACLs: Ensure the private subnet NACL allows outbound traffic and inbound ephemeral return traffic (ports 1024–65535).

Symptom: EKS pods or EC2 instances fail to launch with InsufficientFreeAddressesInSubnet error.

Cause: Modern container workloads (like AWS VPC CNI on EKS) allocate a real VPC IP address to every single pod. A small /24 subnet runs out of addresses within hours under load.

Production Fix:

  1. Do not tear down the VPC! AWS allows adding Secondary CIDR blocks to an existing VPC:
    Terminal window
    aws ec2 associate-vpc-cidr-block \
    --vpc-id vpc-0a1b2c3d4e5f67890 \
    --cidr-block 100.64.0.0/16
  2. Create new subnets inside the secondary CIDR (100.64.1.0/24) and bind them directly to your EKS cluster nodes!

1. Why is deploying a single NAT Gateway across a multi-AZ VPC considered a critical anti-pattern in enterprise production?

Answer: While a single NAT Gateway saves money (~$32/month per gateway), it introduces two fatal issues:

  1. Single Point of Failure (SPOF): If the Availability Zone hosting the single NAT Gateway experiences a hardware failure, all private subnets across ALL other AZs instantly lose internet connectivity.
  2. Cross-AZ Data Transfer Charges: Routing traffic from private instances in AZ-B to a NAT Gateway in AZ-A incurs extra cross-AZ inter-region data transfer fees ($0.01/GB).
2. What is the fundamental difference between an Internet Gateway (IGW) and a NAT Gateway in terms of bidirectional traffic?

Answer:

  • Internet Gateway (IGW): Supports bidirectional 1-to-1 Network Address Translation (NAT). External internet clients can initiate connections to instances with public IPs, and instances can initiate connections out to the internet.
  • NAT Gateway: Supports unidirectional outbound-only Source Network Address Translation (SNAT). Instances in private subnets can initiate outbound requests, but external internet clients can NEVER initiate an inbound connection to private instances behind a NAT Gateway.