Security Groups vs. Network ACLs (NACLs): Deep Dive & Triage
Master AWS dual-layer firewall mechanics: stateful connection tracking in Security Groups, stateless rule evaluation in Network ACLs, ephemeral port allocation, and network triage runbooks.
Security Groups vs. Network ACLs (NACLs): Deep Dive & Triage
Section titled βSecurity Groups vs. Network ACLs (NACLs): Deep Dive & TriageβIn AWS networking, all traffic entering or leaving an Elastic Network Interface (ENI) must traverse two independent, complementary security boundaries: Security Groups (instance/ENI level) and Network Access Control Lists (NACLs) (subnet level).
Confusing these two layersβor misunderstanding how stateful tracking interacts with stateless packet inspectionβis the leading cause of βmysteriousβ packet drops, broken return traffic, and delayed incident responses in enterprise cloud environments.
1. Architectural Comparison Matrix
Section titled β1. Architectural Comparison Matrixβ [ Public Internet / Transit Gateway ] β βΌ βββββββββββββββββββββββββ β Network ACL (Subnet) β <--- Stateless Gatekeeper (Numbered Rules) βββββββββββββ¬ββββββββββββ β βΌ βββββββββββββββββββββββββ β Security Group (ENI) β <--- Stateful Scalpel (Permit Only) βββββββββββββ¬ββββββββββββ β βΌ [ Target EC2 / EKS Pod / RDS ]| Dimension | Security Groups (SG) | Network ACLs (NACL) |
|---|---|---|
| Enforcement Point | ENI / Virtual Network Interface | Subnet Boundary |
| State Tracking | Stateful: Outbound return traffic automatically allowed | Stateless: Inbound and outbound must be explicitly allowed |
| Rule Capabilities | ALLOW rules only (Default Deny All) | ALLOW and DENY rules |
| Evaluation Mechanics | All rules evaluated simultaneously | Evaluated in strict numerical order (lowest number first) |
| Ephemeral Port Handling | Automatic via kernel connection tracking table | Manual: Must explicitly allow ports 1024β65535 outbound |
| Reference Targets | IP CIDRs, Prefix Lists, or Security Group IDs | IP CIDR blocks only (Cannot reference other NACLs or SGs) |
| Scale / Quotas | Up to 5 SGs per ENI, 60 rules per SG (Default) | 1 NACL per Subnet, 20 inbound + 20 outbound rules |
| Operational Analogy | The Security Guard checking ID at your front door | The Military Checkpoint at the entrance to the colony |
2. Stateful Tracking vs. The Stateless Ephemeral Port Trap
Section titled β2. Stateful Tracking vs. The Stateless Ephemeral Port TrapβThe Magic of Stateful Connection Tracking
Section titled βThe Magic of Stateful Connection TrackingβWhen a client sends an HTTP request on port 443 into your EC2 instance, the Security Group inspects its inbound rules. If an inbound allow rule matches, the hypervisor registers the socket in its connection tracking table.
When the server sends the response payload back to the client, the Security Group inspects the tracking table, recognizes the existing connection, and instantly allows the outbound trafficβeven if your outbound Security Group rules have deleted all outbound traffic!
The Stateless NACL Trap & Ephemeral Ports
Section titled βThe Stateless NACL Trap & Ephemeral PortsβNetwork ACLs have zero memory. When an inbound packet arrives on port 443, the NACL checks inbound rule 100 (ALLOW tcp 443 0.0.0.0/0). The packet enters the subnet.
However, when the web server sends the response back to the client, the destination port is no longer 443! The clientβs operating system assigned a temporary random port between 1024 and 65535 (known as an ephemeral port) to receive the response:
- Linux clients / AWS NAT Gateways: Ports
1024β65535 - Windows clients: Ports
49152β65535
[!CAUTION] The #1 Most Common NACL Outage: If your subnetβs outbound NACL rule does not explicitly allow return traffic to ports
1024β65535, the NACL will drop the outbound response packet at the subnet boundary! Your users will see endless connection timeouts, even though your web server is running and port 443 is open inbound!
3. Production Terraform Pattern: Hardened Subnet NACL & Security Group
Section titled β3. Production Terraform Pattern: Hardened Subnet NACL & Security Groupβ# 1. Stateful Application Security Groupresource "aws_security_group" "web_app" { name = "corp-prod-web-app-sg" description = "Allows HTTPS ingress and unrestricted outbound" vpc_id = var.vpc_id
# Inbound: Allow HTTPS from Application Load Balancer ONLY ingress { description = "HTTPS from corporate ALB" from_port = 443 to_port = 443 protocol = "tcp" security_groups = [var.alb_security_group_id] # Referencing SG directly! }
# Outbound: Stateful egress egress { from_port = 0 to_port = 0 protocol = "-1" cidr_blocks = ["0.0.0.0/0"] }
tags = { Environment = "Production" ManagedBy = "Terraform" }}
# 2. Stateless Subnet Network ACLresource "aws_network_acl" "public_subnet_nacl" { vpc_id = var.vpc_id subnet_ids = var.public_subnet_ids
# INBOUND RULE 100: Allow HTTPS from anywhere ingress { rule_no = 100 protocol = "tcp" action = "allow" cidr_block = "0.0.0.0/0" from_port = 443 to_port = 443 }
# INBOUND RULE 110: Allow Ephemeral Return Ports (for outbound downloads) ingress { rule_no = 110 protocol = "tcp" action = "allow" cidr_block = "0.0.0.0/0" from_port = 1024 to_port = 65535 }
# OUTBOUND RULE 100: Allow Ephemeral Return Ports to clients egress { rule_no = 100 protocol = "tcp" action = "allow" cidr_block = "0.0.0.0/0" from_port = 1024 to_port = 65535 }
# OUTBOUND RULE 110: Allow HTTPS outbound (e.g. for API calls) egress { rule_no = 110 protocol = "tcp" action = "allow" cidr_block = "0.0.0.0/0" from_port = 443 to_port = 443 }}4. Real-Time Production Triage Runbook
Section titled β4. Real-Time Production Triage RunbookβIncident: Traffic Drops with βConnection Refusedβ or βTimeoutβ
Section titled βIncident: Traffic Drops with βConnection Refusedβ or βTimeoutββScenario: An EC2 instance or RDS database cannot be reached from an application server in another subnet.
Triage Protocol:
- Determine if itβs a Timeout or Refusal:
- Connection Timed Out: Points to packet filtering by a Security Group or NACL (packets silently dropped).
- Connection Refused: Points to OS level (service is not listening on that port, or local firewall
iptables/ufwis active).
- Inspect VPC Flow Logs for
NODATAvsREJECT:- Query Athena or CloudWatch Insights:
fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, action| filter dstPort = 443 or dstPort = 5432| sort @timestamp desc| limit 20
- If
action = "REJECT", the packet was blocked by an SG or NACL.
- Query Athena or CloudWatch Insights:
- Verify Security Group Reference Chaining:
- If using SG references (
security_groups = [aws_security_group.app.id]), verify that traffic stays inside the same VPC. Security group references do not work across VPC peering if DNS resolution of private IPs is disabled!
- If using SG references (
π§ Architect Knowledge Check
Section titled βπ§ Architect Knowledge Checkβ1. Why is a Network ACL the only AWS native network tool capable of blocking a specific malicious IP address?
Answer: Security Groups support ALLOW rules only (they cannot specify an explicit Deny). If an attacker attacks your application from IP
203.0.113.50, you cannot block that single IP with a Security Group. Network ACLs support explicit DENY rules. You can create Rule50withAction: Deny, CIDR: 203.0.113.50/32. Because NACLs evaluate rules in numerical order, Rule 50 will drop the attackerβs packets immediately before Rule 100 (ALLOW 0.0.0.0/0) is ever evaluated.
2. Can Security Group connection tracking exhaust hardware limits on large EC2 instances?
Answer: Yes. Under extreme distributed denial of service (DDoS) traffic or massive microservice connections, the hypervisorβs state tracking table can fill up (known as Connection Tracking Untracked Drops). To prevent connection tracking exhaustion on high-throughput workloads, configure Untracked Security Groups by adding matching inbound and outbound allow rules for all ports (
0-65535), or offload perimeter filtering to AWS WAF and Network ACLs.