Skip to content
About Me
----

Security Groups vs. Network ACLs (NACLs): Deep Dive & Triage

Architect Key Takeaways

Master AWS dual-layer firewall mechanics: stateful connection tracking in Security Groups, stateless rule evaluation in Network ACLs, ephemeral port allocation, and network triage runbooks.

Security Groups vs. Network ACLs (NACLs): Deep Dive & Triage

Section titled β€œSecurity Groups vs. Network ACLs (NACLs): Deep Dive & Triage”

In AWS networking, all traffic entering or leaving an Elastic Network Interface (ENI) must traverse two independent, complementary security boundaries: Security Groups (instance/ENI level) and Network Access Control Lists (NACLs) (subnet level).

Confusing these two layersβ€”or misunderstanding how stateful tracking interacts with stateless packet inspectionβ€”is the leading cause of β€œmysterious” packet drops, broken return traffic, and delayed incident responses in enterprise cloud environments.


[ Public Internet / Transit Gateway ]
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Network ACL (Subnet) β”‚ <--- Stateless Gatekeeper (Numbered Rules)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Security Group (ENI) β”‚ <--- Stateful Scalpel (Permit Only)
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
β”‚
β–Ό
[ Target EC2 / EKS Pod / RDS ]
Dimension Security Groups (SG) Network ACLs (NACL)
Enforcement Point ENI / Virtual Network Interface Subnet Boundary
State Tracking Stateful: Outbound return traffic automatically allowed Stateless: Inbound and outbound must be explicitly allowed
Rule Capabilities ALLOW rules only (Default Deny All) ALLOW and DENY rules
Evaluation Mechanics All rules evaluated simultaneously Evaluated in strict numerical order (lowest number first)
Ephemeral Port Handling Automatic via kernel connection tracking table Manual: Must explicitly allow ports 1024–65535 outbound
Reference Targets IP CIDRs, Prefix Lists, or Security Group IDs IP CIDR blocks only (Cannot reference other NACLs or SGs)
Scale / Quotas Up to 5 SGs per ENI, 60 rules per SG (Default) 1 NACL per Subnet, 20 inbound + 20 outbound rules
Operational Analogy The Security Guard checking ID at your front door The Military Checkpoint at the entrance to the colony

2. Stateful Tracking vs. The Stateless Ephemeral Port Trap

Section titled β€œ2. Stateful Tracking vs. The Stateless Ephemeral Port Trap”

When a client sends an HTTP request on port 443 into your EC2 instance, the Security Group inspects its inbound rules. If an inbound allow rule matches, the hypervisor registers the socket in its connection tracking table.

When the server sends the response payload back to the client, the Security Group inspects the tracking table, recognizes the existing connection, and instantly allows the outbound trafficβ€”even if your outbound Security Group rules have deleted all outbound traffic!

Network ACLs have zero memory. When an inbound packet arrives on port 443, the NACL checks inbound rule 100 (ALLOW tcp 443 0.0.0.0/0). The packet enters the subnet.

However, when the web server sends the response back to the client, the destination port is no longer 443! The client’s operating system assigned a temporary random port between 1024 and 65535 (known as an ephemeral port) to receive the response:

  • Linux clients / AWS NAT Gateways: Ports 1024–65535
  • Windows clients: Ports 49152–65535

[!CAUTION] The #1 Most Common NACL Outage: If your subnet’s outbound NACL rule does not explicitly allow return traffic to ports 1024–65535, the NACL will drop the outbound response packet at the subnet boundary! Your users will see endless connection timeouts, even though your web server is running and port 443 is open inbound!


3. Production Terraform Pattern: Hardened Subnet NACL & Security Group

Section titled β€œ3. Production Terraform Pattern: Hardened Subnet NACL & Security Group”
# 1. Stateful Application Security Group
resource "aws_security_group" "web_app" {
name = "corp-prod-web-app-sg"
description = "Allows HTTPS ingress and unrestricted outbound"
vpc_id = var.vpc_id
# Inbound: Allow HTTPS from Application Load Balancer ONLY
ingress {
description = "HTTPS from corporate ALB"
from_port = 443
to_port = 443
protocol = "tcp"
security_groups = [var.alb_security_group_id] # Referencing SG directly!
}
# Outbound: Stateful egress
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = {
Environment = "Production"
ManagedBy = "Terraform"
}
}
# 2. Stateless Subnet Network ACL
resource "aws_network_acl" "public_subnet_nacl" {
vpc_id = var.vpc_id
subnet_ids = var.public_subnet_ids
# INBOUND RULE 100: Allow HTTPS from anywhere
ingress {
rule_no = 100
protocol = "tcp"
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 443
to_port = 443
}
# INBOUND RULE 110: Allow Ephemeral Return Ports (for outbound downloads)
ingress {
rule_no = 110
protocol = "tcp"
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 1024
to_port = 65535
}
# OUTBOUND RULE 100: Allow Ephemeral Return Ports to clients
egress {
rule_no = 100
protocol = "tcp"
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 1024
to_port = 65535
}
# OUTBOUND RULE 110: Allow HTTPS outbound (e.g. for API calls)
egress {
rule_no = 110
protocol = "tcp"
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 443
to_port = 443
}
}

Incident: Traffic Drops with β€œConnection Refused” or β€œTimeout”

Section titled β€œIncident: Traffic Drops with β€œConnection Refused” or β€œTimeout””

Scenario: An EC2 instance or RDS database cannot be reached from an application server in another subnet.

Triage Protocol:

  1. Determine if it’s a Timeout or Refusal:
    • Connection Timed Out: Points to packet filtering by a Security Group or NACL (packets silently dropped).
    • Connection Refused: Points to OS level (service is not listening on that port, or local firewall iptables/ufw is active).
  2. Inspect VPC Flow Logs for NODATA vs REJECT:
    • Query Athena or CloudWatch Insights:
      fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol, action
      | filter dstPort = 443 or dstPort = 5432
      | sort @timestamp desc
      | limit 20
    • If action = "REJECT", the packet was blocked by an SG or NACL.
  3. Verify Security Group Reference Chaining:
    • If using SG references (security_groups = [aws_security_group.app.id]), verify that traffic stays inside the same VPC. Security group references do not work across VPC peering if DNS resolution of private IPs is disabled!

1. Why is a Network ACL the only AWS native network tool capable of blocking a specific malicious IP address?

Answer: Security Groups support ALLOW rules only (they cannot specify an explicit Deny). If an attacker attacks your application from IP 203.0.113.50, you cannot block that single IP with a Security Group. Network ACLs support explicit DENY rules. You can create Rule 50 with Action: Deny, CIDR: 203.0.113.50/32. Because NACLs evaluate rules in numerical order, Rule 50 will drop the attacker’s packets immediately before Rule 100 (ALLOW 0.0.0.0/0) is ever evaluated.

2. Can Security Group connection tracking exhaust hardware limits on large EC2 instances?

Answer: Yes. Under extreme distributed denial of service (DDoS) traffic or massive microservice connections, the hypervisor’s state tracking table can fill up (known as Connection Tracking Untracked Drops). To prevent connection tracking exhaustion on high-throughput workloads, configure Untracked Security Groups by adding matching inbound and outbound allow rules for all ports (0-65535), or offload perimeter filtering to AWS WAF and Network ACLs.