AWS – Compute – EC2

1. Amazon EC2 (Elastic Compute Cloud)

EC2 is an IaaS (Infrastructure as a Service) solution where we have full control at the OS level by selecting the desired CPU, memory, and storage options.

Think of an EC2 instance like a Custom-Built Office Space.

• IaaS means you own the keys to the office; you decide the furniture (RAM/CPU) and the security locks (OS Hardening).
• User Data is the Interior Design Team that sets up everything only on the very first day.
• Golden Images (AMIs) are the Master Blueprints—if you need 100 identical offices, you just use the blueprint.
• In DevSecOps, we don’t “fix” a broken office; if a lightbulb breaks or a lock is compromised, we tear down the room and build a brand-new one from the blueprint (Immutable Infrastructure).

1.1. Service Model IaaS – Infrastructure as a Service

• Full Control: – You have “Root” or “Administrator” access. This gives you the power to install security agents (like CrowdStrike or Wazuh) and harden the kernel.
• The Responsibility Shift: – In the AWS Shared Responsibility Model, AWS secures the “Cloud” (Physical infrastructure, Hypervisor), but you must secure what is “In” the Cloud (OS, Data, Networking).

1.2. OS Options & Hardening

AWS supports Windows, Ubuntu, Red Hat, Amazon Linux, and macOS. However, from a security standpoint, not all OS choices are equal:

• Golden Images: Instead of using a raw Public AMI, DevSecOps teams build “Hardened” images. This means pre-installing security patches and removing unnecessary software (like Telnet or legacy drivers) before the application even starts.

1.3. DevSecOps Best Practices for EC2

To be a “Guru,” you must implement these four security pillars for every EC2 instance:

1.4. Deployment Strategy – Immutable Infrastructure

• Instead of logging into a server to “fix” a bug (which is risky and unrecorded), update the script (Terraform/CloudFormation), build a new AMI, and replace the old server.
• Benefit: This ensures that no “Configuration Drift” happens and every server in production is exactly what the security team approved.

Key Terms to Remember

• Hardening: The process of securing an OS by minimizing its surface of vulnerability.
• Least Privilege: Giving the EC2 instance only the specific permissions it needs (e.g., only access to one S3 bucket, not all).
• IMDSv2: The secure version of the Instance Metadata Service that protects against SSRF (Server-Side Request Forgery) attacks.

Real-World Challenges

1. Configuration Drift
   • An engineer logs in “just for 5 minutes” to fix a small bug manually. Now, that server is different from the others.
   • The Impact: When you scale up, the new servers won’t have that fix, causing your app to crash.
   • Fix: Use AWS Config to detect changes and SSM Inventory to track what is actually installed.
2. Secret Leakage in Metadata (IMDSv1)
   • Older instances use IMDSv1 which is vulnerable to SSRF (Server-Side Request Forgery). An attacker can steal your IAM Role credentials with a simple URL command.
   • Fix: Strictly enforce IMDSv2. It requires a “Session Token,” making it nearly impossible for hackers to steal credentials.
3. The “Stateful” Trap
   • Immutable infrastructure is easy for web servers (stateless), but very hard for databases or legacy apps that save files locally.
   • The Impact: If you terminate the instance to “patch” it, you lose your data.
   • Fix: Always separate data from compute. Use EBS Volumes that “Detach/Reattach” or store data in Amazon S3/RDS.
4. Alert Fatigue
   • Tools like Amazon Inspector or GuardDuty can send 1,000 alerts a day.
   • The Impact: Security teams start ignoring the alerts because there are too many “Low Priority” ones.
   • Fix: Use AWS Security Hub to aggregate and filter alerts. Only “High” and “Critical” should trigger a PagerDuty/Slack notification.
5. Kernel Compatibility
   • Moving from old Xen instances to new Nitro instances requires specific drivers (ENA/NVMe).
   • The Impact: If your Golden AMI doesn’t have these drivers, the instance will simply not boot, leading to “Instance Status Check Failed.”
   • Fix: Always test your AMI on a small t3.micro before pushing it to the production fleet.

2. Hardware Configuration Instance Types

AWS EC2 Instance Types decide the hardware configuration of your cloud server.
Each instance type defines CPU, RAM, network speed, storage options, and accelerator hardware.

Choosing an EC2 Instance Type is like choosing the right vehicle for a job. You wouldn’t use a Formula 1 car (Compute Optimized) to move furniture, nor a tiny scooter (T-series) to haul a heavy trailer (Big Data). In DevSecOps, we don’t just look at “speed”; we look at Isolation (Nitro System), Compliance (Dedicated Hosts), and Sustainability (Graviton).

2.1. The Naming Logic: m5zn.large

Understanding the suffixes is a “Guru” skill for optimizing security and performance:

• m = Family: The primary purpose.
• 5 = Generation: Newer generations are more secure. DevSecOps Tip: Always use the latest generation (e.g., m6 instead of m5) because they use the Nitro System, which provides hardware-based root-of-trust.
• Attributes (z/n/g):
  • g = Graviton (AWS Silicon) More secure and 40% better price-performance.
  • n = High Network Crucial for security appliances like firewalls.
  • z = High clock speed.
• large = Size : Vertical scaling (RAM/CPU).

2.2 The Complete EC2 Instance Series Guide

3. AMI (Amazon Machine Image) – The “Golden Image” Strategy

An AMI is a “master blueprint” of your server. It contains the OS, security settings, and your application. In DevSecOps, we don’t just “click and launch.” We follow the Golden Image process: we take a base OS, “harden” it (remove weaknesses), install security agents, and then lock it down as a private AMI for the whole company to use.

3.1. AMI Types & Security Risks

Not all AMIs are safe. As a Guru, you must know where your image comes from:

3.2. The “Golden Image” Lifecycle

In a professional DevSecOps setup, we use EC2 Image Builder to automate this:

1. Base: Start with a clean, official OS (e.g., Amazon Linux 2023, Ubuntu etc).
2. Harden: Disable unused ports, remove default passwords, and set up CIS Benchmarks.
3. Install: Add security agents (CrowdStrike, CloudWatch Agent, Inspector).
4. Scan: Use Amazon Inspector to check the image for vulnerabilities before saving.
5. Distribute: Share the “certified” AMI with other AWS accounts in your organization.

3.3. Key Technical Properties and best practices

• Regional Resource: An AMI created in Mumbai (ap-south-1) cannot be used in US-East (us-east-1) directly. You must Copy the AMI to the new region first.
  • When copying AMIs across regions, always Encrypt the target copy using a KMS key in that region.
• Storage-backed: AMIs are essentially snapshots of EBS volumes stored in S3.
• Launch Permissions: You can make an AMI “Public” (dangerous!) or share it with specific AWS Account IDs (Secure).

• Deregister Old AMIs: Don’t keep old, unpatched AMIs. If an AMI has a “Critical” vulnerability, deregister it so no one can launch new servers from it.
• Enforce IMDSv2: When creating an AMI, configure it to require Instance Metadata Service Version 2. This prevents SSRF attacks from stealing your server’s identity.
• No Hardcoded Secrets: Never create an AMI that has API keys, passwords, or SSH keys “baked in.” Use AWS Secrets Manager to pull these at runtime instead.
• Automated Cleanup: Use a “Recycle Bin” for AMIs to prevent accidental deletion of images currently being used by Auto Scaling Groups.

Real-World Challenges

1. AMI Sprawl (The “Library” Problem)
   • Over time, you end up with hundreds of AMIs (v1, v1.1, v1.2-beta). This wastes S3 storage costs and makes it impossible to know which one is “actually” safe.
   • Fix: Implement an AMI Lifecycle Policy. Automatically Deregister and delete snapshots of any AMI older than 90 days that isn’t currently associated with a running instance.
2. The “Stateless” Struggle
   • If your application stores data inside the AMI (like logs or local DBs), you can’t replace the AMI without losing data.
   • Fix: Force developers to use External Storage (EBS, EFS, or S3). The AMI should be “Disposable.”
3. Pipeline Bottlenecks (Speed vs. Security)
   • A full security scan and bake process can take 30–60 minutes. During an active zero-day exploit, this delay is dangerous.
   • Fix: Use In-Place Patching via SSM Run Command for urgent “Emergency” fixes, but follow up immediately with a new Golden AMI bake to maintain consistency.
4. Cross-Account Sharing Risks
   • Accidentally making a Private AMI “Public” can leak your internal source code and configuration to the world.
   • Enable the “Block Public Access for AMIs” setting at the AWS Account level. This is a “Safety Switch” that prevents anyone from making an image public, even by mistake.
5. Patching “Ghost” Vulnerabilities
   • You patch the OS, but the Application Libraries (like a vulnerable Python package) are still hidden inside the AMI.
   • Fix: Use Software Bill of Materials (SBOM) tools during the bake process to list every library inside the AMI and scan them specifically.

4. EC2 Storage – The Hard Disks

EC2 storage is like the storage on your phone. You have EBS (like an SD card you can move), Instance Store (built-in storage that wipes if you reset), and EFS (like Google Photos/Cloud storage shared by multiple phones). In DevSecOps, we follow the “Encrypted by Default” rule no data should ever sit on a disk without a digital lock (KMS Encryption).

4.1. Storage Types & Security Trade-offs

4.2. EBS Volume Types

In DevSecOps, picking the volume isn’t just about speed; it’s about Cost vs. Security Overhead.

• gp3 (General Purpose): The modern standard. It allows you to provision IOPS and Throughput independently. Perfect for 90% of DevSecOps tools.
• io2 (Provisioned IOPS): The “High-End” choice. Supports EBS Multi-Attach (with Block Express) and provides 99.999% durability. Use this for critical production databases.
• st1 / sc1 (HDD): Low cost, but slow. Only use for big data logs where you don’t need fast random access.

4.3. EBS Encryption

You must emphasize this on your site: Encryption is non-negotiable.

1. Encryption by Default: Go to EC2 Settings → Data Protection → Enable “Always encrypt new EBS volumes.”
2. KMS Integration: Use Customer Managed Keys (CMK) instead of AWS Managed Keys for better audit trails in CloudTrail.
3. Data in Transit: EBS automatically encrypts data moving between the instance and the volume.

4.4. Backup & Recovery

A “Guru” doesn’t just back up; they automate for Compliance.

• Amazon Data Lifecycle Manager (DLM): Great for simple, tag-based automation of EBS snapshots.
• AWS Backup (The “Guru” Choice): Centralized backup service.
  • Cross-Region Copy: Protects against a whole AWS Region going down.
  • Cross-Account Backup: Protects against a “Ransomware” attack where your main account is compromised.
  • WORM (Write Once Read Many): Using AWS Backup Vault Lock to make backups immutable (they cannot be deleted by anyone, even the root user).

DevSecOps Checklist for Storage Security

• [ ] Classify Data: Is it Public, Private, or Secret?
• [ ] Enable Encryption: Is “EBS Encryption by Default” turned on in every region?
• [ ] Snapshot Hygiene: Are you using DLM to delete old snapshots to save costs?
• [ ] Least Privilege: Does the EC2 IAM Role have permissions to only the KMS key it needs?

Real-World Challenges

1. The “Orphaned Volume” Cost Leak
   • When you delete an EC2 instance, the EBS volumes are sometimes left behind (if “Delete on Termination” is unchecked).
   • Impact: You keep paying for disks that are doing nothing. This is a “DevOps” failure.
   • Fix: Use AWS Config Rules to automatically find and delete (or snapshot) “Available” volumes that haven’t been attached for more than 7 days.
2. Encryption “Migration” Headaches
   • You cannot encrypt an existing unencrypted volume directly.
   • Impact: If you find an old unencrypted volume with 10TB of data, you have to take a snapshot, copy that snapshot to an encrypted version, and then create a new volume.
   • Fix: Shift-Left. Enforce “Encryption by Default” from day one so you never have to do this manual migration.
3. Instance Store Data Loss
   • Developers often put important data on the Instance Store because it is “free and fast,” forgetting it is Ephemeral.
   • Impact: The host hardware fails, the instance stops, and the data is gone forever.
   • Fix: Ensure your application replicates data to EBS or S3 continuously if using Instance Store for performance.
4. Shared Storage (EFS) Performance Bottlenecks
   • EFS performance is based on “Burst Credits” or “Provisioned Throughput.”
   • Impact: If your application suddenly grows, the EFS might become extremely slow, making it look like a Denial of Service (DoS) attack.
   • Fix: Monitor the BurstCreditBalance metric in CloudWatch and switch to Elastic Throughput mode for unpredictable workloads.
5. Snapshot Data Leakage
   • Public EBS Snapshots.
   • Impact: If a developer accidentally makes a snapshot “Public,” the entire world can see your files, including passwords and customer data.
   • Fix: Use Service Control Policies (SCPs) at the Organization level to Deny the ec2:ModifySnapshotAttribute action for public sharing.

5. EC2 Instance Lifecycle – Power States

What happens when you start EC2 instance.

DevSecOps Implementation Guide

1. Termination Protection “Anti-Accident” Lock
   • Enable Termination Protection for all “Production” or “Stateful” instances (like Databases). This prevents a developer or a buggy script from accidentally deleting a critical server via the API or CLI.
2. Incident Response: “Stop vs. Terminate”
   • If Amazon GuardDuty alerts you that an instance is compromised:
     • NEVER Terminate immediately. You will lose all evidence (RAM and temp files).
     • Isolate the instance: Change the Security Group to a “Quarantine SG” (No Inbound/Outbound).
     • Hibernate or Stop: Capture the memory and disk snapshots to understand how the hacker got i
3. User Data & Metadata Security – When an instance is in the Pending state, it pulls metadata.
   • IMDSv2 Only: Ensure your lifecycle process forces IMDSv2. This prevents “Metadata SSRF” attacks where a hacker could steal the IAM Role credentials while the server is running.
   • No Secrets in User Data: Never put passwords or API keys in the User Data script. Use AWS Secrets Manager to pull them securely once the instance is Running.
4. Auto Scaling Lifecycle Hooks – use Lifecycle Hooks during Auto Scaling.
   • When an instance is about to be terminated, the hook pauses the process. This gives you time to run a security audit script or export final logs before the instance vanishes forever.

Real-World Challenges

1. The “Ghost Log” Challenge
   • When an instance is terminated suddenly (e.g., during a Spot interruption), any logs not yet synced to CloudWatch are lost forever.
   • Solution: Implement Lifecycle Hooks to give logging agents time to “flush” data before the termination completes.
2. Hibernation Compatibility
   • Hibernation only works if the EBS root volume is encrypted and you have enough space to dump the RAM.
   • Solution: Always enable Encryption by Default at the account level (using KMS) so your instances are always hibernation-ready for forensic use.
3. Bootstrap Failures (The “Zombie” Instance)
   • A User Data script fails halfway through (e.g., a network timeout). The instance shows as Running in the console, but the security agents aren’t installed.
   • Solution: Use CloudFormation WaitConditions or SSM Automation to monitor bootstrap success. If it fails, the instance should be automatically terminated (Fail-Fast).
4. IMDSv2 Migration Friction
   • Legacy applications or older SDKs might break if you suddenly force IMDSv2.
   • Solution: Use the MetadataNoToken CloudWatch metric to identify which applications are still using the insecure IMDSv1 before enforcing the change account-wide using an SCP (Service Control Policy).
5. SSM Agent Connectivity
   • For modern “No SSH” access, the SSM Agent must be active. If the instance is in a private subnet with no NAT Gateway/VPC Endpoint, it becomes a “Black Box.”
   • Solution: Use VPC Endpoints for Systems Manager to ensure secure, private connectivity for maintenance in all lifecycle states.

6. EC2 Networking

Think of EC2 networking like a house in a gated community. The Private IP is your internal intercom address. The Public IP is like a temporary guest pass for the outside world. An Elastic IP is your permanent home address. The ENI is the physical network card you plug into your computer, and ENA/EFA are high-speed “highway” connections for heavy lifting. From a DevSecOps view, we treat networking as code (IaC), ensuring that no “doors” (ports) are left open by accident and that all traffic is monitored and encrypted.

6.1. Private IP Addresses

• Automatically assigned from the IPv4 address range of the subnet.
• It is the primary identity of the instance within the VPC. It persists through stops and starts.
• DevSecOps Perspective:
  • Zero Trust Architecture: Communication between microservices should ideally happen over Private IPs only.
  • Internal Security: Use Security Groups to restrict traffic between Private IPs, ensuring that even if one instance is compromised, the attacker cannot move laterally through the network easily.

6.2. Public IP Addresses

• A dynamic IP address used to communicate with the internet.
• Lost when the instance is stopped; a new one is assigned upon restart.
• DevSecOps Perspective:
  • Attack Surface Reduction: Avoid using Public IPs for database servers or backend logic. Use NAT Gateways instead so internal instances can get updates without being exposed to the internet.
  • Monitoring: Track the creation of public IPs via AWS CloudTrail to ensure no unauthorized public entry points are created.

6.3. Elastic IP (EIP)

• A static, reserved public IPv4 address. Remains associated with your account until you release it. Helpful for masking instance failures by remapping the IP to another instance.
• DevSecOps Perspective:
  • Whitelisting: Essential when your application needs to talk to a third-party API that requires a fixed IP for their firewall “allow-list.”
  • Cost Management: From a “DevOps” efficiency standpoint, EIPs incur costs if they are not attached to a running instance to prevent IP address wasting.

6.4. Elastic Network Interfaces (ENI)

• A logical networking component in a VPC that represents a virtual network card.
• Can include a primary private IP, multiple secondary IPs, a MAC address, and security group memberships.
• DevSecOps Perspective:
  • Network Isolation: You can attach two ENIs to a single instance one for management traffic (restricted) and one for public data traffic.
  • High Availability: In a DevSecOps pipeline, you can programmatically move an ENI from a failed instance to a standby instance to maintain the same IP and security configuration.

6.5. Enhanced Networking (ENA & EFA)

• ENA (Elastic Network Adapter): Supports speeds up to 100 Gbps. Uses SR-IOV to provide high throughput and low CPU utilization.
• EFA (Elastic Fabric Adapter): A specialized ENI for High-Performance Computing (HPC) and Machine Learning. It uses “OS-bypass” to let instances communicate directly with each other.
• DevSecOps Perspective:
  • Performance Bottleneck Security: In heavy workloads like ML or Big Data, network lags can cause system timeouts that look like Denial of Service (DoS) attacks. Using ENA/EFA ensures the “Availability” pillar of the CIA triad (Confidentiality, Integrity, Availability) is maintained under high load.

DevSecOps “Best Practice” Checklist

1. Security Groups (Firewalls): Always follow the Principle of Least Privilege. Only open ports (like 80 or 443) that are strictly necessary.
2. Infrastructure as Code (IaC): Use Terraform or CloudFormation to define your VPC and Subnets. This ensures that networking is “Version Controlled” and can be audited.
3. VPC Flow Logs: Enable these to capture information about the IP traffic going to and from network interfaces. This is your primary tool for “Continuous Monitoring” and detecting suspicious traffic patterns.
4. Reachability Analyzer: Use this tool during the “Test” phase of your DevOps cycle to ensure that your network paths are actually secure and no unintended paths to the internet exist.

Real-World Challenges

1. Security Group “Sprawl”
   • Large organizations often end up with hundreds of Security Groups. Some groups have 50+ rules, making it impossible to audit manually.
   • Impact: A developer might add a “temporary” rule to port 22 (SSH) and forget to delete it, creating a permanent hole.
   • Fix: Use AWS Firewall Manager or AWS Config Rules to automatically flag and delete any SG rule that doesn’t follow your standard “Approved” list.
2. IP Address Exhaustion
   • If you choose a small subnet (like /28), you might run out of Private IPs when your Auto Scaling Group tries to add more servers.
   • Impact: Your app won’t scale up during peak traffic, leading to a crash.
   • Fix: Plan your VPC CIDR carefully. Always use secondary CIDR blocks if you start running out of space.
3. The “Elastic IP” Bill Shock
   • AWS charges for Elastic IPs that are not attached to a running instance.
   • Impact: If a developer terminates a server but forgets to “Release” the EIP, you keep paying for it every hour.
   • Fix: Write a simple Lambda function that runs every night to find “Unassociated EIPs” and sends a bill alert to Slack or deletes them.
4. VPC Flow Log Data “Drowning”
   • Flow logs generate millions of lines of data. Storing and searching through them is expensive and slow.
   • Impact: When a security breach happens, it takes hours to find the “needle in the haystack.”
   • Fix: Send logs to Amazon Athena or OpenSearch and create “Summary Dashboards.” Focus only on REJECT actions to spot hackers trying to scan your ports.
5. Latency in ENA/EFA
   • While ENA is fast, “OS-bypass” in EFA can be tricky to configure for specific apps.
   • Impact: Misconfiguration can actually make your network slower than standard networking.
   • Fix: Use Performance Testing in your staging environment to baseline the “Network Latency” before pushing to production.

7. Security Group and NACL

Think of Security Groups (SG) as your personal security guard who stands right at your front door (the instance); if he lets you in, he remembers you and lets you out automatically.

NACLs are like the main gate of your gated community (the subnet); they don’t remember faces, so they need a list of exactly who can enter and who can leave. In

DevSecOps, we use the SG for fine-tuned control and the NACL for broad “blocking” of known bad actors.

7.1. Security Groups

• Virtual Firewall: Acts as a firewall for your EC2 instances to control inbound and outbound traffic.
• Default Behavior: All inbound traffic is blocked by default. All outbound traffic is allowed by default (can be changed).
• Stateful Nature: If you allow traffic on port 80 (Inbound), the response (Outbound) is automatically allowed, even if no outbound rule exists.

DevSecOps Perspective:

1. Principle of Least Privilege: Never use 0.0.0.0/0 unless it’s a public-facing load balancer. For internal apps, always reference other Security Groups (e.g., “Allow Inbound from SG-App-Tier”) instead of IP ranges.
2. Infrastructure as Code (IaC): Always define SGs in Terraform or CloudFormation. This prevents “Rule Sprawl” where manual changes lead to security holes over time.
3. Security Group Tagging: Use tags like Environment: Production and Compliance: PCI-DSS to automate security audits.
4. Zero-Trust: Treat every instance as potentially compromised. Use SGs to isolate instances even within the same subnet.

Real-World Challenges

1. Rule Sprawl & Limits: SGs have a default limit (usually 60 rules). In large enterprises, you quickly hit this limit.
   • Guru Fix: Use Managed Prefix Lists to group multiple IP ranges into a single “Named List” to save space and simplify management.
2. Overlapping Rules: Attaching 5 different SGs to one instance can create “Swiss Cheese” security—one permissive rule in any group opens the whole instance.
   • Guru Fix: Use IAM Access Analyzer for Network to audit if any instance has unintentional paths to the internet.

7.2. NACL (The Shield)

• Subnet Level Security: Controls traffic entering or leaving the entire subnet.
• Stateless Nature: It does not “remember” sessions. If you allow Inbound traffic on port 443, you must also add an Outbound rule for the ephemeral ports (usually 1024-65535) for the response to reach the user.
• Rule Ordering: Rules are processed from lowest to highest number. As soon as a match is found (Allow or Deny), it stops looking.

DevSecOps Perspective:

1. Hardened Perimeter: Use NACLs as a “Blacklist.” If your logs (VPC Flow Logs) show an attack from a specific IP range, block it at the NACL level so it never even reaches your instances.
2. Compliance Guardrails: Use NACLs to enforce high-level company policies, such as “No SSH (Port 22) traffic allowed into the Database Subnet,” regardless of what an individual developer might set in a Security Group.
3. Ephemeral Port Security: In DevSecOps pipelines, misconfiguring NACL ephemeral ports is a common cause of deployment failures. Always ensure your “Stateless” logic accounts for the return path of the traffic.
4. Forensics & Logging: Since NACLs sit at the boundary, they are the best place to monitor for “Rejection” spikes, which often indicate a port-scanning attempt or a brute-force attack.

Real-World Challenges

1. The “Return Path” Nightmare: Developers often forget the stateless nature of NACLs. They open Inbound 443 but block Outbound, causing the app to “timeout.”
   • Fix: Always add a standard Outbound rule (e.g., Rule #1000) allowing TCP 1024-65535 for any public-facing subnet.
2. Order of Operations: If you have an ALLOW ALL at Rule #100 and a DENY Malicious IP at Rule #200, the Deny rule will never run.
   • Fix: Always leave “gaps” in your rule numbers (10, 20, 30…). Put your DENY rules at the top (lowest numbers) and broad ALLOW rules at the bottom.
3. Visibility Bias: Security Groups don’t tell you what they blocked. NACLs also don’t have a “Log” button in the console.
   • Fix: Enable VPC Flow Logs. This is the only way to see REJECT packets and prove exactly which rule is blocking your traffic.

8. EC2 Instance Connect / Accessing options

Think of accessing your server like entering a high-security vault.

• SSH is like having a physical metal key; if you lose it or someone steals a copy, they can get in.
• SSM Session Manager is like a biometric scan; you don’t need a key, and every single thing you do inside the vault is recorded on camera.
• EC2 Instance Connect is like a digital one-time passcode.
• Instance Connect Endpoint is a special “tunnel” that lets you use your digital passcode even if the vault is hidden deep underground (private subnet) without a public door.

8.1. Traditional SSH / RDP

• The “Legacy” way: Direct connection to the instance. Requires port 22 (Linux) or 3389 (Windows) to be open in the Security Group.
• DevSecOps Perspective:
  • Key Sprawl Risk: Static .pem files are often accidentally uploaded to GitHub or shared over Slack, leading to major leaks.
  • Brute Force: Open ports 22/3389 are “magnets” for botnets.
  • Remediation: If you must use SSH, use Client VPN or a Bastion Host (Jump Box) to limit exposure.

8.2. AWS Systems Manager (SSM) Session Manager

• Uses the SSM Agent (pre-installed on most AMIs) to initiate an outbound connection to AWS. You connect via the browser or CLI.
• DevSecOps Perspective:
  • No Inbound Ports: You can close port 22/3389 entirely in your Security Groups. This is the Gold Standard for DevSecOps.
  • Centralized Logging: Every command typed by an engineer is logged to S3 or CloudWatch Logs. This is essential for compliance (HIPAA, SOC2).
  • MFA Integration: You can require Multi-Factor Authentication (MFA) at the IAM level before a session can start.

8.3. EC2 Instance Connect (EIC)

• How it works: AWS pushes a one-time SSH public key to the instance metadata for 60 seconds. You log in using that temporary key.
• DevSecOps Perspective:
  • Eliminates Static Keys: No more managing .pem files.
  • IAM-Centric: Access is controlled by IAM policies, not by who has a file on their laptop.
  • Restriction: Still requires the instance to have a Public IP and Port 22 open to the EC2 Instance Connect service range.

8.4. EC2 Instance Connect Endpoint (EICE)

• The “No-Bastion” Solution: A VPC endpoint that allows you to SSH into instances in private subnets without needing a Bastion Host or a Public IP.
• DevSecOps Perspective:
  • Cost & Security Efficiency: Traditionally, you had to pay for a “Bastion” instance and secure it. EICE is a managed service that replaces the Bastion, reducing your attack surface and management overhead.
  • Traffic Control: You can use Security Groups on the Endpoint itself to restrict which IP addresses in your office can even attempt to connect to the private instances.

DevSecOps Best Practices for EC2 Access

1. Enforce IMDSv2: Ensure your instances use Instance Metadata Service Version 2 (session-oriented) to prevent SSRF (Server-Side Request Forgery) attacks that try to steal IAM roles.
2. Infrastructure as Code (IaC): Use Terraform to ensure all EC2 instances are launched with an IAM Instance Profile containing the AmazonSSMManagedInstanceCore policy.
3. Automated Key Rotation: If using traditional SSH, automate the rotation of keys using AWS Secrets Manager.
4. Just-In-Time (JIT) Access: In a mature DevSecOps pipeline, no one has permanent access to production. Use IAM Condition Keys to grant access only during approved maintenance windows.

Real-World Challenges

1. SSM Agent Connectivity: For Session Manager to work, the instance must reach the SSM service. If your private instance has no NAT Gateway or VPC Endpoint, the agent cannot “heartbeat,” and the instance will show as “Offline.”
2. IMDSv2 Migration Outages: Moving from IMDSv1 to IMDSv2 (which is more secure) can break legacy scripts or older applications that expect to fetch metadata without a session token.
3. Large Scale Auditing: In an environment with 1,000+ instances, searching through millions of CloudWatch lines to find one specific command can be like finding a needle in a haystack.
4. WebSocket Issues: CLI-based session manager connections sometimes fail if your company’s corporate firewall/proxy blocks WebSockets.
5. KMS Key Permissions: If you encrypt your SSM logs with a custom KMS key, but forget to give the EC2 IAM role kms:Decrypt permissions, the session will terminate instantly with a cryptic error.

9. EC2 Maintenance & Patching

Maintaining an EC2 instance is like servicing a car.

Patching is changing the oil and fixing security leaks. Instead of doing this manually for every car (instance),

SSM Patch Manager as an automated robot that services the whole fleet at midnight.

SSM Session Manager is like a secure remote control that lets you check the engine without needing a physical key (SSH key).

DevSecOps view, we want our servers to be “self-healing” and always up-to-date without humans touching them.

9.1. SSM Patch Manager

• Patch Baselines: A set of rules that define which patches are “approved” (e.g., “Install all security updates with a ‘Critical’ severity after 7 days of release”).
• Patch Groups: Using Tags (e.g., PatchGroup: Production) to associate instances with specific baselines.
• Maintenance Windows: Scheduled time slots where patching occurs to avoid impacting business hours.

DevSecOps Perspective:

• Vulnerability Management: Integrate Patch Manager with Amazon Inspector. If Inspector finds a vulnerability, Patch Manager can automatically trigger a “Scan and Install” to fix it.
• Compliance as Code: Use AWS Config to monitor if instances have the correct Patch Group tags. If an instance is missing a tag, it’s a security risk.
• Immutable Infrastructure: In a mature DevSecOps pipeline, instead of patching a running server (“In-place”), we patch the AMI (Golden Image) and redeploy the entire fleet.

Real-World Challenges

Working on EC2 maintenance at scale isn’t always smooth. Here are the common hurdles:

1. SSM Agent “Heartbeat” Issues
   • Sometimes an instance shows as “Connection Lost” or “Unmanaged.”
   • Why it happens: The SSM Agent crashed, the IAM role is missing AmazonSSMManagedInstanceCore, or the instance has no outbound path to the SSM endpoint (needs NAT Gateway or VPC Endpoint).
   • Fix: Use an Automation Runbook to automatically restart the agent or verify IAM roles upon launch.
2. The “Reboot” Nightmare
   • Many security patches (like Kernel updates) require a reboot to take effect.
   • Why it happens: If you have a single instance (not in an Auto Scaling Group), a reboot causes downtime.
   • Fix: Use Target Tracking and Health Checks. Only patch 10% of the fleet at a time so the application stays online.
3. Broken Dependencies
   • A patch might update a library that breaks your application (e.g., updating Python or Java version).
   • Why it happens: Lack of “Pre-prod” testing.
   • Fix: Always patch Dev/QA environments first. Use Patch Manager’s “Scan” mode to see what would happen before running “Install.”
4. Custom Repositories
   • Patch Manager is great for OS patches but struggles with custom software or third-party apps not in standard yum or apt repos.
   • Fix: Use SSM Distributor to package and deploy custom software updates alongside OS patches.

10. EC2 Boot Strap Options

If building an EC2 instance is like building a smart-home:

User Data is the One-Time Construction Crew. They install the foundation, set up the electricity, and put in the smart-locks once. When the house is built, they leave and never come back.

Cloud-Init is the Maintenance Operating System. It’s the “brain” of the house that reads your custom settings every time the power cycles—ensuring that if a window is accidentally left open (a security gap), it gets closed and locked.

10.1. DevSecOps Integration – Secure by Design

1. Automated Security Agent Deployment – In a DevSecOps pipeline, never want a “naked” instance on the network. Use User Data to pull and install your security stack immediately:
   • EDR/Antivirus: yum install -y crowdstrike-agent
   • Monitoring: curl -sSL https://install.datadoghq.com/ansible-installer.sh | bash
   • Vulnerability Scanning: Installing the AWS Inspector agent.
2. Hardening at Launch (Shift-Left) – Instead of waiting for a manual audit, use Cloud-Init to:
   • Disable Root Login: Set PermitRootLogin no in SSH config.
   • Update Everything: Force a yum update -y or security-only updates before the app starts.
   • Install CIS Benchmarks: Run scripts that align the OS with Center for Internet Security (CIS) standards.
3. Identity and Access (IAM Role instead of Keys)
   • Never pass AWS Access Keys through User Data.
   • DevSecOps Best Practice: Assign an IAM Instance Profile to the EC2. Then, use User Data to simply call the AWS CLI to fetch secrets from AWS Secrets Manager securely.

Real-World Challenges

1. The “Plaintext Secret” Leak
   • Anything you put in User Data can be seen by anyone with ec2:DescribeInstanceAttribute permissions or by running curl http://169.254.169.254/latest/user-data/ on the instance.
   • Solution: Never put passwords or API keys in User Data. Use IMDSv2 (Instance Metadata Service v2) to protect the metadata service and pull secrets from Secrets Manager at runtime.
2. “First Boot Only” Limitation
   • By default, User Data doesn’t run again if the instance restarts. If a security patch is released later, User Data won’t help.
   • Solution: Use MIME multi-part scripts to tell Cloud-Init to run specific security checks every time the instance boots, not just the first time.
3. “Silent Failure” Debugging
   • If your User Data script has a typo, the instance might look “Running” but the security agents aren’t actually installed.
   • Solution: Always redirect your script output to a log file: #!/bin/bash -ex. Check /var/log/cloud-init-output.log to audit the bootstrap success.
4. Large Script Size
   • User Data is limited to 16 KB.
   • Solution: Don’t write a 1,000-line script. Instead, host your main configuration script in an S3 bucket. Use User Data to download and run that one script: aws s3 cp s3://my-security-bucket/harden.sh . && chmod +x harden.sh && ./harden.sh.

11. Auto Scaling Groups (ASG) and Launch Templates

Think of an Auto Scaling Group (ASG) as a Smart Manager. If your shop gets crowded (high CPU/traffic), the manager hires more staff (scales up). if the shop is empty, the manager lets staff go to save money (scales down).

The Launch Template is the Instruction Manual the manager uses to hire new staff it says exactly what they allowed (Security Groups) and what skills and capacity they must have (AMI/Instance Type).

11.2. Launch Templates (The Successor to Launch Configurations)

It’s an instance configuration template for the auto-scaling group, AMI, Instance Type, Key pair and Security group.

• Versioning: You can have “v1” for Production and “v2” for Testing.
• Metadata Options: Supports IMDSv2, which is mandatory for modern security to prevent credential theft.
• Spot + On-Demand: Allows you to define how many “cheap” (Spot) vs “reliable” (On-Demand) instances you want.

DevSecOps Integration: “Self-Healing & Compliance”

1. Immutable Infrastructure – In DevSecOps, we never patch a running ASG.
   • Process: Update your Launch Template with a new “Golden AMI” (patched version).
   • Action: Trigger an Instance Refresh. ASG will replace old instances with new ones automatically.
   • Result: No “Configuration Drift” (where different servers have different settings).
2. Automated Vulnerability Response – integrate Amazon Inspector with ASG. If Inspector finds a critical bug:
   • It sends an alert to EventBridge.
   • EventBridge triggers a Lambda.
   • The Lambda updates the Launch Template and forces the ASG to rotate the “infected” instances.
3. Enforcing IMDSv2
   • Launch Configurations (Legacy) allowed the older, less secure IMDSv1.
   • Launch Templates allow you to strictly enforce IMDSv2 (Session-based). This prevents attackers from stealing the instance’s IAM role via a Server-Side Request Forgery (SSRF) attack.

Real-World Challenges

1. “The Scaling Flap” (Simple Scaling Issue):
   • Simple scaling can lead to “flapping” adding an instance, then immediately deleting it, then adding it again.
   • Fix: Use Target Tracking Scaling or Predictive Scaling. It uses machine learning to look at your traffic history and prepare capacity before the spike happens.
2. “Spot Interruption” (Availability Risk):
   • Spot instances can be taken back by AWS with only a 2-minute warning.
   • Fix: Use Capacity Rebalancing. ASG will proactively launch a new instance as soon as it gets a “rebalance recommendation” from AWS, rather than waiting for the 2-minute “death notice.”
3. “Zombie Instances” (Termination Policy Error):
   • If you set your termination policy is “Newest Instance,” the ASG might accidentally kill a server that just started and is still doing important setup work.
   • Fix: Use Custom Termination Policies (Lambda-based) or stick to “Default,” which balances instances across Availability Zones first to ensure high availability.
4. “Launch Configuration Migration”:
   • Many older companies still use Launch Configurations, which don’t support new instance types (like Mac or Graviton).
   • Fix: AWS has a “Copy to Launch Template” button. As a DevSecOps pro, you should automate this migration to ensure your fleet can use the latest security features.

11.3. Placement Strategy

12. EC2 Monitoring

Think of EC2 monitoring like a Smart Watch for your server. It tracks the “heartbeat” (CPU), “blood pressure” (Network), and “memory” (RAM).

• Standard Monitoring is like a free check-up every 5 minutes.
• Detailed Monitoring is like a premium 1-minute heart-rate monitor.
• Status Checks tell you if the “hardware” (AWS) or the “person” (your OS) has a problem.
• DevSecOps view, we add a “Security Alarm” to this watch. If the server starts talking to a suspicious country or its memory usage spikes suddenly (possible crypto-jacking), the alarm goes off and fixes the issue automatically.

12.1. Metrics and Monitoring Types

12.2. Status Checks: Who is at fault?

You can set a CloudWatch Auto-Recovery alarm on System Status checks. If the physical hardware fails, AWS will automatically move your instance to a new healthy host without changing your Private IP or Volume data.

DevSecOps Perspective: “Monitor, Detect, Respond”

1. Shift-Left Observability
   • In a DevSecOps pipeline, monitoring configuration is Infrastructure as Code (IaC). We don’t manually create alarms. We define them in Terraform/CloudFormation so that every new instance is born with a “Security Guard” (Alarm) attached.
2. IMDSv2: The Metadata Shield – Instance Metadata (IMDS) is a goldmine for attackers.
   • IMDSv1 (Old): Vulnerable to SSRF attacks. A hacker could trick your app into leaking your IAM Role credentials.
   • IMDSv2 (Secure): Requires a Session Token.
   • DevSecOps Best Practice: Disable IMDSv1 account-wide. Use the “Hop Limit” of 1 to prevent credentials from leaving the instance.
3. Automated Incident Response (The “Self-Healing” Loop)
   • Instead of a human getting a 2 AM call:
   • Event: CloudWatch detects a CPU spike > 90% (Potential DoS).
   • Action: CloudWatch triggers an AWS Lambda function.
   • Response: Lambda inspects the traffic, blocks the attacking IP in the NACL, and scales the ASG.

Real-World Challenges

1. The “CloudWatch Bill” Shock
   • Detailed 1-minute monitoring and high-resolution custom metrics are expensive. If you monitor 1,000 instances with 10 custom metrics each, your bill can skyrocket.
   • Fix: Use Metric Streams to send data to cheaper third-party tools or only enable detailed monitoring on “Production” tagged instances.
2. The “SSM Agent” Heartbeat
   • The SSM/CloudWatch agent must be running inside the OS. If the OS crashes or the agent hangs, you lose all RAM and Disk Space visibility.
   • Fix: Use a “Dead Man’s Switch” alarm—if CPUUtilization is 0% or metrics stop arriving, trigger a reboot.
3. Alert Fatigue
   • Too many alarms lead to “ignoring the noise.” If 50 alarms go off for one minor issue, the team misses the real problem.
   • Fix: Use CloudWatch Anomaly Detection. Instead of a fixed “90%” threshold, it uses Machine Learning to learn what is “normal” for your specific app.
4. IMDSv2 Migration
   • Older SDKs or legacy code might break if you force-toggle to IMDSv2 only.
   • Fix: Use the MetadataNoToken CloudWatch metric to identify which applications are still using the old, insecure IMDSv1 before you disable it.

13. Pricing Models & Tenancy

Think of EC2 pricing like Travel Options:

• On-Demand is like a Standard Taxi: Pay as you go, flexible, but expensive.
• Spot is like Standby Tickets: Extremely cheap, but you might be kicked off if someone else pays full price.
• Savings Plans/RI are like a Monthly Pass: You commit to the service and get a massive discount.
• Dedicated Hosts are like Owning the Whole Bus: Only your people are inside, which is great for strict company rules (compliance).
• Placement Groups are about Seating Arrangements: Do you want everyone sitting close for fast talking (Cluster), or spread out so one accident doesn’t hurt everyone (Spread)?

13.1 Pricing Models

13.2 Tenancy Options: “Physical Security”

In DevSecOps, Tenancy is about the “Isolation” pillar.

• Shared (Default): Multi-tenant. Safe, but theoretically vulnerable to “Side-Channel Attacks” (like Spectre/Meltdown), though AWS mitigates this.
• Dedicated Instance: You are the only one on that hardware, but AWS manages the physical placement.
• Dedicated Host: You have full visibility.
• DevSecOps Use Case: Required for apps that need “Host Affinity” or strict regulatory requirements where you must prove your data isn’t physically next to a competitor’s.

14. AWS Hypervisor Types (Xen & Nitro)

A hypervisor is software that creates and manages EC2 virtual servers by sharing hardware resources like CPU, memory, and storage.

Think of the Xen Hypervisor like a shared apartment building where a Security Guard lives in one of the rooms.

The Nitro System is like a high-tech vault where there is no security guard living inside. Instead, all the “heavy lifting” like mail and locks is handled by hidden machinery (Nitro Cards) behind the walls. This makes the building faster (no guard taking up space) and much harder to rob because there is no human manager to target.

Hypervisor = A manager that controls virtual servers in AWS.

AWS mainly uses two hypervisor technologies:

DevSecOps Perspective: Why Nitro is a Security Game-Changer

1. Attack Surface Reduction
   • In Xen, the Dom0 is a general-purpose Linux OS with a shell, drivers, and a network stack. This is a massive attack surface.
   • Nitro’s DevSecOps Advantage: The Nitro Hypervisor has no shell, no networking stack, and no interactive access. It is a tiny, read-only firmware-like component (~50k lines of code). If a hacker breaks out of a VM, there is “nothing to talk to” on the host.
2. Nitro Security Chip (Hardware Root of Trust)
   • Continuous Monitoring: The chip constantly validates the system firmware. If the motherboard firmware is tampered with, the chip prevents the system from booting.
   • Hardware-Enforced Isolation: Security isn’t just a software rule; it’s physically wired into the silicon.
3. Confidential Computing (Nitro Enclaves)
   • Zero-Trust Data Processing: Nitro Enclaves allow you to create an isolated “black box” within your EC2 instance.
   • DevSecOps Use Case: You can process highly sensitive data (like credit card info or private keys) in an environment where even the root user of the main EC2 instance cannot see what’s happening inside the Enclave.
4. Transparent VPC Encryption
   • In-Transit Security: On Nitro-based instances, traffic between instances is automatically encrypted at the hardware level with no performance penalty. This fulfills the “Encryption in Transit” compliance requirement without manual setup.

Real-World Challenges

1. The “Driver” Gap (Xen to Nitro Migration)
   • Xen instances use “Para-virtual” (PV) drivers. Nitro requires NVMe (for storage) and ENA (for networking) drivers.
   • Fix: Before moving an old AMI to a Nitro instance (like C5 or M5), you must install these drivers. If you don’t, the instance will fail to boot and you’ll get a “1/2 Status Check” error.
2. OS Compatibility
   • Very old Linux kernels (pre-2012) or older Windows versions (2003/2008) simply don’t have the “brains” to talk to Nitro hardware.
   • You may be forced to stick with older “T2” or “M3” (Xen-based) instances for legacy apps, which are more expensive and less secure.
3. “Noisy Neighbor” Variance
   • While Nitro reduces the “noisy neighbor” effect, the extreme performance of Nitro can sometimes expose bugs in your code that were “hidden” by the slower, jittery performance of Xen.
   • Fix: Always re-run your performance and stress tests during a migration to ensure your application logic handles the high-speed Nitro I/O correctly.

EC2 common issues and Troubleshooting

EC2 Rescue Mode used to fix the issues

Best Method for OS-level issues:

1. Attach back and start
2. Stop instance
3. Detach disk
4. Attach to rescue EC2
5. Fix errors

EC2 Not Starting / Stuck in Pending

Cause: –

• Availability Zone has no capacity
• Instance stuck on bad hardware
• Corrupted EBS Volume

Fix: –

• Try launching in another Availability Zone
• Stop → Start the instance it moves EC2 to a new hardware host.

Stuck on Boot / Kernel Panic / GRUB Error

Cause

• Wrong OS update
• Corrupted bootloader
• Wrong /etc/fstab
• Missing drivers

Fix (Rescue Method)

1. Stop instance
2. Detach root volume
3. Attach to a rescue EC2
4. Mount volume and fix files:

Fix GRUB / Kernel

sudo grub2-install /dev/xvda
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Fix wrong fstab edit /etc/fstab and fix the entries.

Cannot SSH into EC2 (Linux)

Typical errors:-

• Connection timed out
• Permission denied (publickey)

Fix:-

• Check the username and ipaddress.
• Update Security group inbound rule allow port 22 and NACL if required.
• Update Key pair permissions to 400.
• Lastly sshd service might be issue.

Cannot RDP into EC2 Windows

Cause: –

• Windows firewall blocking RDP
• Port 3389 blocked
• Wrong password

Fix:-

• Check the username and ipaddress.
• Update Security group inbound rule allow port 3389 and NACL if required.
• lastly Use EC2Rescue for Windows to fix firewall rules

High CPU or Memory Usage

Cause: –

• Small instance size
• Application leak
• Malware / runaway process

Fix: –

• Check the instance size and increase.
• Check for application log.
• Patch the OS and check for vulnerabilities

EBS Volume / Disk Full

Symptom: –

• OS freeze
• SSH not working

Fix: –

• Modify EBS size in console
• Reboot instance
• Expand filesystem

Network Connectivity Issues

Cause: –

• Wrong route table
• Missing IGW/NAT
• Wrong SG/NACL
• Bad VPC peering config

Fix: – Check and update the network configurations.

Status Checks Failed (2/2)

1. System Status Check Failed (AWS issue)
   • Hardware failure or Network failure
   • Fix:- Stop → Start instance (moves to new hardware)
2. Instance Status Check Failed (OS issue)
   • Low memory, Kernel panic, Corrupted root disk
   • Fix: – Reboot, Check logs,

EC2 Cannot Reach Internet

Cause: –

• Missing Internet Gateway
• Wrong Subnet
• Public IP not assigned

Fix: – Check VPC configuration, IGW, Route table. and Public IP assigned

Slow Performance (CPU Credits Issue)

Cause: – T2/T3 instance burst credits exhausted.

Fix:- Change instance to Non credit based instance or

Enable unlimited mode:

aws ec2 modify-instance-credit-specification \
--instance-credit-specifications InstanceId=i-123,CpuCredits=unlimited

“Insufficient Capacity” Error

Cause: – AWS does not have that instance type in the selected AZ.

Fix:- Select different AZ