Block & File Storage: EBS Volumes, Snapshots & Amazon EFS Deep Dive
Master AWS block and file storage: Amazon EBS volume types (gp3 vs io2), crash-consistent snapshots, Fast Snapshot Restore, and Amazon EFS multi-AZ POSIX mounts.
Block & File Storage: EBS Volumes, Snapshots & Amazon EFS Deep Dive
Section titled βBlock & File Storage: EBS Volumes, Snapshots & Amazon EFS Deep DiveβCompute instances without durable storage are strictly ephemeral. When an EC2 instance is terminated or suffers a physical host crash, data residing on ephemeral instance store disks disappears forever.
To achieve persistent, durable, and highly available architectures, AWS provides two foundational storage pillars:
- Amazon Elastic Block Store (EBS): High-performance raw block-level storage volumes designed for single-instance attachment.
- Amazon Elastic File System (EFS): Managed, elastic, multi-AZ shared file storage adhering to the POSIX standard for concurrent access across thousands of compute nodes.
1. Amazon EBS Volume Architecture & Performance Tuning
Section titled β1. Amazon EBS Volume Architecture & Performance TuningβAn EBS volume behaves like a physical hard drive connected over the high-speed AWS data center network fabric. Choosing the right volume family dictates your IOPS, throughput, and monthly AWS storage bill.
βββββββββββββββββββββββββββββββββββββββββββ β Amazon EBS Volume Types β ββββββββββββββββββββββ¬βββββββββββββββββββββ β βββββββββββββββββββββββ΄ββββββββββββββββββββββ βΌ βΌ [ Solid-State Drives (SSD) ] [ Hard Disk Drives (HDD) ] β’ gp3: 3,000 IOPS & 125 MB/s Baseline β’ st1: Throughput Optimized β’ gp2: Legacy baseline tied to disk size β’ sc1: Cold HDD Archive β’ io2 Block Express: Up to 256,000 IOPSComplete EBS Volume Family Comparison:
Section titled βComplete EBS Volume Family Comparison:β| Volume Type | Max IOPS | Max Throughput | Latency | IOPS:GiB Ratio | Ideal Production Workload |
|---|---|---|---|---|---|
| General Purpose (gp3) | Up to 16,000 | 1,000 MB/s | Single-digit ms | Up to 500:1 | Default Production Standard: OS boot disks, dev/staging, web backends |
| General Purpose (gp2) | Up to 16,000 | 250 MB/s | Single-digit ms | 3:1 (Tied to size) | Legacy: Requires over-provisioning storage just to get IOPS (Avoid!) |
| Provisioned IOPS (io2) | Up to 256,000 | 4,000 MB/s | Sub-millisecond | Up to 1,000:1 | Mission-critical databases: Oracle, SAP HANA, SQL Server |
| Throughput Optimized (st1) | 500 | 500 MB/s | Milliseconds | N/A (Throughput) | Sequential I/O, Apache Kafka logs, Big Data EMR clusters |
| Cold HDD (sc1) | 250 | 250 MB/s | Milliseconds | N/A (Throughput) | Infrequently accessed file servers, low-cost long-term storage |
[!TIP] FinOps Optimization: The gp2 to gp3 Migration: Migrating from legacy
gp2togp3delivers an immediate 20% cost reduction per GB. Furthermore, gp3 gives you 3,000 IOPS and 125 MB/s baseline throughput for free, completely decoupled from disk size! You can modify volume types live in the AWS console or via CLI without rebooting or unmounting the filesystem.
2. EBS Snapshots, Crash Consistency & Fast Snapshot Restore (FSR)
Section titled β2. EBS Snapshots, Crash Consistency & Fast Snapshot Restore (FSR)βEBS Snapshots are point-in-time backups preserved redundantly across multiple Availability Zones inside Amazon S3:
Initial Snapshot (Snapshot 1) ββ> [ Stores 100 GB Data Blocks in S3 ]Change 5 GB Data Blocks...Second Snapshot (Snapshot 2) ββ> [ Stores ONLY the 5 GB Changed Blocks (Incremental) ]Delete Snapshot 1?S3 Retains Common Blocks ββ> [ Zero data loss; unique blocks are garbage-collected ]The 3 Rules of EBS Snapshot Integrity:
Section titled βThe 3 Rules of EBS Snapshot Integrity:β- Incremental Backup Mechanics: You only pay for changed blocks. Deleting an earlier snapshot does not break subsequent snapshots; AWS automatically consolidates required data blocks in S3.
- Crash Consistency vs. Application Consistency:
- Taking a snapshot while an active database (MySQL/PostgreSQL) is writing dirty pages in memory results in a Crash-Consistent snapshot (equivalent to pulling the power plug).
- For Application Consistency, freeze I/O operations (
xfs_freeze -f /dataor database flush commands) before callingCreateSnapshot.
- Fast Snapshot Restore (FSR):
- Restoring a regular EBS snapshot results in βlazy loadingβ (the first time a block is accessed, it must be pulled over the network from S3, causing I/O latency).
- FSR pre-initializes and warms all data blocks across target AZs, delivering maximum provisioned IOPS the instant the volume is mounted!
3. Amazon EFS: Elastic Multi-AZ POSIX Shared Storage
Section titled β3. Amazon EFS: Elastic Multi-AZ POSIX Shared StorageβWhen microservices in different subnets or Kubernetes pods managed by AWS EKS need simultaneous read-write access to a shared directory, Amazon EFS provides a fully managed, elastic NFSv4.1 file system.
[ Availability Zone A ] [ Availability Zone B ] [ Availability Zone C ] EC2 / EKS Pod-1 EC2 / EKS Pod-2 EC2 / EKS Pod-3 β β β βΌ βΌ βΌ [ EFS Mount Target A ] [ EFS Mount Target B ] [ EFS Mount Target C ] (10.100.11.50) (10.100.12.50) (10.100.13.50) β β β βββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββ βΌ βββββββββββββββββββββββββββββββββββββββββ β Amazon EFS Shared File System β β β’ Petabyte-scale auto-growth β β β’ POSIX permissions (chmod, chown) β β β’ Multi-AZ 11 9's durability β βββββββββββββββββββββββββββββββββββββββββEFS Performance Modes & Storage Classes:
Section titled βEFS Performance Modes & Storage Classes:β- General Purpose Performance: Default mode optimized for web serving, content management (WordPress), and CI/CD tools. Lowest latency.
- Max I/O Performance: Scales to thousands of concurrent compute nodes with higher aggregate throughput, but slightly higher per-operation latency (ideal for Big Data).
- EFS Lifecycle Management: Automatically moves files not read in 14 to 90 days to the EFS Infrequent Access (EFS-IA) tier, reducing storage costs by 92%!
4. Real-Time Production Triage Runbook
Section titled β4. Real-Time Production Triage RunbookβIncident: EBS Volume Full & System Hanging
Section titled βIncident: EBS Volume Full & System HangingβSymptom: SSH connections fail with Write failed: Broken pipe or application crashes with No space left on device.
Production Remediation Protocol (Zero Downtime Elastic Volume Expansion):
- Increase the EBS volume size via AWS CLI without stopping the instance:
Terminal window aws ec2 modify-volume --volume-id vol-0a1b2c3d4e5f67890 --size 100 - Verify the partition table using SSH or SSM Session Manager:
Terminal window lsblk - Expand the underlying partition:
Terminal window sudo growpart /dev/xvda 1 - Expand the filesystem to occupy the newly available space:
Terminal window # If XFS filesystem:sudo xfs_growfs -d /# If ext4 filesystem:sudo resize2fs /dev/xvda1 - Confirm space is available:
df -hT /.
π§ Architect Knowledge Check
Section titled βπ§ Architect Knowledge Checkβ1. Can an Amazon EBS volume be attached to two EC2 instances simultaneously?
Answer: Yes, but only if EBS Multi-Attach is supported and enabled. EBS Multi-Attach is available strictly on io1 and io2 Provisioned IOPS volumes within the same Availability Zone (up to 16 Nitro instances). Crucially, the operating system must use a cluster-aware filesystem (such as GFS2 or OCFS2) to coordinate concurrent writes; attaching a standard ext4 or XFS volume to multiple instances will cause instant filesystem corruption.
2. What is the fundamental difference in Availability between Amazon EBS and Amazon EFS?
Answer:
- Amazon EBS is Single-AZ Scoped: A volume created in
us-east-1acan only be attached to EC2 instances inus-east-1a. If AZ-a fails, the EBS volume is temporarily unreachable.- Amazon EFS is Multi-AZ Scoped: It stores data redundantly across 3 Availability Zones, providing Mount Targets in each AZ. If one AZ suffers an outage, applications in surviving AZs continue reading and writing to EFS with zero downtime.