Kubernetes PV & PVC

Permanent Storage in K8s

Kubernetes Pods are “ephemeral” (temporary). When a Pod dies, any data saved inside it dies too. This is bad for databases like MySQL or MongoDB where you need to keep data forever.

To solve this, Kubernetes uses two things:

1. PV (Persistent Volume): This is the actual piece of storage (like a hard drive, an AWS EBS volume, or an NFS server). It is a cluster-wide resource.
2. PVC (Persistent Volume Claim): This is a request for storage. A developer writes a PVC to say, “I need 10GB of storage.”

The Administrator sets up the hard drives (PV), and the Developer asks for a slice of that hard drive (PVC). Kubernetes connects them automatically.

Key Characteristics to Remember

• “PV is the physical disk; PVC is the ticket to use it.”
• “Pods eat PVCs; PVCs eat PVs.” (Pods use PVC as a volume; PVC binds to the actual PV).
• “PVs are global; PVCs are local to a namespace.”

Feature	Persistent Volume (PV)	Persistent Volume Claim (PVC)
Who creates it?	Cluster Admin (or dynamic provisioner)	Developer / DevOps Engineer
Scope	Cluster-wide (Not inside a Namespace)	Namespaced (Lives inside a specific Namespace)
Role	The actual storage resource (Supply)	The request for storage (Demand)
Lifecycle	Independent of the Pod	Dependent on the user’s need
Connection	Binds to a PVC	Binds to a PV

The Workflow:

1. Provisioning: storage is created.
   • Static: Admin manually creates a PV manifest pointing to real storage (like an NFS path).
   • Dynamic: Admin creates a StorageClass. When a user creates a PVC, the StorageClass talks to the cloud provider (AWS/Azure/GCP) and creates the PV automatically.
2. Binding: The control plane watches for new PVCs. If it finds a PV that matches the PVC’s request (size and access mode), it “binds” them together. It is a 1-to-1 mapping.
3. Using: The Pod uses the PVC as a volume. The cluster looks up the bound PV and mounts the physical storage into the Pod.
4. Reclaiming: When the user deletes the PVC, the PV is handled based on its ReclaimPolicy (Retain, Delete, or Recycle).

• Access Modes:
  • RWO (ReadWriteOnce): Only one node can write to it (like a standard hard drive).
  • ROX (ReadOnlyMany): Many nodes can read, but nobody can write.
  • RWX (ReadWriteMany): Many nodes can read and write at the same time (like NFS or Cloud Filestore).
• Note: You cannot just choose any mode. The underlying physical storage must support it. (e.g., AWS EBS generally does not support RWX; you need EFS for that).

Technical Challenges, Limitations, Common Issues

Issue	Cause	Solution
PVC Stuck in “Pending”	No PV matches the claim (wrong size, wrong access mode, or no dynamic provisioner).	Check kubectl describe pvc <name> -> Events. Ensure StorageClass exists.
“Volume node affinity conflict”	PV exists in Zone A, but Pod is scheduled on a Node in Zone B.	Use WaitForFirstConsumer in StorageClass.
PVC Stuck in “Terminating”	A Pod is still using the PVC (Finalizers blocking deletion).	Delete the Pod first. Or force patch the finalizer (dangerous).
ReadWriteMany Error	Trying to mount an EBS/AzureDisk on two nodes.	Block storage (RWO) cannot do RWX. Switch to NFS/EFS/FileStore.
Size Mismatch	PVC requests 10Gi, PV is 5Gi.	PV must be greater than or equal to PVC request.

https://kubernetes.io/docs/concepts/storage/persistent-volumes

• Static vs. Dynamic Provisioning:
  • Static: You manually create a PV object with specific details (IP, path, size). Then you create a PVC to bind to it. (Old school, hard to manage).
  • Dynamic: You create a StorageClass. When you create a PVC, the provisioner automatically creates the PV for you. (Modern standard).
• Access Modes:
  1. ReadWriteOnce (RWO): Mounted as read-write by a single node. (Common for Block Storage like AWS EBS, Azure Disk).
  2. ReadOnlyMany (ROX): Mounted read-only by many nodes.
  3. ReadWriteMany (RWX): Mounted read-write by many nodes. (Requires file storage like NFS, AWS EFS, or GlusterFS).
  4. ReadWriteOncePod (RWOP): Strict restriction—mounted by only one Pod (K8s v1.22+).
• Reclaim Policies:
  • Retain: When PVC is deleted, PV becomes “Released” but data remains. Admin must manually clean up.
  • Delete: When PVC is deleted, the backend storage (e.g., AWS EBS volume) is also deleted. (Default for dynamic provisioning).
  • Recycle: (Deprecated) Performs a basic scrub (rm -rf) and makes it available again.

---