Fix Kubernetes Persistent Volume Mount Failure: Step-by-Step Guide

1. Introduction

A persistent volume mount failure leaves your pod stuck in ContainerCreating indefinitely, with the kubelet unable to attach or mount the storage volume the pod needs. The pod is scheduled, the PVC is bound, but the actual volume can't be made available on the node — and without it, the container won't start.

Mount failures are some of the most infrastructure-specific Kubernetes problems. The root cause depends entirely on the storage backend: AWS EBS, GCE PD, NFS, Ceph, local volumes, and CSI drivers each have their own failure modes. This guide covers the diagnostic approach that works across all backends, followed by fixes for the most common specific scenarios.

2. What a Mount Failure Means

Volume mounting in Kubernetes happens in two phases: attach (the volume is attached to the node at the block device or network level) and mount (the volume is mounted into the container's filesystem). Either phase can fail independently. A failure in the attach phase leaves the pod waiting on the node; a failure in the mount phase means the device is present but can't be formatted or mounted.

3. Common Causes

• Multi-attach error — block volume already attached to another node
• Availability zone mismatch — EBS volume in us-east-1a, pod scheduled to us-east-1b
• CSI driver not installed or version incompatible with the storage class
• NFS server unreachable or export permissions changed
• PVC is in Pending state — see Fix Kubernetes PVC Pending
• Node doesn't have the required filesystem tools (xfs_repair, e2fsck) for the volume type
• Volume is marked as needing fsck repair (unclean shutdown)
• Access mode mismatch — ReadWriteOnce volume used by multiple pods on different nodes

4. Step-by-Step Diagnosis and Fix

Step 1: Read the pod events

# The error is almost always in the Events section
kubectl describe pod <pod-name> -n <namespace>

# Common mount failure messages:
# "Multi-Attach error for volume ... Volume is already used by pod(s)"
# "Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data]"
# "timed out waiting for the condition"
# "failed to sync <volume>: rpc error: ... No such file or directory"
# "MountVolume.SetUp failed for volume ... (already in use)"

Step 2: Fix multi-attach errors

"># Find which pod is holding the volume
kubectl get volumeattachments
kubectl describe volumeattachment <attachment-name>

# Find the old pod holding the attachment
PV_NAME=$(kubectl get pvc <pvc-name> -n <namespace> -o jsonpath='{.spec.volumeName}')
kubectl get pod -A -o json | jq -r   '.items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == "<pvc-name>") | .metadata.name'

# Force-delete the old pod (if it's stuck)
kubectl delete pod <old-pod> -n <namespace> --force --grace-period=0

# If the VolumeAttachment is stuck, delete it manually
kubectl delete volumeattachment <attachment-name>

Step 3: Fix availability zone mismatches

# Check which AZ the PV/volume is in
kubectl get pv <pv-name> -o jsonpath='{.metadata.labels}'
# Look for: topology.kubernetes.io/zone

# Check which AZ the pod scheduled to
kubectl get pod <pod-name> -n <namespace> -o wide
kubectl get node <node-name> -o jsonpath='{.metadata.labels.topology\.kubernetes\.io/zone}'

# Fix: add node affinity to the pod to match the volume's zone
# OR: use a StorageClass with volumeBindingMode: WaitForFirstConsumer
# WaitForFirstConsumer creates the volume in the same zone as the pod

kubectl get storageclass
# Preferred StorageClass config:
# volumeBindingMode: WaitForFirstConsumer

Step 4: Check and fix CSI driver issues

# Check CSI driver pods
kubectl get pods -n kube-system | grep csi

# Check CSI node pods on the affected node
kubectl get pods -n kube-system   --field-selector spec.nodeName=<node-name> | grep csi

# Check CSI driver logs
kubectl logs -n kube-system <csi-node-pod> -c <container-name> --tail=50

# For AWS EBS CSI driver:
kubectl get pods -n kube-system -l app=ebs-csi-node
kubectl logs -n kube-system -l app=ebs-csi-node -c ebs-plugin --tail=50

Step 5: Fix NFS mount failures

# Test NFS connectivity from the node
kubectl debug node/<node-name> -it --image=ubuntu -- bash
mount -t nfs <nfs-server>:<export-path> /mnt/test

# Common NFS issues:
# 1. Server unreachable — check security group / firewall rules for port 2049
# 2. Export not found — check /etc/exports on NFS server
# 3. Permission denied — check NFS export options (no_root_squash vs root_squash)

# Verify NFS packages on node
apt-get install -y nfs-common   # Debian/Ubuntu
yum install -y nfs-utils        # RHEL/Amazon Linux

5. Verification Steps

# Pod should transition from ContainerCreating to Running
kubectl get pod <pod-name> -n <namespace> -w

# Verify the volume is mounted inside the container
kubectl exec -it <pod-name> -n <namespace> -- df -h
# Should show the mounted volume at its expected mountPath

# Check VolumeAttachment is healthy
kubectl get volumeattachments | grep <pv-name>

6. Common Mistakes

• Force-deleting the old pod without deleting the VolumeAttachment — the attachment can remain even after the pod is gone
• Using ReadWriteOnce volumes with Deployments that have more than 1 replica — only one pod can mount at a time
• Using volumeBindingMode: Immediate in a multi-AZ cluster — volumes get created before pod scheduling, often in the wrong zone
• Not checking if the CSI driver is deployed before creating PVCs with that StorageClass
• Assuming a volume will be immediately available after force-deleting the old attachment — block volumes need 1-6 minutes to detach in AWS

7. Prevention Tips

• Always use volumeBindingMode: WaitForFirstConsumer in multi-AZ clusters — prevents zone mismatches
• Use StatefulSets instead of Deployments for stateful workloads — StatefulSets manage PVC lifecycle correctly
• Design stateless pods where possible — stateless pods can reschedule instantly without volume constraints
• Set terminationGracePeriodSeconds appropriately so pods have time to flush writes and detach gracefully
• Monitor volume attachment counts — AWS EBS has per-instance limits on the number of attached volumes

8. FAQ

9. Summary