Fix Kubernetes Pod Evicted

Q: How do I tell if eviction was caused by memory or disk?

Read the eviction message: kubectl describe pod . The Message field will say exactly which resource triggered eviction and what the threshold and available values were at the time.

Q: Can I prevent eviction for specific pods?

Set them as Guaranteed QoS class (requests equal limits for all resources) and use a high-value PriorityClass. Guaranteed pods are the last to be evicted. Note: you cannot fully prevent eviction — if the node literally runs out of memory and all pods are Guaranteed, the OOM killer will eventually kill processes regardless.

1. Introduction

A Kubernetes pod with STATUS: Evicted was forcibly removed from a node by the kubelet. Unlike a crash or a probe failure, eviction is a deliberate act — the node was under pressure (running out of memory, disk, or inodes) and Kubernetes needed to free resources. The pod didn't fail; it was displaced.

Evicted pods stay in the API server in a terminal state and never restart on their own. The workload controller (Deployment, StatefulSet, DaemonSet) creates a replacement pod, but if the underlying pressure isn't resolved, the new pod will be evicted too — creating a cycle of evictions that can prevent workloads from ever stabilising.

2. What Pod Eviction Means

The kubelet monitors node resources and evicts pods when configurable thresholds are crossed. Eviction can be soft (grace period given) or hard (immediate, no grace period). The kubelet evicts pods in order of Priority Class, then by resource consumption relative to requests.

Common eviction signals: memory.available, nodefs.available, nodefs.inodesFree, imagefs.available. When any of these drops below the configured eviction threshold, the kubelet begins evicting pods starting with Best Effort pods (no requests/limits), then Burstable, then Guaranteed.

3. Common Causes

Node is out of memory — workloads exceeded node capacity
Node disk is full — logs, container layers, or ephemeral storage consuming all space
Pod has no resources.requests set — classified as Best Effort, first to be evicted
Pod has no resources.limits and is consuming far more memory than other pods
Ephemeral storage limit on the pod exceeded
Too many pods scheduled to one node — over-commitment
A memory leak in the application is growing node memory usage over time
ResourceQuota pressure causing failed scheduling elsewhere, concentrating workloads on specific nodes

4. Step-by-Step Diagnosis and Fix

Step 1: Confirm eviction and read the reason

# Find evicted pods
kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted

# Read the eviction reason
kubectl describe pod <evicted-pod> -n <namespace>
# Look for:
# Status:       Failed
# Reason:       Evicted
# Message:      The node was low on resource: memory.
#               Threshold quantity: 100Mi, available: 42Mi

# Delete evicted pods (they don't restart and waste API resources)
kubectl delete pod -A --field-selector=status.phase=Failed

Step 2: Check node resource pressure

# Check node conditions
kubectl describe node <node-name> | grep -A 5 "Conditions:"
# Look for: MemoryPressure, DiskPressure, PIDPressure — True = eviction active

# Check actual node resource usage
kubectl top nodes

# Check kubelet eviction thresholds
kubectl get configmap kubelet-config -n kube-system -o yaml | grep -A 20 eviction

Step 3: Find what's consuming node resources

# Find memory-hungry pods on the node
kubectl top pods -A --sort-by=memory | head -20

# Find pods with no resource requests (Best Effort = first evicted)
kubectl get pods -A -o json | jq -r '
  .items[] |
  select(.spec.containers[].resources.requests == null) |
  "\(.metadata.namespace)/\(.metadata.name)"'

# Check disk usage on the node (requires SSH or exec into debug pod)
kubectl debug node/<node-name> -it --image=ubuntu -- df -h
kubectl debug node/<node-name> -it --image=ubuntu -- du -sh /var/log/pods/*

Step 4: Fix — Set resource requests and limits

Pods without requests are classified as Best Effort and are the first to be evicted. Setting requests also affects scheduling — the scheduler only places pods on nodes that can accommodate them. If memory limits are too generous and a pod has a leak, see Fix Kubernetes OOMKilled for right-sizing guidance.

spec:
  containers:
  - name: my-app
    resources:
      requests:
        memory: "256Mi"
        cpu: "100m"
      limits:
        memory: "512Mi"
        cpu: "500m"

Step 5: Fix — Clean up disk space

# If eviction is due to disk pressure, clean up Docker/container images:
# SSH to the node or use kubectl debug
docker system prune -af      # removes unused images and containers
# For containerd nodes:
crictl rmi --prune

# Clean up old logs
journalctl --vacuum-size=500M
truncate -s 0 /var/log/containers/*.log 2>/dev/null || true

# In Kubernetes: enable log rotation in kubelet config
# containerLogMaxSize: "50Mi"
# containerLogMaxFiles: 5

Step 6: Fix — Set Priority Classes to protect critical pods

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: "Critical application pods — evict last"
---
spec:
  template:
    spec:
      priorityClassName: high-priority

5. Verification Steps

# Check node conditions are back to normal
kubectl get nodes
# All nodes should show Ready, no MemoryPressure or DiskPressure

# Verify no new evictions
kubectl get pods -A --field-selector=status.phase=Failed | grep Evicted

# Confirm replacement pods are running
kubectl get pods -n <namespace> -l app=<your-app>
# All should show Running 1/1

6. Common Mistakes

Deleting the evicted pod without fixing the underlying resource pressure — the new pod will also be evicted
Setting very high memory limits without requests — pods look cheap to schedule but can consume far more than the node has
Not monitoring eviction events — evictions can be silent and only discovered during incidents
Ignoring DiskPressure — it evicts just like MemoryPressure but is often caused by log accumulation, not application workloads
Not using PriorityClasses — without them, critical system pods can be evicted before low-priority batch jobs

7. Prevention Tips

Set resources.requests and limits on every container — it's the single most impactful practice for preventing evictions
Alert on node MemoryPressure and DiskPressure conditions before eviction starts
Use PriorityClass to protect critical workloads — stateful services should have higher priority than batch jobs
Configure log rotation in kubelet (containerLogMaxSize, containerLogMaxFiles) to prevent disk pressure from log accumulation
Use Vertical Pod Autoscaler (VPA) in recommendation mode to right-size requests based on actual usage
Monitor for ResourceQuota exceeded events — they can cause scheduling failures that concentrate pods on a few nodes

8. FAQ

The evicted pod keeps coming back and getting evicted again. How do I stop the cycle?

The Deployment controller recreates the pod, which lands on the same pressure-affected node (or another one also under pressure) and gets evicted again. Fix the root cause first: resolve the node resource pressure, clean up disk if needed, right-size the workload's resource requests. Temporarily cordon the affected node (kubectl cordon <node>) to prevent new pods from scheduling there while you investigate.

How do I tell if eviction was caused by memory or disk?

Read the eviction message: kubectl describe pod <evicted-pod>. The Message field will say exactly which resource triggered eviction and what the threshold and available values were at the time.

Can I prevent eviction for specific pods?

Set them as Guaranteed QoS class (requests equal limits for all resources) and use a high-value PriorityClass. Guaranteed pods are the last to be evicted. Note: you cannot fully prevent eviction — if the node literally runs out of memory and all pods are Guaranteed, the OOM killer will eventually kill processes regardless.

9. Summary

Eviction reason	Signal	Fix
memory.available	Node MemoryPressure=True	Right-size requests; fix memory leaks; add nodes
nodefs.available	Node DiskPressure=True	Clean logs and images; enable log rotation
nodefs.inodesFree	DiskPressure from inode exhaustion	Delete many small files; clean /tmp and log dirs
Pod Best Effort	No requests/limits set	Add resource requests and limits to all containers
Ephemeral storage limit	Pod emptyDir or logs too large	Set ephemeral-storage limits; use PVCs for large data

Explore More in This Category

Explore more in this category: Kubernetes guides. Browse all DevOps Compass articles or jump to: Kubernetes, AWS, CI/CD, Containers, Monitoring, Networking.