How to Fix Kubernetes Node Disk Pressure

TechOps Examples

Hey — It's Govardhana MK 👋

Along with a use case deep dive, we identify the top news, tools, videos, and articles in the TechOps industry.

IN TODAY'S EDITION

🧠 Use Case

• How to Fix Kubernetes Node Disk Pressure

🚀 Top News

• Apple creates Private Cloud Compute VM to let researchers find bugs

📽️ Videos

• How Docker Works - Intro to Namespaces 

• Understanding Linux System Performance - The Top Command

📚️ Resources

• The 4 Focus Areas of Kubernetes Cost Optimization

• EC2 vs. Fargate for Amazon EKS: A Cost Comparison

• Simple Step-By-Step Tutorial on the Terraform Dynamic Block

TOGETHER WITH CODEIUM

Unlock Windsurf Editor, by Codeium.

Introducing the Windsurf Editor, the first agentic IDE. All the features you know and love from Codeium’s extensions plus new capabilities such as Cascade that act as collaborative AI agents, combining the best of copilot and agent systems. This flow state of working with AI creates a step-change in AI capability that results in truly magical moments.

Download It Free Today

🛠️ TOOL OF THE DAY

Inspektor Gadget -  Open source eBPF debugging and data collection tool for Kubernetes and Linux.

• Build and package eBPF programs into OCI images called Gadgets

• Collect and export data to observability tools with a simple command

🧠 USE CASE

How to Fix Kubernetes Node Disk Pressure

Imagine you deploy an application, but after a few days, it starts throwing warnings like this:

Warning  NodePressure  [timestamp]  kubelet  Node [node-name] status is now: NodeHasDiskPressure

Your application slows, pods get evicted, and new ones fail to schedule. This common error in Kubernetes is known as Node Disk Pressure, and if left unchecked, it can severely impact application performance.

What is Kubernetes Node Disk Pressure?

Node Disk Pressure occurs when a node’s filesystem is under strain due to low available disk space or inodes.

Kubernetes automatically detects these low resource conditions and sets a NodeHasDiskPressure status.

This status signals that the node has insufficient disk resources for further scheduling, evicting non-critical pods to prevent critical system disruptions.

How to Check Kubernetes Node Disk Pressure

To verify if your nodes are experiencing Disk Pressure, you can use:

kubectl describe node [node-name]

Look for any nodes with the condition type DiskPressure and status True. In the output, focus on the Conditions section. Here’s an example where techops-node2 is experiencing disk pressure:

techops-node2 Ready worker 14d v1.28.1 DiskPressure=True,MemoryPressure=False,PIDPressure=False,Ready=True

Additionally, use this command to monitor disk usage:

kubectl top nodes

This shows overall CPU, memory, and disk usage for each node, helping you pinpoint where Disk Pressure is affecting your nodes.

Why Should You Care About Node Disk Pressure?

Ignoring Disk Pressure can lead to various issues:

1. Pod Eviction: Kubernetes evicts lower-priority pods to free up disk space, which can cause disruptions in non-critical workloads.

2. Scheduling Failures: New workloads may not deploy if nodes are in a Disk Pressure state.

3. Performance Degradation: Insufficient disk space impacts node performance and can lead to application latency.

How to Fix Kubernetes Node Disk Pressure

Here are some strategies to address Disk Pressure:

Clean Up Disk Space: Clear out unused images and containers, which can take up significant space.

Increase Node Disk Size: If your nodes are in a cloud environment, consider resizing disks. In AWS, for instance - Increase the EBS volume size.

Move Logs and Data to Separate Disks: If your node frequently generates large logs, consider mounting a separate disk for log storage to keep system space free.

Implement Resource Quotas and Limits:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: storage-quota
  namespace: [namespace]
spec:
  hard:
    requests.storage: 10Gi

Monitor with Alerts:
Use Prometheus or another monitoring tool to set up alerts when disk usage exceeds a threshold. This proactive approach helps you intervene before Disk Pressure arises.

Hope this helps in tackling one of the common Kubernetes challenges.

You may even like:

How To Fix OOMKilled 

Kubernetes Node Not Ready - How To Fix It 

Over the past 17+ years, I’ve led millions of dollars' worth of digital and cloud transformation projects for 40+ clients, including:

I can help you kickstart your digital and cloud transformation or optimize your existing systems to make them more efficient, scalable, and future-ready.

Whenever you’re ready for that, book a call.

— Govardhana MK

Looking to promote your company, product, service, or event to 16,000+ TechOps Professionals? Let's work together.