Troubleshooting Storage Saturation in Jenkins

More and more top organizations worldwide rely on Jenkins to streamline entire project lifecycles. From testing prototypes and automating the CI/CD pipeline through to production workloads, Jenkins provides reliability, speed, and cost savings.

When Jenkins disk space issues occur, the entire organization may be affected. It’s therefore critical to diagnose and fix Jenkins problems quickly.

In this article, we’ll look at causes of disk-related problems, fast troubleshooting techniques, and suggested fixes for common problems. 

What Are the Symptoms of Jenkins Disk Space Issues?

Storage problems cause all kinds of unhealthy symptoms in a Jenkins application.

If the issue affects a Jenkins agent, we may see:

• Builds slowing down or appearing to hang;
• Long delays between console outputs;
• Agents intermittently disconnecting or failing to respond to the controller;
• Docker container operations becoming painfully slow;
• Agent queues building up;
• Agents crashing with disk-related error messages.

If the problem is in the Jenkins controller, the symptoms are different. One or more of these indicate possible disk issues.

• Builds on multiple agents are slow to start.
• Delays occur with updating build status and console output.
• Multiple agents are reported as being offline.
• Scheduled jobs aren’t run.
• UI is unresponsive. This is sometimes intermittent.
• Jenkins restarts are slow.

What Causes Disk Space Problems in Jenkins?

When disk saturation occurs, requests for storage reads or writes can’t be completed on time. Applications waste time waiting for I/O, and performance drops accordingly. This can happen for one of several reasons:

• A device is more than 90% full;
• Read/write queues are filling up faster than the hardware can process them. This could be because the hardware is slow, or it could be because too many I/O based processes are running at the same time;
• Bandwidth problems, such as bottlenecks on ports or links;
• A disk is faulty;
• Memory limitations are causing excessive swapping to storage.

The table below shows possible areas where disk space usage in Jenkins controller and agents may be excessive:

Since there are many possible causes, we need to use diagnostic tools to find more information before looking at how to fix the problem. 

How to Diagnose Disk Space and Disk I/O Problems in Jenkins

Let’s now have a look at the tools and tactics we can use to diagnose disk problems, and how we can relate these to Jenkins. We’ll then work through a short case study to illustrate the process.

1. Recommended Strategy for Diagnosing Disk Saturation

If we suspect Jenkins may be affected by storage-related problems, we should find the answers to a series of questions to help us establish the root cause.

These are summarized in the table below, along with the tools we can use to find the answers.

For a deeper look at solving disk issues in Linux, see Troubleshooting Disk Issues in Linux. For Windows-based systems, see Troubleshooting Performance Problems in Windows.

2. Relating Disk Diagnostics to Jenkins

We need to understand where Jenkins is likely to store information. The du command lets us explore disk usage by directory. The tables below relate this back to Jenkins. 

In the controller, Jenkins usually uses the following directories:

The agent is likely to use the following directories:

To relate thread dump findings back to Jenkins, we need to understand the Jenkins package structure. By examining the stack traces of threads stalled on disk-related tasks, we can establish which area of Jenkins is experiencing the problem.

The table below summarizes the high-level breakdown of the Jenkins package structure.

For full details of the package structure, see the Jenkins Javadocs.

3. Troubleshooting Jenkins Disk Space Issues: A Practical Example

Let’s see how we would put all that information together in practice.

To simulate disk overload, we set up a test system as follows:

• We set GC logging to high verbosity and wrote the logs to a USB drive with limited size. To achieve this, we set JAVA_OPTS in the Jenkins configuration as follows:
  • JAVA_OPTS=-Xlog:gc*=trace:file=/media/jill/MyUSB/logs/jenkins.log
• We created a script designed to keep the USB drive busy by filling it with thousands of unnecessary files:

#!/bin/bash                                                                                                                                                                                                                                
for i in {1..10000}; do                                                                                                    
      SUFFIX=$i.$(date +%Y%m%d%H%M%S%N) 
      cp /home/jill/shenandoah.log /media/jill/MyUSB/jenkins/log$SUFFIX 
done

• We set the number of executors to run on the Jenkins controller to 5, to make sure builds with the label WASTE would run in parallel on the controller
• We created a Jenkins job to run the above script, directing it to the WASTE label.
• We enabled Jenkins Thin Backup and directed the output to the USB drive.
• We requested multiple builds of the above job at the same time as initiating a backup.

Not surprisingly, Jenkins first slowed down, then became totally unresponsive. Let’s see what diagnostics showed us when we worked through the questions suggested earlier.

Are any devices close to 100% usage?

We ran the command df to see disk usage by device:

$ df
Filesystem   1K-blocks    Used     	Available    Use%   Mounted on
tmpfs    	343808   	3696     	340112   	2% 	/
/dev/sda2    959786032    89375320 	821585204    10%    /
tmpfs    	1719020  	0        	1719020  	0% 	/dev/shm
tmpfs    	5120     	4        	5116     	1% 	/run/lock
tmpfs    	1719020  	0        	1719020  	0% 	/run/qemu
/dev/sda1    523248   	6232     	517016   	2% 	/boot/efi
tmpfs    	343804   	108       	343696   	1% 	/run/user/1000
/dev/sdb1    1883644  	1796620  	0        	100%    	/media/jill/MyUSB

This showed that /dev/sdb1, which was our USB drive mounted on /media/jill/MyUSB, was 100% full. The hard disk, /dev/sda1I, has plenty of space.

Which directories use the most space? 

Once we’d established we were short of space on the USB drive, we used the command 

du -d2 -c -h media/jill 

to see which top-level directories on this drive were using the most space. Note:

• -d2 specifies we want to explore directories to a depth of 2
• -c requests an overall total
• -h displays the sizes in a form that’s easily readable.

Here’s the output:

$ sudo du -d2 -c -h /media/jill
94M 	/media/jill/MyUSB/logs
314M	/media/jill/MyUSB/jenkins
16K 	/media/jill/MyUSB/lost+found
4.0K	/media/jill/MyUSB/backupj
1002M   /media/jill/MyUSB/jenkinsb
1.4G	/media/jill/MyUSB
1.4G	/media/jill
1.4G	total

This shows three directories that are using a lot of space:

• jenkins is the directory used by the script waste_disk
• jenkinsb is the directory configured for Thin Backup
• logs is the directory configured for GC logging.

Are I/O read/write queues excessive?

We ran the command iostat -p /dev/sda1,/dev/sdb1 -x 20 to check disk activity levels on the devices /dev/sda1 and /dev/sdb1. 

The text below shows a portion of the output, truncated to fit.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
       	2.47	0.01    2.00   66.77	0.00   28.75
 
Device w/s  wkB/s      wrqm/s  %wrqm  w_await wareq-sz d/s   dkB/s drqm/s %drqm d_awt dareq-sz f/s   f_awt aqu-sz %util
sda1     0.00  0.00        0.00       0.00         0.00	       0.00 	    0.00 0.00  0.00       0.00      0.00    0.00         0.00 0.00   0.00    0.00
sdb1    28.85 2526.40  68.35   70.32       32.25      87.57	    0.00 0.00  0.00       0.00      0.00    0.00         0.00 0.00   0.93    93.24

We have a high value for IO waits overall (66.77). /dev/sdb1 is heavily utilized, with wait queues and over 90% utilization.

We would therefore conclude that both disk space and high usage are a problem in this application.

Are there errors in the Jenkins logs?

In this application, journalctl showed no errors in the Jenkins log.

Are there errors in the kernel log? 

In this application, dmesg showed no errors in the kernel log.

Are Jenkins threads stalled on disk-related tasks?

We captured three thread dumps at 10-second intervals to search for threads that were not progressing between dumps. We then loaded these into fastThread, which analyzed the dumps and created a report.

The image below shows the fastThread summary.

Fig: fastThread Summary

This summary shows threads that are not moving in the following methods:

• write0() method in sun.nio.ch.UnixFileDispatcherImpl
• directCopy0() method in sun.nio.fs.LinuxNativeDispatcher

Since both of these are I/O-related, this is highly relevant to our investigation. Clicking on the first link shows us full details of the following thread, including its stack trace:

Fig: Thread Stalled on write() Method

The second link shows this thread:

Fig: Thread Stalled on directCopy() Method

Following the stack traces, we see both are associated with the Thin Backup plugin. This tells us the backup is held up due to disk saturation.

Exploring the fastThread report of threads that are waiting, we see the thread below. The starting point of the stack trace is hudson.model.Executor.run(Executor.java:456). This identifies the thread as one of the executors running a build job. Conveniently, Jenkins names the thread with the name of the job and the build number, in this case Waster #23. The thread dump shows that it is waiting for a PID, which would be a script run by the job.

Fig: Stack Trace for Executor Thread

This makes it easy to tie waiting threads back to actual running jobs.

We’ve established that:

• Disk space is critically short in /dev/sdb1
• Writes are queueing for this drive, causing application threads to wait
• GC logging, Jenkins backups and a Jenkins job are all contending for /dev/sdb1
• Threads associated with the Thin Backup plugin are stalled on disk-related methods
• The job Waster is hung, waiting for a script to complete.

With these findings in mind, we can re-look at the list of common causes of Jenkins disk saturation.

How to Fix Disk Space and Disk I/O Problems in Jenkins

Let’s now refer back to the root causes of disk problems in Jenkins, alongside possible fixes by cause.

Within the Jenkins controller:

Within the Jenkins agent:

How to Monitor and Prevent Jenkins Disk Space Problems

When we manage critical systems like Jenkins, it’s essential to put proper monitoring and alerts in place. 

Administrators commonly use one or more of the following tools to raise alerts in time to proactively prevent problems.

• Jenkins built-in monitoring features.
• Disk monitoring plugin. This allows administrators to analyze usage by task.
• External monitoring systems such as Prometheus.

For Jenkins systems that handle crucial workloads, we recommend automated monitoring with yCrash agent. This low-overhead tool samples many different aspects of a JVM and its environment regularly. If it detects situations that could become problematic, such as disk saturation, it captures a full range of diagnostic information as illustrated in the image below. It submits these to a yCrash server, which uses machine learning to diagnose possible problems and root causes. It raises an alert, and produces a full set of useful diagnostic reports.

Fig: yCrash 360 Degree Diagnostic Artifacts

Conclusion

Jenkins disk space issues play havoc with the tight deadlines DevOps typically have to work with to stay competitive.

It’s essential to be able to identify the root cause and apply the appropriate fix quickly to avoid downtime.

Regular monitoring allows us to pick up developing problems before they affect performance, and correct the problem without affecting critical workflows.