Troubleshooting Blocked Threads in Jenkins

Blocked threads in Java occur when a thread is waiting for a lock that’s currently held by another thread. Until the lock becomes available, the blocked thread can do no work. Tasks stall, or the entire system slows down or hangs altogether. The problem is compounded if several threads are waiting for the lock.

In large organizations, where developers are pressured to produce software updates quickly and efficiently, Jenkins performance is critical. Jenkins blocked threads must be diagnosed and fixed as quickly as possible.

In this article, we’ll look at causes, fixes and diagnostic processes to help us get the system back up to speed as fast as possible.

Common Symptoms of Jenkins Blocked Threads

If Jenkins has too many blocked threads, performance degrades. Common symptoms include:

• Slow builds;
• Builds stuck with a status of ‘Waiting for Next Executor’;
• Delayed build scheduling;
• Slow or unresponsive user interface.

What Causes Blocked Threads in Jenkins? 

In Java, threads can be in one of the following states: NEW, RUNNABLE, BLOCKED, WAITING, TIMED_WAITING, TERMINATED.

For more information on thread states, see Java Thread States.

Threads enter the blocked state for only one reason: they are waiting for a Java lock that is held by another thread. This happens frequently in high-concurrency applications, but the block should only be momentary. However, if the thread that has the lock is delayed while waiting for other resources, such as I/O or the network, it soon develops into a problem. In a busy system, hundreds of threads may block waiting for a single lock, with disastrous effects on performance.

 Java locking methods include the following:

Synchronized Methods and Blocked Threads

public synchronized void getData() {
// Some code
…
}

Only one thread can enter this method at one time. Other threads must wait until the thread that holds the lock completes the method.

How Synchronized Blocks Cause Blocked Threads 

Object lockVariable = new Object();
…
synchronized (lockVariable) {
// Some code
 … 
}

When a thread enters the synchronized block, it obtains a lock on the specified variable. It releases it when the block of code is complete. Any other threads needing to execute this code are blocked until the lock is released.

How ReentrantLock Can Lead to Blocked Threads 

private final ReentrantLock lockVariable = new ReentrantLock();
….
public void doSomething {
   lockVariable.lock();
   try {
   // Critical section of code
   …
   } 
finally {
   lockVariable.unlock();
   }
}

The same thread may re-enter this code more than once, but other threads that require the lock are blocked until the lock is released.

Blocked Threads in Java 

The thread that has a lock is runnable. Any threads waiting on the lock are blocked, and are added to an entry set. Once the lock is free, the JVM chooses a thread from the entry set, awards it the lock and changes its status to RUNNABLE.

Fig: Blocked Threads

In this diagram. Thread A has the lock and is runnable. Threads B, C and D want the lock, and are blocked.

Note: Blocked Threads do NOT occur when a thread is waiting for: Network response I/O Database responsejoin()These threads usually appear as RUNNABLE or WAITING.The problem is that, if these waits occur inside a synchronized section of code, they’re likely to cause a queue of blocked threads who need to acquire the lock. This is one of the most common causes of blocked threads in Java.

For more information on blocked threads, watch Blocked Threads Explained.

Common causes in Jenkins include:

• Queue and scheduling contention;
• Synchronization issues related to plugins;
• Overloading on the agent remoting channel;
• Poorly-configured source code management;
• Excessive or slow credential checking;
• File system or networking issues.

How to Diagnose Jenkins Blocked Threads 

Diagnosing Jenkins blocked threads starts with identifying which thread is holding a lock and why other threads are waiting for it. The following techniques can help you systematically isolate the root cause.

1. Capture a Jenkins Thread Dump 

The first step with any thread-related problem in Java is to take a thread dump, which is a snapshot of the state of all threads in the JVM at the time of the dump. This is a text file, and you can read it manually, but in a large system like Jenkins it is very long and would take time to analyze it. Tools such as fastThread make diagnosis much quicker by presenting the information in the thread dump as a series of graphs and charts, as well as highlighting possible problem areas. 

For more information on thread dumps, see Understanding Java Thread Dumps.

2. Capture Comprehensive Diagnostics Using yc-360 

With blocked threads, the problem could be caused by other factors, such as networking or I/O issues, so it’s a good idea to take a more comprehensive set of diagnostics. The easiest way to do this is to use the free open-source script yc-360. This records comprehensive information about the JVM and its environment, as well as three thread dumps taken at intervals for comparison.

3. Identify the Thread Holding the Lock 

The most important question to ask when investigating blocked threads is, “Which code is holding the lock that other threads are waiting on?”

Let’s demonstrate the process by taking code that deliberately causes blocked threads, and incorporating it into a Jenkins plugin.

The code is as follows:

package com.buggyapp.blockedapp;
public class BlockedAppDemo {
	//private static final Logger s_logger = 
              LogManager.getLogger(BlockedAppDemo.class);
	
	public static void start() {
		
		System.out.println("App started");
		for (int counter = 0; counter < 10; ++counter) {
			// Launch 10 threads.
			new AppThread().start();
		}	
	}
	
	public static void stop() {
		
		System.out.println("Blocked App problem terminated!");
	}
}

This class creates ten identical threads of class AppThread.

package com.buggyapp.blockedapp;
public class AppThread extends Thread {
	@Override
	public void run() {
		
		AppObject.getSomething();
	}
}

The thread calls a synchronized method from the class AppObject. Only one thread will be able to enter this method at one time.

package com.buggyapp.blockedapp;
public class AppObject {
	private static boolean flag = true;
	
	public static void setFlag(boolean newValue) {
		flag = newValue;
	}
	
	public static synchronized void getSomething() {
		// Put the thread to sleep forever. The first
		// thread acquired the lock and went to sleep
		// No other thread would be able to enter this method.
		while (flag) {
			
			try {
				
				Thread.sleep(10 * 60 * 1000);
			} catch (Exception e) {}
		}
	
	}
}

The synchronized method goes to sleep indefinitely, so the lock won’t be released. The first thread will obtain the lock. The other 9 will block waiting for it.

This simulates a Jenkins plugin that holds a lock for too long, which is one of the most common causes of blocked threads.

We used the yc-360 script to take a full range of diagnostics, including three thread dumps. This produces a .zip file that holds the thread dumps along with the output from other JDK and operating system diagnostic commands. These files can be analyzed manually or with your own choice of tools. We chose to load it into yCrash via the ‘Upload Bundle’ facility on the dashboard. This feature carries out a Root Cause Analysis using the set of diagnostics and produces a comprehensive set of reports.

The yCrash summary report is shown below:

Fig: yCrash Summary Report

The software has immediately diagnosed that the application is suffering from a problem with blocked threads, and has identified Thread 14 as the culprit that is holding the lock. Clicking the ‘stack trace’ link next to this message gives us more information about Thread 14, as we’ll see.

If we’d chosen to load a thread dump directly into fastThread, we would also have seen a diagnostic summary at the front of the report:

Fig: fastThread  Problem Detected

Again, if we click the link, we’ll see details of the problematic thread, as shown in the image below.

Fig: Details of Problem Thread

The stack trace tells us what instruction was executing from what class at the time of the dump. It also shows the path the program followed to get to that instruction.

If we scroll down a little further in the report, we’ll see a graph showing which threads are blocked waiting for this thread to release the lock:

Fig: Blocked Threads Graph

Clicking on any of these threads shows its stack trace in a pop-up window on the right:

Fig: Stack Trace of Blocked Threads

4. Use Stack Traces and Package Names to Locate the Root Cause 

The class names in the stack trace are an important clue as to where and why threads are blocked. We can use this to zoom in on the affected area of Jenkins. Jenkins architecture is quite complex, as shown in the diagram below, and it saves us a lot of time if we know where to look for the problem.

Fig: Jenkins Architecture

How does this relate to the class names we see in the stack trace?

We can use the table below as a starting point:

In our example, the package name does not match the Jenkins package prefixes, and it doesn’t contain plugin in the package path. It must, therefore, be a third-party library.

The Javadocs for Jenkins core classes and the official plugins are available from the Jenkins documentation. We can use them to see how the packages of the blocked and blocking threads fit into the Jenkins ecosystem.

Alternatively, searching the internet for the package name and “Javadoc” brings up the documentation for most commonly used libraries, including Jenkins. We can find the modules involved and information about what they do, which gives us a good idea of where the bottleneck might be.

If the classes in the stack trace are network or I/O related, slow data retrieval could be causing threads to hold locks for too long. The yCrash report has tabs where we can look at disk and network activity.

How to Fix Blocked Threads in Jenkins Core and Plugins 

Once we’ve run diagnostics and traced the class that’s causing the issue, we need to get the system up and running and prevent recurrence.

1. How Can Jenkins Administrators Prevent Thread Blocking? 

A restart of Jenkins almost always fixes the problem temporarily, but it’s likely to recur.

Here are details of problems that have been encountered in the past, and their fixes.

If these aren’t applicable, or don’t work:

• Make sure you’re using the latest versions of Jenkins for the controller, remoting, agents and plugins.
• Remove recently-installed plugins to see if that fixes the problem.
• Put in a bug report. The Jenkins documentation gives details of where to submit the report and what to include in it.

2. How Can Plugin Developers Avoid Thread Contention in Jenkins? 

Plugin developers should make sure any locks they hold are released as soon as possible. Avoid holding locks while carrying out slow operations such as:

• Accessing the network
• Querying GitHub
• Talking to Kubernetes
• Accessing a database
• Performing filesystem operations
• Obtaining credentials

Only acquire one lock at a time.

Always test the plugin under a heavy load, and monitor thread dumps to make sure locks aren’t causing other threads to block.

Can Jenkins Blocked Threads Be Detected Early? 

Regular system monitoring can identify locking issues before they actually cause loss of performance.

On a regular basis, take three thread dumps and check the status of the threads, comparing the three reports. fastThread has a useful summary of thread states, as shown below.

Fig: fastThread Thread Status Summary

In a healthy system, threads should only ever be blocked momentarily, and a build-up indicates problems are developing. Any thread that doesn’t unblock between the three thread dumps is holding its lock too long, and should be investigated.

Alternatively, we can set up a yCrash agent to sample performance regularly. If it detects any issues, it raises an alert and creates a full diagnostic report.

Conclusion

Jenkins blocked threads can eat away at performance over time and cause builds to stall. Taking a thread dump lets us look at the stack trace of culprit and victim threads, helping us quickly diagnose and fix the problem.

It’s always a good idea to monitor system health to catch issues before they affect production.