Troubleshooting Spring Boot Thread Leaks

Spring Boot is a widely used framework for building highly transactional Java-based web applications and backend systems. These applications process heavy loads and are often multi-threaded.

Video

Watch this webinar video to learn more about 9 expert tips and tricks to help you master thread dump analysis.

Thread leaks that occur in Spring Boot can lead to a gradual accumulation of threads, consuming system resources and potentially causing performance issues or even application crashes.

What is a thread leak?

Thread leak occurs when applications unintentionally retain references to thread objects after they have finished executing their tasks or when threads do not exit with proper conditions, preventing them from being garbage collected. Here are some common scenarios where Java thread leaks may occur:

1. Failure to Terminate Threads: If threads are not properly terminated or if the termination conditions are not handled correctly, threads may continue running even after their tasks are completed.
2. Thread Pools Mismanagement: Using thread pools is a common practice for managing threads efficiently. However, if a thread pool is not properly managed, it may retain references to threads that are no longer needed. Thread pool mismanagement can include not shutting down the thread pool when it is no longer needed.
3. ThreadLocal Issues: The ThreadLocal class in Java allows variables to be bound to a thread. If not used and cleaned up properly, it can lead to memory leaks. For example, if a ThreadLocal variable holds a reference to an object that should be garbage collected, but the ThreadLocal is not cleared, it can cause a leak.
4. Incorrect Use of Thread.join(): The join() method is used to wait for a thread to complete. If it is not used properly or if it is not called when it should be, it can result in threads not being released properly.

Simulating Thread Leak in SpringBoot

In order to simulate the thread leak problem in Spring Boot, we leveraged the open source BuggyAPI application, a comprehensive Spring Boot service capable of simulating various performance problems. When we launched the Buggy API application, it looked as below:

Here Spring Boot Buggy API services are simulating thread leak:

@Service
public class ThreadLeakDemoService {

	private static final int threadLeakSize = 1250;
	
	private static final Logger log = LoggerFactory.getLogger(ThreadLeakDemoService.class);

	public void start() {

		log.info("Thread App started");
		int threadCount = 0;
		
		while (threadCount < threadLeakSize) {
			log.info("Thread Started : "+threadCount);
			try {
				// Failed to put thread to sleep.
				Thread.sleep(10);
			} catch (Exception e) {
			}
			threadCount++;

			new ForeverThread().start();
		}
	}
}

public class ForeverThread extends Thread {

	@Override
	public void run() {
	
		while (true) {
		
			try {
				
				Thread.sleep(10 * 60 * 1000);
			} catch (Exception e) {}
		}
	}
}

You can notice that the sample program contains the ‘ThreadLeakDemoService’ class. This class has a start() method. In this method, ‘ForeverThread’ is created with many threads where there is the run() method. In this method, thread is put to continuous sleep i.e. thread is repeatedly sleeping for 10 minutes again and again. This will keep the ‘ForeverThread’ alive always without doing any activity. A thread will die only if it exits the run() method. In this sample program run() method will never exit because of the never-ending sleep.

Since the ‘ThreadLeakDemo’ class keeps creating ‘ForeverThread’ infinitely and they never exit. Thus very soon, several thousands of ‘ForeverThread’ will be created. It will saturate memory capacity, ultimately resulting in ‘java.lang.OutOfMemoryError: unable to create new native thread’ problem.

Troubleshooting Thread Leaks

In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before it surfaces in the production environment. Once it predicts outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them and instantly generates a root cause analysis report. Artifacts it captures include Garbage Collection log, Thread Dump, Heap Substitute, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters, disk usage…. 

You can register here and start using the free-tier of this tool.

Below is the report generated by the yCrash tool when the above sample SpringBoot program is executed:

From the report, you can notice that yCrash points out that 1,200+ threads are created, and they have the potential to cause ‘OutOfMemoryError: unable to create new native thread’ problem. Besides the thread count, the tool is also reporting the line of code, i.e. ‘com.ycrash.springboot.buggy.app.service.threadleak.ForeverThread.run(ForeverThread.java:12)‘ in which all the 1,200 threads are stuck. Equipped with this information, one can quickly go ahead and fix the problematic code.

Possible Solution

An appropriate solution needs to be applied based on the specific thread leak problem. Below are some recommended resolutions for thread leak issues:

Recommendation	Description
Properly terminate threads	Properly terminate threads when they are no longer needed.
Use thread pools	Use thread pools and ensure they are properly managed.
Use ThreadLocal	Use ThreadLocal cautiously and ensure proper clean-up.
Handle Exceptions in threads	Handle exceptions properly to avoid leaving threads in an inconsistent state.

Conclusion

In this blog, we looked in detail at troubleshooting thread leak exceptions with an emphasis on detecting the root cause and resolving the thread leak in Spring Boot applications.