Troubleshooting Java Spring Boot Deadlock

Spring Boot is a widely used framework for building highly transactional Java-based web applications and backend systems. These applications process heavy loads and are often multi-threaded, requiring performance analysis for efficient resource utilization.

Spring Boot applications can suffer from Deadlock problem, which can bring the entire application to grinding halt. This blog discusses the troubleshooting of deadlock issues in Spring Boot applications and provides remedies to solve them.

Thread Deadlocks

A thread deadlock in Java occurs when two or more threads are blocked forever, each waiting for the other to release a lock. In other words, a set of threads is in a deadlock state when every thread in the set is blocked, and no thread can proceed.  

A deadlock typically involves two or more threads and two or more locks. Here’s a simple scenario to illustrate a deadlock:

1. Thread A acquires lock 1.
2. Thread B acquires lock 2.
3. Thread A attempts to acquire lock 2 but is blocked because Thread B already holds it.
4. Thread B attempts to acquire lock 1 but is blocked because Thread A already holds it.

Now, both threads are blocked, and neither can proceed because each is holding a lock that the other thread needs. This situation results in a deadlock.

The most common reasons for deadlocks are:

1. No Pre-emption: Resources cannot be forcibly taken away from a thread; they can only be released voluntarily by the thread holding them.
2. Circular Wait: A cycle must exist in the resource allocation graph, where each thread in the cycle is waiting for a resource held by another thread in the cycle.
3. Mutual Exclusion: At least one resource must be held in a mutually exclusive mode.
4. Hold and Wait: A thread holds at least a resource and is waiting to be acquired by other threads.

Troubleshooting Setup

This case study demonstrates how to troubleshoot a Spring Boot thread deadlock issue. There are a few tools available to find deadlock issues. In this blog, we will be using yCrash, a non-intrusive comprehensive tool with 360° data capture and analysis for Spring Boot apps performance tuning and troubleshooting. Below is an overview of the service to troubleshoot Spring Boot services with the yCrash service:

Setup YCrash

Setting up yCrash involves three simple steps:

1. Register with yCrash.
2. Install the yCrash Agent.
3. Set up the yCrash Server.

Register With yCrash 

yCrash offers a free tier. You can register here.

Install yCrash Server

The yCrash server is a web application that provides an incident management report and comes in two variants:

1. Cloud Service – A secured managed service with no installation required.
2. Enterprise Edition – An on-premises instance that offers greater control and adheres to enterprise compliance requirements.

Note: For this tutorial, we will be using the cloud service. You can sign in here. 

Install yCrash Agent

The yCrash agent captures a 360° view of the service and underlying infrastructure. It can be deployed with just a few simple steps. Please refer to the yCrash agent installation guide for more information.

Sample Application

Performance monitoring can be applied to any Spring Boot app. We utilized the Spring Boot BuggyApi application, a comprehensive Spring Boot service capable of simulating various performance problems, to replicate common performance issues that may occur with Spring Boot apps.

You can check out and run the Spring Boot Buggy API service from 

https://github.com/yCrash/springboot-buggyapi

Here SpringBoot Buggy API services is simulating deadlock between 2 threads.

public class DeadLockDemoService {

  

    public void start() {

   	 

   	 System.out.println("App started");

   	 new ThreadA().start();

   	 new ThreadB().start();

    }

}

public class CoolObject {

    public static synchronized void method1() { 

   	 try {

   		 // Sleep for 10 seconds

   		 Thread.sleep(10 * 1000);

   	 } catch (Exception e) {  		 

   	 }	 

   	 HotObject.method2();

    }

}

public class ThreadA extends Thread {

    @Override    

    public void run() {

   	 CoolObject.method1();

    }

}

public class ThreadB extends Thread {

    @Override    

    public void run() {

   	 HotObject.method2();

    }    

}

You can notice the sample program contains the ‘DeadlockDemoService’ class. This class has a start() method. In this method, 2 threads with the name ‘ThreadA’ and ‘ThreadB’ are launched. 

‘run()’ method in ‘ThreadA‘ invokes ‘CoolObject#method1()’. Similarly ‘run()’ method in ‘ThreadB’ invokes ‘HotObject#method2()’.

If you notice both ‘CoolObject#method1()’ and ‘HotObject#method2()’ are synchronized methods. When a method is synchronized, only one thread who has the lock of that object can execute that method. If another thread tries to execute the same method, then it will go to the BLOCKED state, until the first thread completes executing the method. After entering the respective methods, both threads sleep for some time and then continue to invoke other method i.e. ‘CoolObject#method1()’ will invoke ‘HotObject#method2()’ and ‘HotObject#method2() will invoke ‘CoolObject#method1()’.

So lets visualize what happens when above program is executed:

1. ThreadA acquires CoolObject’s lock.
2. ThreadB acquires HotObject’s lock.
3. ThreadA waits for HotObject’s lock.
4. ThreadB waits for CoolObject’s lock.

Thus, both threads will end up in classic Deadlock.

Troubleshooting Deadlocks

Deadlocks can be detected by using a thread dump or by utilizing yCrash. Below, yCrash has raised an incident for a deadlock with a complete set of reports, including thread dump, garbage collection, heap dump, netstat, etc., required for troubleshooting the incidents. Details on deadlock detection and the root cause are provided in the thread view.

Below is the thread dump analysis report from the tool highlighting the two threads and their stack trace which were causing the Deadlock.

You can clearly see yCrash reporting ‘Thread-4’ and ‘Thread-5’ suffering from Deadlock. yCrash also reports the stack trace of ‘Thread-4’ and ‘Thread-5’. From the stack trace you can notice ‘Thread-5 ‘ acquired the lock of ‘CoolObject’, and it is waiting for ‘HotObject’ lock. On the other hand, ‘Thread-1’ acquired the lock of ‘HotObject’, and it is waiting for ‘CoolObject’ lock. Now based on this stacktrace we know the exact line of code that is causing the problem.

The solutions to solve Java thread issues are:

• Timeout Locks: Implement a mechanism to timeout locks. If a thread cannot acquire a lock within a specified time, it releases its acquired locks and retries or reports an error.
• Concurrency Utilities: Such as using the tryLock method with a timeout to prevent threads from waiting indefinitely, provide effective mechanisms for thread management. When explicit locking is required, it is advisable to use ReentrantLock instead of synchronized methods or blocks. These utilities contribute to better control and prevent issues like deadlock.
• Avoid nested locks: Minimize or eliminate nested locks. If a thread holds one lock and then attempts to acquire another, it increases the chances of deadlocks.

Conclusion

In this blog, we looked in detail at troubleshooting deadlocks with an emphasis on detecting the root cause and resolving deadlocks in Spring Boot applications.