Analyzing & Troubleshooting Deadlock Using JFR 

Java Flight Recorder (JFR) provides continuous, low-overhead profiling by capturing runtime execution samples directly within the JVM. In this blog, we will examine Deadlock, a performance issue commonly caused by two or more threads waiting indefinitely on locks held by each other, with neither able to proceed. We will simulate the problem, capture the relevant JFR data, and analyze the recording to identify the underlying issue. Let’s take a closer look.

What is Deadlock?

Fig: Illustration showing how a Deadlock Occurs 

Deadlock is a situation where two or more threads are each holding a resource the other needs, and each is waiting for the other to release it first. Since neither thread will let go of what it’s holding, and neither can get what it’s waiting for, both simply stop permanently, unless something outside the deadlock intervenes (like a thread being forcibly killed). Unlike a slow or blocked thread that eventually resolves, a true deadlock never breaks on its own.

What causes ‘Deadlock’?

Let’s look at the list of causes for this Deadlock issue: 

1. Inconsistent Lock Ordering: When two or more threads acquire the same set of locks in a different order, a deadlock becomes possible. In our example, ThreadA locks CoolObject first and then tries to lock HotObject, while ThreadB locks HotObject first and then tries to lock CoolObject, the exact reverse order, which is what creates the circular wait.
2. Holding a Lock While Waiting on Another: A thread that acquires one lock and holds it while attempting to acquire a second lock is exposed to deadlock risk, since it can’t release the first lock until it’s done, even if releasing it early would let another thread proceed.
3. Nested Synchronized Calls Across Classes: When one synchronized method calls into another class’s synchronized method (as CoolObject.method1() calls HotObject.method2(), and vice versa), the lock dependency isn’t always obvious from reading either class in isolation; the deadlock only becomes visible when you trace the full call chain.

Simulating Deadlock Performance Issue

To understand how Deadlock appears in JFR data, let’s reproduce the issue using a sample Java application. The following program deliberately launches two threads that acquire the same two locks in opposite order, guaranteeing a circular wait.

DeadLockDemo starts the two competing threads.

public class DeadLockDemo {
    public static void start() {
        System.out.println("App started");
        new ThreadA().start();
        new ThreadB().start();
    }
    public static void stop() {
        System.out.println("Unsupported!");
    }
}

ThreadA calls into CoolObject first.

public class ThreadA extends Thread {
    @Override
    public void run() {
        CoolObject.method1();
    }
}

ThreadB calls into HotObject first, the reverse order.

public class ThreadB extends Thread {
    @Override
    public void run() {
        HotObject.method2();
    }
}

CoolObject locks itself, sleeps, then tries to call into HotObject.

public class CoolObject {
    public static synchronized void method1() {
        try {
            Thread.sleep(10 * 1000);
        } catch (Exception e) {}
        HotObject.method2();
    }
}

HotObject locks itself, sleeps, then tries to call into CoolObject, closing the circular dependency.

public class HotObject {
    public static synchronized void method2() {
        try {
            Thread.sleep(10 * 1000);
        } catch (Exception e) {}
        CoolObject.method1();
    }
}

In this program, start() launches two threads: ThreadA, which calls CoolObject.method1(), and ThreadB, which calls HotObject.method2(). Both method1() and method2() are static synchronized, meaning calling either locks the entire class (CoolObject.class or HotObject.class) for the duration of the call. ThreadA acquires the lock on CoolObject, then sleeps for 10 seconds while still holding it, before attempting to call HotObject.method2(), which requires the lock on HotObject. Meanwhile, ThreadB acquires the lock on HotObject, sleeps for 10 seconds while holding it, then attempts to call CoolObject.method1(), which requires the lock on CoolObject. By the time either thread finishes its sleep and tries to proceed, the other thread already holds the lock it needs, and since neither will release its own lock until it completes its call, both threads wait for each other indefinitely. 

Capturing JFR Data for Troubleshooting Deadlock

To capture JFR data for troubleshooting the Deadlock issue, we recommend that you follow the steps below:

If you are interested in learning about the other methods available to capture JFR recordings, we recommend that you read ‘How to Capture Java Flight Recorder (JFR)?’ blog.

Step 1: Start a JFR recording against the running JVM: 

jcmd {PID} JFR.start \
    name=loadTestCapture \
    settings=profile \
    filename=/tmp/tomcat.jfr

Note: Replace <PID> with the Process ID (PID) of your Java application running inside the container. If you don’t know how to find it, refer to our guide on finding the Java application Process ID (PID) for step-by-step instructions.

Step 2: Let it run while the Deadlock is occurring, then stop it manually: 

jcmd {PID} JFR.stop \
    name=loadTestCapture

Step 3: Alternatively, for Fixed-Duration Capture,  you can start a recording that automatically exits after a predefined duration (for example, 15 minutes) in a single step:

jcmd {PID} JFR.start \
    name=loadTestCapture \
    settings=profile \
    duration=15m \
    filename=/tmp/tomcat.jfr

Analyzing Deadlock Using the JFR Data

You can analyze JFR recording using yCrash JFRPlayer by following the steps mentioned below.  

Step 1: Install yCrash JFRPlayer, which is available in both cloud and on-premises versions. Use one of the options below to get started:

• Cloud service: Register and upload your JFR file online.
• On-Premises: Install and run yCrash JFRPlayer on your local machine or within your organization’s environment.

Step 2: Upload the jfr file to your yCrash JFRPlayer. Once JFR file is uploaded, JFRPlayer parses the JFR file and generates an incident report instantly.

Fig: Uploading a standalone JFR file to yCrash JFRPlayer 

Step 3: I’d recommend you review the AI overview section first, as it gives you an executive summary of the issue in plain language, along with the Root Cause Analysis (RCA). In this case, it flags a recurring deadlock between HotObject and CoolObject, one thread blocked waiting for a lock held by the other, and vice versa.

Fig: yCrash JFRPlayer AI Overview identifying a circular deadlock between HotObject and CoolObject

Step 4: Next is to check the “Issues in the Application” section, which flags every problem individually. Unlike CPU Spike and Thread Leak, this is flagged FATAL, with a direct link to the exact threads causing it.

Fig: yCrash JFRPlayer flagging a fatal deadlock, with a direct link to the causing threads

Step 5: Click “Here are the threads” to see the stack traces: Thread-1 holds CoolObject‘s lock while waiting on HotObject; Thread-2 holds HotObject‘s lock while waiting on CoolObject, the circular wait, confirmed.

Fig: yCrash JFRPlayer confirming Thread-1 and Thread-2 in mutual deadlock, each blocked waiting on a lock the other holds

Simple, right? Now that we have analyzed the data, we are equipped with all the information to fix the problem.

If you face any challenges while analyzing a JFR file using yCrash JFRPlayer, check out our FAQ for answers to common questions and troubleshooting guidance.

How to fix Deadlock

The following are potential solutions to fix this issue:

1. Enforce Consistent Lock Ordering: Define a fixed order for acquiring locks across the application, for example, based on a stable identifier like System.identityHashCode(), and ensure every thread acquires CoolObject and HotObject in that same order, regardless of which one it starts from.
2. Avoid Holding a Lock While Calling Into Another Locked Resource: Restructure the code so a thread releases its first lock before attempting to acquire a second, rather than nesting the second lock inside the first, as method1() and method2() currently do.
3. Use Timed Lock Attempts: Replace synchronized blocks with Lock.tryLock() and a timeout. If a thread can’t acquire the second lock in time, it can back off, release what it’s holding, and retry, instead of waiting indefinitely.
4. Detect Deadlock via JFR or Thread Dump: Once a deadlock is suspected, a JFR recording or thread dump will show each thread waiting to lock a monitor already held by the other, making the circular dependency explicit, exactly as it did in this example.

Conclusion

Diagnosing and identifying the root cause of a Deadlock can be challenging, especially in complex production environments where multiple symptoms often overlap. In this example, the JFR recording was analyzed using yCrash JFRPlayer, which automatically identified the key performance bottlenecks and correlated JVM events to pinpoint the underlying problem. The analysis traced the issue to ThreadA and ThreadB each waiting on a lock held by the other: CoolObject’s lock held by ThreadA while it waits on HotObject, and HotObject‘s lock held by ThreadB while it waits on CoolObject. By examining the relevant JFR insights, including thread states, monitor contentions, and lock ownership across threads, we were able to identify the root cause and determine the appropriate fix.