Analyzing & Troubleshooting Thread Leak Using JFR 

Java Flight Recorder (JFR) provides continuous, low-overhead profiling by capturing runtime execution samples directly within the JVM. In this blog, we will examine Thread Leak, a performance issue commonly caused by threads that are created but never terminated, accumulating in memory until the JVM can no longer create new ones. We will simulate the problem, capture the relevant JFR data, and analyze the recording to identify the underlying issue. Let’s take a closer look.

What is Thread Leak?

Fig: Comparison of a normal thread lifecycle and a leaked thread 

Fig: Thread Leak Progression – Leaked threads accumulate over time until the JVM reaches its native thread limit, triggering java.lang.OutOfMemoryError: Unable to create new native thread 

A thread leak occurs when an application creates threads that never terminate and are never cleaned up, so they accumulate over time. Each leaked thread continues to exist in memory even though it isn’t doing any real work, and since every thread consumes memory (for its stack) and counts against the OS’s limit on how many threads a process can create, an unbounded pile-up eventually exhausts that capacity. When there’s no room left to create a new thread, the JVM throws java.lang.OutOfMemoryError: unable to create new native thread, and the application can no longer create threads it needs for anything else.

What causes ‘Thread Leak’?

Let’s look at the list of causes for this thread leak issue: 

1. Thread Leak due to Buggy Code: Due to a bug in the code, the application can inadvertently create a lot of new threads, leading to a buildup of unused threads in memory, eventually exhausting the available native memory.
2. Lack of RAM Capacity: When there is a lack of RAM capacity in the container/device in which the application is running.
3. More Processes in Memory: When other processes are running on the container/device, it leaves less room for threads to be created in the native memory.
4. Kernel Limit: By default, the kernel sets a limit on the number of threads each process can create. When the application creates more threads than the allowed kernel limit, the error is thrown.

Simulating Thread Leak Performance Issue

To understand how Thread Leak appears in JFR data, let’s reproduce the issue using a sample Java application. The following program deliberately launches a new thread every 100 milliseconds, each of which sleeps indefinitely instead of exiting, to drive unbounded thread accumulation.

public class ThreadLeakDemo {
    private static boolean flag = true;
    public static void setFlag(boolean newValue) {
        flag = newValue;
    }
    public static void start() {
        System.out.println("Thread App started");
        while (flag) {
            try {
                // Failed to put thread to sleep.
                Thread.sleep(100);
            } catch (Exception e) {}
            new ForeverThread().start();
        }
    }
    public static void stop() {
        System.out.println("Thread leak problem terminated!");
    }
}

In this program, the start() method runs a loop that continues as long as the static flag remains true. On every iteration, it attempts to sleep for 100 milliseconds, then launches a new ForeverThread. Since nothing in start() ever calls setFlag(false) during normal execution, the loop never exits, meaning a new ForeverThread is created roughly every 100 milliseconds, indefinitely. Each ForeverThread, once launched, sleeps for an extended period and never terminates on its own, so none of the previously created threads are ever cleaned up. As the application continues to run, the number of live threads keeps climbing, consuming more memory for thread stacks and moving the JVM closer to the OS’s limit on how many threads a process can create, until eventually the JVM can no longer create new threads at all.

Capturing JFR Data for Troubleshooting Thread Leak

To capture JFR data for troubleshooting the Thread Leak issue, we recommend that you follow the steps below:

If you are interested in learning about the other methods available to capture JFR recordings, we recommend that you read the ‘How to Capture Java Flight Recorder (JFR)?’ blog.

Step 1: Start a JFR recording against the running JVM:

jcmd {PID} JFR.start \
    name=loadTestCapture \
    settings=profile \
    filename=/tmp/tomcat.jfr

Note: Replace <PID> with the Process ID (PID) of your Java application running inside the container. If you don’t know how to find it, refer to our guide on finding the Java application Process ID (PID) for step-by-step instructions.

Step 2: Let it run while the Thread Leak is occurring, then stop it manually: 

jcmd {PID} JFR.stop \
    name=loadTestCapture

Step 3: Alternatively, for Fixed-Duration Capture,  you can start a recording that automatically exits after a predefined duration (for example, 15 minutes) in a single step:

jcmd {PID} JFR.start \
    name=loadTestCapture \
    settings=profile \
    duration=15m \
    filename=/tmp/tomcat.jfr

Analyzing Thread Leak Using the JFR Data

You can analyze a JFR recording using yCrash JFRPlayer by following the steps mentioned below.  

Step 1: Install yCrash JFRPlayer, which is available in both cloud and on-premises versions. Use one of the options below to get started:

• Cloud service: Register and upload your JFR file online.
• On-Premises: Install and run yCrash JFRPlayer on your local machine or within your organization’s environment.

Step 2: Upload the jfr file to your yCrash JFRPlayer. Once JFR file is uploaded, JFRPlayer parses the JFR file and generates an incident report instantly.

Fig: Uploading a standalone JFR file to yCrash JFRPlayer 

Step 3: I’d recommend you review the AI overview section first, as it gives you an executive summary of the issue in plain language, along with the Root Cause Analysis (RCA) that traces the issue back to the specific method and class responsible, in our case, pointing to ForeverThread. 

Fig: yCrash JFRPlayer AI Overview identifying ForeverThread.run() as the primary cause of the thread leak, with thread count escalating from 614 to 2,412 

Step 4: Next is to check the “Issues in the Application” section, which flags every problem individually. In this case, the two key findings are: a high thread count warning, 2,412 threads, which can lead to java.lang.OutOfMemoryError: Unable to create new native thread, and a report of 2,396 threads all originating from line #18 of com.buggyapp.threadleak.ForeverThread in the run() method, all sitting in a WAITING state with an identical stack trace.

Fig: yCrash JFRPlayer flagging a high thread count and 2,396 threads stuck in ForeverThread.run()

Step 5: Click into the flagged threads’ stack trace (linked directly from the “Issues in the Application” section) to confirm the exact execution path and verify it matches the reported line number in your source code. Here, Thread-1, one of the 2,396 threads sharing this stack trace, shows a TIMED_WAITING state, having been alive for 240 seconds, sitting in Thread.sleep() called from ForeverThread.run(ForeverThread.java:18).

Fig: Stack trace confirming Thread-1 sleeping in ForeverThread.run() at line 18, matching the source code

Simple, right? Now that we have analyzed the data, we are equipped with all the information to fix the problem.

If you face any challenges while analyzing a JFR file using yCrash JFRPlayer, check out our FAQ for answers to common questions and troubleshooting guidance.

How to fix Thread Leaks

The following are potential solutions to fix this issue:

1. Fix the Thread Leak: Analyze the recording and identify the leaking threads. Instrument a fix to ensure threads are properly terminated after completing their tasks.
2. Increase RAM Capacity: Try running your application on a container/device with a larger RAM capacity.
3. Reduce Other Processes: Terminate or move other processes running on the container/device, so there’s more room for the application to create new threads.
4. Reduce Thread Stack Size: Reducing thread stack size (via the -Xss JVM argument) lets your application create more threads within the same memory. Be cautious, though, this can result in StackOverflowError.
5. Change Kernel Thread Limit: If the error is happening because of the kernel’s per-process thread limit, you can increase this using the ulimit -u command.

Conclusion

Diagnosing and identifying the root cause of a Thread Leak can be challenging, especially in complex production environments where multiple symptoms often overlap. In this example, the JFR recording was analyzed using yCrash JFRPlayer, which automatically identified the key performance bottlenecks and correlated JVM events to pinpoint the underlying problem. The analysis traced the issue to the run() method in ForeverThread. By examining the relevant JFR insights, including thread creation events, thread count over time, and stack trace grouping across identical threads, we were able to identify the root cause and determine the appropriate fix.