Detecting thread leakage is important for performance benchmarking for several reasons.
Firstly, thread leakage can lead to resource exhaustion, as each thread consumes memory and other system resources. Over time, if threads are not properly managed and released, they can accumulate and degrade system performance, causing slowdowns and potentially crashing the application or system.
Secondly, excessive thread creation and leakage can impact system scalability and throughput. Threads are often used to handle concurrent tasks, and if threads are leaked, it can limit the system’s ability to handle concurrent requests efficiently. This can lead to decreased throughput and responsiveness, especially in high-concurrency scenarios.
Additionally, thread leakage can indicate underlying issues with thread management, such as not properly terminating threads after they have completed their tasks. Identifying and fixing these issues can lead to more efficient resource utilization and better overall system performance.
Furthermore, in performance benchmarking, it’s crucial to have a clear understanding of resource usage patterns, including thread usage. Detecting thread leakage allows developers to accurately measure and analyze resource utilization metrics, which is essential for optimizing performance, identifying bottlenecks, and ensuring the application or system can scale effectively under varying loads.
Thread Leak in Boomi
We can implement our performance test to simulate a thread leakage condition within Boomi, a popular no-code/low code middleware. In order to simulate the thread leakage scenario, we will make use of a simple open source chaos engineering application called BuggyApp. We need to build a process that ultimately invokes the BuggyApp JAR and execute the process to simulate the thread leakage.
The BuggyApp simulates the thread leakage scenario by instantiating a class called ForeverThread which runs forever. The instantiation is repeated by a wrapper class in an infinite loop.
Within the Dell Boomi Process, we need to create a process that has three shapes/steps:
- A start shape, with no significance other than the start of the process.
- A data process shape (Figure below) which calls the below custom groovy script invoking the BuggyApp class.
- A stop shape which terminates the process successfully.
Fig: Data Processing Shape in Boomi
We use the below script in a data process shape to simulate the thread leakage scenario within Boomi. Note that dataContext is an object from the Boomi runtime that provides access to documents flowing through the process:
import java.util.Properties;
import java.io.InputStream;
import com.buggyapp.threadleak.ThreadLeakDemo;
for( int i = 0; i < dataContext.getDataCount(); i++ ) {
InputStream is = dataContext.getStream(i);
Properties props = dataContext.getProperties(i);
ThreadLeakDemo.start();
dataContext.storeStream(is, props);
}
dataContext is an Atom object. Documentation is not available to customers as its an internal Boomi object. dataContext provides the input stream to each document as well as a properties set for each document in the flow
Fig: Boomi Process Creation – Thread Leakage
BuggyApp ThreadLeakDemo
Here’s a code example that shows how thread leaks occur in Java applications. It demonstrates a scenario where threads are created endlessly, leading to potential resource issues if not managed properly:
package com.buggyapp.threadleak;
import com.buggyapp.cpuspike.CPUSpikerThread;
/**
* Created infinite number of threads
*
* @author Ram Lakshmanan
*/
public class ThreadLeakDemo {
private static boolean flag = true;
public static void setFlag(boolean newValue) {
flag = newValue;
}
public static void start() {
System.out.println("Thread App started");
while (flag) {
try {
// Failed to put thread to sleep.
Thread.sleep(100);
} catch (Exception e) {
}
new ForeverThread().start();
}
}
public static void stop() {
System.out.println("Thread leak problem terminated!");
}
}
package com.buggyapp.threadleak;
public class ForeverThread extends Thread {
private static boolean flag = true;
public static void setFlag(boolean newValue) {
flag = newValue;
}
@Override
public void run() {
while (flag) {
try {
Thread.sleep(10 * 60 * 1000);
} catch (Exception e) {}
}
}
}
This code demonstrates a classic thread leak issue within a Java application. In the ThreadLeakDemo class, the start() method enters an infinite loop, continuously creating and starting new ForeverThread instances as long as the static flag variable is true. Each ForeverThread runs its own infinite loop, sleeping for 10 minutes in each iteration, which prevents the thread from terminating and thus causing a potential resource leak. The setFlag() methods in both classes allow changing the value of flag, which theoretically could stop the loops if set to false, but in practice, this approach doesn’t address the fundamental issue of unbounded thread creation. The stop() method in ThreadLeakDemo is designed to indicate when the thread leak problem should be terminated, but it does not actually stop the ongoing threads.
yCrash Monitoring for Boomi
In order to monitor the performance of the application, I have selected yCrash to help. The yCrash agent must be installed in order to report back performance metrics to the yCrash receiver/dashboard. To install the yCrash agent, follow these instructions.
OutOfMemoryError: unable to create native thread in Boomi Process
The process eventually landed in an error state. The container logs showed that eventually, it could not spawn anymore threads due to memory resources being exceeded:
Mar 26, 2024 9:43:33 AM EDT SEVERE [com.boomi.process.ProcessExecution handleProcessFailure] Unexpected error executing process: java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
Fig: Boomi Process Execution – Thread Leakage
yCrash Diagnostics Boomi Thread Leak
I switched over to yCrash, hoping to get more information about the thread leakage that occurred. I noticed that an incident summary was generated for the scenario I created in Boomi.
Fig: yCrash Incident Summary – Thread Leakage
In the yCrash’s summary page it clearly pointed out that 1980 threads are in WAITING state and they all exhibhit exact stack traces and it needs to be examined. I clicked on the stack trace hyperlink in the error statement. Further information on the incident summary can be found here.
Fig: yCrash Stack Trace – Thread Leakage
yCrash showed the stack trace of the leaking threads. From the stack trace, one could see where the threads are launched, what code execution path is followed and where it finally reached. The stack trace indicated that BuggyApp’s ForeverThread was the culprit for the threads to leak in the Boomi platform.
Conclusion
In this study, we explored the implementation of a thread leakage simulation within Boomi, a no-code/low-code middleware, using the BuggyApp. The simulation revealed at the surface a memory resourcing issue preventing new threads to be created. Using yCrash, a performance monitoring tool, I successfully detected the culprit class that was spawned 1980 times and consuming resources due to being in a sleep state.
Share your Thoughts!