In this blog we are going to simulate a blocked threads scenario from a job within Boomi. Then we will use a tool to diagnose the blocked threads scenario and perform triage. Boomi is a cloud-based integration platform that enables organizations to connect and automate their applications, data, and processes across on-premises, cloud, and hybrid environments. It offers a low-code development environment, pre-built connectors, and features for API management, workflow automation, and master data management, simplifying the complexities of integration and fostering digital transformation.
The environment setup for Boomi performance testing offers flexibility, accommodating either an on-premises instance or leveraging the “Boomi Private Cloud” Environments. In the context of this performance test, I opted to install the Boomi Atom on my local M2 MacBook Pro. This strategic choice enables a controlled testing environment, allowing for meticulous evaluations without impacting live production systems. The localized deployment empowers a focused and efficient exploration of Boomi’s capabilities, ensuring a robust understanding of its performance under specific conditions.
After the Boomi Atom was set up, I also installed the yCrash agent to assist with performance monitoring and diagnostics.
Permanently Blocked Threads
Threads can become in a permanently blocked state and bog down an application. Many Boomi processes that have threads in a prolonged blocked state can slow down or crash Boomi. There are several reasons why threads can be in a blocked state for a prolonged period of time:
1. Contention for Shared Resources:
When multiple threads contend for access to shared resources, such as critical sections of code or shared data structures, contention can lead to blocking. If one thread holds a lock on a resource and another thread attempts to acquire the same lock, the latter will be blocked until the lock is released.
2. Deadlocks:
As discussed earlier, deadlocks occur when two or more threads are blocked forever, each waiting for the other to release a lock. This can happen due to circular waiting, no preemption, hold and wait, or other reasons mentioned in the previous responses.
3. Excessive Synchronization:
Overuse of synchronization mechanisms, such as using synchronized blocks or methods unnecessarily, can lead to contention and blocking. Fine-tuning synchronization is important to prevent unnecessary blocking.
4. I/O Operations:
Threads can become blocked when performing input/output (I/O) operations, especially when waiting for data from external sources like files, network sockets, or databases. Slow or unresponsive I/O operations can cause prolonged blocking.
5. Thread Starvation:
Thread starvation occurs when a thread is unable to get the required resources for an extended period. This can happen in scenarios where certain threads are given higher priority, leaving others waiting for resources for an extended duration.
6. Inefficient Algorithms:
Poorly designed algorithms or data structures may cause threads to spend more time than necessary waiting for resources or completing tasks, leading to prolonged blocking.
7. Lock Contention:
High contention for locks, especially in scenarios where many threads are competing for a small set of locks, can result in prolonged blocking as threads wait for their turn to acquire the lock.
8. Thread Sleep or Wait:
Explicitly calling Thread.sleep() or Object.wait() without careful consideration of the appropriate time to wait can result in prolonged blocking. Threads may sleep for longer than necessary, causing delays in the execution flow.
BuggyApp Blocked Threads Simulation
BuggyApp is an open source chaos engineering program that can simulate various performance problems like Memory Leak, OutOfMemoryError, CPU spike, thread leak, StackOverflowError, deadlock, unresponsiveness.Below is the BuggApp code that simulates BLOCKED threads:
package com.buggyapp.blockedapp;
public class BlockedAppDemo {
//private static final Logger s_logger = LogManager.getLogger(BlockedAppDemo.class);
public static void start() {
System.out.println("App started");
for (int counter = 0; counter < 40; ++counter) {
// Launch 10 threads.
new AppThread().start();
}
System.out.println("App became unresponsive");
}
public static void stop() {
System.out.println("Blocked App problem terminated!");
}
}
public class AppThread extends Thread {
@Override
public void run() {
AppObject.getSomething();
}
}
public class AppObject {
private static boolean flag = true;
public static void setFlag(boolean newValue) {
flag = newValue;
}
public static synchronized void getSomething() {
// Put the thread to sleep forever. Basically first
// thread would have acquired the lock and go to sleep
// No other thread would be able to enter this method.
while (flag) {
try {
Thread.sleep(10 * 60 * 1000);
} catch (Exception e) {}
}
}
}
In the above program, BuggyApp spawns 40 threads. All 40 threads simultaneously call run() method, the method which is called when a thread is instantiated. Inside the run() method there is one method call to the getSomething() (which is both a static and synchronized method) in the AppObject. Inside getSomething() is an infinite while loop encapsulating a Thread.sleep (long period of time) call.
When a method is declared as synchronized, only one thread is allowed to enter that method at a time. Any other thread which attempts to enter the method will be put into BLOCKED state.
When executing, one thread of Forty enters the getSomething() method after being spun up, and it would sleep forever. All other threads trying to enter the getSomething() method will be put into the BLOCKED state. In a production environment, blocked threads will lead to poor application performance.
Blocked Threads in Boomi Server
To trigger the above mentioned Blocked threads problem in Boomi, we constructed a 3 shape/step process:
- First shape is a start shape, this is only significant to indicate the start of the process
- Second is a data process shape that invokes a custom Groovy script below
- Finally a stop shape that terminates the process successfully.
The boilerplate code to execute the blocked thread demo inside the BuggyApp from Boomi is below. Please note that dataContext and storeStream are boilerplate objects and methods within Boomi that obtain the document as a stream and store the document as a stream at the end.
import java.util.Properties;
import java.io.InputStream;
import com.buggyapp.blockedapp.BlockedAppDemo;
for( int i = 0; i < dataContext.getDataCount(); i++ ) {
InputStream is = dataContext.getStream(i);
Properties props = dataContext.getProperties(i);
BlockedAppDemo.start();
dataContext.storeStream(is, props);
}
Executing the process was relatively quick.
Boomi Process Execution
The process executed within one second. Of note, in this simulation, Boomi does not capture the blocked threads that persist after the data process shape/step. If this job was continually run it would eventually crash the application, by stacking up blocked threads.
After executing, I checked several Boomi logging layers such as: container logs, process logs, and enhanced data sections of process reporting for the above job. I was really not able to triage this incident. We need to turn to another tool to assist with capturing and alerting on this performance bottleneck event.
yCrash Diagnostics for Boomi
In comes yCrash, which captures the prolonged thread persistence in blocked state and alerts about it:
I immediately noticed the following alert from the yCrash, after logging into the console:
39 threads were executing com.buggyapp.blockedapp.AppObject.getSomething(AppObject.java:16). It can slow down transactions. Examine their stack trace.
I then clicked into the stack trace of the blocked threads reports to find out more detail about the incident. Below is the screen shown in yCrash:
yCrash first identified the one thread that looped infinitely/stuck and slept for each iteration. It showed the stack trace of that thread (refer the above figure). Subsequently, yCrash also reported all the 39 threads that are in BLOCKED state and their stack trace, indicating exact lines of code, where they are blocked.
Conclusion
We were able to simulate a condition within Dell Boomi where a process creates a permanent thread blocking situation, which persists even after the process is finished executing. The blocking incident is then caught by yCrash and triage/troubleshooting is able to be performed on the incident.
Share your Thoughts!