Deadlock On ServiceNow MID Server

Is your ServiceNow MID Server hitting roadblocks and going silent? 🤔 It might just be caught in a deadlock at the thread level. Fear not! In this post, we’ll guide you through simulating and breaking free from this deadlock. Brace yourself to unveil the insider tips for troubleshooting JVM deadlock issues in your ServiceNow MID server environment, safeguarding the stability and performance of your applications. Ready to unravel the mystery? Let’s dive in!

What can cause Deadlock in ServiceNow?

1. Resource Contention: Deadlock can occur when multiple threads compete for the same resources and each thread holds a resource while waiting for another resource that is held by a different thread, leading to a circular dependency and a deadlock situation.

2. Improper Resource Allocation: Incorrectly managing the allocation of resources, such as locks or database connections, can result in deadlocks. This may happen if resources are not acquired and released in a consistent and coordinated manner across threads.

3. Concurrency Control Issues: Inadequate concurrency control mechanisms, such as improper use of locks or semaphores, can introduce the potential for deadlock situations. For example, if two threads acquire locks in different orders, it can lead to a deadlock if each thread waits for the other to release its lock.

4. Synchronization Errors: Errors in thread synchronization, such as missed notifications or incorrect signaling, can also lead to deadlock. This may occur if threads are not properly synchronized when accessing shared resources or communicating with each other.

5. External Factors: External factors such as system failures or delays in resource availability can exacerbate deadlock situations by prolonging the time that threads hold onto resources, increasing the likelihood of a deadlock occurring.

Simulating Deadlock in MID Server

The Java program given below simulates Deadlock on any machine/container in which it is launched:

public class Deadlock {

    private static final Object lock1 = new Object();
    private static final Object lock2 = new Object();

    public static void main(String[] args) {
        start();
    }

    public static void start() {
        Thread threadA = new Thread(() -> {
            synchronized (lock1) {
                System.out.println("ThreadA acquired lock1");
                try {
                    Thread.sleep(1000); // Simulating some work
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

                synchronized (lock2) {
                    System.out.println("ThreadA acquired lock2");
                }
            }
        });

        Thread threadB = new Thread(() -> {
            synchronized (lock2) {
                System.out.println("ThreadB acquired lock2");
                try {
                    Thread.sleep(1000); // Simulating some work
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

                synchronized (lock1) {
                    System.out.println("ThreadB acquired lock1");
                }
            }
        });

        threadA.start();
        threadB.start();
    }
}

Above ‘Deadlock’ class is designed to intentionally create a deadlock situation as a demonstration. The ‘main()’ method serves as the entry point of the program, calling the start method to initiate the process. Within the start method, two threads called ‘threadA’ and ‘threadB’ are created, each attempting to acquire two shared locks, ‘lock1’ and ‘lock2’, but in the reverse order.

‘threadA’ acquires ‘lock1’ and then attempts to acquire ‘lock2’ after a simulated work delay of one second. Concurrently, ‘threadB’ acquires ‘lock2’ and then attempts to acquire ‘lock1’ after its one second own delay. The staged delays ensure that both threads have enough time to acquire their first lock before trying for the second.

The deadlock occurs when each thread has acquired one lock and is waiting to acquire the other lock, which is already held by the other thread. Neither thread can proceed because they are both waiting for the other to release a lock, creating a classic deadlock where no progress is possible, and both threads remain indefinitely blocked.

So in nutshell let’s visualize what happens when above program is executed:

1. threadA acquires lock1.
2. threadB acquires lock2.
3. threadA waits to acquire lock2.
4. threadB waits to acquire lock1.

Thus, both threads will end up in classic Deadlock.

Note: Above explanation may be a little tricky to understand, if you are reading it very fast. Read it slowly and correlate with the program.

Deadlock In ServiceNow MID Server

Now let’s try to simulate this thread deadlock in the ServiceNow MID Server environment. Let’s create a JAR (Java Archive) file from the above program by issuing below command:

jar cf Deadlock.jar Deadlock.class

Once a JAR file is created, let’s upload and run this program in the ServiceNow MID Server as documented in the MID Server setup guide. This guide provides a detailed walkthrough on how to run a custom Java application in the ServiceNow MID Server infrastructure. It walkthrough following steps:

1. Creating a ServiceNow application
2. Installing MID Server in AWS EC2 instance
3. Configuring MID Server
4. Installing Java application with in MID Server
5. Running Java application from MID server

We strongly encourage you to check out the guide if you are not sure on how to run custom Java applications in ServiceNow MID server infrastructure.

Deadlock diagnosis in ServiceNow using yCrash

yCrash is a light-weight monitoring tool designed to pinpoint performance bottlenecks and provide actionable recommendations within the ServiceNow environment. Infact ServiceNow organization itself internally uses yCrash to troubleshoot their performance problems. 

When the deadlock scenario occurred on ServiceNow’s MID Server, yCrash diligently monitored the micro-metrics of the ServiceNow environment. It swiftly identified the deadlock issue and generated a detailed report on the dashboard, facilitating effective resolution.

Above is the screenshot of yCrash summary report, highlighting the detection of deadlock issue. This view helps in understanding the context and severity of the deadlock situation, allowing users to quickly grasp the impact on system performance and stability. When you click on the ‘Here are the threads’ hyperlink in the summary, details about the threads that are causing the deadlock are rendered. Refer to the screenshot below:

You can notice yCrash pointing out threads ‘threadA’ and ‘threadB’ are causing the deadlock. yCrash also reports the stack trace of ‘threadA’ and ‘threadB’. From the stack trace you can notice ‘threadA‘ acquired the first lock and gets BLOCKED waiting for the second lock. On the other hand, ‘threadB’ acquired the second lock and got BLOCKED waiting for the first lock. Now based on this stacktrace we know the exact line of code that is causing the problem.

Equipped with information, developers can easily isolate the problem and instrument a fix in their code to terminate deadlock amongst the threads. Once the necessary corrections are made, the thread’s execution will be normalized, and the risk of a deadlock will be mitigated, ensuring the stability and performance of the application. To see the real yCrash report for this simulation, you can click here.

If you need to troubleshoot performance issues in your ServiceNow deployment using yCrash, feel free to sign up here to start using the free cloud-based tier.

Alternatively, if your security requirements as a large enterprise prevent you from sending data to the cloud, you can register here to access the on-premises installation of yCrash.

Conclusion

In conclusion, tackling deadlocks on your ServiceNow MID Server is crucial for maintaining smooth operations and optimal performance. By simulating and troubleshooting these issues effectively, you’re not just resolving immediate challenges, but also fortifying your infrastructure for future resilience. Remember to stay vigilant, utilize the right tools, and continuously monitor your environment to preemptively address any potential deadlocks. With these strategies in place, you’re well-equipped to conquer any thread-level hurdles that come your way. You can also read other articles that will help you troubleshoot performance issues on ServiceNow Server here. Happy troubleshooting!