Troubleshooting Memory Leak in ServiceNow Mid Server

Is your ServiceNow MID Server experiencing unexpected halts and silence? 🤔 It might be grappling with a different challenge: insufficient heap space, a frequent headache in JVM setups. But worry not! In this article, we’ll navigate you through diagnosing and resolving this issue. Prepare to discover unique strategies for troubleshooting insufficient heap space in your ServiceNow MID server environment, ensuring your applications run smoothly and efficiently. Are you ready to unravel this puzzle? Let’s embark on the journey together!

Impact of Insufficient Heap Space in Java Applications

Insufficient heap space can have several adverse effects on Java applications:

1. OutOfMemoryError: When the JVM runs out of heap space to allocate new objects, it throws an OutOfMemoryError. This error halts the execution of the application and can lead to unexpected crashes or shutdowns.

2. Performance Degradation: As the heap becomes increasingly full, the garbage collector (GC) needs to work harder to reclaim memory by identifying and removing unused objects. This increased GC activity can result in longer pauses and decreased application performance.

3. Increased Frequency of Full GC: When the heap is nearly full, the JVM may need to perform full garbage collections more frequently to reclaim memory. Full GCs are more time-consuming and disruptive compared to minor garbage collections, further impacting application responsiveness.

4. Application Instability: Insufficient heap space can lead to unpredictable behavior and instability in the application. OutOfMemoryErrors or excessive GC activity may occur sporadically, making it challenging to identify and diagnose the root cause of performance issues.

5. Reduced Scalability: Without enough heap space, the application may struggle to handle increasing workloads or concurrent user requests. This can limit the scalability of the application, preventing it from efficiently serving a growing number of users or processing large datasets.

Simulating Memory Leak problem

The Java program given below simulates Insufficient heap space problem on any machine/container in which it’s launched:

import java.util.ArrayList;
import java.util.List;

public class HeapSpaceExample {

    public static void main(String[] args) {
        try {
            List<byte[]> memoryConsumingList = new ArrayList<>();

            while (true) {
                memoryConsumingList.add(new byte[1024 * 1024]); 
            }
        } catch (OutOfMemoryError e) {
            System.out.println("OutOfMemoryError: Insufficient Heap Space");
            e.printStackTrace();
        }
    }
}

You can observe the ‘HeapSpaceExample’ class, which demonstrates a scenario where memory consumption leads to an OutOfMemoryError due to insufficient heap space. In the main method, the program continuously adds byte arrays of size 1 MB to an ArrayList, memoryConsumingList, within an infinite loop.

As the program consumes more memory by adding elements to the list, the heap space eventually becomes exhausted, resulting in an OutOfMemoryError.

Memory Leak In ServiceNow MID Server

Now let’s try to simulate this insufficient heap space  problem in the ServiceNow MID Server environment. Let’s create a JAR (Java Archive) file from the above program by issuing below command:

jar cf HeapSpaceExample.jar HeapSpaceExample.class

Once a JAR file is created, let’s upload and run this program in the ServiceNow MID Server as documented in the MID Server setup guide. This guide provides a detailed walkthrough on how to run a custom Java application in the ServiceNow MID Server infrastructure. It walkthrough following steps:

1. Creating a ServiceNow application
2. Installing MID Server in AWS EC2 instance
3. Configuring MID Server
4. Installing Java application with in MID Server
5. Running Java application from MID server

We strongly encourage you to check out the guide if you are not sure on how to run custom Java applications in ServiceNow MID server infrastructure.

yCrash’s Memory Leak diagnosis in ServiceNow

yCrash is a light-weight monitoring tool meticulously designed to identify performance bottlenecks and provide actionable recommendations within the ServiceNow environment. ServiceNow organizations heavily rely on yCrash for diagnosing and resolving performance issues.

When facing insufficient heap space scenarios on ServiceNow’s MID Server, yCrash actively monitors the micro-metrics of the environment. It promptly detects instances of insufficient heap space and generates comprehensive reports on the dashboard, offering valuable insights into the impact on system performance. These insights empower ServiceNow administrators/developers to implement effective measures to address heap space limitations and optimize system efficiency.

Here is the actual report generated by yCrash for the above program simulation. Let’s review the report.

You can notice that yCrash is reporting that there is a Memory Leak suspect in the application. Tool points that there is an application thread by name ‘main’. This thread has a local variable which is occupying 99.96% of overall memory.

Above is the excerpt from the ‘Largest Objects’ section of the report. This section shows large memory consuming objects in the application. You can notice that the ‘main’ thread is reported first in the table, occupying 99.96% of overall memory. This thread internally contains an ‘ArrayList’. This ‘ArrayList’ occupies 99.96% of memory. This is the ‘memoryConsumingList’ that we are creating in the above program and adding large byte array objects to it. 

As you can you infer, tool is exactly pointing the leaking objects (i.e. ArrayList), who is holding on to its reference (i.e. main Thread) and the size of those objects (i.e. 99.96%). This fine grained information will facilitate the ServiceNow developers/administrators to understand the origin of the problem in the application and instrument necessary fix. 

Conclusion

In summary, yCrash’s analysis of insufficient heap space provides valuable insights into memory-related issues within the application. The tool’s problem detection capabilities help identify potential causes of memory constraints, while its leak suspect view aids in pinpointing memory leaks and suspect objects. Additionally, yCrash offers visibility into the largest objects in the memory space, facilitating identification of memory-intensive elements contributing to heap space constraints. The integration with ServiceNow streamlines incident reporting and resolution processes, enhancing overall IT service management efficiency and contributing to a more stable and optimized IT environment. If you want to diagnose performance problems in your ServiceNow deployment using yCrash you may register here.