Troubleshooting Java OutOfMemoryError on ServiceNow Servers

Is your ServiceNow MID Server suffering from sluggish performance or abrupt shutdowns? You could be facing the common Java application challenge of an OutOfMemoryError. Don’t worry, though! This guide is designed to help you identify and fix this problem. We’ll provide you with expert strategies for pinpointing and tackling OutOfMemoryError within your ServiceNow MID Server setup, helping to secure and optimize your application’s performance. Are you prepared to tackle this issue directly? Join us as we explore the remedies!

Impact of OutOfMemoryError in Java Applications

OutOfMemoryError in Java applications can result in several adverse effects:

• Application Disruption: Similar to heap space exhaustion, when the JVM encounters OutOfMemoryError due to insufficient memory to allocate objects, the application’s execution is disrupted. This can lead to unexpected crashes or abrupt shutdowns, affecting user experience and system reliability.
• Resource Leaks: OutOfMemoryError can be caused by resource leaks, such as unclosed database connections or file handles. These leaks gradually consume system resources until the JVM runs out of memory, resulting in the error. Identifying and fixing these leaks is crucial for maintaining application stability and performance.
• Degraded Performance: When OutOfMemoryError occurs, the JVM may spend an excessive amount of time performing garbage collection to reclaim memory. This can lead to degraded application performance, increased response times, and potential timeouts, impacting overall system efficiency.
• Data Loss: In some cases, OutOfMemoryError can lead to data loss or corruption if critical operations fail to complete due to insufficient memory. This can have severe consequences, especially in transactional or data-intensive applications, where data integrity is paramount.
• User Frustration: Frequent occurrences of OutOfMemoryError can frustrate users and erode trust in the application. Users may experience disruptions in service, loss of unsaved data, or inability to complete tasks, leading to dissatisfaction and potentially driving them to seek alternative solutions.

Overall, addressing OutOfMemoryError is essential for ensuring the stability, reliability, and performance of Java applications, minimizing downtime, and preserving user satisfaction.

To simulate an OutOfMemoryError crash, you can write a Java program that consumes a large amount of memory. Here is an example program:

package com.test;

import java.util.HashMap;
import java.util.Map;
import java.util.logging.Logger;

public class OOMCrash {

    private static final Logger logger = Logger.getLogger(OOMCrash.class.getName());

    public static void main(String args[]) {
        Map<String, String> map = new HashMap<>();

        long counter = 0;
        try {
            while (true) {
                counter++;
                map.put("key-" + counter, "value-" + counter);

                if (counter % 1000 == 0) {
                    System.out.println("Added " + counter + " elements");
                    logger.info("Added " + counter + " elements");
                }
            }
        } catch (OutOfMemoryError e) {
            System.err.println("OutOfMemoryError occurred after adding " + counter + " elements.");
            logger.severe("OutOfMemoryError occurred after adding " + counter + " elements.");
            e.printStackTrace();
        }
    }
}

The OOMCrash Java class intentionally triggers an OutOfMemoryError, simulating a scenario where continuous memory allocation exhausts available resources. In large applications and servers like ServiceNow, such simulations are valuable for testing memory management, optimizing resources, detecting leaks, tuning performance, planning capacity, and ensuring robust error handling. They proactively address potential issues before affecting real-world usage, contributing to a more reliable and resilient application.

Java OutOfMemory In ServiceNow MID Server

Now let’s try to simulate this thread deadlock in the ServiceNow MID Server environment. Let’s create a JAR (Java Archive) file from the above program by issuing below command:

jar cf OOMCrash.jar OOMCrash.class

Once a JAR file is created, let’s upload and run this program in the ServiceNow MID Server as documented in the MID Server setup guide. This guide provides a detailed walkthrough on how to run a custom Java application in the ServiceNow MID Server infrastructure. It walkthrough following steps:

1. Creating a ServiceNow application
2. Installing MID Server in AWS EC2 instance
3. Configuring MID Server
4. Installing Java application with in MID Server
5. Running Java application from MID server

We strongly encourage you to check out the guide if you are not sure on how to run custom Java applications in ServiceNow MID server infrastructure.

OutOfMemory diagnosis in ServiceNow using yCrash

yCrash is an advanced monitoring tool engineered to detect performance bottlenecks and provide actionable recommendations within the ServiceNow environment. ServiceNow organizations rely on yCrash extensively for diagnosing and resolving performance issues.

When encountering outOfMemoryError scenarios on ServiceNow’s MID Server, yCrash diligently monitors the micro-metrics of the environment. It promptly identifies instances of outOfMemoryError and generates comprehensive reports on the dashboard, offering valuable insights into the impact on system performance. These insights empower ServiceNow administrators to implement effective measures to mitigate outOfMemoryError and optimize system efficiency.

The yCrash report has highlighted several issues with the JVM performance in the ServiceNow MID Server. One of the critical findings is related to the main thread of the application. The analysis indicates that the java.lang.Thread instance, specifically the main thread, is holding local variables that collectively consume 130,730,112 bytes of memory, which accounts for 98.95% of the total local variable memory usage. This excessive memory usage is traced back to a single instance of java.util.HashMap$Node[].

This issue can lead to significant memory retention, causing potential memory leaks or inefficient utilization of memory. As a result, the application may run out of memory or experience high latency due to the large memory consumption. To address this problem, a thorough review of the source code is necessary to understand why the HashMap is holding such a large amount of data. Once identified, optimizing the code to minimize the size of the data held in the HashMap or implementing periodic memory cleanup can help alleviate this issue. Additionally, fine-tuning the GC configuration to handle large memory chunks more effectively can also be beneficial.

Our machine learning (ML) algorithms have identified a critical issue within your application. Specifically, the main thread (java.lang.Thread @ 0xf8a60fc0) is retaining local variables with a total size of 130,730,112 bytes, accounting for 98.95% of the used heap space. This excessive memory usage is localized within an instance of java.util.HashMap$Node[], suggesting a potential memory leak. The report indicates a used heap size of 126.00 MB, with significant object and class counts, and highlights this as a potential cause for Java OutOfMemoryError on ServiceNow servers. By pinpointing this memory retention issue, the yCrash report enables us to take targeted actions to resolve memory inefficiencies and prevent application crashes.

The yCrash report provides detailed insights into memory retention within our application, specifically highlighting the main thread (java.lang.Thread @ 0xf8a60fc0) which occupies 124.67 MB or 98.95% of the retained heap. This concentration of memory usage in a single thread strongly indicates a memory leak, particularly within the java.util.HashMap$Node[] instance. By breaking down the shallow and retained heaps for various classes, the report helps us identify the disproportionate memory consumption and its sources. Knowing that the bulk of memory is being held by the main thread allows us to focus our debugging and optimization efforts on this specific area, directly addressing potential causes of OutOfMemoryError on ServiceNow servers. This focused insight is crucial for implementing effective memory management and improving the overall performance and reliability of the application.

Conclusion

In conclusion, yCrash’s analysis of OutOfMemory errors offers invaluable insights into memory-related challenges within the application. By scrutinizing heap size and thread behavior, particularly those with identical stack traces, the tool facilitates the identification of potential causes behind OutOfMemory occurrences. Moreover, its seamless integration with ServiceNow streamlines incident reporting and resolution workflows, thereby improving the efficiency of IT service management and fostering a more stable and optimized IT environment. To diagnose performance issues in your ServiceNow deployment using yCrash, you can register here.