Diagnosing and Resolving Garbage Collection Pauses in the ServiceNow MID server

Is your ServiceNow MID Server experiencing sudden slowdowns and unexpected pauses? 🤔 It might be encountering challenges related to garbage collection pauses, a common issue in Java applications. But don’t worry! In this article, we’ll guide you through diagnosing and resolving this problem. Get ready to uncover expert tips for troubleshooting garbage collection pauses in your ServiceNow MID Server environment, ensuring the stability and efficiency of your applications. Ready to tackle this issue? Let’s dive into the solution!

Impact of Garbage Collection Pauses

Garbage collection (GC) pauses in Java applications can cause several problems:

1. Slow Performance

Frequent or long GC pauses can slow down your application. This happens because the JVM stops the application briefly to free up memory, causing delays.

2. Freezes and Timeouts

Long GC pauses can make the application freeze for a moment. In systems like the ServiceNow MID server, long gc pauses may lead to timeouts and missed requests, affecting service reliability.

3. Higher Latency

Garbage collection can increase the time it takes for the application to respond. This added latency can affect user experience and slow down important tasks.

4. High Resource Usage

Intensive GC activity can consume significant CPU and memory resources. This occurs because the JVM expends considerable effort managing memory, placing additional load on system resources.

5. Hard to Diagnose

Intermittent GC pauses can be challenging to diagnose. These pauses may not occur frequently but can be difficult to pinpoint without detailed logs and thorough analysis.

6. Scalability Issues

Inefficient garbage collection management can restrict your application’s scalability. As memory handling becomes less effective, managing more requests or larger amounts of data can become increasingly difficult.

Simulating Garbage Collection Pauses in MID Server

The Java program given below simulates garbage collection pauses on any machine or container where it is launched:

import java.util.logging.Logger;
import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class GCPauseSimulator {
    private static final Logger LOGGER = Logger.getLogger(GCPauseSimulator.class.getName());
    private static final int ITERATIONS = 1000; // Number of iterations

    public static void main(String[] args) {
        LOGGER.info("Starting GC Pause Simulator...");

        List<String> garbageList = new ArrayList<>();
        Random random = new Random();
        
        long startTime = System.currentTimeMillis();

        for (int i = 0; i < ITERATIONS; i++) {
            long iterationStartTime = System.currentTimeMillis();
            LOGGER.info("Iteration " + (i + 1) + " of " + ITERATIONS + ": Creating garbage...");
            createGarbage(garbageList, random);

            long iterationEndTime = System.currentTimeMillis();
            LOGGER.info("Iteration " + (i + 1) + " of " + ITERATIONS + " completed in " + (iterationEndTime - iterationStartTime) + " ms");
        }

        long endTime = System.currentTimeMillis();
        LOGGER.info("GC Pause Simulator finished. Total execution time: " + (endTime - startTime) + " ms");
    }

    private static void createGarbage(List<String> garbageList, Random random) {
        int size = 500000; // Number of garbage elements to create
        for (int i = 0; i < size; i++) {
            garbageList.add(new String("Garbage" + random.nextInt(size)));
        }
        
        // Clean up to force GC
        if (garbageList.size() > 5 * size) {
            garbageList.clear();
            LOGGER.info("Garbage list cleared to force GC.");
        }
        
        // Log the completion of garbage creation
        LOGGER.info("Garbage creation complete. List size: " + garbageList.size());
    }
}

GCPauseSimulator: Simulating Garbage Collection in Java

The GCPauseSimulator program demonstrates how garbage collection (GC) pauses can affect Java applications by creating many temporary objects in a loop. It provides insights into memory management by recording the time taken for each iteration and the total execution time.

Execute the program using the default G1 garbage collector settings and examine the results:

It repeatedly generates 500,000 new strings, periodically clearing the list to trigger garbage collection. The program logs the time taken for each iteration and the overall execution time, providing detailed insights into how the system handles memory management. This simulation is especially useful for analyzing garbage collection behavior in environments like the MID Server, helping to identify performance bottlenecks. By monitoring GC pauses, developers can optimize application performance and reduce system delays effectively.

Diagnosing and Resolving Garbage Collection Pauses in the ServiceNow MID server 

Let’s run our application on the ServiceNow MID Server and see how GC pauses look. First, create a JAR file from the program using the command:

jar cf GCPauseSimulator.jar GCPauseSimulator.class

Once a JAR file is created, let’s upload and run this program in the ServiceNow MID Server as documented in the MID Server setup guide. This guide provides a detailed walkthrough on how to run a custom Java application in the ServiceNow MID Server infrastructure. It walkthrough following steps:

1. Creating a ServiceNow application
2. Installing MID Server in AWS EC2 instance
3. Configuring MID Server
4. Installing Java application with in MID Server
5. Running Java application from MID server

We strongly encourage you to check out the guide if you are not sure on how to run custom Java applications in ServiceNow MID server infrastructure.

yCrash’s Garbage collection pauses Analysis 

A GC pause lasting 1 second and 721 milliseconds was triggered by the G1_EVACUATION_PAUSE event, which occurs when the GC copies live objects between regions. This copying process triggers a Young GC if only Young Generation regions are involved, or a Mixed GC if both Young and Tenured regions are involved. Evacuation failures often result from over-tuning, so it’s recommended to simplify memory settings by using only -Xms, -Xmx, and a realistic pause time goal with -XX:MaxGCPauseMillis. If problems persist, consider increasing the heap size with -Xmx, adjusting -XX:InitiatingHeapOccupancyPercent appropriately, or increasing -XX:ConcGCThreads to speed up garbage collection.

The Key Performance Indicators section highlights critical metrics of the application’s performance. Throughput, representing the percentage of time spent running as opposed to GC, is 94.532%. In terms of latency, the average GC pause time is 61.5 milliseconds, with the maximum pause time being 181 milliseconds. The distribution of GC pauses shows that 89.29% of all pauses are under 100 milliseconds, while 10.71% fall between 100 and 200 milliseconds. These metrics provide insight into the efficiency and impact of garbage collection on application performance.

To further enhance our program’s effectiveness, we can switch to the Z Garbage Collector (ZGC). ZGC is designed for low-latency operation and excels at handling large memory workloads. By using ZGC and allocating sufficient memory, we can significantly reduce garbage collection pauses, improving the program’s overall performance. We will run the same program with ZGC to assess its impact and strive for better efficiency.

Run the program with the ZGC garbage collector and compare the results with the previous test:

When we ran the program using the G1 garbage collector, the elapsed time was 35,116 milliseconds. By switching to ZGC, the elapsed time improved to 29,177 milliseconds, and there were no noticeable GC pauses. This demonstrates the efficiency of ZGC in handling large memory workloads and reducing pause times. To further analyze garbage collection performance and logs, yCrash tools can be extremely helpful. They offer detailed insights into GC activity and assist in identifying and resolving performance bottlenecks, ensuring optimal application efficiency. You can see the reports for G1 and ZGC. 

Conclusion

In summary, yCrash’s analysis provides valuable insights into garbage collection pauses within the ServiceNow MID Server environment. The tool’s detailed GC log analysis helps identify potential causes of these pauses, while its comprehensive reporting features aid in understanding the root issues affecting performance. Additionally, yCrash offers visibility into memory usage patterns, facilitating the identification of inefficient memory allocations that contribute to GC pauses. Its integration with ServiceNow enhances incident reporting and resolution processes, improving overall IT service management efficiency and contributing to a more stable and optimized IT environment. If you want to diagnose performance problems in your ServiceNow MID server deployment using yCrash, you may register here.