Troubleshooting Memory Leaks in Jenkins

Does your application behave like a Ferrari on start-up, then slowly degrade into a chugging old Morris? If your code leaks memory like a dripping tap, this may well be the problem.

A Jenkins memory leak is highly frustrating, and can cause costly delays. Jobs slow down, queues build up, the GUI becomes unresponsive. Just what you don’t need when you have a tight deadline for your next release.

Memory leaks occur when objects are retained beyond their useful lifespan, causing the garbage collector to work harder and harder to clear enough memory for incoming memory requests. 

In this article, we’ll look at the symptoms and possible causes of memory leaks in Jenkins. We’ll also include a worked example of how to use diagnostic tools to find the root cause of the problem, and see how to use this to determine the most appropriate fix.

Common Symptoms of a Jenkins Memory Leak

When an application retains unnecessary memory, the heap size gradually increases, causing frequent resource-hungry GC cycles, degrading overall performance.

If Jenkins is suffering from a leak, we may observe:

• Builds taking longer than usual to run;
• Queues building up;
• Jobs disappearing due to an agent crashing;
• GUI functions becoming less and less responsive over time;
• Temporary recovery, followed by recurrence of the problem;
• Individual tasks failing to complete due to a thread that crashed with an OutOfMemoryError.

 Restarting Jenkins may sometimes temporarily solve the problem. However, since Jenkins is designed to continue build tasks after a restart,  it’s possible that the problem can instantly recur as the job continues where it left off..

Memory leaks drain resources even when the heap and metaspace configurations are optimal, since leaks continue to consume memory until one or more threads crash with an OutOfMemoryError.

Understanding the Root Causes of Memory Leaks in Jenkins

If the heap is configured correctly, the garbage collector (GC) should always be able to clear enough memory for new requests by cleaning out objects that are no longer needed. If it can’t, it means the application contains program code that retains references to memory it no longer needs. To understand how GC identifies objects that can safely be discarded, read How Garbage Collection Works.

If we look at a heap dump of an application with a memory leak, we’re likely to see one of the following:

• A collection that keeps growing indefinitely;
• A proliferation of similar objects.

Not all memory leaks affect the heap. The heap is only one of the JVM’s runtime memory pools. This video, JVM explained in 10 minutes, describes Java memory in detail. 

Fig: JVM Memory Pools

In this article, we’ll concentrate on the two memory areas most likely to be affected by Jenkins memory leaks:

• The Heap, highlighted in red in the diagram above;
• The Metaspace, highlighted in yellow in the diagram.

The heap is a common storage area for objects belonging to all classes that make up the application. Growing collections or objects retained from previous tasks are the main causes of memory leaks in the heap.

The metaspace (or the PermGen in older versions of Java) stores information about the classes themselves. Metaspace leaks occur when too many classes are loaded into memory, for example, a large number of Groovy scripts.

For a full coverage of memory issues in the JVM, and the memory pools affected, see 9 Types of OutOfMemoryErrors in Java.

Some of the coding errors that may cause memory leaks are listed below. Whether you’re writing pipeline scripts in Groovy, or developing Jenkins plugins, you need to keep these in mind.

• Large static collections. 
• Event listeners that are never deregistered.
• Executors or threads retaining references, especially ThreadLocal variables.
• Caches without an eviction policy.
• Variables created within loops without a valid termination condition.

Issues that have caused memory leaks in Jenkins in the past include:

• Groovy variables defined in too broad a scope.
• Threads retaining variables too long.
• Agents reconnecting frequently.
• Caches growing without limits.
• Complex Groovy scripts creating too many dynamic classes.
• A Plugin update without a restart occasionally fails to unload old classes.

The first four fill up the heap by retaining too many objects. The last two affect the metaspace, because too many classes have been created.

Objects that are never released for GC are the cause of true memory leaks. However, memory-hogging practices can behave just like a leak, although they usually resolve themselves once a task is complete. In Jenkins, well-known memory hogs include:

• Long-running scripts.
• Huge Groovy variables in pipelines, e.g. loading entire files into memory.
• A large build history, especially if builds contain action items or huge test results.
• Excessive build queues.
• Very long console logs.

How to Diagnose Memory Leaks in Jenkins

Let’s look at the artifacts we need to trace a memory leak problem to its root cause, and a worked example showing how to use them.

1. Sample Code Showing a Memory Leak

For this exercise, we took some buggy code and incorporated it into a copy of a sample plugin taken from the Jenkins Plugin Development Tutorial. This causes a memory leak whenever we activate the plugin.

Here is the code.

import java.util.HashMap;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
/**
 * 
 * @author Ram Lakshmanan
 */
public class MapManager {
// This HashMap grows indefinitely
    HashMap<Object, Object> myMap = new HashMap<>();
    ScheduledExecutorService scheduler =
               Executors.newSingleThreadScheduledExecutor();
    long counter = 0;
// If this method is called by the plugin, it will add a String
// to the HashMap every 10 milliseconds
    public void grow() {
	  scheduler.scheduleAtFixedRate(() -> {
	    myMap.put("key" + counter, 
            "Large stringgggggggggggggggggggggggggggg"+
            "ggggggggggggggggggggggggggggggggggggggggggggggggggggg" +
            counter);
            counter++;			
            }, 0, 10, TimeUnit.MILLISECONDS);
	}
}

We used the ScheduledExecutorService to run the leaky code. This lets us add an entry to the HashMap at fixed intervals. It also makes it easy to adjust the code to cause either a fast-acting leak, or a slow leak.

2. The Jenkins Log

With any software issue, we should always begin by looking at the logs. Depending on the operating system and how Jenkins was installed, there are several ways of viewing Jenkins logs.

Since Jenkins has many threads running concurrently, an OutOfMemoryError won’t necessarily crash the system: other tasks may continue to run. If the logs contain an OutOfMemoryError, it’s a valuable clue as to where to start with troubleshooting. Look for the errors listed below. The links give more information about how to solve each of these errors.

• OutOfMemoryError: Java Heap Space
• OutOfMemoryError: GC Overhead Limit Exceeded
• OutOfMemoryError: Metaspace

When we ran the plugin with the buggy code on a fast timer, the thread ran out of memory, and the Jenkins log contained the following:

Fig: Section of the Jenkins Log Showing OutOfMemoryError

Caution: The thread that throws the error need not necessarily be the one causing the memory leak. Once the GC is unable to clear memory, the next instruction to request memory for a new variable will crash with an OutOfMemoryError.

Memory leaks degrade performance long before throwing OutOfMemoryErrors, however, so we often don’t have these clues to work with.

3. GC Logs

The best way to tell whether we have a memory leak is to make sure GC logging is enabled, and submit the logs to a GC Log Analyzer tool. In our examples, we’ll use GCeasy to analyze the logs. GCeasy provides, among other things, a graph of heap usage over time. The image below shows a comparison between GCeasy graphs of a healthy application and an application with a memory leak. 

Fig: Memory Leak Pattern vs Healthy GC Pattern: Graphs by GCeasy 

The red markers represent full GC events. Notice that in the healthy GC pattern, heap usage increases as the program carries out its tasks, but the GC is always able to bring it back to a similar level. In the memory leak pattern, although memory is reduced each time the GC runs, it’s never brought back to the same level, and the bottom line keeps increasing as the leak consumes more and more memory. GC events run more frequently as time goes on, trying to clear enough memory for other tasks.

Look at the bottom line: if it’s increasing in spite of GC events, it’s likely to be a leak. Also look for GC events running more and more frequently over time.

The GC report may show a warning like the one below if GC events are running too often:

Fig: GCeasy Warning Message

In Jenkins, the pattern may not be this clear-cut, since memory usage varies depending on what jobs are running. Also, Jenkins usually runs with the G1GC algorithm, which relies less on full GC cycles.  When we ran the bug using the long timer, our heap usage looked like this:

Fig: Heap Usage with Slow Jenkins Memory Leak

Memory spikes as jobs are run, but the GC is able to clear the memory after the job is finished. However, because of the memory leak, the bottom heap usage level is gradually increasing over time, and GC events are more frequent.

When we sped up the timer, we saw an acute memory leak, ending in a sudden drop in heap usage as the thread terminated with an OutOfMemoryError. At this point, all variables belonging to the thread were released for GC, so the heap recovered. This example was actually run with the plugin invoked three times, creating three separate memory leaks.

Fig: Acute Jenkins Memory Leak

Notice that GC events run back to back just before the thread crashes. Other threads continued running normally once the culprit threads were terminated.

This pattern is typical of heap-related memory leaks. If the leak had been in the metaspace, we would have seen a different GC pattern. When the metaspace fills up due to either under-configuration or a metaspace leak, GC events run back to back trying to release memory without success. The heap usage is unaffected and may even be very low. The image below is taken from the logs of an application with a metaspace leak.

Fig: Metaspace Leak Pattern

4. The Heap Dump

Once we’ve established from the GC logs that we have a memory leak, and determined whether it’s in the heap or in the metaspace, our next step is to capture a heap dump. This is a snapshot of the heap’s contents at a given moment. Since it’s a large binary file, we’ll need a heap dump analyzer such as HeapHero or Eclipse MAT to explore its contents. For this example, we used HeapHero.

If the leak affected the heap, the most useful section of the report is the Largest Object section. If the leak involves an object that’s continually growing, like the HashMap in our example, we will be likely to find the offending object among the three or four items at the top of the Largest Objects table.

We took the image below from the HeapHero report for our buggy plugin example.

Fig: HeapHero Largest Object Report

Objects are sorted inversely by their retained heap size (the space occupied by this object and all objects that descend from it in the dominator tree). Here, the top object is a Thread, which is often the case. Clicking the arrow to the left of it shows its child objects. We can drill down until we find the actual object that is using most of the space: in this case, an object of class MapManager. 

This is what we would expect to find when running the sample bug. The important thing when working with Jenkins is to note the package name of the culprit. As we’ll see later, the package is an important clue.

Another useful thing to look at when troubleshooting Thread objects is the actual values of the variables, since they help us trace the problem back to its stack trace. When we click on the Thread object, the right-hand pane of the report shows the values:

Fig: Values Within Thread Object

Using the thread name, we can explore the thread information. Further down the report, there is a Threads section, which lets us explore threads. In this section of the report, we find the current stack trace:

Fig: Stack Trace

This is highly useful when debugging our own plugins, or when putting in a bug report.

For more information on how to use heap dumps to identify memory issues, you may like to watch How to Analyze a Heap Dump Fast.

5. Metaspace Memory Leaks

If the leak had been in the metaspace, we would use a different troubleshooting technique. Also within the HeapHero report is the class histogram, which lists all loaded classes, ordered by the number of objects in each class. Using this as a starting point, watch this video on troubleshooting metaspace issues.

Fig: HeapHero Class Histogram

How to Fix Memory Leaks in Jenkins Core and Plugins

Once we know what classes are leaking, we can use this information to find the probable cause of the problem and decide on an appropriate fix.

The table below shows the most likely causes of Jenkins memory leaks. For each, it shows the names of the packages that we’re most likely to see in the problem areas of the heap dump, and a suggested fix.

Jenkins Memory Leaks: Causes and Fixes

These are true memory leaks.

Other issues that can cause similar symptoms are detailed below.

Jenkins Excess Memory Usage: Causes and Fixes

Can We Predict and Prevent Jenkins Memory Leaks?

In almost all cases, we can identify memory leaks before they actually cause problems in the live system.

Monitoring the garbage collection logs regularly is the best way to proactively prevent memory leaks. Check for GC patterns, making sure the heap usage after GC is not steadily rising.

GCeasy also shows key performance indicators such as throughput and latency. If throughput decreases or latency increases over time, it’s an indication that the GC is struggling.

Another useful tactic is to set up automatic monitoring with yCrash, which alerts us if it detects performance degradation.

Conclusion

Memory leaks may start as a minor irritation, but they worsen over time, causing frustration, missed deadlines, and system crashes.

Regular monitoring can prevent a minor problem from developing into a major outage.