JVM Tuning Best Practices for Jenkins

Critical DevOps workflows need to be backed by stable and reliable CI/CD software. Many organizations choose Jenkins for this very reason. It’s renowned for high availability and cost-effectiveness – provided it’s managed correctly.

Jenkins JVM tuning is an important part of system management, but it’s surprising how often administrators get it wrong. Why?

Let’s look at a few reasons.

• They don’t set clear goals;
• They base decisions on generalized tuning suggestions. Each installation is different, and tuning must be the result of reliable diagnostics.
• They fail to understand the JVM memory model, and therefore don’t include non-heap memory in tuning decisions;
• They retain settings taken from a different server or JVM platform;
• They don’t monitor and fine-tune the results.

Now we know what not to do; let’s look at how we should approach Jenkins tuning.

Why Is Jenkins JVM Tuning Important?

The direct results of inefficient JVM tuning are:

• Increased footprint: the application uses more memory and/or CPU time than it actually needs;
• Increased latency: Latency is the time when application threads are paused during critical stop-the-world garbage collection tasks;
• Decreased throughput: Throughput is the percentage of time spent on carrying out actual work compared to the time spent carrying out JVM internal tasks.
• Decreased Stability: The application is prone to system crashes, such as OutOfMemory errors, resulting in failed tasks or system downtime.

Can we quantify actual costs for these issues in Jenkins? We’d need to take into account:

• CI/CD pipelines taking longer, resulting in potentially delayed releases. For an excellent article on how to put a price tag on this disadvantage, see Economic Cost of Delays in Software Delivery.
• Increased infrastructure costs, especially when Jenkins is hosted in the cloud. Excess memory and CPU requirements translate to higher bills. Slow applications require more time, and therefore incur more costs.
• Reduced developer efficiency due to delays and frustration;
• DevOps time spent troubleshooting system glitches.

All this can add up to your organization spending a lot of money, simply because Jenkins is not running efficiently.

How to Configure JVM Options for Jenkins

Java performance tuning is controlled by command line options when the JVM is invoked. How do we set them in Jenkins?

Remember, Jenkins consists of a controller, which communicates with the user and manages tasks, as well as one or more agents, which actually run the tasks. Both of these are JVMs and need to be tuned.

The process of setting JVM options is different, depending on the underlying operating system, and whether Jenkins was installed as a service or as a stand-alone application. The table below summarizes how to set the options for each scenario, with links to the Jenkins documentation for more detailed information.

How to Size the Jenkins JVM Heap Correctly

Sizing the heap is easy. If you’re having JVM problems, just increase the value of the command line argument -Xmx, right?

Wrong – but it’s amazing how many systems administrators try to tune it this way. 

We’ll never get the best performance from any Java application if we don’t understand the JVM memory model. The heap is only one of the JVM’s many memory areas, although yes, it’s usually the biggest and has the most effect on performance. We can visualize the JVM like this:

Fig: JVM Memory Model

The JVM uses heap memory to store all objects. Most, but not all, garbage collection (GC) algorithms split the heap into the Young Generation (newly-created objects) and the Old Generation (objects that have survived several cycles of GC). They do this because most applications create a lot of short-lived objects, so it makes sense to place new objects in a smaller space that is cleaned regularly.

Heap memory is managed by the garbage collector, and its initial and maximum size are controlled by the command-line arguments -Xms and -Xmx, respectively.

Native memory, on the other hand, is managed by the operating system. As you can see, native memory is part of the JVM’s overall footprint, but it’s not included in the memory reserved by the -Xmx argument. We can tune the requirements of some of the native memory areas: the metaspace, the thread area, and the direct buffer area. We’ll talk more about those in a later section. For more information on JVM memory, see JVM Explained in 10 Minutes.

When we decide on the best heap size for an application, we always have to remember that, in addition to the heap, we need memory for other things:

• Native JVM memory;
• Memory required by the operating system for its own use;
• Memory for any other applications that we plan to run in the same device or container. In Jenkins, this is particularly important if we’re running agents on the same device as the controller. This is often the case in smaller organizations.

The Jenkins documentation recommends that we configure heap memory as not more than:

• 50% of total container memory when running in a container or
• 60% of total device memory when running on a freestanding device.

Jenkins documentation tells us that exceeding this value makes the application unstable.

Obviously, if we’re running other applications on the same platform, we’d have to subtract the memory they use before we do our calculation.

Jenkins also recommends scaling horizontally rather than vertically when our workload increases, and this value is no longer sufficient. In other words, we should consider adding more Jenkins instances rather than throwing more and more resources at the same instance.

You may like to read this article: Sizing Your Heap Correctly. Important concepts it emphasizes include:

• The heap shouldn’t be configured too large, because unused heap is simply wasted. Cloud costs are calculated on the amount of memory an application reserves, not on the amount it actually uses.
• It shouldn’t be configured too small, because GC then has to work overtime to service incoming requests. This uses more CPU time and therefore increases costs. There is also the danger of the system crashing with OutOfMemoryErrors.
• The only way to accurately configure the heap is to:
  • Use a test system, beginning with the heap configured fairly high;
  • Monitor the GC logs to measure performance under a typical workload;
  • Adjust the heap size and re-monitor until we find the sweet spot where performance is not affected, and memory is not wasted.

Through the rest of this article, we’ll use a worked example from a small Jenkins test system.

The instance is running under Ubuntu on a machine with 4GB of memory. Some of this memory is retained by the hardware for uses such as on-board graphics, so Linux only ‘sees’ 3.3GB.

$ free -h                  
         total     used     free   shared  buff/cache   available  
Mem:     3.3Gi    402Mi    1.8Gi     28Mi       1.1Gi       2.6Gi  
Swap:    4.0Gi    172Mi    3.8Gi 

We’ll assume no other applications are running on this device, so our starting point is 60% of 3.3 GB. 

Jenkins is running as a service on OpenJDK version 21. 

Modern Java HotSpot JVMs have four settings that affect the heap size:

We used sudo systemctl edit jenkins to set the JAVA_OPTS environment variable. Since we want to use 60% of available memory as our starting point, we used percentages rather than fixed values. We also included the option to enable GC logging, so we can monitor the results as we change tuning parameters. At this early stage, we didn’t set any other options.

Environment="JAVA_OPTS=-Xlog:gc*:file=/var/log/jenkins/gc.log -XX:MinRAMPercentage=60 -XX:MaxRAMPercentage=60"

We then ran Jenkins for around 30 minutes under a normal workload. In real life, you’d run it for much longer than this, making sure the period includes both peak and off-peak times. 

At the end of this time, we stopped Jenkins and loaded the GC logs into GCeasy for analysis. These diagnostics form a baseline for further tuning. Here is an analysis of memory usage taken from the GCeasy report:

Fig: Allocated vs Peak Memory Usage

As you can see, peak usage is considerably less than the value we allocated, so we’re effectively wasting space.

Another useful chart from GCeasy is the heap usage over time graph, taken before GC events.

Fig: Heap Usage Before Garbage Collection

The peak usage is a little less than 800 MB. As a rule of thumb, we usually get good results by setting the heap size to about 85-90% of its peak value. For our next test, we’ll set it to 750 MB and monitor results.

GCeasy also includes an analysis of key performance indicators:

Fig: GCeasy Key Performance Indicators

This is invaluable, because we can monitor KPI after each tuning change to see what effect it has on performance.

Before we decide on the final size, we need to do some further tuning. In particular, GC efficiency has a big impact on heap usage, so we need to find the best configuration for our unique instance.

Which Jenkins JVM Arguments Should You Keep or Remove?

When you’re tuning a Java JVM, the first thing to remember is the KISS principle – keep it simple.  The JVM allows more than six hundred different command line arguments. They’re not fixed: they change depending on the platform, the version, and the GC algorithm. It’s tempting to find a new, exciting-sounding argument, and enable it hoping it will improve performance. It probably won’t; in fact, it’s highly likely it will make things worse.

Newer Java versions and GC algorithms are largely self-tuning: they make decisions at runtime depending on the environment, and automatically adjust tuning as the workload changes. Setting an argument you don’t need often hinders this process, and performance drops. JVM arguments should only be set when you have good reasons based on actual diagnostic data.

If you’ve upgraded either Jenkins or your Java version, or moved your instance to a new server, never keep old JVM options: start from scratch. It’s highly likely that outdated options or tuning set for a different environment will adversely affect runtime efficiency.

So which arguments should you remove? All of them.

Having said that, there are a few options you do need to understand. Let’s look at these by category.

1. Recommended JVM Options for Jenkins

• Enable GC Logging. GC logs are invaluable for troubleshooting and performance tuning, and they add almost no overhead to the application. For Java 9 and above, the example below is usually sufficient. For earlier versions of Java, and for more logging options, see GC Logging Best Practices.

-Xlog:gc*:file=/var/log/jenkins/gc.log

• Obtain a heap dump if an OutOfMemoryError occurs. We need to enable the option and set a path to a directory where the JVM creates the dump, as shown below.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/jenkins

• Run a diagnostic script if an OutOfMemoryError occurs. You can write your own script to extract useful artifacts about the JVM and its environment, or use the free open-source yc-360 script. It’s fine to configure both the script and the heap dump to be triggered.

-XX:OnOutOfMemoryError="/usr/local/bin/oom-handler-yc-360.sh

• Enable String Deduplication. This option is available from Java 9 onwards, and can save a huge chunk of JVM memory. It does add a small CPU overhead to the application, so for critical applications you may like to test performance with and without this switch.  

-XX:+UseStringDeduplication

• Set overall default network timeouts. Application code frequently communicates with external services, such as REST or JDBC. Developers don’t always set reasonable timeouts on these calls, and they can sometimes hang Jenkins. These options prevent network calls from waiting indefinitely.

Dsun.net.client.defaultConnectTimeout -Dsun.net.client.defaultReadTimeout

2. Jenkins Garbage Collection (GC) Tuning

Modern versions of Jenkins running under newer JVM versions need very little GC tuning. If you’re not familiar with how GC works, I’d suggest reading this article before you change any configurations: What is Garbage Collection? It explains different GC algorithms and has links to detailed tuning guides for each one.

The first step is to choose the right GC algorithm for your situation. In almost all cases, Jenkins works well with the default algorithm. Before Java 9, this was Parallel GC, but as of the time of writing, all subsequent Oracle and OpenJDK versions default to G1GC. G1 is a good all-round option, and we only need to change it when we either need extremely low latency or have a very large heap (greater than 32 GB). 

Jenkins tasks are better tuned for throughput rather than latency, so G1GC is ideal. For very large installations, it may be worth experimenting with either ZGC or Shenandoah to see if they give better performance.

G1 has an option to set the preferred maximum pause time:   -XX:MaxGCPauseMillis. Since G1 is largely self-tuning around this parameter, it’s often the only parameter used for this algorithm. The default is 200 ms. Since Jenkins is not a latency-critical system, this default is usually fine, and we wouldn’t include this switch.

G1 GC splits the heap into equal-sized regions, designating some for young generation (YG) and some for old generation (OG). If the region size is not configured, the JVM creates 32 equal-sized regions. 

It has several different collection phases:

• Young: Application threads are briefly stopped while a YG region is cleaned;
• Concurrent marking: This is carried out in parallel with application threads in the background;
• Mixed: Both YG and OG regions that have been identified by the marking phase as having a high percentage of garbage are cleaned;
• Full: All regions are cleaned when not enough space is reclaimed by other phases. If possible, we need to avoid full GC events, as they are time-consuming.

We’d usually start with the G1 defaults, as we did in the example where we set the heap size, then monitor the GC logs to see if any tuning is needed. We need to look at what kind of GC events are occurring, and what’s triggering them.

Here are a couple of tables that show the most common causes of GC events. 

For Normal GC Operations

This table shows triggers that, according to the Oracle documentation, are normal operations:

For Possible Causes for Concern

This table shows triggers that may be causes for concern:

Oracle describes normal phases of G1 as:

• Young collections 
• Mixed collections 
• Concurrent marking 

The four causes shown in the first table fall into that category. We don’t need to worry about them, unless they are happening too frequently or taking too long. Let’s look at the GCeasy chart of GC causes for our example.

Fig: Analysis of GC Triggers

We’d use this as the starting point to see if GC needs tuning. Looking at each in turn:

• G1 Evacuation Pause: This is normal, and we don’t need to worry about it unless the pauses are too frequent or take too long. If they are taking too much time, we could consider increasing the heap size. As it is, they look fine.
• GCLocker Initiated GC: This is unusual, but it simply means that a GC cycle was blocked because native code was in a critical phase, and was then initiated as soon as the native code completed. Seeing this occasionally is not a problem, but if it happens frequently, we could look at whether a Jenkins plugin is not working efficiently.
• Metadata GC threshold: GC occurred because the metadata has filled up to the point where the GC attempts to clear it. We’ll talk more about this in a later section.
• G1 Humongous Allocation: If any object is bigger than 50% of the region size, it’s counted as being Humongous. The GC places each humongous object in one or more adjacent regions. These are then referred to as Humongous Regions. No other objects are included in the same region. This makes memory usage less efficient, as space may be wasted. Also, humongous regions aren’t collected in YG cycles, so they may occupy memory for longer than necessary.  With a smallish heap, an object doesn’t have to be all that big to be counted as humongous. If we have a 2GB heap, any object greater than (2GB / 32) /2 is a humongous object. This equates to approximately 32MB. Many objects in Jenkins exceed this, so it’s worth tuning the region size to be larger.

Looking at the GC causes in our example, GC is generally working well, and the only change we should make based on current evidence is to increase the G1 region size. Fewer objects will then be classified as humongous, reducing GC overhead.  We added the following JVM option:

-XX:G1HeapRegionSize=4m

With GC tuning, as with adjusting the heap size, we should follow an iterative process: change what’s obviously necessary, run under a test load, analyze performance diagnostics from GC logs, and, if necessary, repeat.

G1GC has several options available for fine-tuning. On the whole, we seldom need them with Jenkins. If your diagnostics show unhealthy GC causes, see the G1GC Tuning Guide, which goes deeper into GC triggers and suggested tuning adjustments.

3. How to Tune Jenkins Metaspace

The metaspace (or permgen in Java versions 7 and below) holds class metadata, including definitions of the class’s fields and methods and its class hierarchy. Each class loader retains only one set of metadata for any class it holds, and the information remains there until the class loader becomes eligible for GC. 

Jenkins is likely to need plenty of metaspace because:

• Jenkins plugin-related architecture allows us to add and remove features as and when we need them. Each plugin has its own class loader because different plugins may depend on different versions of software libraries. This means more than one copy of a class library may be loaded at runtime.
• Jenkins Groovy Scripting creates dynamic classes on the fly. Metaspace requirements therefore differ widely between Jenkins instances, depending on the type of workload.

By default, the metaspace is allowed to grow indefinitely, requesting more native memory whenever it’s full. This is never ideal, because a memory leak in the metaspace would eventually result in either the device or the container freezing or crashing when there’s no longer enough memory for the operating system. We should always set the maximum metaspace.

Let’s relook at a section of a previous image that showed memory sizes:

Fig: JVM Memory Usage

The peak metaspace usage is 104.94 MB. We need to allow plenty of room for expansion, in case more plugins or Groovy scripts are added. We therefore set the cap as per the example below.

-XX:MaxMetaspaceSize=250m

Should we set a minimum metaspace size as well, as per the example below?

-XX:MetaspaceSize=150m

First, we need to understand that this is not a minimum size automatically reserved for the metaspace on startup.  It’s actually a directive to the JVM that when the metaspace reaches this size, GC should begin trying to clear metaspace. In Jenkins, which as we’ve seen is metaspace-heavy, it is, in fact, worth setting it to a figure slightly higher than the peak metaspace usage shown in the GC logs. This saves time by not forcing GC while the metaspace size is within its expected limits.

4. How to Configure Jenkins Thread Stack Size

We can’t directly set the size of the thread stack. It’s managed by the operating system. If it tries to expand beyond the available resources, Jenkins will throw the following error: java.lang.OutOfMemoryError: unable to create native thread.

The only thing we can control is the size of the area allocated to each thread for its stack. For example, to set the maximum stack size of stacks to 512 KB:

-Xss512k

If an application thread’s stack needs to grow larger than the stack size limit, the JVM throws a StackOverflowError. Most of the time, this is caused by buggy code, but some applications do need a bigger stack, so we don’t want to set -Xss too low.

On the other hand, if an application has a huge number of threads, they can take up a lot of memory. If we have 50 thousand threads with a size of 1 MB each, the stack would occupy 50 GB. Unless we have plenty of RAM, that will crash the system. We may therefore sometimes set the stack size smaller so the thread space requires less RAM. 

Should we set this in Jenkins? Only if the diagnostics tell us that the default setting (usually 1MB) is likely to become problematic.

While Jenkins is running under a normal workload, capture a thread dump. You can then use a thread dump analyzer such as fastThread to extract diagnostics from it. What we need to know is the number of threads and the maximum thread stack size.

 Here are the relevant fastThread charts taken from our sample Jenkins instance.

The first shows the number of threads currently created in the JVM:

Fig: Number of Threads Reported by fastThread

The second gives a breakdown by size category of the threads:

Fig: Breakdown by Stack Trace Size

In this Jenkins instance, the number of threads is not abnormally large, and the stack sizes are all in the small or medium category. This indicates that there’s no need at the moment to change the value of -Xss.

This is something we should monitor from time to time. Changing workloads can affect the thread space size dramatically:

• Jenkins creates threads as needed to deal with build requests and user commands;
• The type of build affects the stack size. Stack size can be large if the workload includes:
  • Deeply nested function calls.
  • Recursive code in plugins or pipeline-related logic.
  • Extensive nested closures.
  • Complex Groovy processing.
  • Plugins that perform synchronous operations for extended periods.

Unfortunately, we can’t calculate the current size of the thread space from the number of threads and their number of frames. Too many run-time variables affect this calculation. The only way to accurately see the size of native memory areas is to enable native memory tracking on the Jenkins command line:

-XX:NativeMemoryTracking=summary

We would not normally keep this option permanently in production, since it adds overhead to the application, but it’s useful for diagnostics.

While Jenkins is running, we can then dump the tracking data to a file using jcmd.

jcmd <PID> VM.native_memory summary > jenkinsnmt.txt

This produces a text file. However, it can be tedious to analyze. If we load the file into Gceasy,  it summarizes the data into useful charts:

Fig: Gceasy NMT Summary

The thread space is small: only 5.46 MB. We can therefore leave out the -Xss parameter, since the default is working fine.

5. Should You Configure Jenkins Direct Buffer Size?

Java NIO allows us to create buffers in native memory for more efficient I/O transfers. The JVM allows us to configure the maximum size of the direct buffer area if we need to.

Jenkins uses direct buffers, but it’s almost unheard of for it to exceed the default size. We would only need to increase the maximum if:

• We’re seeing OutOfMemory errors related to direct buffers;
• Native memory analysis shows a very large direct buffer area.

6. How to Disable Explicit GC in Jenkins

Way back when Java was new, the books advised us to use the System.gc() command to ensure garbage was collected on time. This is known as Explicit GC.

This was not good advice. System.gc() requests the GC to initiate a full GC event, which is time-consuming and is seldom necessary. The GC algorithms are very efficient and know exactly when a full GC is needed.

Outside of benchmarking or specialized situations such as synchronizing remote and local GC, we should never use explicit GC. Unfortunately, Jenkins and its plugins include a large number of third-party libraries, and some of them may include this instruction. If we’re seeing ‘Explicit GC’ as a GC cause in the logs, we know this is a problem in our instance. Fortunately, the JVM allows us to disable this feature using the following option:

-XX:+DisableExplicitGC

We recommend always keeping this set in Jenkins.

Sample Jenkins JVM Tuning Configuration

We applied these principles to our sample Jenkins system, and set the command line options as follows:

Environment="JAVA_OPTS=\-Xlog:gc*:file=/var/log/jenkins/gc.log \-Xms750m \-Xmx750m \-XX:+HeapDumpOnOutOfMemoryError \-XX:HeapDumpPath=/home/jill/shared \-XX:MaxMetaspaceSize=250m \-XX:MetaspaceSize=150m \-XX:G1HeapRegionSize=4m \-XX:+UseStringDeduplication \-XX:+DisableExplicitGC"

How to Validate Jenkins JVM Tuning

As we said earlier, JVM tuning should be an iteration of adjusting parameters, running for a test period under a normal workload, and examining diagnostic reports to see the effect of the changes.

Once we’d changed the JVM configuration for our test Jenkins instance, we reran it under the same workload for a similar period, then loaded the GC logs into GCeasy.

We can now look at the two sets of reports together to see if our changes have improved performance and haven’t introduced any new problems.

In this case, performance was already excellent, but since we lowered the heap size, we need to make sure the KPI are still acceptable.

The image below shows the KPI from the original run side by side with the new test.

Fig: Comparing KPI

Throughput has actually improved slightly after reducing the heap size, which confirms our tuning choices were good. CPU time has reduced on the second run, and latency is negligible on both runs. The maximum pause time has increased on the second run, but it’s still well below acceptable limits. We have fewer GC events in the second run.

Let’s now compare the GC causes between the two tests.

Fig: Comparing Causes of GC

We have successfully eliminated the unhealthy GC causes, and we’re left with only G1 Evacuation Pause, which is normal.

If memory constraints are very tight in the container, we could try reducing the heap size further and retest it to see whether it has adversely affected performance. If it has, we can set the heap back to the size that gave the best throughput.

We then checked overall performance in both the JVM and the container using the yCrash tool. This utility analyzes a full range of diagnostics relating to both the JVM and its environment, and uses machine learning to identify any performance issues that could become problematic.

There are two ways to use yCrash:

• As an agent that runs in the background, sampling performance data from the JVM and the operating system. We recommend running this in production so we can identify whether the system is developing any problems or needs retuning.
• Via the free open-source yc-360 script, which we can run manually to capture a wide range of diagnostics. We can either interpret the diagnostics manually or load them into a yCrash server for analysis.

We used the yc-360 script. The image below shows yCrash’s diagnostic summary.

Fig: yCrash Summary

Since it hasn’t found any tuning-related issues, we conclude that our system is now healthy. It has, however, identified that a Jenkins thread is looping. On investigation, we found that this is normal: it’s the controller waiting for communication from a running agent.

In a real-life situation, we would need to look into the issue identified in the device, which indicates that the operating system is using more than 50% of the available memory. We would consider adding more RAM to the device or container.

Jenkins JVM Tuning Best Practices: Key Takeaways

Jenkins JVM tuning should never be a matter of guesswork. We should begin by setting the heap size to around 60% of the device’s memory, testing under a normal workload, and analyzing GC logs to identify areas that need further tuning. Thread dumps can also be helpful, and native memory tracking allows us to make sure overall memory usage is acceptable.

JVM tuning is never a one-off job: the system is likely to need retuning as workloads change. We recommend regular monitoring with yCrash to pick up any developing issues before they affect our CI/CD deadlines.