Your Jenkins isn’t performing as it should. Builds take longer, the GUI hangs while processing requests, or you experience unexpected system crashes. It’s becoming more difficult to get important software upgrades out on time, and your job satisfaction has gone down to zero.
You’ve tried all the recommended fixes, and pored over diagnostics until you’re almost cross-eyed.
If this happens, it’s time to look at some of the less common causes of Jenkins production issues. In this article, we’ll look at three obscure problems that cause hard-to-find performance problems in Jenkins:
- Network lag;
- File connector leaks;
- Excess Direct Buffer Memory usage.
Jenkins Network Latency Between Controller and Agent
The Jenkins controller regularly communicates with its agents. If this communication is slow, unreliable, or unavailable, it’s known as network latency.
What Causes Network Latency in Jenkins?
Network lag occurs when communication between the Jenkins controller and an agent is delayed. This can be caused by:
- High latency.
- Packet loss.
- Network congestion.
- Unreliable connectivity.
How Network Latency Affects Jenkins Performance
The agent may still execute build commands, but performance in both the agent and the controller is affected. Network traffic is an essential part of the Jenkins workflow. The controller must stay in touch with the agent to:
- Dispatch work;
- Exchange status updates;
- Provide console output;
- Return results.
Common Symptoms of Network Latency in Jenkins
If the network is faulty, we’re likely to see some or all of the following symptoms:
- Builds take too long to start, or are slow to complete;
- Pipeline steps appear to stall;
- Console output is delayed and may arrive in irregular bursts;
- Agents intermittently disconnect and reconnect;
- Remote operations respond slowly;
- Builds fail with communication or channel-closure errors.
Memory, CPU, and disk resources appear healthy on diagnostic reports.
How to Diagnose Network Issues in Jenkins
The Internet has several excellent tutorials already that teach us how to find and resolve network problems. Rather than reinventing the wheel, in this article, I’ll just link to a step-by-step guide I’ve found particularly useful: Network Troubleshooting Steps.
In Java systems such as Jenkins, an additional tool that can speed up diagnostics is a thread dump analyzer, such as fastThread. As an example, we used the following Linux command to disrupt the Docker network while Jenkins Docker agents were busy:
sudo tc qdisc add dev docker0 root netem delay 500ms 100ms loss 10%
This simulates typical network problems on the network link docker0:
- 500ms delay
- 100ms jitter
- 10% packet loss
We then captured three thread dumps at intervals of about 10s. This is useful for diagnosing slow performance, as we can see which threads are not moving forward between dumps. If the problem is network lag, we’d expect to see network-related tasks stalled while waiting for a response. The image below is taken from a fastThread comparison of the three dumps:

Fig: Thread Dump Comparison
The thread highlighted is of particular interest: the Chanel Reader thread. Clicking on it shows the stack trace, as shown below:

Fig: Stack Trace of Hung Thread
We can see that three threads are all hanging on socket reads. This confirms the performance problem is network-related.
Jenkins File Descriptor Leaks from Plugins and Groovy Scripts
Operating systems maintain objects known as file descriptors, handles or file connectors to link applications with actual resources, such as files, network sockets or pipes. This is seldom problematic, but as with any object, poor coding can cause leaks, where resources either proliferate inside an unterminated loop, or are retained beyond their useful lifespan.
What Causes File Descriptor Leaks in Jenkins?
The operating system allocates file connectors, otherwise known as file descriptors (FD), to identify open connections. File connection leaks occur when a resource is opened but never closed.
In the Jenkins controller, this can happen because of a poorly coded plugin or Groovy script. In the Jenkins agent, it’s likely to happen within an application included in a build.
Over time, open descriptors accumulate until the operating system’s limit is reached, causing the application to fail with an error such as ‘Too many Open Files’.
The example below shows Groovy code that is vulnerable to File Connection leaks:
def reader = new FileInputStream("/tmp/example.txt")// Read the file, but never close the stream
There is no guarantee the stream will be closed, especially if an exception occurs.
The example below is safer because the resource is automatically closed as soon as the block completes, even if it throws an exception.
new FileInputStream("/tmp/example.txt").withCloseable { stream -> println stream.text}
Within Java applications or plugins, we should also ensure files are always closed on completion.
This code is vulnerable to file connection leaks:
BufferedReader reader = new BufferedReader(new FileReader("hello.txt"))) {… Some code}
Again, there is no guarantee the file will ever be closed.
We can prevent leaks by recoding this as follows, using the try-with-resources construct introduced in Java 7: This ensures the file will always be closed.
try (BufferedReader reader = new BufferedReader(new FileReader("hello.txt")))) {… Some code}
How File Descriptor Leaks Affect Jenkins Performance
If the leak persists, the number of open connections creeps up towards the operating system’s limit. In Windows, there is an overall limit on handles (connections). In Linux, in addition to the overall limit, there is a configurable per-process limit on file connectors.
Opening additional files or network connections then becomes a problem, causing application failures or hangs.
In Jenkins, this causes failed requests, delayed processing, logging failures, and interrupted communications. CPU usage may spike due to repeated attempts to open a connection, and response times degrade.
Although file connectors don’t directly occupy heap memory, the open files may retain heap resources, causing memory-related issues.
Common Symptoms of File Descriptor Leaks in Jenkins
Symptoms include:
- Intermittent build failures;
- Logs include errors such as ‘Too many open files’;
- The application fails when accessing a file or a socket;
- Network connection failures;
- Agent disconnects;
- The Jenkins controller becomes unstable.
How to Diagnose File Descriptor Leaks in Jenkins
File leaks could be in the controller itself, caused by a faulty plugin or a poorly coded Groovy script. They may also be within an application that’s being tested as part of the build.
In Linux, the most useful tool for detecting excess file connections is lsof, which lists open files. We included the above faulty code in a small program and ran it as part of a Jenkins build. The lsof -p command shows a list of open files for a given PID. By piping the output to the word count (wc) command, we can see how many files the process has open at a given moment. We ran this repeatedly to see if the number of files was growing, and saw the following output:
~$ sudo lsof -w -p 52004 | wc -l380~$ sudo lsof -w -p 52004 | wc -l411~$ sudo lsof -w -p 52004 | wc -l433~$ sudo lsof -w -p 52004 | wc -l453~$ sudo lsof -w -p 52004 | wc -l502~$ sudo lsof -w -p 52004 | wc -l524~$ sudo lsof -w -p 52004 | wc -l558~$ sudo lsof -w -p 52004 | wc -l573~$ sudo lsof -w -p 52004 | wc -l620~$
If the number of files is steadily growing, as in this case, the application is likely to have a file descriptor leak.
We next obtained a list of open files held by the application using the command sudo lsof – -p 52004.
Part of the output is shown below.
java 52004 jill mem REG 8,2 18504 52207616 /usr/lib/x86_64-linux-gnu/libcom_err.so.2.1java 52004 jill mem REG 8,2 39792 52301982 /usr/lib/jvm/java-21-openjdk-amd64/lib/libjimage.sojava 52004 jill mem REG 8,2 240936 52166832 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2java 52004 jill 0u CHR 136,1 0t0 4 /dev/pts/1java 52004 jill 1u CHR 136,1 0t0 4 /dev/pts/1java 52004 jill 2u CHR 136,1 0t0 4 /dev/pts/1java 52004 jill 3r REG 8,2 140863940 52302002 /usr/lib/jvm/java-21-openjdk-amd64/lib/modulesjava 52004 jill 4r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 5r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 6r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 7r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 8r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 9r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 10r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 11r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 12r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 13r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 14r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 15r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 16r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 17r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 18r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 19r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 20r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 21r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 22r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txtjava 52004 jill 23r REG 8,2 12 48496824 /home/jill/git/jilltest/hello.txt
As you see, the file ‘hello.txt’ is open many times, giving a clue to the fact that the file is being opened repeatedly without being closed.
Jenkins Direct Buffer Memory Issues and Java NIO
What is Java NIO (New I/O)? The name is misleading, since it’s been around since Java 1.4, with new features introduced from time to time.
It offers more efficient IO than the original java.io package and is able to use off-heap memory for large transfers. If this memory is overused, pinpointing the issue can be tricky.
What Causes Java NIO and Direct Buffer Memory Issues?
Java NIO includes:
- Channels
- Selectors
- Buffers.
Direct Buffers are an important feature of NIO, saving both memory and time by using native memory rather than heap memory. For a breakdown of how the JVM memory is allocated, see JVM Explained in 10 Minutes.
As with any memory-related code, poor development practices can result in leaks and wastage in the direct buffer area. Since they affect off-heap memory, these issues can be tricky to diagnose. Heap memory may appear healthy, but overall memory within the device is high. In extreme cases, this leads to OutOfMemoryError: Direct Buffer Area, or, in some cases, java.lang.OutOfMemoryError: Cannot reserve x bytes of direct buffer memory. Pressure on device memory can also lead to the OS or container manager killing the process.
Java NIO issues may occur with tasks that use file or network I/O extensively, especially if several concurrent processes are using resources at the same time.
Groovy scripts can also make use of direct buffers:
import java.nio.ByteBuffer// Allocate 10 MB outside the normal Java heapdef buffer = ByteBuffer.allocateDirect(10 * 1024 * 1024)// Write some databuffer.put("Jenkins direct buffer test".getBytes("UTF-8"))
For situations that really do need large direct buffers, we can configure the JVM on the command line to allocate more direct buffer space. For example, to reserve 2 GB for direct buffers, the command line option is -XX:MaxDirectMemorySize=2G.
How Java NIO Issues Affect Jenkins Performance
If Jenkins or its plugins consume too much native memory, there’s less memory available for the operating system and other processes. This in turn causes excess paging, which slows down the entire device.
Common Symptoms of Direct Buffer Memory Problems in Jenkins
If off-heap memory pressure is high in Jenkins, builds slow down, and there are often delays in controller-agent communication. Eventually, the application crashes with an OutOfMemoryError. In extreme cases, the operating system or container manager kills the process to prevent the entire system from freezing.
How to Diagnose Java NIO and Direct Buffer Memory Issues
As an illustration, we created an application that leaks direct buffer memory, as shown below:
import java.nio.ByteBuffer;import java.util.ArrayList;// This program causes a memory leak in the Direct Buffer areapublic class BuggyProg17 { public static void main(String[] args) { ArrayList al = new ArrayList(); while (true) { // Allocate a large direct ByteBuffer ByteBuffer directBuffer = ByteBuffer.allocateDirect(1000000); al.add(directBuffer); try{Thread.sleep(500);} catch(Exception e){} } }}
The program includes an infinite loop that creates a large direct byte buffer. To prevent the buffers from being released, it adds references to them into an ArrayList, which is never cleared.
Since direct buffers aren’t created in the heap, standard tools such as heap dump analyzers won’t pick up the problem. Even if there’s a serious memory leak in the direct buffer area, heap usage remains normal, and garbage collection may be healthy.
The first thing to check is whether the Java application as a whole is using excessive memory. Operating system tools such as the Linux top command are useful here:
Tasks: 215 total, 1 running, 214 sleeping, 0 stopped, 0 zombie%Cpu(s): 1.4 us, 2.8 sy, 0.0 ni, 95.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 stMiB Mem : 3357.5 total, 544.7 free, 1646.4 used, 1166.4 buff/cacheMiB Swap: 4096.0 total, 4079.9 free, 16.0 used. 1440.8 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4427 jill 20 0 4709788 1.1g 22652 S 0.0 33.3 0:03.27 java 1285 gdm 20 0 4055028 118408 53448 S 0.0 3.4 0:57.42 gnome-shell 1309 root 20 0 2173856 73380 28928 S 0.0 2.1 0:23.47 dockerd 1171 gdm 20 0 535900 45996 22876 S 0.0 1.3 0:03.46 Xorg 900 root 20 0 2023988 23872 8696 S 0.0 0.7 0:22.45 containerd 779 root 20 0 2076456 22944 8132 S 0.0 0.7 0:08.27 snapd 1144 gdm 9 -11 899528 21352 14160 S 0.0 0.6 0:00.34 pulseaudio 1526 gdm 20 0 2660980 20068 14968 S 0.0 0.6 0:00.09 gjs 1390 gdm 20 0 2660848 19816 15012 S 0.0 0.6 0:00.11 gjs
Our Java application is the highest memory user on the list. It’s using 1.1GB of resident memory, and 0.47GB of virtual memory. Using jcmd, we can see how much of this is heap memory:
$ jcmd 4427 GC.heap_info4427: garbage-first heap total 57344K, used 2364K [0x00000000cb800000, 0x0000000100000000)region size 1024K, 2 young (2048K), 0 survivors (0K) Metaspace used 132K, committed 320K, reserved 1114112K class space used 7K, committed 128K, reserved 1048576K
As you can see, the heap only occupies 57K of memory.
To analyze native memory usage, we first need to enable native memory tracking (NMT) on the command line. We also set the direct buffer area to a maximum of 2GB.
$ java -XX:NativeMemoryTracking=summary -XX:MaxDirectMemorySize=2G BuggyProg17
Once NMT is enabled, we use jcmd to extract native memory usage to a file:
jcmd 4427 VM.native_memory summary >> BuggyProg17.txt
If we run this at fixed intervals, we have a record over time of native memory usage. Since the output is quite long, the quickest way to analyze it is using the GCeasy tool, which presents the results in a series of easily-read charts.
The chart below is the GCeasy summary of native memory usage by category:

Fig: Native Memory Summary by GCeasy
The JVM includes direct buffers in the ‘Other’ category when extracting NMT data. As you can see, the ‘Other’ category is occupying almost all of the total committed memory.
We can zoom in on this category to see a graph of usage over time:

Fig: Usage by Category Over Time
Memory is steadily increasing in this category, which is a strong indication of a memory leak.
Finally, check the Jenkins logs for errors like this one. This error occurs when the direct buffer area tries to grow beyond the limit.
Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 1000000 bytes of direct buffer memory (allocated: 2147000000, limit: 2147483648) at java.base/java.nio.Bits.reserveMemory(Bits.java:178) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:111) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:360) at BuggyProg17.main(BuggyProg17.java:13)
Quick Reference: Common Jenkins Performance Issues and Their Symptoms
In summary, here is a table showing uncommon causes of performance issues in Jenkins:
| Issue | Typical Symptoms | Diagnostic tools |
| Network latency between Jenkins controller and agent | Slow job execution despite low CPU usage; delays transferring build logs or files; intermittent agent disconnects; slow remote command execution; builds delayed while waiting for controller responses. | Networking Tools e.g. ping; mtr; iperf3; ss -ti; Jenkins Monitoring, fastThread |
| File descriptor leaks | Increasing number of open file descriptors; ‘Too many open files’ errors; failures opening files, sockets or pipes; plugin or agent connection failures; Jenkins becoming intermittently unresponsive. | lsof; /proc/<PID>/fd; /proc/<PID>/limits; repeated FD-count sampling; fastThread (for supporting evidence) |
| Direct buffer / Java NIO memory problems | High native memory use despite a relatively healthy Java heap; ‘OutOfMemoryError: Direct buffer memory’; increasing direct or mapped buffer usage; I/O degradation. | jcmd VM.native_memory summary (if NMT enabled at startup) in conjunction with GCeasy; Java Flight Recorder (JFR) |
How yCrash Helps Diagnose Jenkins Performance Issues
Jenkins performance problems can have a huge impact on an organization. Software upgrades may fail to be released on time, developers’ time is wasted waiting for slow responses, and administrators spend far too much time troubleshooting.
Can we reduce the chances of Jenkins performance problems?
The yCrash suite allows us to proactively monitor system performance and raise alerts before Jenkins issues escalate to the point where they affect our workflow.
This range of tools saves time and money when Jenkins experiences performance issues.
Let’s look at the tools it offers.
1. Analyze Jenkins Thread Dumps With fastThread
The fastThread utility analyzes thread dumps, identifying deadlocks and high CPU usage, and allows the user to explore stack traces, locks, and hot methods.
2. Analyze GC Logs and Native Memory Usage With GCeasy
GCeasy analyzes GC logs and NMT traces. The GC logs are often the best advance warning of future performance issues. Micrometrics such as throughput, latency, and footprint identify issues before they affect production.
3. Investigate Java Heap Memory With HeapHero
HeapHero creates an interactive report from a heap dump, allowing the user to browse, sort, and search the contents of the heap, as well as explore the dominator tree to find out why objects are not being garbage collected.
4. Collect JVM Diagnostics and Detect Performance Issues With yc-360
The free yc-360 script can be used in two ways:
- To gather a full range of diagnostic artifacts related to the JVM and its environment when problems surface;
- To regularly sample performance micrometrics and transmit them to the yCrash server for analysis. If problems are detected, it raises an incident report with full diagnostic information for further investigation.
5. Root Cause Analysis With yCrash Server
The yCrash server, which uses machine learning to generate root cause and comparison reports, allows quick and efficient troubleshooting. yCrash can feed curated data into AI tools such as Claude or OpenAI for further analysis. It can receive data from:
- A bundle of artifacts created by the yc-360 script;
- A JFR recording.
Conclusion
Jenkins is a highly complex system. In addition to the Jenkins core, each installation has its own unique combination of plugins and Groovy scripts, any of which can cause performance problems if not coded according to best practices.
Jenkins production issues cause wide-ranging problems for an organization that relies on efficient job scheduling to meet business targets.
A good range of troubleshooting tools is essential, especially when we encounter obscure performance issues.
FAQ
1. What Causes High Network Latency Between a Jenkins Controller and its Agents?
Communication problems between a controller and its agents may be due to several different causes. These include:
- Network congestion;
- Overloaded network interfaces;
- Packet loss;
- DNS resolution delays;
- Firewall inspection;
- VPN overhead;
- Poorly configured virtual or container networks.
Monitor network performance between the controller and agent, and investigate whether delays occur consistently or only during periods of heavy build activity.
2. How do I fix the Too Many Open Files error in Jenkins?
This error can be caused by:
- A poorly-coded plugin;
- A poorly-coded Groovy script
- Too many file-hungry builds running at once.
First, establish what builds or tasks were running when the error occurred. Next, check the Jenkins log file to get the exact error message and stack trace.
Below is an excerpt from the Jenkins log when running a Groovy script that opens multiple files without closing them:

The error message gives the name of the file that could not be opened: in this case, /dev/null. The subsequent stack trace is the most important clue. In this example, we see Groovy-related classes, which tells us the problem is in a Groovy script.
Problems associated with a plugin usually show ‘plugin’ as part of the class path.
If the problem is in a plugin:
- If the plugin was written locally, review the code to make sure files will always be closed, even if an exception is thrown. Check for unterminated loops.
- If the plugin was downloaded from elsewhere, remove the plugin for the time being and lodge a bug report with the developers.
If the problem is within a Groovy script, ascertain what scripts were running at the time of the error, and whether they completed successfully. Check the code to make sure there are no unterminated loops, and files are always closed.
Otherwise, check what builds and activities were running at the time. If there are a lot of file-hungry builds, schedule them so they don’t all run together.
3. What causes Direct Buffer Memory OutOfMemoryError in Java?
Direct buffer memory is allocated outside the normal Java heap, commonly for fast Java NIO operations and network or file I/O. Extremely large buffers, long retention, or too many buffers in use at one time can cause this problem.
Some applications may need large buffer areas, and in this case, the solution is to increase the amount of memory allocated for them using the -XX:MaxDirectMemorySize argument, for example -XX:MaxDirectMemorySize=5G.
