Java NIO (New Input/Output) is high-performance networking and file handling API that facilitates you to do non-blocking IO. Non-blocking I/O provides following advantages:
- Concurrency: NIO enables handling multiple connections simultaneously without blocking threads, leading to better concurrency.
- Asynchronous Programming: Asynchronous programming allows the application to perform other tasks while waiting for I/O operations to complete, improving overall efficiency.
- Performance: Non-blocking I/O can manage more connections with fewer threads, reducing the resources required for handling concurrent requests.
One of our applications was leveraging this NIO, however it suffered from frequent ‘java.lang.OutOfMemoryError: Direct buffer memory’ when we were running in Java 11. However when we upgraded to Java 17 frequency of the occurrence of ‘java.lang.OutOfMemoryError: Direct buffer memory’ got reduced dramatically. In this post we would like to share our findings and resolution to fix this problem.
Simple Java NIO Client
To demonstrate our case, we have built a simple Spring Boot application that asynchronously uploads images. This application was leveraging Spring WebClient to connect with REST APIs. Spring WebClient underlyingly uses Java NIO technology to handle connections. Below is the source code of this application.
public void webHeavyClientCall(Integer id,String url, String imagePath) {
// Create a WebClient instance
WebClient webClient = WebClient.create();
// Prepare the image file
File imageFile = new File(imagePath);
// Perform the POST request with the image as a part of the request body
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(imageFile));
System.out.println("Starting to post an image for Id"+id);
webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData(body))
.retrieve().bodyToMono(String.class).subscribe(response -> {
System.out.println("Response Id"+id+ ":" + response);
});
}Java 11 NIO Memory Leak
We executed the above code in Java 11. After around 15 iterations this simple application started to throw ‘java.lang.OutOfMemoryError: Direct buffer memory’. Below is the output printed in the console.
Starting to post an image for Id0
Starting to post an image for Id1
Starting to post an image for Id2
Starting to post an image for Id3
Starting to post an image for Id4
Starting to post an image for Id5
Starting to post an image for Id6
Starting to post an image for Id7
Starting to post an image for Id8
Starting to post an image for Id9
Starting to post an image for Id10
Starting to post an image for Id11
Starting to post an image for Id12
Starting to post an image for Id13
Starting to post an image for Id14
2023-12-06 17:21:46.730 WARN 13804 --- [tor-http-nio-12] io.netty.util.concurrent.DefaultPromise : An exception was thrown by reactor.ipc.netty.FutureMono$FutureSubscription.operationComplete()
reactor.core.Exceptions$ErrorCallbackNotImplemented: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdown
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na]
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na]
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) ~[na:na]
at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[na:na]
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[na:na]
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[na:na]
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) ~[na:na]
at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:418) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final]
... 18 common frames omitted
Java 17 NIO Memory Optimization
We executed the same program in Java 17. However to run this program in Java 17, we had to make some minor modifications. Below is the revised code that runs on Java 17 that simulates the same behaviour as above.
public void webHeavyClientCall(Integer id,String url, String imagePath) {
// Create a WebClient instance
WebClient webClient = WebClient.create();
// Prepare the image file
File imageFile = new File(imagePath);
// Perform the POST request with the image as a part of the request body
MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
body.add("file", new FileSystemResource(imageFile));
System.out.println("Starting to post an image for Id"+id);
webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData(body))
.retrieve().bodyToMono(String.class).subscribe(response -> { System.out.println("Response Id"+id+ ":" + response);
});
}There was an improvement in memory usage after the upgrade. Java 17 was able to handle at least twice as many NIO connections compared to Java 11. Below is the output from the console. You could see the application was able to iterate until 50 connections before it struck with ‘java.lang.OutOfMemoryError’. On the other hand Java 11 failed with ‘java.lang.OutOfMemoryError’ right after 15 connections.
Starting to post an image for Id38
Starting to post an image for Id39
Starting to post an image for Id40
Starting to post an image for Id41
Starting to post an image for Id42
Starting to post an image for Id43
Starting to post an image for Id44
Starting to post an image for Id45
Starting to post an image for Id46
Starting to post an image for Id47
Starting to post an image for Id48
Starting to post an image for Id49
2023-12-12 14:49:38.421 WARN 59559 --- [ctor-http-nio-4] r.netty.http.client.HttpClientConnect : [bfc8b2c8, L:/127.0.0.1:57435 ! R:localhost/127.0.0.1:8090] The connection observed an error
reactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Cannot reserve 4096 bytes of direct buffer memory (allocated: 202956, limit: 204800)
Caused by: java.lang.OutOfMemoryError: Cannot reserve 4096 bytes of direct buffer memory (allocated: 202956, limit: 204800)
at java.base/java.nio.Bits.reserveMemory(Bits.java:178) ~[na:na]
at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[na:na]
at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[na:na]
at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:634) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:397) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final]
at org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71) ~[spring-core-5.3.15.jar!/:5.3.15]
at org.springframework.core.io.buffer.DataBufferUtils$ReadCompletionHandler.request(DataBufferUtils.java:945) ~[spring-core-5.3.15.jar!/:5.3.15]
Troubleshooting ‘OutOfMemoryError: Direct buffer memory’
In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before it surfaces in the production environment. Once it predicts outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them and instantly generates a root cause analysis report. Artifacts it captures include Garbage Collection log, Thread Dump, Heap Substitute, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters, disk usage….
You can register here and start using the free-tier of this tool.
The yCrash server analyzed the sample application and provided clear indications of issues with recommendations. Below is the incident summary report that yCrash generated for the SpringBoot WebClient application.You can notice yCrash clearly pointing out the error with necessary recommendations to remediate the problem.
Garbage Collection analysis Report
yCrash’s Garbage Collection (GC) analysis report revealed that Full GCs were consecutively running (see screenshot below). When GC runs, the entire application pauses and no transactions will be processed. Entire application would become unresponsive. We observed the unresponsiveness behaviour before the SpringBoot WebClient application crashed with OutOfMemoryError.
Logs analysis reporting OutOfMemoryError: Direct buffer memory
yCrash’s application log analysis report revealed that application was suffering from ‘ java.lang.OutOfMemoryError: Direct buffer memory’ (see the screenshot below) which causing the application to crash
Why Java NIO application suffering from OutOfMemoryError?
Java NIO objects are stored in the ‘Direct Buffer Memory’ region of JVM’s native memory. (Note: There are different memory regions in JVM. To learn about them, you may watch this video clip). When we executed the above two programs, we had set the Direct Buffer Memory size as 200k (i.e. -XX:MaxDirectMemorySize=200k). Under 200k Direct Buffer Memory allocation Java 11 was able to do only 15 iterations, whereas Java 17 was able to go till 50 iterations. It clearly indicates the optimization JDK team has done in Java 17 version.
On java 11 or below versions increase -XX:MaxDirectMemorySize
Thus if your application is leveraging Java NIO and running on Java 11 or below versions and experiencing ‘java.lang.OutOfMemoryError: Direct buffer memory’, there are couple of solutions in front of you:
- Consider allocating higher Direct Buffer Memory Size.
- Consider upgrading to Java 17 or higher version
Since upgrading Java 17, requires more dependencies, we increased the direct memory size to a higher value using the JVM argument -XX:MaxDirectMemorySize=1000k. After making this change, Java 11 version of the application was able to run successfully without any errors.
Conclusion
In this post we discussed ‘java.lang.OutOfMemoryError: Direct buffer memory’ caused by Java NIO in Java 11 and potential solutions to fix the same. We hope you find it helpful.