Java NIO (New Input/Output) is high-performance networking and file handling API that facilitates you to do non-blocking IO. Non-blocking I/O provides following advantages:
- Concurrency: NIO enables handling multiple connections simultaneously without blocking threads, leading to better concurrency.
- Asynchronous Programming: Asynchronous programming allows the application to perform other tasks while waiting for I/O operations to complete, improving overall efficiency.
- Performance: Non-blocking I/O can manage more connections with fewer threads, reducing the resources required for handling concurrent requests.
One of our applications was leveraging this NIO, however it suffered from frequent ‘java.lang.OutOfMemoryError: Direct buffer memory’ when we were running in Java 11. However when we upgraded to Java 17 frequency of the occurrence of ‘java.lang.OutOfMemoryError: Direct buffer memory’ got reduced dramatically. In this post we would like to share our findings and resolution to fix this problem.
Simple Java NIO Client
To demonstrate our case, we have built a simple Spring Boot application that asynchronously uploads images. This application was leveraging Spring WebClient to connect with REST APIs. Spring WebClient underlyingly uses Java NIO technology to handle connections. Below is the source code of this application.
public void webHeavyClientCall(Integer id,String url, String imagePath) { // Create a WebClient instance WebClient webClient = WebClient.create(); // Prepare the image file File imageFile = new File(imagePath); // Perform the POST request with the image as a part of the request body MultiValueMap<String, Object> body = new LinkedMultiValueMap<>(); body.add("file", new FileSystemResource(imageFile)); System.out.println("Starting to post an image for Id"+id); webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData(body)) .retrieve().bodyToMono(String.class).subscribe(response -> { System.out.println("Response Id"+id+ ":" + response); });}
Java 11 NIO Memory Leak
We executed the above code in Java 11. After around 15 iterations this simple application started to throw ‘java.lang.OutOfMemoryError: Direct buffer memory’. Below is the output printed in the console.
Starting to post an image for Id0Starting to post an image for Id1Starting to post an image for Id2Starting to post an image for Id3Starting to post an image for Id4Starting to post an image for Id5Starting to post an image for Id6Starting to post an image for Id7Starting to post an image for Id8Starting to post an image for Id9Starting to post an image for Id10Starting to post an image for Id11Starting to post an image for Id12Starting to post an image for Id13Starting to post an image for Id142023-12-06 17:21:46.730 WARN 13804 --- [tor-http-nio-12] io.netty.util.concurrent.DefaultPromise : An exception was thrown by reactor.ipc.netty.FutureMono$FutureSubscription.operationComplete()reactor.core.Exceptions$ErrorCallbackNotImplemented: io.netty.channel.socket.ChannelOutputShutdownException: Channel output shutdownCaused by: java.lang.OutOfMemoryError: Direct buffer memory at java.base/java.nio.Bits.reserveMemory(Bits.java:175) ~[na:na] at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:118) ~[na:na] at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:318) ~[na:na] at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) ~[na:na] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:164) ~[na:na] at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:130) ~[na:na] at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:496) ~[na:na] at io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:418) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final] at io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:934) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final] ... 18 common frames omitted
Java 17 NIO Memory Optimization
We executed the same program in Java 17. However to run this program in Java 17, we had to make some minor modifications. Below is the revised code that runs on Java 17 that simulates the same behaviour as above.
public void webHeavyClientCall(Integer id,String url, String imagePath) { // Create a WebClient instance WebClient webClient = WebClient.create(); // Prepare the image file File imageFile = new File(imagePath); // Perform the POST request with the image as a part of the request body MultiValueMap<String, Object> body = new LinkedMultiValueMap<>(); body.add("file", new FileSystemResource(imageFile)); System.out.println("Starting to post an image for Id"+id); webClient.post().uri(url).contentType(MediaType.MULTIPART_FORM_DATA).body(BodyInserters.fromMultipartData(body)) .retrieve().bodyToMono(String.class).subscribe(response -> { System.out.println("Response Id"+id+ ":" + response); }); }
There was an improvement in memory usage after the upgrade. Java 17 was able to handle at least twice as many NIO connections compared to Java 11. Below is the output from the console. You could see the application was able to iterate until 50 connections before it struck with ‘java.lang.OutOfMemoryError’. On the other hand Java 11 failed with ‘java.lang.OutOfMemoryError’ right after 15 connections.
Starting to post an image for Id38Starting to post an image for Id39Starting to post an image for Id40Starting to post an image for Id41Starting to post an image for Id42Starting to post an image for Id43Starting to post an image for Id44Starting to post an image for Id45Starting to post an image for Id46Starting to post an image for Id47Starting to post an image for Id48Starting to post an image for Id492023-12-12 14:49:38.421 WARN 59559 --- [ctor-http-nio-4] r.netty.http.client.HttpClientConnect : [bfc8b2c8, L:/127.0.0.1:57435 ! R:localhost/127.0.0.1:8090] The connection observed an errorreactor.netty.ReactorNetty$InternalNettyException: java.lang.OutOfMemoryError: Cannot reserve 4096 bytes of direct buffer memory (allocated: 202956, limit: 204800)Caused by: java.lang.OutOfMemoryError: Cannot reserve 4096 bytes of direct buffer memory (allocated: 202956, limit: 204800) at java.base/java.nio.Bits.reserveMemory(Bits.java:178) ~[na:na] at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[na:na] at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[na:na] at io.netty.buffer.UnpooledDirectByteBuf.allocateDirect(UnpooledDirectByteBuf.java:104) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.UnpooledDirectByteBuf.<init>(UnpooledDirectByteBuf.java:64) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:41) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:634) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:397) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116) ~[netty-buffer-4.1.73.Final.jar!/:4.1.73.Final] at org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71) ~[spring-core-5.3.15.jar!/:5.3.15] at org.springframework.core.io.buffer.DataBufferUtils$ReadCompletionHandler.request(DataBufferUtils.java:945) ~[spring-core-5.3.15.jar!/:5.3.15]
Troubleshooting ‘OutOfMemoryError: Direct buffer memory’
In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before it surfaces in the production environment. Once it predicts outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them and instantly generates a root cause analysis report. Artifacts it captures include Garbage Collection log, Thread Dump, Heap Substitute, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters, disk usage….
You can register here and start using the free-tier of this tool.
The yCrash server analyzed the sample application and provided clear indications of issues with recommendations. Below is the incident summary report that yCrash generated for the SpringBoot WebClient application.You can notice yCrash clearly pointing out the error with necessary recommendations to remediate the problem.
Garbage Collection analysis Report
yCrash’s Garbage Collection (GC) analysis report revealed that Full GCs were consecutively running (see screenshot below). When GC runs, the entire application pauses and no transactions will be processed. Entire application would become unresponsive. We observed the unresponsiveness behaviour before the SpringBoot WebClient application crashed with OutOfMemoryError.
Logs analysis reporting OutOfMemoryError: Direct buffer memory
yCrash’s application log analysis report revealed that application was suffering from ‘ java.lang.OutOfMemoryError: Direct buffer memory’ (see the screenshot below) which causing the application to crash
Why Java NIO application suffering from OutOfMemoryError?
Java NIO objects are stored in the ‘Direct Buffer Memory’ region of JVM’s native memory. (Note: There are different memory regions in JVM. To learn about them, you may watch this video clip). When we executed the above two programs, we had set the Direct Buffer Memory size as 200k (i.e. -XX:MaxDirectMemorySize=200k). Under 200k Direct Buffer Memory allocation Java 11 was able to do only 15 iterations, whereas Java 17 was able to go till 50 iterations. It clearly indicates the optimization JDK team has done in Java 17 version.
On java 11 or below versions increase -XX:MaxDirectMemorySize
Thus if your application is leveraging Java NIO and running on Java 11 or below versions and experiencing ‘java.lang.OutOfMemoryError: Direct buffer memory’, there are couple of solutions in front of you:
- Consider allocating higher Direct Buffer Memory Size.
- Consider upgrading to Java 17 or higher version
Since upgrading Java 17, requires more dependencies, we increased the direct memory size to a higher value using the JVM argument -XX:MaxDirectMemorySize=1000k. After making this change, Java 11 version of the application was able to run successfully without any errors.
Conclusion
In this post we discussed ‘java.lang.OutOfMemoryError: Direct buffer memory’ caused by Java NIO in Java 11 and potential solutions to fix the same. We hope you find it helpful.
[…] to be running on a version less than Java 17, upgrade it. Here is a case study that showcases the performance optimization to Direct Buffer Memory region in Java […]
[…] Upgrade to Java 17 (or above version): Enhancements have been made in Java 17 to use the Direct Buffer Memory region in an effective manner. Thus, if you happen to be running on a version less than Java 17, upgrade it. Here is a case study that showcases the performance optimization to Direct Buffer Memory region in Java 17. […]
[…] (for inspecting thread dumps) which can quickly pinpoint issues like memory leaks resulting in OutOfMemoryError, flag objects consuming excessive heap space, thread contention, or excessive garbage […]
[…] Upgrade to Java 17 (or above version): Enhancements have been made in Java 17 to use the Direct Buffer Memory region in an effective manner. Thus, if you happen to be running Jenkins on a version less than Java 17, upgrade it. Here is a case study that showcases the performance optimization to Direct Buffer Memory region in Java 17. […]
[…] First off, we got to learn about off-heap vs. heap and moved on into where it is used- NIO, Netty, and Spring WebClient. Potential issues were mentioned, including cases when direct buffers behave […]