Analyzing & Troubleshooting Java Heavy I/O Using JFR 

Java Flight Recorder (JFR) provides continuous, low-overhead profiling by capturing runtime execution samples directly within the JVM. In this blog, we will examine Heavy I/O, a performance issue commonly caused by a performance issue commonly caused by an application performing excessive, unthrottled file read/write operations across multiple threads, saturating disk I/O capacity. We will simulate the problem, capture the relevant JFR data, and analyze the recording to identify the underlying issue. Let’s take a closer look.

What are Heavy I/O?

Fig: Continuous disk I/O from multiple threads saturates the disk, increasing I/O wait times 

Heavy I/O occurs when an application performs a large volume of file or disk operations in a short period, without any pacing between them. Each read or write operation takes time to complete and competes with every other thread doing the same thing for the same underlying disk resource. When many threads perform I/O continuously with no delay between operations, the disk can become the bottleneck, slowing down every thread waiting on it, not just the ones directly responsible for the load.

What causes ‘Heavy I/O’?

Let’s look at the list of causes for this Heavy I/O issue: 

1. Unthrottled Read/Write Loops: A loop that performs file operations continuously, with no pause between iterations, can generate far more I/O volume than the underlying disk can comfortably handle.
2. Multiple Concurrent Threads Performing I/O Simultaneously: When several threads each run their own I/O loop at the same time, their combined disk activity compounds, turning what any single thread would consider a modest workload into significant aggregate pressure.
3. Small, Frequent Operations Instead of Batching: Writing or reading data in many small operations, rather than combining them into fewer, larger ones, increases overhead per byte transferred and multiplies the number of disk accesses required.

Simulating Heavy I/O Performance Issue

To understand how Heavy I/O appears in JFR data, let’s reproduce the issue using a sample Java application. The following program deliberately launches multiple threads that continuously write to and read from their own file with no pause between operations, generating sustained I/O load.

public class IODemo {
    public void start() {
        for (int counter = 1; counter <= 5; ++counter) {
            new IOThread("fileIO-" + counter + ".txt").start();
            System.out.println("Starting to write to fileIO-" + counter + ".txt");
        }
    }
    public static void stop() {
        System.out.println("Heavy IO activity terminated!");
    }
}

public class IOThread extends Thread {
    public String fileName;
    public static final String CONTENT = "Hello World! We are building a simple chaos engineering product here. \n" /* repeated */ ;
    public IOThread(String fileName) {
        this.fileName = fileName;
    }
    private static boolean flag = true;
    public static void setFlag(boolean newValue) {
        flag = newValue;
    }
    public void run() {
        int counter = 0;
        while (flag) {
            FileUtil.write(fileName, CONTENT);
            FileUtil.read(fileName);
            if (++counter == 1000) {
                System.out.println("Read & write 1000 times to " + fileName);
                counter = 0;
            }
        }
    }
}

In this program, IODemo.start() launches five IOThread instances, each assigned its own file. Every IOThread runs an unbounded while (flag) loop, writing the same block of content to its file and immediately reading it back, with no pause between iterations and no upper limit on how many times this repeats. Since flag is never set to false during normal execution, all five threads continue this write-then-read cycle indefinitely and simultaneously. As the threads keep running, five concurrent, unthrottled I/O loops place sustained pressure on the disk, and because none of the operations are batched or spaced out, the volume of small read and write calls compounds quickly across all five files. 

Capturing JFR Data for Troubleshooting Heavy I/O

To capture JFR data for troubleshooting the Heavy I/O issue, we recommend that you follow the steps below:

If you are interested in learning about the other methods available to capture JFR recordings, we recommend that you read ‘How to Capture Java Flight Recorder (JFR)?’ blog.

Step 1: Start a JFR recording against the running JVM:

jcmd {PID} JFR.start \
    name=loadTestCapture \
    settings=profile \
    filename=/tmp/tomcat.jfr

Note: Replace <PID> with the Process ID (PID) of your Java application running inside the container. If you don’t know how to find it, refer to our guide on finding the Java application Process ID (PID) for step-by-step instructions.

Step 2: Let it run while the Heavy I/O is occurring, then stop it manually:

jcmd {PID} JFR.stop \
    name=loadTestCapture

Step 3: Alternatively, for Fixed-Duration Capture,  you can start a recording that automatically exits after a predefined duration (for example, 15 minutes) in a single step:

jcmd {PID} JFR.start \
    name=loadTestCapture \
    settings=profile \
    duration=15m \
    filename=/tmp/tomcat.jfr

Analyzing Heavy I/O Using the JFR Data

You can analyze JFR recording using yCrash JFRPlayer by following the steps mentioned below.  

Step 1: Install yCrash JFRPlayer, which is available in both cloud and on-premises versions. Use one of the options below to get started:

• Cloud service: Register and upload your JFR file online.
• On-Premises: Install and run yCrash JFRPlayer on your local machine or within your organization’s environment.

Step 2: Upload the jfr file to your yCrash JFRPlayer. Once JFR file is uploaded, JFRPlayer parses the JFR file and generates an incident report instantly. 

Fig: Uploading a standalone JFR file to yCrash JFRPlayer 

Step 3: The AI Overview points to five threads looping on sun.nio.fs.UnixNativeDispatcher.open0(), driving CPU utilization to 11.5%, with java.io.BufferedReader.implReadLine() confirmed as a hot method consuming 24.55% of CPU across 135 samples in com.buggyapp.io.IOThread.

Fig: yCrash JFRPlayer AI Overview identifying the file I/O hot path and CPU impact 

Step 4: The Issues in the Application panel flags four issues, five threads looping on open0() in sun.nio.fs.UnixNativeDispatcher, one thread looping on emitEvent(), and the same BufferedReader.implReadLine() hot method at 24.55% CPU across 135 samples. 

Fig: yCrash JFRPlayer’s Issues in the Application panel showing the flagged threads and hot method 

Step 5: Thread-5’s stack trace, RUNNABLE with 1910.74ms CPU time, traces from sun.nio.fs.UnixNativeDispatcher.open0() down through Files.write() into com.buggyapp.util.FileUtil.write() at line 46, called from com.buggyapp.io.IOThread.run() at line 53.

Fig: Stack trace confirming FileUtil.write() and IOThread.run() as the source of the file I/O loop

Simple, right? Now that we have analyzed the data, we are equipped with all the information to fix the problem.

If you face any challenges while analyzing a JFR file using yCrash JFRPlayer, check out our FAQ for answers to common questions and troubleshooting guidance.

How to fix Heavy I/O

The following are potential solutions to fix this issue:

1. Introduce Pacing Between Operations: Adding a small delay or batching writes together, rather than performing a read/write cycle on every loop iteration, reduces the volume of disk operations without changing what the program accomplishes.
2. Batch Small Operations Into Larger Ones: Writing accumulated data in fewer, larger operations, rather than many small ones, reduces per-operation overhead and the total number of disk accesses.
3. Limit Concurrent I/O Threads: Running fewer threads performing I/O at the same time, or coordinating them through a shared queue, avoids multiple threads competing for the same disk resource simultaneously.
4. Use Buffered or Asynchronous I/O: Buffering reads and writes, or moving to asynchronous I/O APIs, can reduce the number of direct disk accesses and let the OS optimize how operations are scheduled.

Conclusion

Diagnosing and identifying the root cause of Heavy I/O can be challenging, especially in complex production environments where multiple symptoms often overlap. In this example, the JFR recording was analyzed using yCrash JFRPlayer, which automatically identified the key performance bottlenecks and correlated JVM events to pinpoint the underlying problem. The analysis traced the issue to the analysis traced the issue to IOThread.run() performing continuous, unthrottled read and write operations across five concurrent threads. By examining the relevant JFR insights, including file read and write events, thread activity, and I/O duration per thread, we were able to identify the root cause and determine the appropriate fix.