Everyone has at least once seen a failing application due to OutOfMemoryError. This error happens because the application cannot allocate the required memory.

The specific reason for this problem may differ depending on the severity and heap section affected. However, in any case, we would like to have some actions prepared for this. In this article, we’ll learn how to make the analysis easier when encountering OutOfMemoryErrors.

Heap Dumps

In general, automatic heap dumps on OutOfMemoryErrors are good practice, as they have almost no downsides. The dump happens automatically, and it’s an excellent way to analyze the problem afterward. On the negative side, the dump might occupy resources and space, which we should consider while using this. 

Using a JVM Parameter

While starting an application, we can pass the parameter -XX:+HeapDumpOnOutOfMemoryError, which would create a heap dump each time the application fails with the OutOfMemoryError. 

Another helpful parameter for heap dumps is -XX:HeapDumpPath. By default, a heap dump is created in the current working directory using a simple naming convention: java_pid[PID].hprof.

We cannot specify the name but provide a custom path for our heap dumps. 

Setting An Automatic Heap Dump on Running Application

It’s not always possible to configure this on the startup. We can turn on automatic heap dumps even on the running application. JVM exposes HotSpotDiagnosticMBean through JMX, which we can use to configure the VM options. Let’s check the example using JConsole:

Using this bean, we can fetch the value of the required VM option:

Also, we can set it similarly:

Note that while using the HotSpotDiagnostic MBean, we should use HeapDumpOnOutOfMemoryError as the name of the option. We can provide HeapDumpPath in a similar fashion.

Additionally, we can use any tool that allows us to use JMX. For example, JMXTerm enables us to use scripts to automate interactions with the MBeans.

Running a Script With OnOutOfMemoryErrors

At the same time, it’s possible to set up a script that runs on the OutOfMemoryError. Running scripts might be helpful if we need to restart the app, do cleanups, or implement some basic notification system.

#!/bin/bash



java -XX:OnOutOfMemoryError="start.sh" -jar app.jar 

However, use restarting scripts cautiously because it might restart a faulty application indefinitely. 

Also, the script can gather additional information about the system during the failure, which might help further investigation. For example, thread dumps:

#!/bin/bash





# Generate the thread dump

jstack $1

echo "Thread dump saved to thread_dump.txt"

We can pass the PID of the Java process by using the %p placeholder in the XX:OnOutOfMemoryError:

-XX:OnOutOfMemoryError="sh thread-dump.sh %p"

The script might not capture the entire thread dump because JVM may kill the process before finishing the thread dump. 

At the same time, we can combine two ideas and use a script that would not only produce a heap dump but also send it to the HeapHero using its API for analysis:

#!/bin/bash



# Check if a directory is provided

if [ "$#" -ne 1 ]; then

    echo "Usage: $0 /path/to/directory"

    exit 1

fi



# Set your API key here

API_KEY="your-api-key"



# Directory path

DIR_PATH="$1"



# Change to the directory

cd "$DIR_PATH" || exit 1



# Iterate over each file in the directory

for file in *; do

    # Ensure it's a file and not a subdirectory

    if [[ -f "$file" ]]; then

        # Compress the file

        echo "Compressing: $file"

        zip "${file}.zip" "$file"



        # Send the zip file to the endpoint

        echo "Sending ${file}.zip to the endpoint..."

        curl -X POST --data-binary "@${file}.zip" "https://api.heaphero.io/analyze-hd-api?apiKey=${API_KEY}&Content-Encoding=zip" --header "Content-Type:text"

        echo "${file}.zip has been sent!"



        # Delete the zip file after sending

        rm "${file}.zip"

    fi

done

We also, can have the same code for Windows:

@echo off

SETLOCAL ENABLEDELAYEDEXPANSION

REM Check if a directory is provided

IF "%~1"=="" (

    echo Usage: %0 \path\to\directory

    exit /b 1

)

REM Set your API key here

SET API_KEY=your-api-key

REM Directory path

SET DIR_PATH=%~1

REM Change to the directory

cd %DIR_PATH% || exit /b 1

REM Iterate over each file in the directory

FOR %%i IN (*) DO (

    REM Ensure it's a file and not a subdirectory

    IF EXIST %%i (

        REM Compress the file

        echo Compressing: %%i

        REM Note: You will need a zip command-line tool, such as 7-Zip's 7z.exe, to perform this action.

        7z a "%%i.zip" "%%i"

        REM Send the zip file to the endpoint

        echo Sending %%i.zip to the endpoint...

        REM Note: You'll need the curl command for Windows to execute this.

        curl -X POST --data-binary "@%%i.zip" "https://api.heaphero.io/analyze-hd-api?apiKey=!API_KEY!&Content-Encoding=zip" --header "Content-Type:text"

        echo %%i.zip has been sent!



        REM Delete the zip file after sending

        del "%%i.zip"

    )

)



ENDLOCAL

Depending on the JVM, the execution order for XX:HeapDumpOnOutOfMemoryError and XX:OnOutOfMemoryError might differ. That’s why using Python would provide us with more capabilities and more stable and cross-platform scripts. Modern LLMs can easily translate these scripts into any language we would like. 

Additionally, we can implement notifications in the yCrash account or the code. For example, to send emails, Slack messages, SMS, etc. This way, we either can get a JSON response that would contain the analysis information or check it directly on the platform:

360-degree Script

Getting information about the application and its surroundings is a great way to start troubleshooting. One of the killer features of yCrash is its agent, which can collect lots of data, which might be pretty helpful. 

The agent has several different modes, but for this article, we’ll consider only one: the on-demand mode. The main benefit of doing so is that it can provide us with greater insight into the systems’ state and simplify our debugging process. Let’s start with the script:

#!/bin/bash



# Generate the 360-degree data

./yc-agent-latest/mac/yc -j /usr/bin/jvm -onlyCapture -p $1 -hd



echo "360-degree data is saved"

We can pass the PID of the Java process by using the %p placeholder in the XX:OnOutOfMemoryError:

-XX:OnOutOfMemoryError="sh thread-dump.sh %p"

One of the problems with running the agent on the OutOfMemoryError is that it might not capture all the information, as JVM may kill the process before finishing the capture. However, we are still able to get at least some of the following metrics:

As HeapDumpOnOutOfMemoryError, running the yCrash agent on OutOfMemoryError is primarily free and provides a significant upside. While troubleshooting a problem, additional information may help to identify and resolve an issue faster. 

However, it’s always better to prevent a problem than deal with it afterward. yCrash agent can help us with this as well. Micro-metrics monitoring produces a constant flow of analytics, which can notify us about issues long before we must deal with their repercussions. 

Conclusion

Automatic actions on OutOfMemoryError is a good practice that not only helps with the investigation of the problem but also helps to update all interested parties. It’s mostly free, and we don’t need to redeploy or even reboot our system to configure this. 

Feel free to try out the scripts or implement custom ones. Automation and notifications might help decrease the application’s downtime and improve overall health. As a result, we’ll get information about failed applications and detailed reports about the incident.