Monitoring Metaspace and its memory consumption is crucial for the health and stability of Java applications. Metaspace stores metadata related to classes, methods, and other structural elements of Java code. If Metaspace usage exceeds its allocated capacity, it can lead to OutOfMemoryError exceptions, resulting in application crashes or unexpected behavior. By closely monitoring Metaspace usage, developers can receive early warnings when memory usage approaches critical levels, enabling proactive measures to prevent errors and ensure application stability.

Detecting memory leaks is another important aspect of monitoring Metaspace usage. Memory leaks occur when objects are retained in memory longer than necessary, gradually exhausting available memory resources. By tracking Metaspace usage over time, developers can identify abnormal growth patterns that may indicate memory leaks and take corrective actions to address them. Monitoring Metaspace usage provides valuable insights into memory management practices and helps maintain application reliability and performance.

Metaspace usage is directly influenced by class loading activities in Java applications. Monitoring Metaspace usage patterns enables developers to understand how classes are loaded and unloaded dynamically at runtime. By analyzing these patterns, developers can optimize class loading strategies, reduce unnecessary class loading overhead, and improve overall application performance. 

Effective monitoring and management of Metaspace memory allocation are essential for resource optimization, ensuring that sufficient memory resources are available to support the application’s runtime requirements and minimizing resource contention issues. By fine-tuning JVM configuration settings based on insights from Metaspace monitoring, developers can optimize memory usage and enhance the stability and performance of Java applications in production environments.

Today I will demonstrate a Boomi Process which excessively fills up the Metaspace, and then I will use a diagnostic tool like yCrash to triage and identify the scenario. We will perform the tests on my Macbook Pro M2, hosting a single Boomi Atom.

Boomi Implementation

In order to simulate filling up the Metaspace, I have decided to create an unlimited/infinite amount of dynamic classes defined during runtime and loaded into the JVM. The loop to create the dynamic classes is infinite so eventually, the Metaspace will fill up. Below is the code to create our desired scenario:

import java.util.Properties
import java.io.InputStream
import com.boomi.execution.ExecutionUtil
import groovy.lang.GroovyClassLoader

def logger = ExecutionUtil.getBaseLogger()
def classLoader = new GroovyClassLoader()

for (int i = 0; i < dataContext.getDataCount(); i++) {
    InputStream is = dataContext.getStream(i)
    Properties props = dataContext.getProperties(i)

    try {
        while (true) {
            // Generate a unique class name using the current timestamp
            def className = "DynamicClass_${System.currentTimeMillis()}"

            // Define the Groovy source code for the class dynamically
            def classSourceCode = """
                class $className {
                    void sayHello() {
                        println 'Hello from dynamic class!'
                    }
                }
            """

            // Parse and load the class dynamically
            def dynamicClass = classLoader.parseClass(classSourceCode)
        }
    } catch (Exception ex) {
        logger.info(ex.getMessage())
    }

    dataContext.storeStream(is, props)
}

The above script initializes the Groovy Class Loader. Then in the following document for loop, it instantiates an infinite loop, inside of which a dynamic class is defined and then loaded into the JVM. Finally, a method is invoked from the dynamic class proving that it is loaded into the JVM runtime. The script successfully demonstrates a scenario where the Metaspace memory fills up within the Boomi ecosystem. 

I also created a JavaScript version of the script. The Boomi process itself is a 3 shape/step process consisting of a “no data” start shape, a data processing shape to load the script, and a stop shape to successfully terminate the process. This one ensures that all the program does is load a dynamic class into Metaspace:

Fig: Boomi Package and Deploy Screen

The following code is designed to simulate and analyze OutOfMemoryError scenarios in a Boomi process:

function dynamicClass() {
    // Generate a unique class name using the current timestamp
    var className = 'DynamicClass_' + Date.now();

    // Define the class dynamically
    var classDefinition = 'function ' + className + '() {\
        console.log("Instance of dynamic class created");\
    };\
    ' + className + '.prototype.sayHello = function() {\
        console.log("Hello from dynamic class!");\
    };';

    // Load the class using eval()
    eval(classDefinition);

    // Return the class name for reference
    return className;
}

for (var i = 0; i < dataContext.getDataCount(); i++) {
    var is = dataContext.getStream(i);
    var props = dataContext.getProperties(i);

    try {
        while (true) {
            dynamicClass();
        }
    } catch (error) {
        console.error('Metaspace limit reached:', error.message);
    }

    dataContext.storeStream(is, props);
}

The Boomi Process to run the above script consists of three shapes/steps: A no-data start shape to signify the start of the process, a data process shape referencing the above custom groovy script, and finally a stop shape to normally terminate the process. Subsequently , I deployed the process to the Boomi Atom hosted on my Macbook Pro. 

Fig: Boomi AtomSphere IDE – Creation of [Java] MetaSpaceFill Process

Boomi Process Execution

Inside the Process Reporting Screen, the process executed and never finished, it remained in a pending state. 

Fig: Boomi Process Reporting – [Java] MetaSpaceFill Execution

The Atom/Container logs explain more about what happened during the running of this process:

2024_03_07.container.log:Mar 7, 2024 9:32:00 PM EST WARNING [com.boomi.container.config.ResourceManager updateMemoryStatus] Low memory: init = 134217728(131072K) used = 371059712(362363K) committed = 536870912(524288K) max = 536870912(524288K)

2024_03_07.container.log:Mar 7, 2024 9:38:00 PM EST WARNING [com.boomi.container.config.ResourceManager updateMemoryStatus] Low memory: init = 134217728(131072K) used = 376913408(368079K) committed = 536870912(524288K) max = 536870912(524288K)

Results From The Two Tools

I used two tools, Atom and yCrash, to compare the results of the OutOfMemoryError. Here are their findings: 

Boomi Results

The Atom detected an OutOfMemoryError exception, as a result it restarted itself. The job itself responsible for the causing of this scenario eventually failed out due to Java heap space. I would imagine some delay as the server responsible for submitting the specs on the job had restarted.

yCrash Diagnostics

Flipping over to yCrash, the tool I am using to monitor JVM diagnostics, I was able to see that an incident was created. The incident was tied explicitly to the Boomi process.

Fig: yCrash Boomi Incident Report – Application Issue – Metaspace

yCrash correctly identified the memory leak in the metaspace region of the JVM. 

Fig: yCrash GC Intelligence Report – Memory Leak in MetaSpace

yCrash makes it very easy to identify the overallocated Metaspace further down in the report. 

Fig: yCrash – GC Intelligence Report – JVM Memory Size

The report can be retrieved here.

Conclusion

yCrash played a pivotal role in uncovering the root cause behind the memory leak, which was adversely affecting the application’s performance. While Boomi’s alert initially indicated a heap issue, it did not specifically pinpoint the problem to PermGen/Metaspace. It was through our meticulous examination of the container logs that we were able to detect these alerts. Notably, yCrash not only promptly identified the Metaspace memory leak issue but also went a step further by offering actionable insights to mitigate the problem. 

Moreover, in the GC Intelligence report, yCrash provided comprehensive visual representations, including diagrams, illustrating the extensive allocation to Metaspace. These visual aids effectively showcased the disproportionate magnitude of Metaspace utilization compared to other components of the JVM’s memory, such as Old Gen or New Gen.