Spring Boot is a widely used framework for building highly transactional Java-based web applications and backend systems. Metaspace, or Permgen (before version 8), refers to the memory space where metadata related to classes and methods is stored. Spring Boot includes numerous boilerplate libraries that require a significant amount of Metaspace, often leading to Metaspace OutOfMemory issues. 

What is a MetaSpace OutOfMemory Issues?

An OutOfMemoryError related to the Metaspace (java.lang.OutOfMemoryError: Metaspace) in Spring Boot applications indicates that the JVM is unable to allocate more memory for the Metaspace area. A Metaspace is part of the native memory used by the Java Virtual Machine (JVM) to store class metadata and other runtime information. Here are common reasons for Metaspace OutOfMemory Errors:

1. High Class Metadata Usage: If your application dynamically generates, loads, or unloads a large number of classes, particularly during runtime or scenarios like hot code replacement, can lead to high Metaspace usage and potential issues.
2. Leaked Classes: Leaked classes that are not properly unloaded or released can contribute to Metaspace exhaustion. This might happen if classes are loaded and retained unnecessarily.
3. Inadequate Metaspace Size Configuration: If the Metaspace size is not appropriately configured for your application’s needs, it may run out of space. You can adjust the Metaspace size using JVM options, such as -XX:MaxMetaspaceSize. 
4. Use of Reflection and Dynamic Proxies: Excessive use of reflection or dynamic proxies in your code can lead to increased Metaspace consumption.

Note: Here are some crucial JVM Arguments that can be used to configure your Java Applications that can come in handy. 

Simulating MetaSpace out-of-memory in Spring Boot

In order to simulate the MetaSpace OutOfMemory problem in Spring Boot, we leveraged the open source BuggyAPI application, a comprehensive Spring Boot service capable of simulating various performance problems. When we launched the Buggy API application, it looked like this:

Fig: Spring Boot Buggy API Service

Here Spring Boot Buggy API services are simulating metaspace out of memory issues:

@Service
public class MetaspaceLeakService {

    public  void start() throws Exception {
        
    	long startTime = System.currentTimeMillis();
    	
        ClassPool classPool = ClassPool.getDefault();
        for (int i = 0; i < 750_000; i++) {
            
        	if (i % 50_000 == 0) {
        		
                System.out.println(i + " new classes created");
            }
        	
        	// Keep creating classes dynamically!
        	classPool.makeClass("com.buggyapp.metaspaceleak.MetaspaceObject" + i).toClass();
        }
        
        System.out.println("Program Exited: " + (System.currentTimeMillis() - startTime) / 1000 + " seconds");
    }
}

This program leverages the ‘ClassPool’ object from the opensource javassist library. This ‘ClassPool’ object is capable of creating new classes at runtime. Please take a close look at the above program. You’ll notice this program keeps on creating new classes. Below arethe sample class names generated by this program:

com.buggyapp.metaspaceleak.MetaspaceObject69101 new classes created
com.buggyapp.metaspaceleak.MetaspaceObject69102 new classes created
com.buggyapp.metaspaceleak.MetaspaceObject69103 new classes created
com.buggyapp.metaspaceleak.MetaspaceObject69104 new classes created
com.buggyapp.metaspaceleak.MetaspaceObject69105 new classes created
com.buggyapp.metaspaceleak.MetaspaceObject69106 new classes created
com.buggyapp.metaspaceleak.MetaspaceObject69107 new classes created

Whenever a new class is created, its corresponding class metadata definitions are created in the JVM’s Metaspace region. This causes the Metaspace to grow. When the maximum metaspace size is reached, the application will experience ‘java.lang.OutOfMemoryError: Metaspace’

Troubleshooting Metaspace OutOfMemory Exceptions

In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before they surface in the production environment. Once it predicts outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them and instantly generates a root cause analysis report. Artifacts it captures include garbage collection log, thread dump, heap substitute, netstat, vmstat, iostat, top, top -h, dmesg, kernel parameters, disk usage…. 

You can register here and start using the free-tier of this tool.

Below is the report generated by the yCrash tool when the above sample Spring Boot program is executed:

Fig: yCrash GC showing OutOfMemory issues with suggestions

When Metaspace runs out of memory, the GC analysis also shows continuous garbage collection cycles during out-of-memory events as below.

Fig: yCrash tool showing continuous GC cycles

Possible Solutions

An appropriate solution needs to be applied based on the specific problem. Below are some recommended resolutions for Metaspace OutOfMemory issues: 

Recommendation	Description
Optimize Class Loading	Reduce unnecessary class loading by optimizing your application’s class loading process. Minimize the use of dynamic class loading if possible.
Increase Metaspace Size	Adjust the Metaspace size by increasing the MaxMetaspaceSize parameter. You can set it in the Java Virtual Machine (JVM) options using the -XX:MaxMetaspaceSize flag.
Garbage Collection Tuning	Adjust garbage collection settings to better suit your application’s needs. You can experiment with different garbage collection algorithms and parameters to find the optimal configuration. For example, you might use the -XX:+UseG1GC option to enable the Garbage-First (G1) collector.

Conclusion

In this blog, we looked in detail at troubleshooting Metaspace OutOfMemory exceptions with an emphasis on detecting the root cause and resolving Metaspace OutOfMemory in Spring Boot applications.