OQL vs Heap Dump Analyzers: How to Use OQL for JVM Troubleshooting 

Is OQL a necessary skill for Java professionals? Don’t heap dump analyzers give us everything we need to solve memory problems?

Recently, yCrash hosted a webinar: OQL for JVM Troubleshooting: Querying Heap Dumps Like a Database, where I was a speaker. 

We had a great session, but the best part was the short Q&A session at the end. Our audience included real pros in the Java troubleshooting world, and the questions they asked were extremely thought-provoking. Not only thought-provoking, but they also highlighted the pros and cons of Object Query Language vs the standard reports we get from a good heap dump analyzer.

In this article, we’ll pick up on a few questions that weren’t fully answered during the session and investigate them to see how OQL complements heap dump analyzers like HeapHero or Eclipse MAT.

We’ll see that every troubleshooting tool and artifact has its own place in determining root causes, and investigate how they work together to streamline problem solving.

What Is OQL and When Should You Use It for Heap Dump Analysis? 

 For hard-to-find bugs or performance problems, especially those that are related to memory, capturing a heap dump is invaluable. It’s a snapshot of the heap at a given moment, but since it’s a very large binary file, we’d find it almost impossible to analyze it manually. Luckily, we have several excellent tools that allow us to explore the file’s contents. In this article, we’ll be using HeapHero for this purpose. For a quick and effective lesson on using a heap dump analyzer to locate memory issues, watch How to Analyze a Heap Dump Fast.

Tools such as HeapHero contain a wealth of useful interactive reports. They provide the information we need to pinpoint 90% of memory issues, and include:

• Analytical problem report;
• Overview, including total heap usage and information about the runtime environment;
• Largest Object Report. In most cases, we find the culprit that’s causing memory problems in the top three or four largest objects.
• Class Histogram, listing all classes currently loaded in memory, along with the number of instances and total heap usage.
• Thread Analysis
• Duplicate Classes
• GC Roots
• Unreachable Objects
• System properties
• Objects awaiting finalization.

We can expand the reports to see incoming and outgoing references, and the path of each object to its GC root. We can also search the largest object report by class, and group it in several different ways. In fact, the developers of the tool have anticipated most of what we’re likely to search for in a heap dump, and made the search process simple.

So why do we need OQL?

We estimated that a heap dump analyzer is sufficient in 90% of cases, but there’s always that pesky 10%. For those cases, we need to dig a bit deeper, or view the heap in ways the developers didn’t anticipate. This is where the flexibility of OQL comes in.

Here are a few examples of situations where OQL is useful:

• View the retained heap size of every object created by the third-party library com.image

SELECT @displayName, @retainedHeapSize FROM "com\.image\..*" s 

• View all strings whose length is greater than 1000.

SELECT * FROM java.lang.String s WHERE value.@length > 1000

• View all threads with a given name pattern.

SELECT * FROM INSTANCEOF java.lang.Thread t where toString(name) like ".*nio.*"

Let’s run this last query on the heap dump of a small Tomcat server. We loaded the dump into HeapHero, and typed the query int the OQL section of the report:

Fig: OQL Query Section of HeapHero Report

HeapHero runs the query, and returns the results as an interactive report:

Fig: Query Results as an Interactive Report

From this report, we can explore further, seeing the actual data, and exploring incoming and outgoing references, as well as tracing the object’s path to its GC root.

OQL, therefore, complements the facilities offered by the heap dump analyzer, allowing for more flexible searches.

This gives us the answer to the initial question: Heap Dump Analyzer vs OQL query: Which is better? Neither is better; they work together to cater for a wide range of troubleshooting needs. OQL allows us to filter classes by search criteria, so we can pull out the exact objects we need to examine.

DevOps need a variety of tools for analyzing performance and troubleshooting. OQL is just one of them.

OQL Queries for JVM Troubleshooting: Questions and Practical Examples 

Let’s now work through a few of the questions from the webinar. These illustrate that OQL is just one of several troubleshooting tools, and how these tools work together.

1. Can OQL identify all threads in the heap dump? Can we tell which ones are running? (Question by Pradeep Kumar)

Let’s break this down and take a look at it, shall we? 

Identifying Threads in a Heap Dump

It’s simple to identify all the threads in the heap dump with OQL.

If we run this query:

SELECT * FROM INSTANCEOF java.lang.Thread t 

We get a list of all objects created from the Thread class and any of its subclasses.

Fig: List of Threads

We can then explore further to see field values and references.

Identifying Running Threads

However, identifying which threads are running is not so simple. Thread state representation is not the same in every version of Java. Looking at the values of a Thread object from a Java 17 heap dump, we see:

Fig: Thread Object Values: Java 17

Can we use the attribute threadStatus to determine if the thread is running? Could we filter on this attribute to return all running threads?   Yes, if we know that the dump was definitely taken with this version of Java, and if we know what the status codes represent. Unfortunately, the status codes may differ between JVM versions. If we’re writing a Java program, and we need to know the state of a thread, we’d use the getState() method of the Thread class.  This  caters for differences in the actual underlying hidden field. The JVM developers are therefore under no obligation to keep the contents of the private field ThreadStatus consistent across versions.

To make it more difficult, later versions of Java relocated some of the fields from the Thread class into an inner class java.lang.Thread$FieldHolder (highlighted in the image below) when they developed virtual threads. If we look at the values of a Thread object in Java 26, we see it looks quite different:

Fig: Thread Object Values: Java 26

If we navigate to the FieldHolder object and inspect its values, we see the following fields, including threadStatus.:

Fig: FieldHolder Values

As you can see, accessing and decoding the thread status to find all the running threads is complex and likely to change between JVM releases.

To troubleshoot concurrency issues, the heap dump is not our best diagnostic artifact, although we can certainly use it to find thread-related information. A much better strategy is to capture a thread dump. It’s a text file, so we can analyze it manually, but it’s much quicker to use a tool such as fastThread. This produces an interactive report, including a summary of thread states, as shown in the diagram below:

Fig: Thread State Summary

To see details of each individual thread, click “View Details” under its category. Each thread’s details look like this:

Fig: Thread Details

It’s even more valuable to track what’s happening to each thread over a short period. Is the thread progressing? Are several threads hung at the same place? 

To do this, take at least three thread dumps at intervals of about ten seconds. fastThread has the option to compare these dumps. The image below is a comparison summary of three dumps.

Fig: Thread State Comparison

The report lists each thread by name, comparing its state over the three dumps. We can click on a thread to see more details.

One thing to note: just because a thread is in the RUNNABLE state, it’s not necessarily running. We usually have more threads than CPU cores, and the operating system allocates CPU time between them. To see which threads are actually running, and how much CPU they’re using, we would use an operating system command, such as Linux top – H -p <PID> . PID is the process of the Java application. This shows an interactive screen as shown below. The screen updates itself over time, so we can watch what threads are doing.

Fig: Output of top -H for a Tomcat Server

For this situation, OQL and even the heap dump analyzer can only give us limited information. Our best tactic is to explore thread dumps in conjunction with the output of the top command.

2. What OQL Query Can Find Objects Still Retaining Memory Referenced by a Thread? (Question by Jalaj Asher)

This question is a bit ambiguous. In fact, we could interpret it four ways. Let’s see what they are and answer each of those separately: 

Interpretation 1. How do we find objects that the thread once referred to and has released, but are still retaining memory because they are held by another object?

We can’t. The heap is a snapshot at a given moment, so we have no way to trace references that existed historically.

Interpretation 2. How do we find objects that are no longer referenced anywhere, but are still holding memory because they have not yet been garbage collected?

I experimented with this one, but I couldn’t find a way to do this. Fortunately, we don’t need to: HeapHero has a section of the report that lists exactly these objects. See an example of the Unreachable Objects chart below:

Fig: Unreachable Object Report

Interpretation 3: How do we find objects that a thread refers to via weak, soft or phantom references, which are still occupying memory because something else holds a strong reference?

There could be several ways of doing this. Let’s simulate it in a sample program.

import java.lang.ref.WeakReference;
// ===========================================================
// Used for creating a heap dump where Thread1
// has a weak reference and Thread2 has strong ref to a String
// ===========================================================
public class BuggyProg20 {
    private static final String VALUE = "ABCDEF";
    public static void main(String[] args) {
        Thread thread1 = new Thread(new WeakReferenceTask(), "Thread-1");
        Thread thread2 = new Thread(new StrongReferenceTask(), "Thread-2");
        thread1.start();
        thread2.start();
    }
    static class WeakReferenceTask implements Runnable {
        @Override
        public void run() {
            WeakReference<String> weakRef =
                    new WeakReference<>(VALUE);
            System.out.println("Thread-1: " + weakRef.get());
            sleep();
        }
    }
    static class StrongReferenceTask implements Runnable {
        @Override
        public void run() {
            String myString = VALUE;
            System.out.println("Thread-2: " + myString);
            sleep();
        }
    }
    private static void sleep() {
        try {
            Thread.sleep(10 * 60 * 1000L);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
    }
}

This creates two threads: Thread-1 and Thread-2. Thread-1 creates a weak reference to a String named VALUE, whereas Thread-2 holds a strong reference to the same String. The program sleeps for a long period to allow time to take a heap dump. We loaded the dump for this program into HeapHero, and ran the following query:

select * FROM INSTANCEOF java.lang.ref.Reference r WHERE toString(dominatorof(r)).contains("Thread-1")

This returns any object whose class extends java.lang.ref.Reference that is owned by Thread-1.

We can expand the result set to see the referent, which is the actual String it refers to.

Fig: Expanding the Result Set

If we click ‘more’ against the referent, we get a menu that lets us explore incoming references. Selecting this option shows a new report for this String object. We can expand it to see all its incoming references:

Fig: Exploring the Result Set

This lets us find everything that is preventing the referent of the WeakReference from being garbage collected. This is a good example of where we can use OQL in conjunction with the heap analyzer reports.

Q4: How do we find objects that are no longer referenced, but are still in memory because they’re waiting for finalization?

It’s certainly possible to do this with OQL, since we can use it to explore finalizer queues. In practice, we wouldn’t do this, because HeapHero already contains a report of all objects waiting for finalization, as shown below.

Fig: Objects Waiting for Finalization

3. Can OQL Identify the Thread That Caused an OutOfMemoryError? (Question by Rahul Naraniya)

I thought this might be possible via an Exception object thrown by the thread. This turned out to be an exercise in chasing fairies: there is no Exception object in the heap, because it’s a static class.

The best way to find out which thread threw the exception is not via the heap dump. The application log should tell us which thread threw the error, like this:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to BuggyProg18.hprof ...
Heap dump file created [5400260 bytes in 0.076 secs]
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "clutterThread"

In this case, the heap dump is not the right artifact for the task. We can find the answer in the application log.

4.Can we Identify Which Heap Space (eg., Eden, OG or Humongous) an Object Resides In? (Question by Gokul L)

The answer to this is almost always ‘No’. This information is stored by the Garbage Collector in its own working memory, which resides in native memory, not in the heap.

The only way it might sometimes be possible, if the answer is really important, is to use OQL to find the memory address of the object, and then use a low-level memory analyzer such as the HotSpot Serviceability Agent (HSA) to find out the memory boundaries of Eden and other GC spaces.

This OQL query gives us the Hex memory address of the String VALUE in the previous sample program:

SELECT toHex(getObjectAddress()) from java.lang.String s where toString(s) ="ABCDEF"

Using HSA is quite complex, and beyond the scope of this article, but we can use it to get GC-specific information from a running program or from a core dump. However, modern garbage collectors such as G1GC and ZGC use an elaborate memory arrangement, where memory is split into regions. This would make it very difficult to track which space a given memory address actually falls into.

Conclusion

For deep analysis, we need a variety of artifacts and tools. 

The simplest way to make sure we have all the diagnostic information we need is to use the yc-360 script, which we can download for free from yCrash’s Github repository. With this information and a good set of analysis utilities, troubleshooting becomes much easier.

OQL is one of many useful gadgets worth keeping in your well-stocked troubleshooting toolbox.