Troubleshooting Spring Boot DB Connection Leaks

All Spring Boot applications connect with storage solution, Database or cache. Database connection leak is commonly observed in modern applications, which can result in  production outages. In this post, let’s discuss how to debug and identify the root causes of database connection issues.

What is a DB Connection Leak?

A database connection leak in Spring Boot occurs when a Java application fails to close or release database connections after they are no longer needed. Each open database connection consumes system resources, and if these connections are not properly closed, they can accumulate over time, leading to resource exhaustion and degraded performance. 

Here are some common reasons why database connection leaks may occur in Spring Boot applications:

• Failure to Close Connections
• Failing to return a connection to the pool

Simulating DB Connections Leak in Spring Boot

In order to simulate the DB connection leaks in Spring Boot application, we leveraged the open source BuggyAPI application, a comprehensive Spring Boot service capable of simulating various performance problems. When we launched the Buggy API application, it looked as below:

Here Spring Boot Buggy API services are simulating DB connection leaks:

public void leakDBConnection(String datasourceUrl, String dbUsername,
    String dbPassword, String tableName) {
  Connection connection = null;
  try {
    connection = getConnection(datasourceUrl, dbUsername, dbPassword);
    // Perform database operations using the connection
    String msg = "Connecting to DB and querying " + tableName;
    log.debug(msg);
    PreparedStatement statement =
        connection.prepareStatement("SELECT * FROM  " + tableName);
    ResultSet resultSet = statement.executeQuery();
    resultSet.close();
    statement.close();
  } catch (SQLException e) {
    e.printStackTrace();
    printLog(e);
    e.printStackTrace();
  } finally {
    try {
      // closeConnection(connection);
    } catch (Exception e) {
      printLog(e);
      e.printStackTrace();
    }
  }
}

public Connection getConnection(String datasourceUrl, String dbUsername,
    String dbPassword) throws SQLException {
  return DriverManager.getConnection(datasourceUrl, dbUsername, dbPassword);
}

public void start(String datasourceUrl, String dbUsername, String dbPassword,
    String tableName) {
  while (true) {
    leakConnection(datasourceUrl, dbUsername, dbPassword, tableName);
  }
}

You can notice that the ‘leakDBConnection()‘ method calls the ‘getConnection()‘ method which in turn opens a connection using JDBC driver manager ‘DriverManager.getConnection(datasourceUrl, dbUsername, dbPassword)’. However, this connection is not closed. When this connection is repeatedly called from the ‘start()’ method, it results in many leaked connections and eventually leads to many connection errors.

Troubleshooting Spring Boot Blocked Threads using yCrash

In order to troubleshoot this problem, we leveraged the yCrash monitoring tool. This tool is capable of predicting outages before it surfaces in the production environment. Once it predicts outage in the environment, it captures 360° troubleshooting artifacts from your environment, analyses them and instantly generates a root cause analysis report. Artifacts it captures include Garbage Collection log, Thread Dump, Heap Substitute, netstat, vmstat, iostat, top, top -H, dmesg, kernel parameters, disk usage… You can register here and start using the free-tier of this tool.

When the above program was run, yCrash automatically detected the problem and reported an incident in it’s dashboard. To see the live root cause analysis incident report generated by yCrash for this problem, click here. Below is the summary excerpt from the report:

From the summary you can notice that yCrash is pointing out that 23 times ‘MySQLNonTransientConnectionException: Too many connections’ errors occurred in the application. It’s also giving a hyperlink to ‘see the details’ of the problem. When the hyperlink is clicked, the below report opens up.

AppLog Report

This application log report shows all the errors that occurred in the application log file. It also provides a detailed stack trace of the error, which points out the code path which was creating the connections and not closing it. Equipped with this information users identify the source of the connection leak and fix the problem.

Network Report

yCrash incident also reports the network connection related metrics. From the network report one can notice there are several connections in ESTABLISHED state to the backend Database. A high and growing number of connections to a database server can indicate a leak and require further investigation.

Possible Solutions

An appropriate solution needs to be applied based on the specific connection leak issue. Below are some recommended resolutions for DB connection leak issues: 

Recommendation	Description
Use try-with-resources	Use try-with-resources statement, which automatically closes resources like Connection, Statement, and ResultSet when they are no longer needed. This helps reduce the chances of connection leaks.
Close connections in a finally block	Make sure to close the Connection in a finally block to ensure it is closed, even if an exception occurs.
Use Connection Pools	Consider using connection pooling libraries instead of managing connections manually. Connection pools automatically manage connection acquisition and release.

Conclusion

In this article, we explored the risks of DB connection leaks in Spring Boot applications. These leaks can lead to performance issues and ‘out of connections’ errors. We discussed automated tools such as yCrash to detect leaks. Addressing DB connection leaks is vital to maintaining application stability and reliability. For more insights into Spring Boot troubleshooting articles, click here.