Site icon yCrash

How to Troubleshoot Slow Client (Slowloris) in Node.js

Mahesh Devda

Production issues in Node.js applications can be challenging to diagnose, especially when different problems exhibit similar symptoms. To make troubleshooting easier, we’re introducing a Node.js Production Troubleshooting Series, where each article focuses on one specific production issue, explores its underlying causes, and demonstrates a practical, step-by-step approach to identifying the root cause.

Slow Client is one of those production issues that looks like a socket leak. The client opens a connection, sends only part of an HTTP request, and then stops sending data.

The server keeps the connection open while waiting for the rest of the request. The event loop can continue running, so the application may not appear to be stuck. However, more and more connections can remain open and consume server resources.

If this continues, the server can run out of available connections, leaving legitimate requests waiting for a connection to become available.

In this blog, we’ll walk through how a Slow Client (Slowloris-style) attack shows up in a Node.js application, what typically causes it, and how to trace it back to the exact root cause. We’ll also simulate the issue in a sample application, capture diagnostic data as it happens, and analyze it step by step to pinpoint what’s going wrong.

What is a ‘Slow Client’ in Node.js?

HTTP works by sending a request and then receiving a response. The problem starts when a client connects but takes a very long time to finish sending the request. For details on how Node.js handles HTTP servers and requests, see the Node.js HTTP API documentation.

For example, a client might send a few HTTP headers and then stop. The server keeps the connection open while waiting for the rest of the request. The same thing can happen when a client sends the request body very slowly.

Node.js keeps the net.Socket open until the request is complete. If hundreds of clients do this at the same time, the server can end up with a large number of open connections, even though the application itself is barely using the CPU.

What Causes a ‘Slow Client’ problem in Node.js?

Here is the usual list of causes for the ‘Slow Client’ problem.

  1. Incomplete HTTP headers: The client sends headers such as Host and Content-Length, but never sends the blank line that tells the server the headers are complete. The server keeps waiting for the rest of the request.
  2. Slow body: A client may declare a large Content-Length but send only a few bytes every second. Node.js keeps waiting for the remaining data while the connection stays open.
  3. No request/headers timeout: If `headersTimeout` or `requestTimeout` is not configured appropriately, a client that stops sending data can keep the connection open for a long time.
  4. No connection cap: If the server allows a large number of connections without suitable limits, hundreds or thousands of slow clients can consume available sockets and other resources.

How to Simulate a Slow Client in Node.js

To understand how a Slow Client appears in the diagnostic data, let’s reproduce the issue using a sample Node.js application.

The following program starts a normal http.Server, then opens five raw TCP clients every 500 ms. Each client writes a partial POST header and stops. There is no final blank line, so the server never sees a complete request.

``` JS Code (slow-client.js)
const http = require('http');
const net = require('net');
class SlowClientDemo {
    constructor() {
        this.server = null;
        this.clients = [];
        this.intervalId = null;
    }
    start() {
        if (this.intervalId) return;
        this.server = http.createServer((req, res) => {
            req.on('data', () => {});
            req.on('end', () => res.end('OK'));
        });
        this.server.listen(0, () => {
            const port = this.server.address().port;
            this.intervalId = setInterval(() => {
                for (let i = 0; i < 5; i++) {
                    const client = new net.Socket();
                    client.connect(port, () => {
                        // Partial headers. No final blank line. Server waits.
                        client.write(
                            'POST / HTTP/1.1\r\n' +
                            `Host: 127.0.0.1:${port}\r\n` +
                            'Content-Length: 100000\r\n'
                        );
                    });
                    client.on('error', () => {});
                    this.clients.push(client);
                }
                console.log(`Slow client: ${this.clients.length} half-open requests`);
            }, 500);
        });
    }
    stop() {
        clearInterval(this.intervalId);
        this.clients.forEach((c) => { try { c.destroy(); } catch (_) {} });
        this.clients = [];
        if (this.server) this.server.close();
    }
}
module.exports = new SlowClientDemo();
```

How to Capture Diagnostic Data for Troubleshooting Slow Client

yCrash is a diagnostic tool that captures runtime performance data from a live Node.js process, including CPU usage, event loop lag, worker thread activity, and call stacks, and turns that data into a report that points to the root cause. To set up yCrash for your Node.js application, follow the steps below:

1. Install yCrash

yCrash is available in both Cloud and On-premises versions. Use one of the options below to get started:

2. Set up the yCrash Node.js hook first

Please note that this is a one-time setup. The hook must be loaded into your Node.js application before yCrash can capture the Node.js internals data that matters for diagnosis, things like worker thread CPU, event loop lag, method profiling, and call stacks. To set this up, follow the Hook Mode steps here: Node.js Diagnostic Capture. This hook setup adds very minimalistic, almost zero overhead to your application.

3. Launch yc-360 script

Configure and start the yc-360 script to monitor your Node.js process. The full setup and configuration options specific to your deployment are available here: Micro-metrics Monitoring (M3) Mode

When the yc-360 script detects a problem, it captures  360° diagnostic artifacts (GC log, Event loop lags, Worker threads, CPU Profile, Unhandled rejections, Call stack, Active handles, Process, Storage, Kernel, Network, etc.). It uploads them to the yCrash server automatically. yCrash server analyzes the artifacts using advanced pattern recognition and ML algorithms to identify the root cause of the problem.

How to Analyze Slow Client Using the Diagnostic Data

In this section, we’ll see how to analyze a Slow Client incident using the diagnostic data captured by yCrash. We’ll start with the incident on the calendar dashboard, check the RCA Summary to understand what’s going on, open Active Handles to see inbound TCP sockets climbing, and then check Event Loop Lag to confirm the main thread was not stalled.

Step 1: Open the incident from the yCrash dashboard

When the yc-360 script detects any performance degradation in the Node.js application, it automatically captures 360° diagnostic data and transmits it to the yCrash server. The yCrash server analyses this data and creates an incident in the yCrash dashboard as shown in the screenshot below. Open that incident from the yCrash calendar dashboard to view the reports.

Fig: yCrash dashboard, incident list view

Step 2: Start with the RCA Summary Page

Before diving into details, let’s start with the RCA Summary. yCrash analyzes the Node.js diagnostic data and brings the key findings together in simple language with their severity levels.

In the RCA Summary, we will focus on two sections:

AI Overview

At the top of the RCA summary report page, you’ll see an AI Overview section. It gives you an AI-generated summary of the incident, which would make even a non-technical person understand the root cause of the problem. Within the AI Overview section, below the AI summary, you can also click on the ‘Dive Deeper in AI Mode‘ button to open a chat window to ask any question to AI about this incident.

Fig: RCA Summary highlighting the AI Overview

As shown in the AI Overview above, the report first highlights an event listener leak, which increased from 4,642 to 7,911. It then reports the TCP socket leak, which increased from 700 to 1,200.

During the capture, the number of listeners increased by 3,269, while TCP sockets increased by 500. These extra listeners are callbacks attached to the open HTTP connections. They are a side effect of the slow clients, not a separate listener leak.

Issues in application and device

Fig: RCA Summary highlighting the issues in application and device

As shown in the above screenshot, 3 application-level issues and 1 device-level issue are reported. The one that matters for Slow Client is:

The Fatal listener row is a secondary problem. We can ignore the other 2 warnings shown in the above screenshot as they don’t cause the slow client issue.

Step 3: Confirm inbound sockets on Handle Growth

Click the “detail here” link present at the end of the TCP socket leak problem on the RCA summary page. This takes you to the Active Handles & Requests Growth Trend section in the Node.js Internals report page.

Fig: Active Handles showing TCP sockets at 1,200 under a 7,911 listener line


The last sample shows TCP sockets at 1,200 and Event listeners at 7,911. The total is 9,116. Both the TCP socket and EventListener counts continue to increase, while pipes, TCP servers, and timers remain flat.

How to Fix Slow Client in Node.js

The following are the potential solutions for this issue:

  1. Set `headersTimeout` and `requestTimeout` on `http.Server` so a silent client is dropped.
  2. Cap concurrent inbound connections. A `Slowloris` works by filling the slot table, not by using CPU.
  3. Put a reverse proxy or load balancer in front with its own idle and header timeouts.
  4. When Issues lists both a listener leak and a TCP socket leak, and App CPU is about 1%, think inbound half-requests before you start deleting `EventEmitter.on()` calls.

Conclusion

Diagnosing and identifying the root cause of a Slow Client incident can be challenging when the RCA page matches HTTP Agent Exhaustion. In this example, the diagnostic data was captured using the yc-360 script and analyzed using the yCrash server, which helped correlate the Fatal TCP climb with inbound sockets that never finished HTTP.

The analysis traced the issue to partial requests that held the server-side connection open. TCP sockets went from 700 to 1,200.

Timing out incomplete headers and capping how many connections you accept address the underlying problem and help prevent the issue from recurring.

Exit mobile version