Causes, configuration checks, and troubleshooting recommendations for yCrash incident findings
Your application communicates with databases, APIs, microservices, and other systems over TCP. When connection establishment, reuse, or cleanup is disrupted, you may experience slow responses, timeouts, failed requests, and resource pressure. TCP states reveal which part of a connection’s lifecycle is underway. A state is not itself an error; unusual counts, persistence, and related diagnostic evidence determine whether a finding requires action.
yCrash identifies potentially abnormal connection patterns in your incident analysis. Review any correlated application-log, thread, or runtime findings first. Use this guide to understand the TCP state and decide which concrete application or infrastructure settings to inspect next. The checks below give you concrete setting names to review; change them only if your incident evidence supports the adjustment.
1. Understanding TCP Connection States
| TCP state | What it means |
|---|---|
| ESTABLISHED | TCP handshake completed; both endpoints can exchange application data. |
| CLOSE_WAIT | The remote peer initiated closure; your application has not yet closed its local socket. |
| TIME_WAIT | TCP temporarily retains state after closing to handle delayed packets and acknowledgments. |
| SYN_SENT | An outgoing connection has sent SYN, but its TCP handshake is incomplete. |
| SYN_RECV | An incoming connection has received SYN and sent SYN-ACK; handshake completion is pending. |
| FIN_WAIT1 | Your endpoint sent FIN and is waiting for the TCP close sequence to advance. |
| FIN_WAIT2 | Your FIN was acknowledged; your endpoint is waiting for the peer’s FIN. |
| LAST_ACK | Your endpoint sent its final FIN and is waiting for acknowledgment. |
| CLOSING | Both endpoints initiated closure; acknowledgment of your local FIN is pending. |
Use this quick reference to identify the state, then jump to the corresponding section for causes and settings to inspect. A TCP state alone does not prove a defect.
2. CLOSE_WAIT — Connections closed by the peer, not yet closed locally
What Does It Mean?
The remote peer has sent a FIN and your TCP stack has acknowledged it, but the local application has not yet completed closing its socket. The connection remains in this state while your application finishes its side of the shutdown. Brief transitions can be normal, but persistent accumulation may point to delayed resource cleanup, blocked execution, or a connection leak.
Common Causes
- Sockets are not closed after use or on error paths.
- Threads, tasks, or asynchronous operations stall before releasing network resources.
- A connection pool retains unusable sockets after peer-initiated closure.
Recommended Fixes
1. Ensure database connections, HTTP responses, and other network resources are released after each operation, including when requests fail or are cancelled.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Java | JDBC Connection / Statement / ResultSet try-with-resources; Apache HttpClient CloseableHttpResponse.close() (version dependent); release/consume response entities; HttpURLConnection response/error stream closure; Socket.close() when using raw sockets |
| .NET | Dispose() / DisposeAsync(); HttpResponseMessage.Dispose() |
| Node.js | net.Socket.end() for graceful shutdown; net.Socket.destroy() when necessary; HTTP request/response cleanup |
| Python | socket.close(); socket context managers; requests.Response.close(); requests.Session.close() |
| Go | net.Conn.Close(); http.Response.Body.Close() |
2. Review HTTP client connection pools to ensure idle, stale, or remotely closed connections are detected and retired appropriately.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Java | Apache HttpClient evictIdleConnections(…), validateAfterInactivity (API location varies by version); OkHttp ConnectionPool keep-alive duration and max idle connections |
| .NET | SocketsHttpHandler.PooledConnectionIdleTimeout; PooledConnectionLifetime |
| Node.js | http.Agent keepAlive, maxSockets, maxFreeSockets; agent free-socket timeout where supported |
| Python | requests.Session; urllib3.PoolManager and connection-pool lifecycle |
| Go | http.Transport.IdleConnTimeout; MaxIdleConns; MaxIdleConnsPerHost |
3. Review database connection pool validation, lifetime, and idle-retirement settings to detect peer-closed connections and prevent unnecessary retention; also check for connections not returned to the pool.
| Platform | Database pool settings and APIs to inspect |
|---|---|
| Java | HikariCP maxLifetime (set below applicable database/infrastructure connection limits), keepaliveTime (below maxLifetime), leakDetectionThreshold; Tomcat JDBC testOnBorrow, validationQuery |
| .NET | SqlClient / Npgsql Connection Lifetime (connection-string setting); Npgsql Connection Idle Lifetime; close or dispose borrowed database connections |
| Node.js | pg Pool idleTimeoutMillis; mysql2 pool idleTimeout; verify clients are released to the pool |
| Python | SQLAlchemy pool_recycle, pool_pre_ping; close sessions/connections after use |
| Go | database/sql DB.SetConnMaxLifetime(), DB.SetConnMaxIdleTime(); close sql.Rows and return connections promptly |
Important: Pool idle/lifetime validation can reduce stale pooled connections, but it does not fix a leaked checked-out connection. HikariCP maxLifetime does not retire an in-use connection until it is returned.
4. Review correlated yCrash findings for blocked threads, tasks, locks, or application errors that could delay socket cleanup; fix confirmed application issues before changing unrelated network settings.
5. Review response/read timeouts and request deadlines when a stalled operation prevents cleanup; a timeout cannot repair a missing close or resource-disposal call.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Java | Socket.setSoTimeout(); HTTP-client response/read timeout |
| .NET | HttpClient.Timeout; cancellation tokens |
| Node.js | socket.setTimeout(); client request timeout; explicitly handle timeout-triggered cleanup |
| Python | socket.settimeout(); requests timeout=(connect, read) |
| Go | http.Client.Timeout; http.Transport.ResponseHeaderTimeout; context deadlines |
3. SYN_SENT — Incomplete outgoing connection setup
What Does It Mean?
Your endpoint has sent a SYN to initiate an outgoing TCP connection, but the handshake has not finished. It is waiting for the handshake to progress, typically for a SYN-ACK from the remote endpoint. Accumulation can accompany remote-service outages, network filtering, or connection attempts that repeatedly time out.
Common Causes
- Remote host is down or the configured destination IP is wrong or unreachable.
- A firewall, routing rule, or security policy drops packets.
- SYN/SYN-ACK packets are dropped, or the remote host is unavailable.
Recommended Fixes
1. Verify the destination IP, DNS resolution, and routing. Check the configured port as well, distinguishing a fast connection refusal (TCP reset) from a handshake timeout (silently dropped or unanswered packets).
2. Inspect the connection establishment timeout and bounded retry policy so unreachable services do not accumulate pending attempts.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Java | Socket.connect(endpoint, timeout); HttpClient.Builder.connectTimeout() |
| .NET | SocketsHttpHandler.ConnectTimeout; retry handler/policy |
| Node.js | net.connect() timeout handling; selected HTTP-client connect timeout and retry policy |
| Python | socket.create_connection(…, timeout=…); requests timeout=(connect, read) |
| Go | net.Dialer.Timeout; http.Transport.DialContext; context deadlines |
3. Review network access rules. Check outbound and return-path firewalls, security groups, routing tables, load balancers, and relevant network errors recorded by yCrash.
4. SYN_RECV — Incomplete incoming handshakes
What Does It Mean?
Your server has received a client SYN and sent a SYN-ACK, but has not completed the incoming handshake. The final acknowledgment has not yet been processed, so the connection is not fully established. High counts can reflect incomplete client handshakes or elevated inbound connection activity; they do not by themselves prove a SYN flood or queue overflow.
Common Causes
- Clients or intermediaries do not complete the handshake.
- Incoming connection rate is elevated or traffic is hostile.
- Packet loss or server-side queue pressure affects connection establishment.
Recommended Fixes
1. Inspect the listening socket backlog and TCP handshake queue only if the incident shows incomplete incoming connections or queue pressure.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Linux: net.ipv4.tcp_max_syn_backlog | Pending, incomplete TCP handshakes retained by the kernel |
| Linux: net.core.somaxconn | Upper limit for the application listen() backlog |
| Linux: net.ipv4.tcp_syncookies | SYN-cookie behavior during SYN queue pressure |
| Application listen backlog | Listener/framework-specific backlog setting |
| TCP counters | ListenOverflows, ListenDrops, SYN-cookie counters; confirm evidence before tuning |
2. Inspect ingress devices. Check inbound firewall, load balancer, routing, and return-path handling for dropped SYN-ACK or ACK packets.
3. Review traffic patterns. Examine the affected local listening IP:port and connection rate, not only individual client IPs. Assess SYN flood protections only when corroborating evidence exists.
5. FIN_WAIT1 — Local closure is still progressing
What Does It Mean?
Your endpoint initiated closure and sent its FIN, but the TCP close sequence has not advanced to the next completed stage. It may still be awaiting acknowledgment of its FIN or other closing-sequence progress. Persistent accumulation can accompany retransmissions or delays in the closing handshake.
Common Causes
- Loss or delay of closing-sequence packets.
- Remote endpoint or network-path responsiveness issues.
- Very frequent connection creation and shutdown.
Recommended Fixes
1. Inspect transport evidence. Review TCP retransmission, packet-drop, and network-error evidence correlated with the affected endpoint.
2. Check whether a network intermediary expires TCP sessions or half-closed connections before their closing packets complete.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Firewall/load balancer TCP idle timeout | Lifetime for inactive TCP sessions |
| Half-closed TCP timeout | Lifetime for sessions after one side begins closing |
| TCP retransmission / packet-drop evidence | Confirm whether FIN or ACK delivery is disrupted |
3. Reduce unnecessary churn. Review connection reuse and keep-alive in the relevant client, particularly if sessions are repeatedly opened and closed.
6. FIN_WAIT2 — Waiting for remote closure
What Does It Mean?
The remote peer has acknowledged your local FIN, meaning your application has completed the sending side of the close. The TCP stack is waiting for the remote side to send its own FIN. Accumulation suggests the other endpoint is slow to finish its side of the connection shutdown, although continued data transfer or processing can sometimes be legitimate.
Common Causes
- The remote application continues work or still has data to send.
- The remote application does not complete socket shutdown promptly.
- Remote-side connection lifecycle policies retain sessions longer than intended.
Recommended Fixes
1. Inspect remote-side cleanup. Check close/dispose calls, cancellation paths, and unfinished work on the remote application when you control it.
2. Check whether the remote endpoint is retaining connections after you have closed your sending side.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Remote application idle timeout | How long an inactive connection remains open |
| Remote request or operation deadline | Whether unfinished work prevents remote socket closure |
| Remote pool maximum lifetime / retirement | When pooled connections are retired |
| Linux: net.ipv4.tcp_fin_timeout | Applies to orphaned FIN_WAIT2 sockets only, not a fix for application-owned sockets |
7. LAST_ACK — Awaiting final FIN acknowledgment
What Does It Mean?
Your endpoint received the peer’s FIN and sent its own FIN in response. Your TCP stack is awaiting acknowledgment of that final FIN before it completes local closure. Persistent entries may indicate delayed or lost final acknowledgments, not necessarily an application leak.
Common Causes
- Final FIN acknowledgment is delayed or lost.
- Connectivity to the peer is intermittent.
- A firewall or other intermediary prematurely expires TCP state.
Recommended Fixes
1. Check retransmissions. Review repeated FIN retransmissions and relevant network diagnostics where available.
2. Check whether intermediary timeout policies prematurely discard the final TCP close acknowledgment.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Firewall/load balancer TCP idle timeout | Expiration of inactive established sessions |
| Closing-state or half-closed TCP timeout | Expiration of sessions during shutdown |
| FIN retransmissions / packet-loss evidence | Corroborate missing final acknowledgments before changes |
3. Inspect endpoint health. Check the peer and path for intermittent loss or disruption before tuning operating-system retries.
8. CLOSING — Both sides initiated shutdown
What Does It Mean?
Both endpoints initiated connection shutdown before either completed the usual close sequence. Your TCP stack has received the peer’s FIN but is still waiting for acknowledgment of its own FIN. Simultaneous closure is valid TCP behavior; occasional entries require no correction.
Common Causes
- Both endpoints close at nearly the same time.
- Acknowledgments are delayed or retransmitted.
- An intermediate network device interferes with connection teardown.
Recommended Fixes
1. Check whether it persists. Prioritize repeated or accumulating entries rather than isolated observations.
2. Review retransmission evidence. Inspect packet loss and delayed acknowledgments before attributing the state to the network.
3. If connections remain in CLOSING, inspect evidence of delayed closing acknowledgments and intermediary session expiration.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| TCP retransmission counters | Look for repeated close-packet retransmissions |
| Firewall/load balancer half-closed TCP timeout | Whether connection state expires during shutdown |
| TCP session idle timeout | Compare with observed duration of affected connections |
9. TIME_WAIT — Normal cleanup or connection churn
What Does It Mean?
Your TCP stack has completed the active side of the TCP closing sequence and temporarily retains connection state. This helps handle delayed packets and retransmit the final acknowledgment when necessary before the entry expires. Large counts can be entirely normal for busy servers and do not alone prove a socket leak or ephemeral-port exhaustion.
Common Causes
- High volume of short-lived connections.
- Client connections are not reused when reuse would be appropriate.
- Idle policies or upstream intermediaries frequently recycle otherwise healthy connections.
Recommended Fixes
1. Review connection reuse and pooling so the application avoids unnecessarily opening and closing short-lived outbound connections.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
|---|---|
| Java | HTTP-client connection pooling, keep-alive, idle eviction |
| .NET | SocketsHttpHandler.PooledConnectionIdleTimeout; PooledConnectionLifetime |
| Node.js | HTTP/HTTPS Agent keepAlive, maxSockets, maxFreeSockets |
| Python | requests.Session; urllib3.PoolManager |
| Go | http.Transport.MaxIdleConnsPerHost; IdleConnTimeout |
2. Differentiate connection direction. A high count on a listening port can reflect normal inbound-server behavior. Investigate outbound local-port pressure only with evidence of affected destinations and port availability.
3. If the incident includes EADDRNOTAVAIL / “Cannot assign requested address,” inspect actual outbound local-port pressure, not all TIME_WAIT sockets.
Settings and APIs to inspect
| Platform / Setting | Specific API or what to inspect |
| Linux: net.ipv4.ip_local_port_range | Configured ephemeral port range |
| Local IP:port and remote IP:port | Identify outbound destination concentration and port usage |
| Client connection pool / keep-alive | Reduce unnecessary new outbound TCP connections |
4. Avoid blunt timer changes. Do not decrease TCP TIME_WAIT lifetime or enable aggressive reuse options without verifying the actual workload and platform semantics.
