How ‘yCrash Log’ uses AI & ML?

Application logs are often the first artifact engineers use to troubleshoot the production incident. Application log analysis plays a critical role in modern observability and incident management practices, enabling engineers to understand system behavior during failures. They are also heavily used to diagnose functional issues, performance degradations, and unexpected behavior in applications. 

We all know and see everyday, in large-scale production environments, applications generate thousands to millions of log entries daily, making manual log inspection both time-consuming and error-prone. These log data is largely unstructured and varies widely in format across different frameworks, services, and infrastructure components, which is a major challenge. Without effective application log analysis, identifying the root cause of incidents can significantly increase mean time to resolution (MTTR) and impact system availability. This is where yCrash Log comes into play. In this post, we will discuss how ‘yCrash Log’ tool leverages AI & ML to analyze application log files.

If you are interested in how this yCrash log works, this documentation will give you a step-by-step instruction as to how to use the yCrash Applog feature. 

Video

Watch the webinar below for an in-depth walkthrough of how to analyze application logs precisely and easily using yCrash.

Key Capabilities of ‘yCrash Log’

In this section let’s learn key feature of the yCrash Log tool and see how AI & ML algorithms are used in them.

1. Transform Unstructured 🡪 Structured Data

Log file is basically an unstructured data which comes in various different formats. Here are different log formats:

2025-06-24 00:43:54,919; INFO; c.t.f.h.HeadlineManager; ; 
DF01AE861737699EEF760FC566B04A55; 6bd8ccf3-47cf-4d38-a13a-7358382c8d13; 
http-nio-5001-exec-4; No of days license left: 227

Fig: Log line delimited by semicolon

2026-02-02T15:11:14,805|[default
task-43]|INFO|GET|/bg_api/v3/data/geography/status|200|0||Mozilla/5.0 (iPhone; CPU 
iPhone OS 18_7 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/26.2 
Mobile/15E148 
Safari/604.1|49a12468f16af9698e9200b61c644853|jboss-vm1-2vuo0f_0000|bc5051cd9659f9
c6b487f99924ab6d1e|false|15:11:14,805(55ms)

Fig: Log Line delimited by pipe

2015-07-29 19:27:11,866 - WARN 
[SendWorker:188978561024:QuorumCnxManager$SendWorker@679] - Interrupted while 
waiting for message on queue

Fig: Log line without any delimiter

If you notice, there is no standard structure to a log file. We need to transform this log file into a structured table format as shown in the below figure so that searching, filtering & sorting can be applied to them for effective log analysis.

In the ‘yCrash Log’ tool, we follow below three-steps to achieve this transformation:

1. Identifying Delimiter Sequence: We have developed a sophisticated algorithm based on analyzing thousands of log formats. This algorithm reads the initial few hundred lines of log file to identify the delimiter sequence.
2. Parsing using Delimiter Sequence: Once this delimiter sequence is identified then this sequence is applied to the log file to parse and generate a structured table format of the data.
3. Detecting Data Types: Once data is parsed in to table, tool then uses a  set of predefined regular expressions to identify what are the data type of each column in the table like timestamp, Log Level, Thread Name, Class Name, Message… 

2. Error Detection

Identifying the errors in the application log and isolating the root cause for them is one of the primary use cases for using the application logs. To facilitate it, ‘yCrash Log’ tool provides following capabilities:

a) Identify Errors & Exceptions: Application log file contains hundreds of application and system errors. yCrash log analysis tool, combs through entire log file and identifies all the errors and exceptions that have occurred in the log file and reports them as a summary in the dashboard as shown in the below screenshot.

This is not a simple search of the words ‘error’ or ‘exception’ that is performed on the log file. When you perform simple search, it brings back false positive incidents as well. Example consider this log statement:

2025-06-24 00:43:54,919; INFO; http-nio-5001-exec-4; IllegalArgumentException 
handled successfully!

Here engineer has printed that ‘IllegalArgumentException handled successfully!’. This is a positive affirmation, however it has the word ‘IllegalArgumentException’. If someone performs a search on the word ‘Exception’, above log statement would be matched. However, there is absolutely no error in this statement. Thus, we scan for positive sentiments in these statements using the Named Entity Recognition (NER) model. We use this hybrid approach to identify positive sentiments, if they are encountered, we will eliminate them from our result set.

b) Error Severity Identification: There are certain critical exceptions like ‘OutOfMemoryError‘, ‘StackOverflowError‘… when these sorts of exceptions happen, it can hurt the entire application’s availability. They should be bubbled up as high priority errors and bought the user’s attention. ‘yCrash Log’ tool maintains a library of high priority errors. When such high priority errors surface in the application log, they are caught and reported as serious problems. Besides that, tool also exposes a configuration file, where engineers can configure their own set of high priority errors. When those configured errors are found in the log file, it will also be highlighted to the user.

c) Categorization of Error based on their stack trace: Application could be suffering from multiple flavors of NullPointerException i.e. NullPointerException could be originating from different parts of the application. It should not be categorized under one single bucket, because fix for each flavor of NullPointerException would be different. It will be effective for to show each flavor of NullPointerException by their count separately, so that triaging the problem can become easier. yCrash Log tool performs this operation effectively as shown in the below figure:

3. AI Solution to resolve Identified Errors

Once Errors have been isolated next step in the troubleshooting process is to find the solution to the error. Best way to educate one on the solution is through integration with LLM. To facilitate this ‘yCrash Log’ tool leverages AI assistant yCrash Buddy, which directly integrates with your corporate standardized LLM (OpenAI, Gemini, Copilot, Claude AI…). yCrash Buddy makes API call to your LLM and provides the necessary solution. Using the yCrash Buddy you can directly interact with the LLM to come up with optimal solution for the error. 

4. Repeating Statements

In large application log files, few statements may repeat multiple times – a hundred or even a thousand! These repetitions are indications of underlying issues in the application – e.g., recurring failures, misconfigurations, infinite loops, or threads repeatedly executing the same logic.

If only these repeating statements were easy to identify! These log lines are not always exact text matches. They are a combination of dynamic values such as timestamps, IDs, or memory addresses, and static text. While the variable portions change, the underlying message pattern remains the same. The ‘yCrash – Repeating Statements’ tool will analyze such log lines, based on their structural pattern, using tokenization and cosine similarity, along with textual similarity using levenshtein distance, and cluster similar patterns together with their counts.

Using this feature enables engineers to easily identify abnormal patterns in their application and focus on patterns that may indicate systemic issues, by getting rid of the noise from meaningful log lines. The tool significantly reduces the effort required to manually scan and correlate thousands of log statements in application logs.

5. Time Gap Analysis

Let’s say the backend SOR to which your application connects has some problems, then when your application thread makes a backend call, it’s going to get stuck waiting for the response. It will be reflected in the log. There will be a time gap between the log statements that were printed before the backend call and after the backend call. Similarly let’s say a method takes very long time to complete, then there will be a time gap between the log statements that were printed before the method call and after the method call. One has to carefully look at the timestamp of each log statement and observe the time difference. If there is a considerable time difference, it might be indicative of bottleneck in the application. However application log file has thousands of statements, it’s manually impossible to observe each log statement and identify the time gap between them. ‘yCrash Log’ tool, scans the entire log file and isolates the log statements that show the time gap between them as shown in the below screenshot.

6. PII Leakage

Sometimes developers might accidentally print Personally Identifiable Information (PII) data such as Social Security Number (SSN), Email Address, IP Address, Credit Card Numbers, … in the log file. These are classified as confidential data in several organizations, and it shouldn’t be printed in the application log file. ‘yCrash Log’ use Regex to identify the PII data. But it can result in false positive detection of PII data as well. Thus after filtering with Regex, we use Named Entity Recognition (NER) model to eliminate the false positive detection. Below is the report generated by yCrash log analysis tool that highlights PII data that leaked in an application log file.

7. Dynamic Graphing Capability

You can also select certain range of data in the yCrash log analysis table and generate dynamic column, bar, pie, area … graphs as shown in the screenshot below. You can also use sophisticated aggregate functions such as: sum, count, average, min, max… to graph the data. 

How yCrash Log is different?

There are well-established wonderful log analysis tools in the market like ELK, Splunk…  However, yCrash log analysis doesn’t compete with these tools, it complements these tools and solves different set of problems. Let’s closely examine their differences:

1. Log Aggregation vs Point in Time Tool

 Above mentioned log analysis tools constantly stream all the log from all your servers to a central server, consolidate them and presents in a single dashboard to consume. On the other hand, yCrash Log analysis tool doesn’t stream all the log contents from all your servers. When yc-360 open-source script is triggered, it captures last 10,000 log lines from the log file along with 16 other diagnostic artifacts and transmits them to yCrash server for analysis. yCrash log analysis tool facilitates you to identify the root cause of the production problem. 

2. Automating Root Cause Analysis

Based on the earlier discussion, you can find that ‘yCrash Log’ tool, parses the log file and tries to identify the root cause of the problem through the ‘Error Analysis’, ‘Time Gap Analysis’, ‘Repeating Statements’ features. On the other hand, in the above-mentioned tool, isolating problems is still a manual process. You need to search, filter for the errors.

The following table provides a quick comparison of how yCrash Log complements traditional log aggregation platforms:

Conclusion

We are truly blessed to develop such a lightweight, highly useful tool for the industry using cutting-edge technologies! It was an amazing learning experience for us. We are delighted to share them through this post. We hope you would also enjoy this article as much as we enjoyed it while developing it.