Introduction
In this article, we will examine the significance of integrating a GC API with the Jenkins pipeline. The objective of this article is to leverage the capabilities of Jenkins CI/CD pipeline to correctly measure the stability of the product for every major release.
This integration can give you more insights about the well-being of the application at the time of merging, or at any suitable time when it is required to run the pipeline.
When this pipeline runs, it will conduct the performance tests on the target application and analyze the GC log using the GC REST API.
Let’s dig deeper into this process!
Introduction to Jenkins
Jenkins is a well known CI/CD tool. It has got amazing features, and hence developers from all over the world love it. This tool can control the build behavior in many different ways with the help of plugins.
For example, it is very easy to start building the application and deploying it to a server using a rather trivial pipeline. In order to achieve this,the Jenkins pipeline should include commands to run tools such as Git and Maven, and deploy jar files.
Another example of where Jenkins is invaluable is to run unit test cases after building the project, and find out the coverage.
These are only a few examples, but this CI/CD server is used in many different ways.
yCrash GCeasy Setup
GCeasy is a powerful GC log analyzer, which provides GC metrics and identifies potential issues. It is part of the yCrash suite, and provides REST APIs to automate GC monitoring.
In order to invoke the GCeasy REST APIs, it is required to set up GCeasy locally. All you have to do is access the link below and download the latest version.
Unzip the file and start the server by clicking on either the `launchserver.sh` or the `launchServer.bat` file. This will start the application on port 8080.
Then you can invoke all the REST endpoints using `localhost`.
At this point, you can invoke all the REST APIs directly. However, there are some scenarios where we need to safeguard these REST APIs. Some environments need very restricted access to the resources, such as protecting URLs from public access. To do that, you need to add a file called `saml.xml` in the root of the yCrash server (where launchServer.bat or launchServer.sh files are located).
In order to access the GCeasy REST APIs, you would then need to generate an access token. This token will be validated by the yCrash server. In order to do so, add your token validation endpoint to the saml.xml file and restart the server.
The following link describes in detail how to do this.
Next time you invoke GCeasy REST APIs, you will need to pass a bearer token along with the REST endpoint. For example:
Authorization: Bearer <access_token>Garbage Collection Reporting
We need to mention the location of the GC log in the pipeline when it starts the application, so that it will generate it in the Jenkins workspace.
Jenkins Script
Let’s take the pipeline script below:
#... now invoke the GC easy REST API with the gc log file generated by the previous step
Copy-Item C:\workspace\rest\myapp-gc.log C:\workspace\rest\myapp-gc-tmp.log -Force
$logFilePath = "C:\workspace\rest\myapp-gc-tmp.log"
$outputFilePath = "C:\workspace\rest\gc.json"
$apiUrl = "http://localhost:8081/tier1app/analyzeGC?apiKey=e094a34e-c3eb-4c9a-8254-f0dd107245cc&graphs=true"
Write-Host "Going to invoke GCeasy REST API"
Invoke-WebRequest -Uri $apiUrl `
-Method Post `
-Headers @{"Content-Type" = "text/plain"} `
-InFile $logFilePath `
-OutFile $outputFilePath
# we are done with the load testing, so stop this tomcat instance
cd "C:\_tools\Tomcat-9\apache-tomcat-9\apache-tomcat-9\bin"
.\shutdown.bat
Write-Host "Tests complete, stopping Tomcat custom server.."
Write-Host "GCeasy REST API execution finished.. Parsing the response json file"
#.. Preparing GC easy graph
Write-Host "Parsing rest json file and display the graph"
$jsonContent = Get-Content -Path "c:/workspace/rest/gc.json" -Raw
$jsonObject = $jsonContent | ConvertFrom-Json
$problem = $jsonObject.isProblem
$avgPauseTime = $jsonObject.gcKPI.averagePauseTime
$maxPauseTime = $jsonObject.gcKPI.maxPauseTime
$gcThroughPut = $jsonObject.gcKPI.throughputPercentage
$objectCreationRate = $jsonObject.gcStatistics.avgAllocationRate
$cpuTime = $jsonObject.gcKPI.cpuTime
$heapAfter = $jsonObject.graphs.heapAfterGCGraph
$heapBefore = $jsonObject.graphs.heapBeforeGCGraph
$gcDuration = $jsonObject.graphs.gcDurationGraph
$gcPauseDuration = $jsonObject.graphs.gcPauseDurationGCGraph
$htmlContent = @"
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>GC Easy Rest API Result</title>
<link href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/6.0.0-beta3/css/all.min.css" rel="stylesheet">
<style>
body {
font-family: 'Arial', sans-serif;
background-color: #f4f4f9;
color: #333;
margin: 0;
padding: 20px;
text-align: left; /* Align all text to the left */
}
h1 {
text-align: center;
font-size: 36px;
color: #2d3e50;
margin-top: 30px;
padding-bottom: 15px;
border-bottom: 3px solid #3498db;
width: 50%;
margin-left: auto;
margin-right: auto;
}
h3 {
color: #2980b9;
font-size: 24px;
text-align: center;
margin-top: 30px;
}
p {
font-size: 18px;
line-height: 1.6;
margin: 15px 0;
text-align: center;
position: relative;
}
.data-container {
margin-top: 20px;
max-width: 80%;
counter-reset: list-item; /* Reset only here */
}
.data-container p {
font-size: 18px;
margin-bottom: 10px;
color: #2c3e50;
display: flex;
align-items: center;
counter-increment: list-item; /* Increment counter for each p element */
margin-left: 0;
padding-left: 150px;
}
/* Bullet point style for "Problem:" */
.data-container p strong::before {
content: counter(list-item) ""; /* Add the number before the bullet */
display: inline-block;
width: 30px;
height: 30px;
margin-right: 8px;
border-radius: 50%;
background-color: #000000; /* Black circle */
color: white;
font-weight: bold;
text-align: center;
line-height: 30px;
}
.tooltip {
position: relative;
display: inline-block;
cursor: pointer;
font-weight: normnal;
color: #008000;
margin-left: 8px;
}
.tooltip .tooltiptext {
visibility: hidden;
width: 200px;
background-color: #000;
color: #fff;
text-align: center;
border-radius: 5px;
padding: 10px;
position: absolute;
z-index: 1;
bottom: 125%;
left: 50%;
margin-left: -100px;
opacity: 0;
transition: opacity 0.3s;
}
.tooltip .tooltiptext::after {
content: '';
position: absolute;
top: 100%;
left: 50%;
margin-left: -5px;
border-width: 5px;
border-style: solid;
border-color: #333 transparent transparent transparent;
}
.tooltip:hover .tooltiptext {
visibility: visible;
opacity: 1;
}
.graph-container {
text-align: center;
margin-top: 20px;
}
.graph-container img {
margin: 10px;
border: 3px solid #3498db;
border-radius: 8px;
}
.footer {
text-align: center;
margin-top: 50px;
font-size: 16px;
color: #7f8c8d;
}
.footer a {
color: #2980b9;
text-decoration: none;
}
.footer a:hover {
text-decoration: underline;
}
</style>
</head>
<body>
<h1 onclick="testFunction()">Key Performance Indicators</h1>
<div class="data-container">
<p><strong>Problem
<span class="tooltip"><i class="fas fa-info-circle"></i>
<span class="tooltiptext">Indicates whether there was a problem during the GC process. True means there's a problem, otherwise no problem</span>
</span>
:
</strong>$problem</p>
<p><strong>Object Creation Rate<span class="tooltip"><i class="fas fa-info-circle"></i>
<span class="tooltiptext">The rate at which objects are created during the execution of the program.</span>
</span>:</strong>$objectCreationRate
</p>
<p><strong>GC Throughput<span class="tooltip"><i class="fas fa-info-circle"></i>
<span class="tooltiptext">The percentage of time spent in processing real transactions vs time spent in GC activity.Higher percentage is a good indication that GC overhead is low.One should aim for high throughput</span>
</span>:</strong>$gcThroughPut %
</p>
<p><strong>CPU Time<span class="tooltip"><i class="fas fa-info-circle"></i>
<span class="tooltiptext">Total CPU time consumed by the Garbage collector(sum of User Time and Sys Time).</span>
</span>:</strong> $cpuTime
</p>
<p><strong>Latency:</strong>
<table style="padding-left:150px;" border="0px;">
<tbody>
<tr>
<td style="background-color: #000;color: #fff;"><strong>Avg Pause GC Time</strong>
<span class="tooltip"><i class="fas fa-info-circle"></i>
<span class="tooltiptext">This is the average amount of time taken by one Stop the World GC.</span>
</span>
</td>
<td>$avgPauseTime ms</td>
</tr>
<tr>
<td style="background-color: #000;color: #fff;"><strong>Max pause GC Time</strong>
<span class="tooltip"><i class="fas fa-info-circle"></i>
<span class="tooltiptext">This is the maximum amount of time taken by one Stop the World GC to run.</span>
</span>
</td>
<td>$maxPauseTime ms</td>
</tr>
<tbody>
</table>
</p>
</div>
<h3>Garbage Collection Graphs:</h3>
<div class="graph-container" style="padding-left:150px;">
<table>
<tr>
<td><strong>Heap Before GC</strong></td>
<td><strong>Heap After GC</strong></td>
<td><strong>GC Duration time</strong></td>
</tr>
<tr>
<td>
<img src='$heapAfter' width="256" height="256" alt="Heap Usage After GC">
</td>
<td>
<img src='$heapBefore' width="256" height="256" alt="Heap Usage Before GC">
</td>
<td>
<img src='$gcDuration' width="256" height="256" alt="GC Duration">
</td>
</tr>
</table>
</div>
<script>
function testFunction(){
console.log("clicked on heading")
}
</script>
</body>
</html>
"@
In this script, the GC log file is copied to another file named myapp-gc-tmp.log.
Then the Jenkins script calls the GCeasy REST API to analyze the GC log. This will result in a JSON response file, and the Jenkins PowerShell script will parse this, and create an HTML file with the data. This HTML file will be displayed on the Jenkins dashboard.
GCeasy API Response
This pipeline will pull out the project from the GitHub repository and start building the application. It will be using a Maven tool to build the project. The next step is to start the application and perform the load testing using JMeter binaries. What is observed after the load testing is the garbage collection metrics using the GC REST API. This API will be called from the pipeline after the load testing.
The GC REST API returns a JSON response as follows:
{"isProblem":false,"jvmHeapSize":{"youngGen":{"allocatedSize":"164 mb","peakSize":"152 mb"},"oldGen":{"allocatedSize":"90 mb","peakSize":"14 mb"},"metaSpace":{"allocatedSize":"51.56 mb","peakSize":"51.13 mb"},"total":{"allocatedSize":"254 mb","peakSize":"150 mb"}},"gcStatistics":{"totalCreatedBytes":"705 mb","totalPromotedSize":"19 mb","measurementDuration":"4 sec 373 ms","avgAllocationRate":"161.22 mb/sec","avgPromotionRate":"4.34 mb/sec","minorGCCount":"18","minorGCTotalTime":"119 ms","minorGCAvgTime":"6.59 ms","minorGCAvgTimeStdDeviation":"4.60 ms","minorGCMinTIme":"0","minorGCMaxTime":"17.7 ms","minorGCReclaimedSize":"683 mb","minorGCIntervalAvgTime":"232 ms","fullGCCount":"0"},"gcKPI":{"throughputPercentage":97.755,"averagePauseTime":4.9097495,"maxPauseTime":10.0,"cpuTime":"20.0 ms"},"gcDurationSummary":{"groups":[{"start":"0","end":"0.001","numberOfGCs":2},{"start":"0.001","end":"0.002","numberOfGCs":1},{"start":"0.002","end":"0.003","numberOfGCs":8},{"start":"0.003","end":"0.004","numberOfGCs":2}If there is any problem in the GC report, then the isProblem field will be set to true. In that case, the pipeline will flag a build FAILURE event.
However, it is also possible to consider the other fields in the JSON response, such as throughput, CPU usage etc., as the factors used in deciding if the build failed. If throughputPercentage is less than 95% in the GC report, then that metric will be considered as a build failure, so that the product owners can take necessary actions.
Take a look at the console of the Jenkins pipeline in the following images.
Fig. Jenkins GCeasy API Response
Fig: Jenkins GCeasy Pipeline Status
These images show a ‘Build Success’ scenario, as there are no problems reported while analyzing the GCeasy REST APIs.
Render GCeasy JSON Result
We received the response as JSON format, and we need to parse it. But what is the information we need to extract from the above JSON file?
These are the JSON keys we need to extract from the file:
- isProblem
- averagePauseTime
- maxPauseTime
- throughputPercentage
- avgAllocationRate
- cpuTime
Let’s understand the significance of these elements. If isProblem returns true, then there can be a problem in the application which needs to be looked into.
averagePauseTime and maxPauseTime are related to the garbage collection, and tell us the average and maximum time the application pauses during the GC events
The throughputPercentage is the ability for the application to handle the user transactions, and avgAllocationRate tells us the average number of objects created per second by the application.
The cpuTime is the time taken by the garbage collector.
We can use Windows Powershell in the Post-Build action of the Jenkins job to parse this JSON file and create an HTML file from it. Follow the steps below:
$jsonContent = Get-Content -Path "c:/workspace/gc.json" -Raw
$jsonObject = $jsonContent | ConvertFrom-Json
$problem = $jsonObject.isProblem
$avgPauseTime = $jsonObject.gcKPI.averagePauseTime
$maxPauseTime = $jsonObject.gcKPI.maxPauseTime
$gcThroughPut = $jsonObject.gcKPI.throughputPercentage
$objectCreationRate = $jsonObject.gcStatistics.avgAllocationRate
$cpuTime = $jsonObject.gcKPI.cpuTime
$heapAfter = $jsonObject.graphs.heapAfterGCGraph
$heapBefore = $jsonObject.graphs.heapBeforeGCGraph
$gcDuration = $jsonObject.graphs.gcDurationGraph
$gcPauseDuration = $jsonObject.graphs.gcPauseDurationGCGraph
$htmlContent = @"
<html>
<body>
<h1>GC easy Rest API Result</h1>
<p>Problem: $problem</p>
<p>Avg Pause Time: $avgPauseTime</p>
<p>Max Pause Time: $maxPauseTime</p>
<p>GC Throughput: $gcThroughPut</p>
<p>Object Creation Rate: $objectCreationRate</p>
<p>CPU Time: $cpuTime</p>
<p><h3>Garbage Collection graph: </h3></p>
<p><img src='$heapAfter' width="256" height="256">
<p><img src='$heapBefore' width="256" height="256">
<p><img src='$gcDuration' width="256" height="256">
<p><img src='$gcPauseDuration' width="256" height="256">
</body>
</html>
"@
# Write the HTML report to a file
$htmlContent | Out-File -FilePath "c:/workspace/index.html"
Fig: Windows PowerShell Script
The next step is to install an HTML Publisher plugin in the Jenkins server for publishing the HTML file generated in the previous step, so that it can be viewed on the Jenkins dashboard.
Fig: Jenkins HTML Publisher Interface
Fig: GCEasy Test Results
Upon clicking the above result link, it will open the HTML report we created earlier. Please find the screenshots of the report below:
Fig: Garbage Collection KPI
Fig: Heap Usage after Garbage collection
Fig: Heap Usage before Garbage Collection
Summary
The objective of this article is to understand the possibilities of using the Jenkins server to analyze GC logs by calling yCrash’s REST APIs. This way we can make sure of the health of the application even before it reaches the customers. There are a lot of ways we can improve the script to meet different requirements. One scenario is to capture a thread dump and use the yCrash REST API to analyze it.
Making use of the combination of Jenkins and yCrash is a powerful way to ensure potential problems don’t reach production.