CPU consumption in Unix/Linux operating systems are studied using 8 different metrics: User CPU time, System CPU time, nice CPU time, Idle CPU time, Waiting CPU time, Hardware Interrupt CPU time, Software Interrupt CPU time, Stolen CPU time. In this article let us study about ‘System CPU time’.
What is ‘system’ CPU time?
In order to understand ‘System CPU Time’, one should understand ‘User CPU time’ as well, since they go hand in hand. User CPU time is the amount of time the processor spends in running your application code. System CPU Time is the amount of time the processor spends in running the operating system(i.e., kernal) functions connected to your application. Let’s say your application is manipulating the elements in an array; then, it will be accounted as ‘User CPU’ time. Let’s say your application is making network calls to external applications. To make network calls, it has to read/write data into socket buffers which is part of the operating system code. This will be accounted as ‘System CPU’ time.
How to find ‘system’ CPU time?
System CPU time can be found from the following sources:
a. You can use web-based root cause analysis tools like yCrash to report ‘system’ CPU time. Tool is capable of generating alerts if ‘system’ CPU time goes beyond the threshold (configurable).
b. ‘system’ CPU time is also reported in the Unix/Linux command line tool ‘top’ in the field ‘sys’ as highlighted in the below image.
Fig: sys time in top
How to simulate high ‘system’ CPU time?
To simulate high ‘system’ CPU reporting let’s use BuggyApp. BuggyApp is an opensource java project which can simulate various sort of performance problems. When you launch BuggyApp with following arguments, it will cause the ‘system’ CPU consumption to spike up on the host.
java -jar buggyApp.jar PROBLEM_IO
Fig: You can see the system CPU time spike up to 13.3%
Now let’s see the source code in the BuggyApp which is causing the ‘system’ CPU time to spike up.
public class IODemo {
public void start() {
for (int counter =1; counter <= 5; ++counter) {
// Launch 5 threads.
new IOThread ("fileIO-" + counter + ".txt").start();
}
}
}
public class IOThread extends Thread {
public String fileName;
public static final String CONTENT =
"Hello World! We are building a simple chaos engineering product here. \n" +
"Hello World! We are building a simple chaos engineering product here. \n" +
"Hello World! We are building a simple chaos engineering product here. \n" +
"Hello World! We are building a simple chaos engineering product here. \n"
"Hello World! We are building a simple chaos engineering product here. \n" +
"Hello World! We are building a simple chaos engineering product here. \n" +
"Hello World! We are building a simple chaos engineering product here. \n" +
"Hello World! We are building a simple chaos engineering product here. \n"
public IOThread(String fileName) {
this.fileName = fileName;
}
public void run() {
int counter = 0;
// Loop infinitely trying to read and close the file.
while (true) {
// Write the contents to the file.
FileUtil.write(fileName, CONTENT);
// Read the contents from the file.
FileUtil.read(fileName);
}
}
}
Here you can see that BuggyApp is launching 5 ‘IOThreads’ in ‘IODemo’ class. You can notice that ‘IOThread’ is going on an infinite while loop. In that loop it’s writing the content in to a file and reading the same content from the file. It is doing these 2 steps repeatedly again and again. Since writing contents and reading contents from the disk is a heavy I/O intensive operation ‘waiting’ CPU time is spiking up to 75.9%. Similarly writing and reading needs to exercise kernel level code, that’s why ‘system’ CPU time is spiking up to 13.3%.
How to resolve high ‘system’ CPU time?
- Short-term/Tactical solution is to restart the high CPU consuming process.
- It could be a bug in the application. Application might try to access a resource (like File, network connection, etc), which doesn’t exist repeatedly again & again instead of issuing an error. OR application could be doing a lot of I/O activity (i.e., lot of disk read/write, network read/write). In that circumstance System CPU time will spike up. You can use tools like yCrash to identify the root cause of the problem.
- Make sure you are running on the latest OS version with proper patches installed. If you are running on an old version of OS but new hardware, then this problem will be more pervasive.