Diagnosing Disk Space Issues in Dell Boomi

From time to time, applications may over consume disk space. Detecting such scenarios where a program can fill up all usable disk space is crucial for several reasons:

Reason	Description
Prevention of System Failure	When a program consumes all available disk space, it can lead to system failure or instability. Critical system processes may be unable to write logs, save temporary files, or perform necessary operations, which can disrupt the functioning of the system and potentially lead to crashes.
Maintaining System Performance	A system with limited disk space might experience significant performance degradation as the disk fills up. Processes that require disk space, such as caching and virtual memory, may not function properly, leading to slow system response times and decreased overall performance.
Preventing Data Loss	If disk space is completely consumed, it can result in the loss of valuable data. For example, if a program is writing data to a file and runs out of disk space midway, the file may become corrupted or incomplete, leading to data loss or inconsistency.
Ensuring System Stability	Operating systems and other critical software components rely on having sufficient disk space to perform routine tasks and updates. If disk space is exhausted, it may prevent essential updates from being applied, leaving the system vulnerable to security threats and software bugs.
Avoiding Denial of Service (DoS) Attacks	Malicious actors may intentionally attempt to fill up disk space on a system as part of a denial-of-service attack. By detecting and mitigating such scenarios, system administrators can prevent service disruption and maintain system availability.
Improving Resource Management	Detecting programs that consume excessive disk space allows system administrators to identify resource-intensive applications and take appropriate actions to optimize resource usage, allocate additional storage capacity, or enforce resource usage limits.

Today, we’ll create a program in Boomi that generates a file filled with bytes on the JVM’s native filesystem. I also plan to use a monitoring tool like yCrash to detect that the program in Boomi is creating a really large file with consecutive write commands or that the surrounding system environment has become unstable due to the lack of disk resources.

We will perform the tests on my Macbook Pro M2, hosting a single Boomi Atom.

Boomi Implementation

I borrowed some code from the buggyApp and ported it into a Custom Groovy Script. The biggest change I did was hard coding the number of iterations to a large number that would produce a sufficiently large file to generate the out of disk space scenario.

import java.util.Properties;
import java.io.InputStream;
import com.buggyapp.diskspace.DiskSpaceDemo;
import com.boomi.execution.ExecutionUtil;

import java.io.FileOutputStream;
import java.io.IOException;
import java.nio.file.FileStore;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Random;

logger = ExecutionUtil.getBaseLogger();

directory = ExecutionUtil.getDynamicProcessProperty("Directory");
percent_full = ExecutionUtil.getDynamicProcessProperty("PercentFull");

Integer ONE_MB = 1024 * 1024;

void deleteFile(String drive) {
	String filePath = drive + "/tier1testfile";
	Path path = Paths.get(filePath);
	try {
		// Delete the file
		Files.delete(path);
		logger.info("File deleted successfully.");
	} catch (IOException e) {
		logger.info("File not available "+filePath);
	}
}

byte[] generateRandomData(int size) {
	byte[] data = new byte[size];
	new Random().nextBytes(data);
	return data;
}

long getAvailableSpace(String drive) {
	try {
		FileStore fileStore = Files.getFileStore(FileSystems.getDefault().getPath(drive));
		return fileStore.getUsableSpace();
	} catch (IOException e) {
		logger.info("Error getting available space: " + e.getMessage());
		return -1;
	}
}
	
for( int i = 0; i < dataContext.getDataCount(); i++ ) {
    InputStream is = dataContext.getStream(i);
    Properties props = dataContext.getProperties(i);

    String filePath = directory + "/tier1testfile";
	// Create a File object with the specified file path
	deleteFile(directory);
	// Check available space on the drive
	long availableSpaceInBytes = getAvailableSpace(directory);
	long fileSizeInBytes = (long) ((availableSpaceInBytes * Integer.valueOf(percent_full)) / 100);
	byte[] data = generateRandomData(ONE_MB);

    logger.info("availableSpaceInBytes " + availableSpaceInBytes + " fileSizeInBytes " + fileSizeInBytes);

	Integer noOfInterations = 30000;
	logger.info("Number of Iterations: " + noOfInterations);
	
	for (Integer iteration = 0; iteration < noOfInterations + 1; iteration++) {
		FileOutputStream fos = null;
		try {
		    fos = new FileOutputStream(filePath, true);
			fos.write(data);

		} catch (IOException e) {
			logger.info("Error " + e.getMessage());
		} finally {
		    fos.close();
		}
	}
	logger.info("File created successfully occupying " + percent_full + "%");
   
    dataContext.storeStream(is, props);
}

The code is a script designed to fill up disk space by repeatedly writing data to a file until 30000 iterations of writing have been completed. Breaking the code down:

1. Initialization:

The script initializes necessary variables and retrieves dynamic process properties (global variable) such as the directory where the file will be created and the percentage of disk space to fill.

2. deleteFile() Method:

This method deletes a file located at the specified path. It’s used to ensure that the file doesn’t already exist before writing new data to it.

3. generateRandomData() Method:

This method generates an array of random bytes of a specified size. It’s used to create the data that will be written to the file.

4. getAvailableSpace() Method:

This method retrieves the available disk space on the specified drive using Java’s NIO (New I/O) APIs. 

5. Main Loop:

The script iterates through the data context, which seems to handle input data streams. For each iteration, it calculates the target file size based on the available disk space and the specified percentage to fill.

6. File Writing Loop:

Within each iteration, the script opens a FileOutputStream and writes random data to the file until the specified number of iterations is reached.

7. Logging:

Throughout the script, logging statements are used to provide information about the progress and status of the script’s execution.

The Boomi process contains four shapes/steps to trigger the issue. They are: 

1. The first being a start shape of type “No Data”. 
2. The second is an initialization of properties shape which sets two dynamic global variables/also known as dynamic process properties. 
3. The third is the script above. 
4. The final shape is a stop shape which terminates the document flow, normally. The script above will for 30,000 iterations append a byte array of 1 MB.

Boomi Process Execution

Process was executed without error.

Results

Now, let’s take a closer look at the findings using both Boomi and yCrash.

Boomi Findings 

The operating system generated an error. I issued the ‘df’ command below, the low Avail would trigger system warnings:

(base) rkrule@Roberts-MacBook-Pro-2 /tmp % df -kh 
Filesystem       Size   Used  Avail Capacity iused   ifree %iused  Mounted on
/dev/disk3s1s1  228Gi   29Gi  559Mi    99%  501730 5721600    8%   /
devfs           346Ki  346Ki    0Bi   100%    1198       0  100%   /dev
/dev/disk3s6    228Gi  3.0Gi  559Mi    85%       3 5721600    0%   /System/Volumes/VM
/dev/disk3s2    228Gi  537Mi  559Mi    50%     889 5721600    0%   /System/Volumes/Preboot
/dev/disk3s4    228Gi  663Mi  559Mi    55%     255 5721600    0%   /System/Volumes/Update
/dev/disk1s2    500Mi  6.0Mi  481Mi     2%       1 4929520    0%   /System/Volumes/xarts
/dev/disk1s1    500Mi  7.4Mi  481Mi     2%      38 4929520    0%   /System/Volumes/iSCPreboot
/dev/disk1s3    500Mi  444Ki  481Mi     1%      47 4929520    0%   /System/Volumes/Hardware
/dev/disk3s5    228Gi  193Gi  559Mi   100% 2248671 5721600   28%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0       0  100%   /System/Volumes/Data/home

Below is the `ls` command output that shows the tier1testfile taking up the majority of space:

(base) rkrule@Roberts-MacBook-Pro-2 /tmp % ls -lrt
total 54029352
drwx------  3 rkrule  wheel           96 Feb 12 07:41 com.apple.launchd.WeAOz77NA6
-rw-r--r--  1 rkrule  wheel            5 Feb 18 12:16 i4jdaemon__Applications_Boomi_AtomSphere_Atom_Atom_rk_laptop_bin_atom
-rw-r--r--  1 rkrule  wheel  27663024128 Feb 18 13:09 tier1testfile

yCrash Findings

Flipping over to yCrash, I found that the system eventually detected that the file mount was full. However, it was not an instant finding.

I clicked into the first alert on disk usage, which also indicated in the Storage Report that ‘devfs’ was full.

Conclusion

In conclusion, our simulated Boomi process successfully demonstrated a scenario where the application consumed excessive disk space, leading to potential system failure, degraded performance, and the risk of data loss. Despite the Boomi process completing without error, the operating system reported a critical disk space shortage, triggering system warnings. yCrash, our performance monitoring tool, identified the issue retrospectively, emphasizing the importance of proactive monitoring to prevent adverse impacts on system stability and functionality. To read more articles about troubleshooting performance issues in Dell Boomi Server, click here.