The software applications that run today need to combine the strengths of multiple programming languages, with each language doing what it does best.
It’s common to see Python/R programs doing mathematical calculations on huge volumes of data, Java running the enterprise applications and the ingrained business logic, and JavaScript frameworks taking care of the GUI and client side operations.
In this post we explore the latest Java tools and techniques that enable Java programs to call other programs written in a different language.
We will use the Jextract tool. This tool parses the header files (.h) of native libraries such as Python or C++, and generates Java classes or Bindings. These can then be used by your Java programs to call functions of these native libraries. The binding classes generated by Jextract act as a vehicle to call programs written in other languages, pass parameters and then receive the output inside your Java programs.
Calling Python Programs Via Java
As this blog is code intensive, we will now quickly jump to the implementation steps.
STEPS for calling Python programs from your Java code:-
1. Create your Python program.
2. Generate Java bindings from the Python header file, Python.h, using Jextract.
3. Use the generated code to run a Python script in a Java application.
4. Verify or receive the results inside your Java program.
STEP 1 : Creating the Python program to be called by a Java class.
The Python program below tries to calculate the value of PI , using the Monte-Carlo simulation.
The Monte-Carlo simulation uses repeated random sampling to obtain the value of PI nearest to the actual value. PI is defined as the ratio between a circle’s circumference and its diameter.
As PI is both an irrational as well as a transcendental number; one can never reach an exact, finite decimal representation of PI. Hence we try to come closest to the actual value.
To understand the theory behind this estimation, read Wikipedia’s article: Monte Carlo simulation.
Fig: Monte Carlo Demonstration
Source: https://upload.wikimedia.org/wikipedia/commons/d/d4/Pi_monte_carlo_all.gif
We consider a quadrant(circular sector) inscribed in a unit square. The ratio of the areas of the unit square:quadrant is π/4.
So, we generate a very large number of points and uniformly scatter them inside the unit square. Then we count (via a Python program) the number of points inside the quadrant and call it inside-count. Next, the ratio of inside_count to the total number of points generated (total number of points inside the unit square), is a good estimation of the ratio between their respective areas, and that should be nearly equal to π/4 or PI/4.
Finally we multiply this ratio by 4 to get the value of π.
The Python code looks like this:
import numpy as np
import multiprocessing
import time
# Monte Carlo Estimation of PI, where the ratio of points inside the circle to the total points is approximately π/4.
def monte_carlo_pi(num_samples):
inside_circle = 0
for _ in range(num_samples):
x, y = np.random.uniform(-1, 1, 2) # Generate random point in [-1,1] x [-1,1]
if x**2 + y**2 <= 1: # Using formula (x² + y² ≤ 1), check if the point is inside the unit circle.
inside_circle += 1
return (inside_circle / num_samples) * 4 # π is approximately = 4 × (inside_circle / total_samples),
# For testing only, our MAIN function where execution is started by Python runtime
if __name__ == "__main__":
NUM_SAMPLES = 10**7 # 10 million samples
LARGE_DATASET_SIZE = 10**8 # 100 million data points
print("Starting Monte Carlo simulations...")
start_time = time.time()
# Parallel computation for Monte Carlo PI estimation
with multiprocessing.Pool(processes=4) as pool:
num_samples_per_core = NUM_SAMPLES // 4 # Distribute workload
results = pool.map(monte_carlo_pi, [num_samples_per_core] * 4)
estimated_pi = sum(results) / len(results)
print("Estimated π: {estimated_pi}")
print(f"Total Execution Time: {time.time() - start_time:.2f} seconds")We can convert this Python code to a .SO file on Linux using gcc , or to a .DLL on Windows using Py2EXE
gcc -shared -o CalculatePI.so -fPIC CalculatePI.py
OR
from distutils.core import setup
import py2exe
setup(console=["CalculatePI.py"])STEP 2: Generate Java Bindings From the Python Header file, Python.h ; Using Jextract.
To let Java call the Python program you need some intermediary classes, which act as a stub.
We use the Jextract tool for this.
Here, Jextract generates Java bindings from native library headers internally using the Foreign Function and Memory API.
We use the following switches:-
- -l (lowercase L) Specify the shared library that should be loaded by the generated header class.
- -I ( uppercase i ) Specify the header file search directories.
- –output <path> Folder to store the generated files .
- -t –target-package <package> Specify the package to which the generated class will belong.
Example:
jextract -l "C:\Users\MyUserName\AppData\Local\Programs\Python\Python311\libs"
-I "C:\Users\MyUserName\AppData\Local\Programs\Python\Python311\Lib"
-I "C:\Users\MyUserName\AppData\Local\Programs\Python\Python311\include"
-I "C:\Users\MyUserName\AppData\Local\Programs\Python\Python311"
-I "C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\ucrt"
-I "C:\Windows\System32" -I "C:\Program Files (x86)\Windows Kits\10\Source\10.0.19041.0\ucrt\inc"
-t org.python "C:\Users\MyUserName\AppData\Local\Programs\Python\Python311\include\Python.h" These bindings should be in your classpath or the folder that will hold the Java program we will create in Step 3.
If running this program on Windows, it’s better to have Visual Studio Code set up on your machine, so as to not miss any required header files.
Step 3 Use the Generated Code to Run a Python Script in a Java Application.
We can now write the Java program:
import java.lang.foreign.*;
import java.lang.invoke.MethodHandle;
import java.nio.file.Path;
public class MonteCarloViaPYthon {
public static void main(String[] args) {
try (Arena arena = Arena.ofConfined()) {
var monteCarloLib = Linker.nativeLinker().defaultLookup().lookup(Path.of("CalculatePI.so"));
MethodHandle monteCarloFunc = Linker.nativeLinker().downcallHandle(
monteCarloLib.lookup("monte_carlo_pi").get(),
FunctionDescriptor.of(ValueLayout.JAVA_DOUBLE, ValueLayout.JAVA_INT)
);
int iterations = 1000000;
// We directly call the monte_carlo_pi function, and provide it a parameter
double piEstimate = (double) monteCarloFunc.invokeExact(iterations); // Above, we verify/receive the results in our Java variable
System.out.println("Estimated value of Pi: " + piEstimate);
} catch (Throwable e) {
e.printStackTrace();
}
}
}Compile the above program using the –enable-preview switch, as the Foreign Function and Memory APIs are preview features for now.
javac --enable-preview -source 23 MonteCarloViaPython.java When you run this Java program, you will need to provide the –enable-native-access=ALL-UNNAMED switch. This avoids warnings or problems:
- While calling other native programs on your machine via Java code.
- If the Jextract generated stub classes or Python calls use restricted Java methods.
E.g.
WARNING: java.lang.foreign.SymbolLookup::libraryLookup has been called by org.python.Python_h_3 in an unnamed module
java --enable-native-access=ALL-UNNAMED MonteCarloViaPython The output of our Java program looks like this:
Starting Monte Carlo simulations...
Estimated π: 3.1416348
Total Execution Time: 292.47 secondsSo, our Java program calculates the value of PI using a Python function, and the results are pretty good.
Also, you can see that the Monte-Carlo method of estimating π is very close to the actual value.
Conclusion
We processed a huge dataset of about 100 million data points, but did not need to load those data points on the Java heap. Instead we delegated it to Python, which is good at numerical and statistical calculations. We have also demonstrated how to call existing Python programs from Java code, thereby increasing interoperability and reusability.
Share your Thoughts!