Introduction
This document describes how to perform parallel computations on Matlantis using the joblib Python parallel computing library. In the following program, parallel computations are performed for multiple input parameters.
Caution
When performing parallel computations on Matlantis, it is recommended to use backend="threading". This is because with the default loky backend, when a background job is temporarily stopped, the calculation appears to stop but actually continues, consuming tokens.
Source Code
from joblib import delayed, Parallel
# settings
n_jobs = 10
verbose = 10
backend = "threading" # "threading" is recommended to prevent issues with background jobs.
parameters_for_function = [300, 400, 500, 600]
def function(input_key):
# Write the code you want to execute here.
# Example: Data loading, simulation parameter settings, simulation execution, data saving, etc.
...
# Convert the function so that it can be executed in a delayed manner
delayed_function = delayed(function)
# Set up parallel computation
parallel_calculation = Parallel(n_jobs=n_jobs, verbose=verbose, backend=backend)
# Execute
results = parallel_calculation(delayed_function(params) for params in parameters_for_function)
Program Overview
n_jobs: Specifies the number of processes to use for parallel computation. In this sample program, 10 parallel processes are specified.
verbose: Sets the level of detail for displaying the progress of parallel computations (explained later).
parameters_for_function: The list of parameters to pass to the function you want to run in parallel. List the input files, simulation temperatures, etc., that you want to compute.
The function function is where you describe the actual calculations or processes you want to perform. Please write the contents you want to calculate in each case. If the function has a return value, it will be stored in results at the end.
Note 1: About the verbose Setting
The verbose parameter used in joblib’s Parallel sets how detailed the progress information should be. The higher the value of verbose, the more detailed the progress information displayed.
Here, we briefly explain the verbose setting values and their output content:
For more detailed information, please refer to the official website.
verbose=0: No progress information is displayed. Use this if you do not want to enlarge the output area.
verbose=10: Information on task start and end is displayed. Use this if you want to periodically check the progress.
Note 2: About Delayed Execution
Delayed execution is a mechanism that prepares a function call for execution later, instead of executing it on the spot. By using this technique, you can perform multiple calculations sequentially in parallel.
Using joblib's delayed function, you can delay function calls. Delayed functions are deferred until the actual parallel computations are performed.