In this tutorial, you will build a PDI-enabled application step-by-step from a PDI-free base. You will end-up building the C version of the example provided with PDI for the the Trace, Decl'HDF5, and Pycall plugins. Additional examples are available for the other plugins.
To run this hands-on tutorial, you will of course first need to install PDI and setup your environment.
On Juwels, PDI is pre-installed as part of the training2022 environment. To load it, just run:
Once PDI is installed, you can proceed with getting the sources for the hands-on tutorial from github:
On Juwels, you can just copy the content from
Before compilation, configure the tutorial by detecting all dependencies:
For each exercise, once you've modified it, you can compile it by running the following command:
? is the number of the exercise.
In order to speedup the execution on Juwels, it's a good idea to use
salloc to reserve one node:
You can run each exercise with the following command:
? is the number of the exercise and 4 represents the number of MPI processes to use.
In order to generate clean logs into a file however, the command should be expanded as follow (for example for ex2.):
Now you're ready to work, good luck!
Ex1. implements a simple heat equation solver using an explicit forward finite difference scheme parallelized with MPI. The code uses a block domain decomposition where each process holds a 2D block of data.
Locally, each process holds its local block of data with one additional element on each side for ghost zones.
In the following exercises however, PDI will only be used to decouple I/O operations. There is no need to fully dive in the core of the solver described in the PDI example algorithm and implemented in the
The specification tree in the
.yml files and the
main function are the locations where all the I/O-related aspects will be handled and the only ones you will actually need to fully understand or modify.
This example gets its configuration from a file in the YAML format:
ex1.yml file. If you're not familiar with YAML, please have a look at our quick YAML format documentation to understand it. The example uses the paraconf library to read this file.
ex1.ymlto be able to run the code with 4 MPI processes.
Ex2. is the same code as that of ex1. with PDI calls added in
main function. In our YAML file (
ex2.yml), a new sub-tree has been added under the
pdi key. This sub-tree is the PDI specification tree passed to PDI at initialization. Here, the PDI Trace plugin is used to trace PDI calls.
PDI_reclaimcalls to match the output of
ex2.logfile (only the lines matching
[Trace-plugin]have been kept). You only need to change the
ex2.cfile. You can easily check if the files are the same by running the command:
In this exercise, the code is the same as in ex2. No need to touch the C code here, modification of the YAML file (
ex3.yml) should be enough.
pcoordvariables in addition to
dsizeto match the content expected as described in the
ex3.h5dumptext file (use the
h5dumpcommand to see the content of your HDF5 output file in the same format as the
.h5dumpfile). You can easily check if the files are the same by running the command:
To achieve this result, you will need to fill 2 sections in the YAML file.
datasection to indicate to PDI the datatype type of the fields that are exposed.
decl_hdf5section for the configuration of the Decl'HDF5 plugin.
ex3.h5file before, otherwise the data will not be changed.
In this exercise each MPI process will write its local 2D array block contained in the
main_field variable to a separate HDF5 file. Once again, this can be done by modifying the YAML file only, no nee to touch the C file.
Look at the number of blocks, you will have to use the correct number of MPI ranks to run the example.
Notice that in the YAML file, a list was used in the
decl_hdf5 section with multiple write blocks instead of a single one as before in order to write to multiple files.
Also notice that this example now runs in parallel with two processes. Therefore it uses "$-expressions" to specify the file names and ensure we do not write to the same file from distinct ranks.
Unlike the other fields manipulated until now, the type of
main_field is not fully known, its size is dynamic. By moving other fields in the
metadata section, you can reference them from "$-expressions" in the configuration file.
Unlike the other fields manipulated until now,
main_field is exposed multiple times along execution. In order not to overwrite it every time it is exposed, you need to add a
when condition to restrict its output.
main_fieldat the second iteration (when
ii=1) and match the expected content as described in
In ex4., two variables were written to
ex4-data*.h5, but the file was opened and closed for each and every write. Since Decl'HDF5 only sees the data appear one after the other, it does not keep the file open. Since
main_field are shared in an interlaced way, they are both available to PDI at the same time and could be written without opening the file twice. You have to use events for that, you will modify both the C and YAML file.
main_fieldare shared. With the Trace plugin, check that the event is indeed triggered at the expected time as described in
ex5.log(only the lines matching
[Trace-plugin]have been kept).
on_eventmechanism to trigger the write of
main_field. This mechanism can be combined with a
whendirective, in that case the write is only executed when both mechanisms agree.
main_fieldat iterations 1 and 2, in two distinct groups
iter2. Your output should match the content described in
As you can notice, the PDI code is quite redundant. In this exercise, you will use
PDI_multi_expose to simplify the code while keeping the exact same behaviour. For once, there is no need to modify the YAML file here, you only need to modify the C file in this exercise.
PDI_exposethat is the exact equivalent of a
PDI_sharefollowed by a matching
This replacement is not possible for interlaced
PDI_reclaim with events in the middle. This case is however handled by
PDI_multi_expose call that exposes all data, then triggers an event and finally does all the reclaim in reverse order.
PDI_multi_exposes and ensure that your code keeps the exact same behaviour by comparing its trace to
ex6.log(only the lines matching
[Trace-plugin]have been kept). You can easily check if the files are the same by running:
In this exercise, you will only write a selection of the 2D array in memory excluding ghosts to the HDF5 file. Once again, you only need to modify the YAML file in this exercise, no need to touch the C file.
As you can notice, now the dataset is independently described in the file.
ex7.h5dump. You can easily check if the files are the same by running:
You can achieve this by using the
memory_selection directive that specifies the selection of data from memory to write.
In this exercise, you will once again change the YAML file to handle a selection in the dataset in addition to the selection in memory from the previous exercise. You will write the 2D array from the previous exercise as a slice of 3D dataset including a dimension for time. Once again, you only need to modify the YAML file in this exercise, no need to touch the C file.
Notice how the dataset is now extended with an additional dimension for three time-steps.
main_fieldat iterations 1 to 3 inclusive into slices at coordinate 0 to 2 of first dimension of the 3D dataset. Match the expected output described in
ex8.h5dump. You can easily check if the files are the same by running:
You can achieve this by using the
dataset_selection directive that specifies the selection where to write in the file dataset.
Running the code from the previous exercises in parallel should already work and yield one file per process containing the local data block. In this exercise you will write one single file with parallel HDF5 whose content should be independent from the number of processes used. Once again, you only need to modify the YAML file in this exercise, no need to touch the C file.
mpi plugin was loaded to make sharing MPI communicators possible.
communicatordirective of the Decl'HDF5 plugin to switch to parallel I/O and change the file name so that all processes access the same file.
Match the output from
ex9.h5dump, that should be independent from the number of processes used. You can easily check if the files are the same by running:
In this exercise, you will once again modify the YAML file only and use python to post-process the data in situ before writing it to HDF5. Here, you will write the square root of the raw data to HDF5 instead of the data itself.
Notice that the Decl'HDF5 configuration was simplified, no memory selection is applied, the when condition disappeared. The dataset name is however explicitly specified now because it does not match the PDI variable name anymore, you will instead write a new variable exposed from python.
pycall section has been added to load the Pycall plugin. It executes the provided code when the "loop" event is triggered. The
with section specifies the variables (parameters) to pass to Python as a set of "$-expressions". The provided code again exposes its result to PDI and multiple blocks can be chained this way.
withblock to let the Python code process the data exposed in
ex10.h5dump. You can easily check if the files are the same by running:
.pyfile on the side.