Repository navigation
Home
Using NEMO-PDAF obviously requires that you have downloaded a version of NEMO, PDAF, and the interface code coupling NEMO with PDAF, which also includes the user-routines for using PDAF.
You can obtain PDAF from https://github.com/PDAF/PDAF. PDAF is compiled as a library separately from NEMO.
For compiling PDAF follow the instructions in the README file or on the PDAF web site https://pdaf.awi.de.
When compiling PDAF it is important that you ensure that the same compiler and MPI-library is used as for the compilation of NEMO.
You can obtain the PDAF intreface and user code from https://github.com/larsnerger/NEMO-PDAF. This code has to be compiled with the build process of NEMO.
Check the NEMO website about how to download NEMO.
The NEMO-PDAF user codes include modules that are provided by PDAF. To enable the compilation in NEMO's build process one needs to tell the configuration that these module are not to be build by NEMO. Required is the following:
In NEMO's directory mk add the following lines to the files bld.cfg, bld_preproagr.cfg, and bldxag.cfg:
For PDAF2:
bld::excl_dep use::pdafomi
bld::excl_dep use::pdaf_interfaces_module
For PDAF3:
bld::excl_dep use::pdaf
An example of the adaptions described here is shown in the file run/arch-linux_mpifort_PDAF.fcm.
As usual for compiling NEMO one needs to specify an arch file. For compiling NEMO-PDAF one needs to adapt this file to include the include path to PDAF and the linking of the PDAF library. A possibility is to add the following lines
%PDAF_HOME [PATH_TO_PDAF_ROOT]
%PDAF_INC -I%PDAF_HOME/include
%PDAF_LIB -L%PDAF_HOME/lib -lpdaf-d
and then to adapt the settings of %USER_INC and %USER_LIB e.g. to read
%USER_INC %XIOS_INC %NCDF_INC %PDAF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB %PDAF_LIB
Because PDAF uses the BLAS and LAPACK linear algebra library, one also needs to specify these.
Important: The PDAF user code uses parallel NetCDF for file writing. Thus, one needs to specify a parallel NetCDF library. (Most likely you use such already for XIOS)
NEMO's build process allows to specify a single external directory with adapted code. The NEMO-PDAF model-binding is distributed into two directories:
MY_SRC_pdaf contains the PDAF user routines, which are independent of the NEMO version
MY_SRC_nemoXX, with XX indicating the NEMO version, contains modified NEMO codes including the calls to the PDAF interface routines.
To prepare the compilation one can copy the content of MY_SRC_nemoXX according to the NEMO version into MY_SRC_pdaf and later specify MY_SRC_pdaf as the directory contaiing the external code.
Note: The NEMO-PDAF code provides modifued NEMO files for NEMO 4.0 and 4.0.7. If you use another 4.0.X version, please you might need to adapt the NEMO codes youself. For this you can diff the provided files with the original files in the NEMO src/ directory. You can find the adaptions for interfacing the PDAF when searching for 'PDAF'. They are all enclosed with a preprocessor check for key_USE_PDAF.
After these preparation you should be ready to compile NEMO with activated PDAF coupling. For this use makenemo as common for building NEMO, but specify the external source directory and adding the preprocessor key key_USE_PDAF. For example one can compile as follows
./makenemo -n NEMO-PDAF_test -r REFERENCE_CONFIG -m YOUR_ARCH_FILE -e 'PATH_TO_MY_SRC_pdaf' add_key 'key_USE_PDAF' -j 8
We usually run NEMO-PDAF outside of the NEMO directory. An example run script used with SLURM and OpenMPI is in run/run_NEMO-PDAF_N4.sl. This is an example, and it is unlikely that works with your setup without changes.
Important aspects of the run script are:
- We recommend to run each NEMO task in a separate directory. This avoids conflicts in the file handling. The run script generates up to 99 numbered subdirectores and then links the necessary input files into these directories. Then each NEMO task is started in a separate directory, but within a single 'srun' command so that they all share the same MPI work communicator.
- We usually deactivate the NEMO file writing (except restart files) and let only the PDAF user code write the ensmeble mean and variance into files
- In any case, XIOS needs configuration files. The directory
run/config/contains the configuration files used by the run script (only those which are modified for ensemble output). These are examples of the configuration of XIOS (tested only with XIOS 2.0) for ensemble output (Generally XIOS writes all outputs into the directory001) but needs different xml configuration files for each NEMO ensemble task.