- Created by Glenn Carver, last modified on Sep 23, 2020
Prerequisites
You should have completed: the install GRIB library and downloaded and unpacked the OpenIFS anciliary data files and OpenIFS source code.
Set compilation environment
If your system uses the 'module' command to load software such as compilers, make sure the required modules are loaded before attempting to build OpenIFS.
Make sure the same compiler family is used for the libraries (e.g. netCDF, MPI etc) that OpenIFS is compiled with. Otherwise problem linking/running may arise.
Configuring OpenIFS: oifs-config
If this is the first time compiling OpenIFS, the first step is to setup the environment for compilation and selecting OpenIFS configuration options. A successful compilation relies on correctly specifying the location of some system and user libraries (e.g. netCDF and ecCodes).
Find the file oifs-config.editme.sh
and make a copy to edit using your favourite editor:
% cd oifs43r3v1 % cp oifs-config.editme.sh oifs-config.sh # .. then edit oifs-config.sh
OpenIFS configuration options
For a complete list of options that can be set see make/oifs.fcm
which sets defaults. Most options in this file can be overridden by the user by setting an environment variable of the same name.
There are number of variables defined in the oifs-config.sh file that control the behaviour and capabilities of OpenIFS. The most commonly altered ones are listed in the oifs-config.sh
file.
Edit the oifs-config.sh
file and set your required options. Most of the options can be left to their defaults.
export OIFS_HOME=<set this to the top level directory of your OpenIFS installation> export OIFS_DATA_DIR=<set this to the top level directory of the previous downloaded 'ifsdata' files> export OIFS_LAPACK=<system or openifs> # default is OpenIFS will use LAPACK library available on your system. If not available set to openifs to use provided code. export OIFS_FFTW=<disable or enable> # FFTW is a FFT library that improves performance. Set to enable if it's available on your system. export OIFS_XIOS=<disable or enable> # XIOS is a parallel I/O library that provides netCDF output. Set to enable if available and you intend to use it.
OpenIFS compilation options
The key variables in oifs-config.sh
to set that affect compilation are (see below for more information):
export OIFS_GRIB_DIR="<set this to the location of your ecCodes installation>" export OIFS_NETCDF_DIR="<set this to the directory containing the netcdf installation>" export OIFS_COMP=gnu # this chooses the compiler, either: gnu, intel, or cce (Cray compiler) export OIFS_BUILD=opt # this chooses the type of build, either 'opt' for optimized, or 'noopt' for un-optimized
See the comments in the 'oifs-config.sh
' file for more information about the variables.
Set environment options for compilation
Once you have made desired edits to the oifs-config
file, 'source
' it to set the environment variables:
% source ./oifs-config.me.sh Environment for OpenIFS: CY43R3 OpenIFS home directory: /home/users/auser OpenIFS top-level data directory: /home/users/auser/ecmwf/ifsdata OpenIFS initial data directory : <editme> OpenIFS GRIB directory: /home/users/auser/ecmwf/eccodes OpenIFS netCDF directory: /usr/local/lib64 OpenIFS default compiler and build set to: gnu opt Added OpenIFS FCM and GRIB to user path Creating command aliases for OpenIFS: omake, omakenew, oenv
Subsequent compilations
With the oifs-config.sh
file correctly set, always 'source
' this file to set the environment variables before any compilation of OpenIFS:
% source ./oifs-config.sh
If you prefer not to use the oifs-config.sh
, we suggest editing the oifs.fcm
directly or making a copy and use that on the fcm command line instead.
FCM
OpenIFS uses software called 'FCM' for the compilation of the code. FCM is contained in the OpenIFS source tarball downloaded from the ftp site.
If you are not using the oifs-config.sh
file to set the environment, before trying to compile, make sure the FCM directory is added to your PATH environment variable so the fcm
command can be found (if you already have fcm available on your system, we still recommend using the distributed version as this is tested and known to work):
export PATH=$PATH:$HOME/ecmwf/oifs/fcm/bin
If you are not using oifs-config.sh
, you will also need to make sure the required OIFS_* environment variables to locate the GRIB and netCDF libraries are set otherwise the compilation will fail.
Build configuration
This is a brief introduction on compiling the OpenIFS model. For a more detailed description of the build system for OpenIFS and the different ways to configure it, see Customizing the OpenIFS build.
Before compiling the model, the build configuration must first be set by several environment variables:
OIFS_COMP - This sets the choice of compiler. The default is 'gnu' which means the gfortran/gcc compilers will be used. Other choices are 'intel' and 'cce' (Cray compiler).
OIFS_BUILD - This sets the type of build.
These environment variables directly correspond to the names of the FCM configuration files in the make/fcm/
directory in the source. Please see this directory for the choices provided.
In OpenIFS version prior to 43r3, this directory was named 'make/cfg' rather than 'make/fcm'.
Build types: OIFS_BUILD sets the type of build and there will be one FCM configuration file for each type. Build types provided are:
- opt - Recommended optimized compile settings for this compiler. Tested to provide best performance for this platform & compiler.
- noopt - Debugging options. No optimization set for all of the code. Suitable for use with a debugger. OpenMP is disabled.
- nansC - Debugging options with extra checks. Array bound checking enabled and variables initialized to NaN (not-a-number). Use of a variable set to NaN will trigger a floating point exception. This option is useful for debugging purposes but will run the slowest. It may not work in all cases.
e.g.
export OIFS_COMP=intel export OIFS_BUILD=opt
means use the Intel ifort compiler and the model will be compiled with full optimization compiler settings. So the FCM build system will expect to find a file:
make/fcm/intel-opt.fcm
Setting instead:
export OIFS_COMP=gnu export OIFS_BUILD=noopt
means FCM will expect to find a file:
make/fcm/gnu-noopt.fcm
If using GNU on a Mac, please see MacOS X known issues.
Building for other systems
As the environment variables refer to the corresponding filename in the make/fcm
directory it is straightforward to create a build configuration for other environments. Take a copy of an existing file and modify it as necessary. It's also possible to alter compile options by the use of additional environment variables (see below). Which approach you use depends on your personal preference.
Be aware however that the optimization compiler flags in the provided configuration files are the recommended ones. It is impossible to test all combinations of compiler flags so the model may become unstable, results differ or performance may reduce if higher optimizations are tried.
Setting the location of libraries
The directory paths to the libraries OpenIFS needs for compilation are normally set in the 'oifs-config.sh
' file (as described above). If you choose not to use 'oifs-config.sh
', you'll need to set the environment variables that the FCM configuration files expect to be defined.
The model requires a number of libraries (see 5.2 Prerequisites) to compile: the GRIB library (ECMWF's ecCodes), the netCDF (43r3+ only), and the LAPACK & BLAS numerical libraries are required. An MPI library is also needed but this is assumed to be provided by compiler wrappers in the MPI implementation (e.g. the command mpif90
rather than gfortran
).
To set the location of the GRIB, netCDF & LAPACK libraries, other environment variables need to be defined as the defaults in the FCM configuration files will most likely not be appropriate.
OIFS_GRIB_DIR - Defines the location of the GRIB library. For more details see '5.3 Install ecCodes GRIB library for OpenIFS'.
OIFS_NETCDF_DIR - Defines the location of the directory containing the netCDF libraries. This varies from system to system but typically /usr/local/lib
. n.b. OpenIFS versions before 43r3 did not use the netCDF library.
OIFS_LAPACK_LIB - Defines the location of the directory containing the liblapack.a
and libblas.a
files. This varies from system to system but typically /usr/local/lib
.
If you do not have LAPACK installed on your system or find it does not work, OpenIFS comes with all the necessary LAPACK code to run. It is disabled by default, see the README.lapack
file in the OpenIFS source, or contact openifs-support@ecmwf.int for instructions on how to enable it.
Or if you prefer you can either edit the default value in the appropriate FCM configuration file in make/fcm
directory or make a copy of the supplied configuration file (.fcm) and use that.
Serial (non-MPI) execution
Although the recommended way of using OpenIFS is with MPI parallel execution, we aim to provide a compilation option without MPI. This significantly affects performance and is not recommended except for specialized applications or testing. Only the lower resolutions will be realistic with a serial execution.
In a serial (non-MPI) case, the normal compiler wrappers (e.g. mpif90) are not used and the compiler (e.g. ifort) is called directly. A set of dummy subroutines is used to satisfy the linking of the model for the MPI calls.
To enable serial mode, set the following environment variable or edit oifs-config.sh directly. This enables the dummy MPI library and reset the compiler names (see READMEs/README_mpi for more details):
export OIFS_MPI=serial
Compiling OpenIFS
Once the environment variables are set, OpenIFS can be compiled (make sure you have edited your PATH environment variable to add the FCM installation bin
directory). Assuming you are in the directory where the OpenIFS source code was unpacked:
% cd make % fcm make -f oifs.fcm
The command 'fcm make
' starts the compilation. The -f option specifies the location of the master configuration file for OpenIFS.
For more information about fcm command options, try: fcm --help
.
This generates output similar to:
[init] make # 2012-12-20 19:59:59Z [init] make config-parse # 2012-12-20 19:59:59Z [info] config-file=...../make/oifs.fcm [info] config-file= - ...../make/fcm/gnu-opt.fcm [info] ............ [done] make config-parse # 0.3s [init] make dest-init # 2012-12-20 19:59:59Z [info] mode=new [done] make dest-init # 0.3s [init] make build # 2012-12-20 20:00:00Z [info] sources: total=2194, analysed=2194, elapsed-time=11.7s, total-time=32.2s [info] target-tree-analysis: elapsed-time=9.8s [info] compile targets: modified=2073, unchanged=0, total-time=1095.5s [info] compile+ targets: modified=628, unchanged=0, total-time=1.6s [info] ext-iface targets: modified=1247, unchanged=0, total-time=21.9s [info] install targets: modified=120, unchanged=0, total-time=0.3s [info] link targets: modified=1, unchanged=0, total-time=18.3s [info] TOTAL targets: modified=4069, unchanged=0, elapsed-time=316.3s [done] make build # 329.1s [done] make # 329.8s
Parallel compilation
If you have a multicore processor available, the -j <nthreads> option to 'fcm make
' can be used to significantly speed up the compilation of the code.
% fcm make -j 4 -f oifs.fcm
Tests show that compilation time reduces with a diminishing amount as the number of threads is increased. Set the number of threads to a value less than the available processor cores on your system.
Verbose compilation
The default output of FCM is quite terse. To get more verbose output use the '-v' option. e.g.
fcm make -v -j 4 -f oifs.fcm
will produce a more detailed report and use 4 processes to analyse and build the model code in parallel.
Successful compilation
After a successful compilation, the OpenIFS executable, master.exe
, can be found in the 'make/gnu-opt/oifs/bin
' subdirectory (or whichever compiler you choose):
% ls fcm gnu-opt % ls gnu-opt/oifs bin include o % ls gnu-opt/oifs/bin master.exe
The 'gnu-opt' subdirectory is generated by FCM using the values of the OIFS_COMP
& OIFS_BUILD
environment variables. The oifs/include
directory contains all the files referenced in the code by a '#include
' statement. This includes the Fortran interface block files which are auto-generated by FCM and end in .intfb.h
. The oifs/o
directory contains all the compiled object files. The OpenIFS build does not place any additional files in the src
directory (known as building 'out of source').
If using the intel compiler then the directory would be 'make/intel-opt
' and so on.
Compilation problems
If you get the message:
fcm make -f oifs.fcm /bin/ksh: fcm: not found [No such file or directory]
you need to add the 'bin' directory of the FCM installation to your PATH environment variable. FCM is supplied with OpenIFS and is in the directory 'fcm'.
We recommend editing and using the oifs-config.sh file to store all the correct settings of the OpenIFS environment variables.
On this page ...