This page contains a guide for building and using OpenIFS 48r1 in a linux based container using Docker.
The instructions presented assume that docker is installed on the system of interest. If docker is not installed please speak with local system administration to find out if installation is permitted and how to do it. The specifics of this page have been tested with docker desktop on ubuntu 22.04 and Apple Macbook M1
While this page focuses on the containerised installation, the details presented, particularly the list of libraries to install, can be used as a recipe for the underlying dependencies required to install and run OpenIFS 48r1 on a linux system.
If you have access to the ECMWF HPCF (ATOS hpc2020) and are planning to run OpenIFS on that system please refer to OpenIFS 48r1 User Guide.
Contents:
1. Creation of a docker build directory
The first step is to make a directory from which the OpenIFS container can be created and cd
into that directory, e.g.
mkdir <openifs-docker-build-dir> cd <openifs-docker-build-dir>
Once inside the <openifs-docker-build-dir>
download the OpenIFS release package from https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz, e.g.
# download the openifs-48r1.tar.gz model package from https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz to <openifs-docker-build-dir> and untar : tar -xvzf openifs-48r1.tar.gz
Then also download the SCM package to <openifs-docker-build-dir> from
https://sites.ecmwf.int/openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.gz .
# download the scm_openifs_48r1.tar.gz https://sites.ecmwf.int/openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.gz to <openifs-docker-build-dir> and untar : tar -xzvf scm_openifs_48r1.tar.gz
At present, the SCM package is required for the build of the container.
Once OpenIFS 48r1 and the SCM package have been extracted to your <openifs-docker-build-dir>
, then copy the Dockerfile
from the release to <openifs-docker-build-dir>
cp <openifs-docker-build-dir>/openifs-48r1/scripts/docker/gcc-docker-48r1.1/Dockerfile <openifs-docker-build-dir>/.
This cp
step is important because the Dockerfile
and the OpenIFS release and SCM package, i.e., openifs-48r1 and scm_openifs
, need to be in the same directory to build the container.
2. Build the OpenIFS docker image
Before starting to build the container, it is necessary to set the OIFS_EXPT environment variable in oifs-config.edit_me.sh, i.e.
open <openifs-docker-build-dir>/openifs-48r1/oifs-config.edit_me.sh
with a text editor, then change the following line
export OIFS_EXPT=${PERM}
to
export OIFS_EXPT=/home/openifs/openifs-expt
The above change will ensure that the defined experiment directory is consistent to the experiment directory set-up during the build of the container.
The following command builds the docker image from the <openifs-docker-build-dir>
, with the docker command executed in the same directory as the Dockerfile
.
docker build --tag "openifs-48r1" .
Note the trailing '.' to build in the current dir, which is required.
This command runs the build process for the OpenIFS image using gcc:11.2.0-bullseye as the base image, which is a debian Linux distribution with gcc 11.2 (at the time of writing, similar to the gcc on the ATOS). After downloading the image, the Dockerfile
installs the following software
apt install -y git && \ apt install -y cmake && \ apt install -y python3 python3-ruamel.yaml python3-yaml python3-venv && \ apt install -y libomp-dev && \ apt install -y libboost-dev libboost-date-time-dev libboost-filesystem-dev libboost-serialization-dev libboost-program-options-dev&& \ apt install -y netcdf-bin libnetcdf-dev libnetcdff-dev && \ apt install -y libatlas-base-dev && \ apt install -y liblapack-dev && \ apt install -y libeigen3-dev && \ apt install -y bison && \ apt install -y flex && \ apt install -y vim && \ apt install -y wget bc ## Install python3-pip and the dependencies to run ## plotscm.py with SCM output apt install -y python3-pip && \ pip3 install numpy && \ pip3 install netcdf4 && \ pip3 install matplotlib && \ pip3 install pandas && \ pip3 install xarray ## Download and build openmpi https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz
Once these repository packages and the python librarie are installed, open-mpi
is downloaded and built. Then the openifs
user is created and the openifs-48r1
directory is copied to the image.
On a Macbook pro (M1) the initial build of the container, takes about 5 minutes. If you decide to change the Dockerfile
following a build, you can just execute the same command and, depending on what you change, subsequent builds can be a lot quicker.
- At the time of writing open-mpi is downloaded and built as part of the image creation. This is quite slow, so in the future we will investigate the use of a standard library
- If a fresh build is required but an image has already been built, then execute the following command, which is the same as above but with no-cache
docker build --no-cache --tag "openifs-48r1" .
- The tag name does not need to be provided but it is useful for identifying images.
3. Running OpenIFS-test using the docker image
Once a build has completed, the image can be checked by typing docker images
, e.g.,
docker images REPOSITORY TAG IMAGE ID CREATED SIZE openifs-48r1 latest f72ea92f010f 5 hours ago 2.15GB
And the container can be run using the following
docker run -it 'openifs-48r1'
This command will open an interactive session in the new container, in which the entry directory is the openifs-48r1 directory, e.g.
docker run -it 'openifs-48r1' OpenIFS environment variables are: ------------------------------------------------------ OIFS_ARCH=./arch/ecmwf/hpc2020 OIFS_CYCLE=48r1 OIFS_DATA_DIR=/perm/openifs/oifs_data/48r1/48r1 OIFS_EXEC=/home/openifs/openifs-48r1/build/bin/ifsMASTER.DP OIFS_EXPT=/home/openifs/openifs-expt OIFS_HOME=/home/openifs/openifs-48r1 OIFS_LOGFILE=/home/openifs/openifs-48r1/oifs_test_log.txt OIFS_RUN_SCRIPT=/home/openifs/openifs-48r1/scripts/exp_3d OIFS_TEST=/home/openifs/openifs-48r1/scripts/build_test SCM environment variables are: ------------------------------------------------------ SCM_EXEC=/home/openifs/openifs-48r1/build/bin/MASTER_scm.DP SCM_LOGFILE=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1/scm_run_log.txt SCM_PROJDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects SCM_RUNDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1 SCM_TEST=/home/openifs/openifs-48r1/scripts/scm SCM_VERSIONDIR=/home/openifs/openifs-expt/scm_openifs/48r1 openifs@d1bd89ccc47f:~/openifs-48r1$ ls CHANGES COPYING NOTICE arch ifs-source oifs-config.edit_me.sh scripts CITE LICENSE README bundle.yml ifs-test openifs-bundle
- In the above code block the openifs config (
oifs-config.edit_me.sh
) is sourced automatically so the container has the correct environment for OpenIFS
Once inside the container, the openifs-test.sh
script can be run, e.g.,
openifs@d1bd89ccc47f:~/openifs-48r1$ $OIFS_TEST/openifs-test.sh -cbt -j 2 --arch=""
where :
-c
creates source
directory in $OIFS_HOME
, which is used to bring all the sources, e.g. ifs-source, ifs-test and any packages in the bundle (ecbuild, eccodes etc.), together in preparation for the build
-b
builds the source. This step creates the directory build
in $OIFS_HOME
, which is used to build and store
- the OpenIFS double and single precision master executables (ifsMASTER.DP and ifsMASTER.SP, respectively) , which are used to run 3-D OpenIFS. The executables are located in
$OIFS_HOME/build/bin
.- The location and name of the executable for OpenIFS is defined in the platform configuration file (oifs-config.edit_me.sh) as
$OIFS_EXEC
.
- The location and name of the executable for OpenIFS is defined in the platform configuration file (oifs-config.edit_me.sh) as
- the double and single precision Single Column Model (SCM) executables (MASTER_scm.DP and MASTER_scm.SP, respectively), which are used to run the SCM derived from OpenIFS. The executables are located in
$OIFS_HOME/build/bin
.- The location and name of the executable for the SCM is defined in the platform configuration file (oifs-config.edit_me.sh) as
$SCM_EXEC
.
- The location and name of the executable for the SCM is defined in the platform configuration file (oifs-config.edit_me.sh) as
-t
will run the ifs-test t21 tests, which comprise of
- 21 3-D OpenIFS forecast-only tests with and without chemistry
- 1 SCM test (based on TWP-ICE)
-j
is an optional argument that allows the user to define the number of threads used to create and build openifs, with the above example using 2 threads. It should be noted that
- If
-j
is not supplied as a command line argument, as in the above example, the number of threads will default to 8 threads, which can be problematic for older and lower spec systems. For example, it will not work with a 4 core intel m3. The above example uses 2 threads, which should work on most systems. - The number of threads can be increased to speed up the build step, but how much the number of threads can be increased to will depend on the system spec. Hence, please speak the local system administration for guidance about appropriate settings.
--arch="" means no arch file is required because all the libraries and paths have been set-up locally in the docker container.
For more information on the openifs-test option please refer to Getting started and the linked page Openifs-test - build options.
On a Macbook pro (M1) the build of OpenIFS, takes about 10 minutes, while successful completion of ifstest takes just over 3 minutes.
4. Running the SCM in the docker container
Once the above has been completed, you are ready to run the SCM. The scm_openifs
contains 3 testcases, each representative of different cloudy regimes
- DYCOMS - marine stratocumulus case
- BOMEX - trade-wind cumulus case
- TWPICE - A multi-day deep convective case
The SCM is run using the callscm
script, which is a wrapper for the main run.scm
. Both scripts can found in $SCM_TEST
, which is set in the oifs-config.edit_me.sh
file to ${OIFS_HOME}/scripts/scm
.
callscm
includes default settings, which are the all cases, with a 450 s timestep and an experiment name of ref-oifs-scm
. To run with these settings, enter the following
cd $OIFS_HOME $SCM_TEST/callscm
callscm
(with defaults, i.e. no arguments) will run the DYCOMS case with the SCM and create an output directory in $SCM_RUNDIR/scmout_DYCOMS_ref-oifs-scm_450s
, which contains the diagnostic output from the SCM. In addition, the file scm_run_log.txt
will be created in $SCM_RUNDIR
. This file contains the print output from the SCM, which is useful for checking all the sources and paths for a simulation.
A user can change the defaults by using the available command-line options
callscm -h -c <case_name or list of case_names> -t <timestep or list of timesteps> -x <expt_name> where : -h is help which returns basic usage options and exits -c case_name or list of case_names (space delimited) of the case study used for namelist and output directory. Default list is "DYCOMS BOMEX TWPICE" -t timestep or list of timesteps in seconds. The default is 450s. An example of a list is "1800 900 300" -x expt_name shortname to identify experiment. Default is ref-oifs-scm
For example, if a user wanted to run the BOMEX case with timesteps of 1800 s and 900 s and an experiment name of "bomex_test", they would enter the following
$SCM_TEST/callscm -c BOMEX -t "1800 900" -x "bomex_test"
This command results in the following output directories $SCM_RUNDIR/scmout_BOMEX_bomex_test_900s
and scmout_BOMEX_bomex_test_1800s
.
5. Example plotting script
Once the SCM has successfully, it will produce output in $SCM_RUNDIR/scmout_<casename>_<expt_name>_<timestep>, which contains some netcdf files.
As part of the scm_openifs
download package we provide a python script $SCM_RUNDIR/plotscm.py
, which can be used to plot some profiles at a certain time, 2-d time height plots and some scalar timeseries. This script will work without any changes for the DYCOMS case and the default settings of callscm by type the following
cd $SCM_RUNDIR python3 plotscm.py
For the defaults, the above will produce plot_DYCOMS
directory, which contains png plots
To plot the other cases or experiments with different names and/or timesteps to the defaults, at present, the user will need to edit plotscm.py appropriately.
6. Basic docker commands and functionality
6.1. Check for existing containers
From a terminal on the host system, it is possible to list running and exited containers with the following
List running containers
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d1bd89ccc47f openifs-48r1 "bash" 15 hours ago Up 5 seconds beautiful_pasteur
If no container is running on your system, then only CAPITAL headings are returned with
docker ps
List all containers (running and exited)
$ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d1bd89ccc47f openifs-48r1 "bash" 15 hours ago Exited (0) 4 seconds ago beautiful_pasteur
Notice that the
STATUS
isExited
, rather thanUp
, as in (1)
6.2. Start an existing container
If docker ps -a
shows an exited container it can be restarted using the following
$ docker start -i <Container ID> # e.g. $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d1bd89ccc47f openifs-48r1 "bash" 15 hours ago Exited (0) 4 seconds ago beautiful_pasteur $ docker start -i d1bd89ccc47f
6.3. Exiting the container
- The container can be exited by typing
exit
from the active container. A container can be stopped from a terminal on the host system
docker stop <Container ID>
It is also possible to stop multiple containers at once by adding more than one ID.
Using
docker stop
will attempt a clean shutdown. It is also possible to stop the container by killing itdocker kill <Container ID>
6.4. Removing containers and images
Remove a container with the following command
docker rm <Container ID>
It is possible to remove multiple containers at once by adding more than one ID.
It is worth noting that if a container is running it cannot be removed without being forced or being stopped first
Removing a container will result in a loss of all data and any code changes etc from the container