This page contains a guide for building and using OpenIFS 48r1 in a linux based container using Docker. 

The instructions presented assume that docker is installed on the system of interest. If docker is not installed please speak with local system administration to find out if installation is permitted and how to do it. The specifics of this page have been tested with docker desktop on ubuntu 22.04 and Apple Macbook M1

While this page focuses on the containerised installation, the details presented, particularly the list of libraries to install, can be used as a recipe for the underlying dependencies required to install and run OpenIFS 48r1 on a linux system.

Contents:

1. Creation of a docker build directory

The first step is to make a directory from which the OpenIFS container can be created and cd into that directory, e.g.

mkdir <openifs-docker-build-dir>
cd <openifs-docker-build-dir>

Once inside the <openifs-docker-build-dir>  download the OpenIFS release package from https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz,  e.g.

# download the openifs-48r1.tar.gz model package from https://sites.ecmwf.int/openifs/openifs-data/src/48r1/openifs-48r1.tar.gz to <openifs-docker-build-dir> and untar :
tar -xvzf openifs-48r1.tar.gz

Then also download the SCM package to <openifs-docker-build-dir> from https://sites.ecmwf.int/openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.gz . 

# download the scm_openifs_48r1.tar.gz https://sites.ecmwf.int/openifs/openifs-data/scm/48r1/scm_openifs_48r1.tar.gz to <openifs-docker-build-dir> and untar :
tar -xzvf scm_openifs_48r1.tar.gz

At present, the SCM package is required for the build of the container.

Once OpenIFS 48r1 and the SCM package have been extracted to your <openifs-docker-build-dir> ,  then copy the Dockerfile from the release to <openifs-docker-build-dir> 

cp  <openifs-docker-build-dir>/openifs-48r1/scripts/docker/gcc-docker-48r1.1/Dockerfile <openifs-docker-build-dir>/.

This cp step is important because the Dockerfile  and the OpenIFS release and SCM package, i.e., openifs-48r1 and scm_openifs, need to be in the same directory to build the container.

2. Build the OpenIFS docker image

Before starting to build the container, it is necessary to set the OIFS_EXPT environment variable in oifs-config.edit_me.sh, i.e. 

open <openifs-docker-build-dir>/openifs-48r1/oifs-config.edit_me.sh  with a text editor, then change the following line

export OIFS_EXPT=${PERM}

to 

export OIFS_EXPT=/home/openifs/openifs-expt

The above change will ensure that the defined experiment directory is consistent to the experiment directory set-up during the build of the container. 

The following command builds the docker image from the <openifs-docker-build-dir> , with the docker command executed in the same directory as the Dockerfile .

docker build --tag "openifs-48r1" . 

Note the trailing '.' to build in the current dir, which is required.

This command runs the build process for the OpenIFS image using gcc:11.2.0-bullseye as the base image, which is a debian Linux distribution with gcc 11.2 (at the time of writing, similar to the gcc on the ATOS). After downloading the image, the Dockerfile installs the following software

    apt install -y git && \
    apt install -y cmake && \
    apt install -y python3 python3-ruamel.yaml python3-yaml python3-venv && \
    apt install -y libomp-dev && \
    apt install -y libboost-dev libboost-date-time-dev libboost-filesystem-dev libboost-serialization-dev libboost-program-options-dev&& \
    apt install -y netcdf-bin libnetcdf-dev libnetcdff-dev && \
    apt install -y libatlas-base-dev && \
    apt install -y liblapack-dev && \
    apt install -y libeigen3-dev && \
    apt install -y bison && \ 
    apt install -y flex && \
    apt install -y vim && \
    apt install -y wget bc 
## Install python3-pip and the dependencies to run 
## plotscm.py with SCM output
    apt install -y python3-pip && \
    pip3 install numpy && \
    pip3 install netcdf4 && \
    pip3 install matplotlib && \
    pip3 install pandas && \
    pip3 install xarray
## Download and build openmpi
https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz 

Once these repository packages and the python librarie are installed, open-mpi is downloaded and built. Then the openifs user is created and the openifs-48r1 directory is copied to the image.

On a Macbook pro (M1) the initial build of the container, takes about 5 minutes. If you decide to change the Dockerfile following a build, you can just execute the same command and, depending on what you change, subsequent builds can be a lot quicker. 

docker build --no-cache --tag "openifs-48r1" .

3. Running OpenIFS-test using the docker image

Once a build has completed, the image can be checked by typing docker images , e.g.,

docker images

REPOSITORY          TAG       IMAGE ID       CREATED       SIZE
openifs-48r1      latest    f72ea92f010f   5 hours ago   2.15GB

And the container can be run using the following

docker run -it 'openifs-48r1'

This command will open an interactive session in the new container, in which the entry directory is the openifs-48r1 directory, e.g.

docker run -it 'openifs-48r1'

OpenIFS environment variables are:
------------------------------------------------------
OIFS_ARCH=./arch/ecmwf/hpc2020
OIFS_CYCLE=48r1
OIFS_DATA_DIR=/perm/openifs/oifs_data/48r1/48r1
OIFS_EXEC=/home/openifs/openifs-48r1/build/bin/ifsMASTER.DP
OIFS_EXPT=/home/openifs/openifs-expt
OIFS_HOME=/home/openifs/openifs-48r1
OIFS_LOGFILE=/home/openifs/openifs-48r1/oifs_test_log.txt
OIFS_RUN_SCRIPT=/home/openifs/openifs-48r1/scripts/exp_3d
OIFS_TEST=/home/openifs/openifs-48r1/scripts/build_test


SCM environment variables are:
------------------------------------------------------
SCM_EXEC=/home/openifs/openifs-48r1/build/bin/MASTER_scm.DP
SCM_LOGFILE=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1/scm_run_log.txt
SCM_PROJDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects
SCM_RUNDIR=/home/openifs/openifs-expt/scm_openifs/48r1/scm-projects/ref48r1
SCM_TEST=/home/openifs/openifs-48r1/scripts/scm
SCM_VERSIONDIR=/home/openifs/openifs-expt/scm_openifs/48r1

openifs@d1bd89ccc47f:~/openifs-48r1$ ls
CHANGES  COPYING  NOTICE  arch        ifs-source  oifs-config.edit_me.sh  scripts
CITE     LICENSE  README  bundle.yml  ifs-test    openifs-bundle

Once inside the container, the openifs-test.sh  script can be run, e.g., 

openifs@d1bd89ccc47f:~/openifs-48r1$ $OIFS_TEST/openifs-test.sh -cbt -j 2 --arch=""

where :

-c   creates source directory in $OIFS_HOME, which is used to bring all the sources, e.g. ifs-source, ifs-test and any packages in the bundle (ecbuild, eccodes etc.), together in preparation for the build

-b  builds the source. This step creates the directory build in $OIFS_HOME, which is used to build and store

• the OpenIFS double and single precision master executables (ifsMASTER.DP and ifsMASTER.SP, respectively) , which are used to run 3-D OpenIFS. The executables are located in  $OIFS_HOME/build/bin.
  • The location and name of the executable for OpenIFS is defined in the platform configuration file (oifs-config.edit_me.sh) as $OIFS_EXEC.

• the double and single precision Single Column Model (SCM) executables (MASTER_scm.DP and MASTER_scm.SP, respectively), which are used to run the SCM derived from OpenIFS. The executables are located in  $OIFS_HOME/build/bin.
  • The location and name of the executable for the SCM is defined in the platform configuration file (oifs-config.edit_me.sh) as $SCM_EXEC.

-t will run the ifs-test t21 tests, which comprise of

• 21 3-D OpenIFS forecast-only tests with and without chemistry
• 1 SCM test (based on TWP-ICE)

-j is an optional argument that allows the user to define the number of threads used to create and build openifs, with the above example using 2 threads. It should be noted that 

• If -j is not supplied as a command line argument, as in the above example, the number of threads will default to 8 threads, which can be problematic for older and lower spec systems. For example, it will not work with a 4 core intel m3. The above example uses 2 threads, which should work on most systems.
• The number of threads can be increased to speed up the build step, but how much the number of threads can be increased to will depend on the system spec. Hence, please speak the local system administration for guidance about appropriate settings. 

--arch="" means no arch file is required because all the libraries and paths have been set-up locally in the docker container. 

For more information on the openifs-test option please refer to Getting started and the linked page Openifs-test - build options.

On a Macbook pro (M1) the build of OpenIFS, takes about 10 minutes, while successful completion of ifstest takes just over 3 minutes.

4. Running the SCM in the docker container

Once the above has been completed, you are ready to run the SCM. The scm_openifs  contains 3 testcases, each representative of different cloudy regimes

• DYCOMS - marine stratocumulus case
• BOMEX - trade-wind cumulus case 
• TWPICE - A multi-day deep convective case

The SCM is run using the callscm script, which is a wrapper for the main run.scm. Both scripts can found in $SCM_TEST , which is set in the oifs-config.edit_me.sh file to ${OIFS_HOME}/scripts/scm . 

callscm  includes default settings, which are the all cases, with a 450 s timestep and an experiment name of ref-oifs-scm . To run with these settings, enter the following

cd $OIFS_HOME
$SCM_TEST/callscm

callscm  (with defaults, i.e. no arguments) will run the DYCOMS case with the SCM and create an output directory in $SCM_RUNDIR/scmout_DYCOMS_ref-oifs-scm_450s , which contains the diagnostic output from the SCM. In addition, the file scm_run_log.txt will be created in $SCM_RUNDIR . This file contains the print output from the SCM, which is useful for checking all the sources and paths for a simulation. 

A user can change the defaults by using the available command-line options

callscm -h -c <case_name or list of case_names> -t <timestep or list of timesteps>
        -x <expt_name>
where :
-h is help which returns basic usage options and exits
-c case_name or list of case_names (space delimited) of the case study
   used for namelist and output directory. Default list is
   "DYCOMS BOMEX TWPICE" 
-t timestep or list of timesteps in seconds. The default is 450s. An
   example of a list is "1800 900 300" 
-x expt_name shortname to identify experiment. Default is ref-oifs-scm

 For example, if a user wanted to run the BOMEX case with timesteps of 1800 s and 900 s and an experiment name of "bomex_test", they would enter the following

$SCM_TEST/callscm -c BOMEX -t "1800 900" -x "bomex_test"

This command results in the following output directories $SCM_RUNDIR/scmout_BOMEX_bomex_test_900s  and scmout_BOMEX_bomex_test_1800s.

5. Example plotting script

Once the SCM has successfully, it will produce output in $SCM_RUNDIR/scmout_<casename>_<expt_name>_<timestep>, which contains some netcdf files. 

As part of the scm_openifs  download package we provide a python script $SCM_RUNDIR/plotscm.py , which can be used to plot some profiles at a certain time, 2-d time height plots and some scalar timeseries. This script will work without any changes for the DYCOMS case and the default settings of callscm by type the following

cd $SCM_RUNDIR
python3 plotscm.py

 For the defaults, the above will produce plot_DYCOMS  directory, which contains png plots

To plot the other cases or experiments with different names and/or timesteps to the defaults, at present, the user will need to edit plotscm.py appropriately. 

6. Basic docker commands and functionality

6.1. Check for existing containers

From a terminal on the host system, it is possible to list running and exited containers with the following

1. List running containers

   $ docker ps
   
   CONTAINER ID   IMAGE            COMMAND   CREATED        STATUS         PORTS     NAMES
   d1bd89ccc47f   openifs-48r1   "bash"    15 hours ago   Up 5 seconds             beautiful_pasteur

    If no container is running on your system, then only CAPITAL headings are returned with docker ps 

2. List all containers (running and exited)

   $ docker ps -a
   
   CONTAINER ID   IMAGE               COMMAND       CREATED        STATUS                     PORTS     NAMES
   d1bd89ccc47f   openifs-48r1      "bash"        15 hours ago   Exited (0) 4 seconds ago             beautiful_pasteur

   Notice that the STATUS  is Exited, rather than Up, as in (1)

6.2. Start an existing container

If docker ps -a shows an exited container it can be restarted using the following

$ docker start -i <Container ID> 
# e.g. 
$ docker ps -a

CONTAINER ID   IMAGE               COMMAND       CREATED        STATUS                     PORTS     NAMES
d1bd89ccc47f   openifs-48r1      "bash"        15 hours ago   Exited (0) 4 seconds ago             beautiful_pasteur

$ docker start -i  d1bd89ccc47f

6.3. Exiting the container

1. The container can be exited by typing exit from the active container.

2. A container can be stopped from a terminal on the host system

   docker stop <Container ID>

   It is also possible to stop multiple containers at once by adding more than one ID.

3. Using docker stop  will attempt a clean shutdown. It is also possible to stop the container by killing it 

   docker kill <Container ID>

6.4. Removing containers and images

Remove a container with the following command

docker rm <Container ID>

It is possible to remove multiple containers at once by adding more than one ID. 

It is worth noting that if a container is running it cannot be removed without being forced or being stopped first