Login to ECMWF Cray

Each participant will have a training user account on the ECMWF Cray system. This is different from the user account on the classroom computers.

Begin by logging onto the Cray login nodes:

ssh troifsXX@ccb                        <<< use your Cray user id
Password: xxxxxxxx

troifs@ccb-login2:~> 

The contents of the account should look like:

troifs1@ccb-login2:~> ls
bin  oifs38r1  scratch

If any are missing please let us know.

Directory: oifs38r1

This directory contains the OpenIFS source code and OpenIFS executable.

For the purpose of this workshop, the model has already been compiled to avoid delay and adding unnecessary load to the login nodes.

Directory: scratch

This is the directory in which we will run OpenIFS.

Creating the OpenIFS ensemble experiment

An ensemble experiment consists of a 'control' and multiple 'perturbed' forecasts, or 'ensemble members'. For the operational ECMWF ensemble forecast, 50 members are used.

For this exercise, to save computer time, each participant will run a single, different 'ensemble member' and we will create the complete forecast ensemble from the individual ensembles run by the class as a whole.

The exercises show how to create the experiment directories, prepare the jobs and submit them to the ECMWF Cray.

createENS command

Use the command createENS to create the ensemble experiment:

createENS -d 2013102400 -m 1

createENS --help

usage: createENS [-h] -d <startdate/enddate/hr_inc> [-m <members>]
Set up ensemble forecast for the OpenIFS 2015 workshop
optional arguments:
  -h, --help            show this help message and exit
  -d <startdate/enddate/hr_inc>, --date <startdate/enddate/hr_inc>
                        List of date(s) to retrieve, format: YYYYMMDDHH. The
                        only dates available are: 2013102400/2013102700/24
  -m <members>, --members <members>
                        Number of ensemble members to create e.g. 3 (default 1)

Exercise

Change to the experiment directory labelled by the date chosen:

cd scratch/ENFO/2013102400/

There are two experiment directories: cf000 and pf001. cf000 is for the control forecast which is not used here. pf001 is the single perturbation forecast.

cd pf001
ls

Understand what the different files in the experiment directory are for.

Initial files

ICM* : these are the initial starting files. ICMGG* are the initial gridpoint files, ICMSH* are the initial spectral fields.

Note that part of the file name is 'oifs'. This is known as the 'experiment id' (exptid). It is also encoded into the Grib file itself.

Use the 'grib_ls' command to examine the initial files:

grib_ls ICMGGoifsINIT
grib_ls ICMSHoifsINIT

Fortran namelist

The file: namelistfc is the model 'NAMELIST'. It contains a list of variable settings or 'switches' that control what the model does. These variables are grouped into separate fortran namelists.

The model will read this file when it starts up. There are many options to control the model. For more information, it is best and recommended to check the comments in the code. Please ask if you are interested.

Creating the OpenIFS namelist

Set the ensemble member

In IFS, each ensemble member uses the stochastic physics schemes to generate uncertainty. A random number 'seed' is used by the stochastic scheme to generate a different forecast. This random number seed is changed by altering the ensemble member value in the model's namelist. Each ensemble member must have a unique number and therefore random number seed, in order to produce a different forecast.

Exercise

cp namelistfc mynamelist

gedit mynamelist        # or use your favourite text editor; vi, emacs etc.

and make the following changes to the namelist variables:

CTYPE="pf",   ! change the type of forecast to 'perturbed' from 'control'

NENSFNB=2,    ! change the ensemble member number to your assigned value. 

LSTOPH_SPBS=true,  ! this enables the stochastic backscatter scheme in the model dynamics

LSPSDT=true,       ! this enables the stochastic scheme for the physics tendencies

The variable NENSFNB is important as this determines the random number seed used by the stochastic schemes, by altering the ensemble member value.

diff mynamelist namelistfc

Create and submit Cray batch job

The next steps are to create and submit a batch job to run on the ECMWF Cray XC30 system.

To create the batch job, use the oifs_run command. This creates a small batch job file ready to submit.

Exercise

Use the oifs_run command to create the job.

The exptid to use with the '-e' options passed to oifs_run must be "oifs". It is part of the filename of the initial files for the model (the files starting with "ICM").

The '-l' option is the name of the fortran namelist file you edited above.

oifs_run -e oifs -l mynamelist

This will create a file 'job1'.

Submit this job:

qsub job1

Check the job status:

qstatu

These forecasts complete in about 15mins.

During the course of the model run, files with names like ICMGGoifs+00000, ICMGGoifs+00012 etc will appear in the experiment directory.

These are the model output files. There is 1 per output interval. In these experiments the output interval is every 3 hours.

Checking the model output

After the qstatu command shows that the job has completed, there will be a 'batch log file' in the pf001 directory:

oifs_troifs1.o1481792

Exercise

Using your favourite text editor, check this file to make sure the model has run correctly.  It's usually best to start from the bottom of the file and scroll up.

If you see:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
   OPENIFS JOB FAILED
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

the model run has failed.

The output from each model run goes into a directory named 'output1'.

If the model fails, there are 2 files to look at in the 'output' directory:

NODE.001_01 : this is the output from the model as it is running (i.e. all the print/write fortran statements). This is a large file, so in case of errors, start from the bottom up!

oifs.log : this is where the model writes any error messages.

Preparing files for Metview

The next step is to take the output of the successful run and prepare it for the Metview based exercises.

As the data files and Metview scripts have been optimized by selecting specific areas, we have made a script available to do the work.

 oifs_to_mv -e pf001

This will generate files in scratch/metview.

This task is only to prepare the output fields for Metview. The tutor will then use these files and combine them with the other ensemble members to create the final files that will be used in the plotting exercises.

If errors are encountered please let us know. If the command completes successfully (it will take a while), this task is complete. There is no need to inspect the files.