- Created by Glenn Carver, last modified by Marcus Koehler on Feb 05, 2024
This page describes how to extract forcing data to run the Single Column Model from the ECMWF data archive.
This page is out of date and will shortly be taken offline. The description is no longer applicable since the move to the new HPC system in Bologna.
Note that you need:
- an account on the ECMWF 'ecgate' system
- access to the ECMWF HPCF system
granted by your member state Meteorological Service to be able to run the steps below.
Initial setup
There are a number of required steps before the extraction can be done. These are steps that only need to be done once.
Access to 'ifs' group
You need to be in the 'ifs' unix group in order to run the scripts to generate the SCM forcing.
To check you have access, login
to ecgate,
and type:
groups
If you do not see 'ifs'
listed in the output, please contact ECMWF Service Desk (servicedesk@ecmwf.int) and ask to be added, explaining why.
ssh keys
Make sure you have set up your ssh_keys so that you don't need to use password to go from ecgate to cca and from cca to ecgate:
ssh-keygen (no passphrase) ssh-copy-id -i ~/.ssh/id_rsa.pub name_of_remote_host
Install workflow suite
The forcing data is created using a 'workflow' developed at ECMWF.
A workflow is a defined order in which a number of separate scripts are run. The workflow software used at ECMWF is called 'sms'. SMS is a 'workflow manager' that a user interacts with to control the executation of scripts (or 'tasks') in the workflow.
The workflow 'suite', the sequence of the scripts that will be run, is defined in a .def
file.
The following steps explain how to install and customize this workflow and it's scripts for your use.
Download the workflow suite: sms.tar.gz (last updated Jan/2018) and unpack it to your ecgate user home directory:
cd $HOME mkdir sms tar zxvf sms.tar.gz
Choose either ECMWF operational analysis (OPER-cray.def) or the ERA-Interim analyses (ERAI-cray.def) and copy to 'cray.def'.
e.g. for operational analysescp OPER-cray.def cray.def
Edit cray.def to correct it for your user details:
i) replace all'/home/ms/se/usr'
by your actual home directory path where'usr'
is your username (e.g. /home/ms/se/batman)
ii) replace 'acnt' by your billing units account.
To find your account, type the command:account -l your_user_name
where
your_user_name
is replaced by your username (e.g. batman).
iii) replace'usr'
inOWNER
andUSER
to your username.
iv) setLOGDIR
to a directory on the HPC (where this job runs) where you want to store the log files.In the directory, '
sms/cray
', edit the files:sms/cray/getdata_scm.sms
andsms/cray/getmars_scm.sms
to set the unix directory paths to your user account:Change these lines in both files:export SCRATCH=/ws/scratch/rd/rdx # only useful for $SCMDATA definition export SCMDATA=$SCRATCH/pafv/data # will define the place your resulting profiles are stored TMPDIR=/scratch/rd/pafv/scm_cray # working directory on cca
It is important that these directories are set the same in both
getdata_scm.sms
andgetmars_scm.sms.
In the directory
sms/include_cray,
you need copy the script files from either'OPER'
if you want to use operational analyses, or'ERAI-0_12'
if you want to use ERA-Interim.
Copy the files into thesms/include_cray
directory:e.g. for operational analysescd sms/include_cray cd OPER # we will use the ECMWF operational analysis to create the forcing files cp getini1c* .. # copy the files to sms/include_cray
Edit the files:
sms/include_cray/getini1c_setup_cycle_validation_SCM_FC_cray*,
and replace the text'usr'
in"EXPVER=usr"
to your real username.Invoke the
sms
server to find your 'sms identifier', which is the number in the command output after thePROGNUM
parameter, needed to log in to ecgate and set up server./usr/local/apps/sms/current/bin/sms_start
This command starts up your personal workflow manager on ecgate.
Set up server in the xcdp viewer. This is a visual view of the workflow and the tasks as they execute.
xcdp & # run the workflow viewer 'xcdp' in the background
Once the xcdp window appears, add your server to the list of servers known to xcdp by the menus:
Edit → Preferences → Servers
Add entry to
'Name'
(e.g. myserver), add entry to'Host'
(e.g. ecgate) and setPROGNUM
value (step 7) for the'Number'
field.
After configuring the server, select it's name from the Servers menu. You may need to click the red circle arrow top right, to load the suite.
Set up the job to retrieve forcing
Define the starting day(s) of your experiment(s) by adapting the
YMD
variable insms/cray.def
. The current setting uses the starting date of 1st of April 2016:define rcmd { repeat string YMD 20160401 }
If you want to do multiple experiments simply add other dates in the following way:
define rcmd { repeat string YMD 20160401 20160402 20160403 }
Edit following two files:
sms/include_cray/getini1c_setup_cycle_validation_SCM_FC_cray sms/include_cray/getini1c_setup_cycle_validation_SCM_FC_cray.dataonly
between the lines labelled as "definition area" to reflect your desired points.
If using only one time step keep the forward slash '/' afterwards for correct use of sed command.
If you request fields at 1 hour frequency, some fields won't be available from the MARS archive. It is better to use a 3 hour interval for the operational forecast.
Note the start time has to be either 00 or 12 for the operational forecast archive. While multiple dates can be specified, there is no way to have multiple starting time in the similar way.
getini1c_mars_run_validation_SCM_FC
has information on which variables are read from MARS archive
Run the job to retrieve forcing
Make sure the sms server is already running by using the following command. If it is, this command will do nothing:
/usr/local/apps/sms/current/bin/sms_start
Load your suite into the SMS server and start running it:
cd ~/sms # change to the directory with the SMS scripts cdp # start the workflow manager 'cdp' CDP> set SMS_PROG your_sms_identifier # the command prompt will change to 'CDF'; set the SMS_PROG value
where
your_sms_identifier
is given in the the output after invoking the sms server, look for variable calledPROGNUM.
login -t 60 ecgate your_user_name 314159 # password 314159, can also be 1 instead of 314159
If errors occur, likely need to invoke sms server
Run (or 'play') the workflow suite:
CDP> play -r /cray<cray.def # load the workflow suite into the workflow manager CDP> begin /cray<cray.def # start running the suite CDP> ctrl-D # CTRL-D to quit the CDP workflow manager
To cancel a running application:
CDP> cancel -y -f /cray
before entering the
play
command.Start the
xcdp
viewer, a graphical display of the running tasks:xcdp & # run the viewer in the background
In the viewer, xcdp window, expand the branch of jobs on the host by middle mouse click on the symbol to right. Get further information on the job by right mouse button.
Color coding for xcdp tasks
QUEUING | RUNNING | COMPLETED | ABORTED |
More information on errors can be found in either getmars_scm.*
or getmars_scm.*
in folder LOGDIR
on cca (cray) depending on which process crashes.
Restart failed tasks
Before restarting a failed job ('task') check the job is not still running by the command:
qstat -u your_user_name
which will list all batch jobs for your user name.
If the job is still running, kill it by:
qdel process_id
where the process_id
should be replaced by from the list returned by qstat.
To restart a failed task, select the task, right click and select 'Restart'. All scripts in tree below will also be restarted and executables again copied to the Cray (cca) (changes included on ecgate will be activated). However, there is no restart of cray.def file needed. If a date change is required, login to cdp to change the date.
Retrieving data
After all the tasks are completed (shown as yellow in the xcdp window), your forcing data is retrieved successfully and you should find your results in the directory specified in $SCMDATA.
Download the data from the web interface to ecgate, http://ecaccess.ecmwf.int. For instructions see: ECMWF User Docs: File Transfer.
The data can be download from: Browsing → ECscratch files → data → click on symbol to left (page with red arrow) to download.
Adapting extraction to other operational systems
The scripts in this workflow are configured for the most recent operational IFS operational since March 2016 (IFS cycle 43r1).
Changes to the following scripts are required for dates prior to this:
sms/include_cray/getini1c_setup_cycle_validation_SCM_FC_cray sms/include_cray/getini1c_setup_cycle_validation_SCM_FC_cray.dataonly
For example, retrieving forcing data from the period June 2013 – March 2016, the operational model was IFS cycle 40r3. Modify the scripts by setting:
CYCLE=40r3 dataid2="t1279_2010" CLIMVERSION=climate.v011 GTYPE=l_2
and substitute the script:
$TMPDIR/run_parallel /home/rd/nag/bin/getini1c_TCo1279_geostr
by the script (as used for ERA-I):
$TMPDIR/run_parallel /home/rd/nag/bin/getini1c_lowres_geostr
Extracting operational forecasts or analyses further back in time needs different changes to the two include_cray/getini1c_setup_cycle*
scripts.
IFS operational updates
Date | Horizontal resolution | Vertical resolution |
---|---|---|
21 Nov 2000 | Tl511 | L60 |
1 Feb 2006 | Tl799 | L91 |
26 Jan 2010 | Tl1279 | L137 |
8 March 2016 | Tco1279 | L137 |
Changing horizontal resolution:
- change
ZDELTA
with a coarser grid so that the point is found in circle around desired location. - choose large enough but not to large so that point further away is detected.
TSTEP
is not used in getini1c script, not important to set.
Changing vertical resolution:
- change the value of the variable
LEVELS.
- change option
-l 137
ingetini1c_mars_run_validation_SCM_FC
call ininclude_cray/getini1c_setup_cycle_validation_SCM_FC_cray.dataonly
according to new number of levels (default 137).
Extracting forcing from ERA-Interim
Use the scripts in directory include_cray/ERA-I
instead and also ERAI-cray.def.
Data availability | ||
---|---|---|
00 | forecast step | 0, 3, 6, 9, 12 |
06 | analysis step | 0 |
12 | forecast step | 0, 3, 6, 9, 12 |
18 | analysis step | 0 |
Acknowledgements
The following people contributed to the development and documentation: Filip Váña (ECMWF), Nils Wedi (ECMWF), Paul Dando (ECMWF), Kerstin Hartung (University of Stockholm) and Glenn Carver (ECMWF).
1 Comment
Jareth Holt
XCdp does not seem to want to run in the background for me; appending & does not stop it from occupying the terminal. The following works though:
>> xcdp &>/dev/null &
Also, with the "sms-updated" tar file there doesn't appear to be anything that needs editing in sms/include_cray/getini1c_setup_cycle_validation_SCM_FC_cray.dataonly.