FAQ: Frequently Asked Questions

A. All OpenIFS users will be given full access to this website. Access to other ECMWF systems may not always be required and your needs should be discussed with the OpenIFS team or User Support. Member state users should always apply for an account via their national Met Service representative. If you want to contact ECMWF prior to applying to your national Met Service then please see this list for the appropriate ECMWF User Support contact to ask: http://www.ecmwf.int/en/computing/get-support/user-support-contact-points

If you are a user from a non ECMWF member state you should contact openifs-support@ecmwf.int to discuss your needs.

A. Information about the changes between IFS cycles can be found on the main website 'Evolution of the ECMWF analysis and forecast system'. The ECMWF newsletter publish details of changes to the operational system.

A. This is usually caused by a firewall at the user site blocking ftp connections. There are two types of ftp connections, so-called 'active' and 'passive'. The essential difference is that 'active' mode tries to open multiple connections to the ECMWF ftp server. This behaviour is undesirable for sites running firewalls and is usually blocked. Although passive mode ftp is usually the default, if you see this error try using the '-p' option 'ftp -p' or, if it's available the command 'pftp' which invokes a passive mode only ftp.

A. If using the 'lftp' command, this supports ftp with and without SSL (secure connection) enabled. By default, it tries to use with SSL enabled.  There are several ways around this:

1. Edit the lftp config file to permanently disable this option for all ftp connections. Use a text editor and add the following to either in either the ~/.lftprc or ~/.lftp/rc file in your home directory:

set ftp:ssl-allow false

2. To disable SSL for a connection on the command line:

% lftp -e "set ftp:ssl-allow off;" ftp.ecmwf.int

This error occurs because some modules are missing from the perl installation on your computer. We use the FCM software to compile OpenIFS which uses Perl.  Please see the Prerequisites page for more information about this and how to resolve the error.

 A. When compiling this error message appears:

[FAIL] make oifs           # 94.3s
[FAIL] make                # 94.8s
[FAIL] master.exe: target not found after an update:
[FAIL] /nfs01/n02/n02/emgdc/git/openifs/make/noopt/oifs/bin/master.exe: expect target file

Although make finished, the executable file 'master.exe' was not created. This means an error occurred at the final 'linking' stage when all the compiled object files are combined to make the final executable.

To see what actually caused the problem, look in the logfile generated by the fcm make command. It will be file make/$OIFS_BUILD/.fcm-make/log. Note the '.' in front of the .fcm-make directory name. This directory is 'hidden'. e.g. If you have: OIFS_BUILD=noopt, then the log file from the build will be in make/noopt/.fcm-make/log.

If you look in the logfile, go down to the bottom and look for any errors such as 'undefined reference' which prevented the final link step. A undefined reference means a subroutine/function could not be found either because a library is not included or some code files are missing.

A. This is because the FCM configuration files assume the use of a 'compiler wrapper' which is typically provided by MPI (e.g. OpenMPI or MPICH) installations. Make sure you have a flavour of MPI installed and you have the 'bin' directory included on your PATH environment variable. For example, on linux use 'apt-get install libopenmpi-dev' to get OpenMPI. HPC machines may require you to use a 'module' command e.g 'module load openmpi': check your local help information.

A. FCM has found an include statement: "#include cpg1c.intfb.h" in the source code but can't find the source file cpg1c.F90. Make sure the file exists or FCM is configured to look at all the sources.

A. FCM performs a dependency check on all the code to make sure the compilation is correct. For external libraries, such as ecCodes or MPI, we have to tell FCM they are external and not part of the OpenIFS source code. To do this, edit the file make/cfg/oifs-depend.cfg and edit these lines near the top:

build.prop{no-dep.f.module,class} = eccodes ifcore
build.prop{no-dep.include,class} = mpif.h

These lines let FCM know that the .mod files for eccodes, ifcore (Intel debugging) and .h file for MPI should be treated as external libraries and there are no compilation dependencies on them.

To add your own, taking mod_oasis.mod as an example, add it to the list:

build.prop{no-dep.f.module,class} = eccodes ifcore mod_oasis
build.prop{no-dep.include,class} = mpif.h

A. FCM queries the TMPDIR environment variable. If set it uses that location, otherwise it defaults to /tmp. On HPC systems, /tmp may not be the best choice if it is in memory.

A. The $HERE variable is a special variable that FCM interprets as the directory in which the configuration file resides. Unlike most of the other variables in the FCM configuration files, $HERE cannot be overridden by an environment variable of the same name.

A. For more information on how to correctly set your environment for compiling OpenIFS, see: 5.5 Building OpenIFS in the OpenIFS 43r3 User Guide.

It varies depending on your MPI implementation and your local environment. As an example, for OpenMPI, set the environment variables:
export OMPI_FC=pgf90
export OMPI_CC=pgcc
to use the Portland compilers instead of the default GNU (gfortran) compilers. Check the 'man page' of the mpif90 command to see what other options are possible. Some implementations allow you to use an option -fc to set the underlying compilers.

For Intel installations there are several ways in which the compiler can be set. Normally, the mpif90 command will invoke the gfortran compiler, the mpiifort will invoke the ifort compiler and the mpifc command can be used with a specified compiler on the command line. However, this behaviour can be changed by setting environment variables which these wrapper scripts will read. For more details consult the Intel compiler documentation.

Some HPC centres provide the 'module' command for loading and unloading particular compiler suites. Consult your local documentation for more details.

A. This can arise with the Intel compiler. The MKL library is Intel's version of the LAPACK and BLAS libraries which should be supplied with the compiler. The .so file extension means this is a shared library found at runtime. In order to find this library, the environment variable LD_LIBRARY_PATH must be set correctly. You will need to find the location of this library and add the directory to the LD_LIBRARY_PATH. Normally this is done for you by a small script supplied with the Intel compiler script. Please check your support information.

A. These routines come from the LAPACK and BLAS libraries. You may have one or other of these libraries missing from your .cfg config (or OIFS_LAPACK_LIB environment variable). Alternately you have specific both libraries but the compiler can't find them; check the pathnames to the libraries specified on the compile link carefully.

A. These are references to the JASPER library and arise because the eccodes library referred to in the OpenIFS build configuration was compiled with the Jasper library, but the LD_LIBRARY_PATH environment variable does not list the directory containing the library so the compiler can't find it. Either amend the LD_LIBRARY_PATH or add the -ljasper library to your compilation options.

A. This error is most likely because the version of grib-api is too old. For example, OpenIFS 40R1 requires a minimum version of grib-api version 1.11.0. Please upgrade your version of grib-api, to the latest grib-api release or use the ecCodes library which has now replaced the grib-api library from ECMWF.

A. It's recommended that the same compiler suite is used to compile OpenIFS, your MPI installation and LAPACK. This may not always be practical but it's strongly recommended as some compilers use a different default size of LOGICAL variables which can cause crashes due to memory alignment problems in subroutine calls.

A. The log output from compiling OpenIFS can be found in the 'make' directory, then in the build directories 'opt', 'noopt' etc. The FCM software keeps the log in a hidden directory called '.fcm-make'. So if OIFS_BUILD=opt was set, the logfile for this compilation would be : make/opt/.fcm-make/log.

A. The 'CY' refers to the model 'cycle'. The 'R' refers to the IFS model release. The 'v01' refers solely to OpenIFS and is the version of OpenIFS based on that IFS cycle and release.

ECMWF and Meteo-France jointly develop the IFS model. A new cycle is formed when the two centres merge their code changes, typically about once a year. A model release is specific to ECMWF (the 'R' actually represents 'Reading,UK') and there are usually at most 2 or 3 releases of the model before a new cycle is produced from the next merge with Meteo-France.

The version number refers just to OpenIFS. As OpenIFS model versions have a much longer lifetime than IFS versions, OpenIFS is periodically updated with: technical changes for new compiler versions, new hardware, bugfixes from more recent IFS releases, new functionality and so on.

Changes to the version number will not alter the model's results. Only changes to the cycle and release number will alter the scientific results from the model.

A. The 'T' refers to the spectral truncation of the model and the number following is the highest retained wavenumber. A spectral method, based on spherical harmonic expansion is used in the model, for example to compute some horizontal derivatives. The 'T' means the model uses a 'triangular' truncation in spectral space to decide the retained spherical harmonics. The number following the 'T' is the highest retained wave number. For more information on the spectral truncation and resolution of the model and the model grid, the IFS documentation is recommended reading.

T159 is approximately 125km resolution, T255 - 80km, T511 - 40km, T799 - 25km, T1279 - 16km.

A. This depends on the grid used. For medium and high resolutions, the model uses a 'reduced Gaussian grid' with reduced numbers of gridpoints along a latitude circle towards the poles. Full quadratic and linear grids are also possible. The following table shows the correspondence between spectral resolution and lat/lon resolution. 'TL' means linear grid with a triangular spectral truncation. 'T' means a quadratic grid with triangular spectral truncation. 'N' is the number of latitudes between equation and poles. The grid definitions can also be found here: reduced gaussian grid definitions.

A. The OpenIFS model is based on the IFS code though it is only capable of forecasts, no data assimilation is provided. The first OpenIFS version was based on cycle 38 release 1 (known as CY38R1). More information is found on the page comparing OpenIFS .v. IFS. OpenIFS is released less frequently than the IFS so model versions are not always consecutive.

Information on scientific and technical changes for operational cycles is available on the main ECMWF website.

A. The model writes its output in GRIB format using WMO FM-92 GRIB edition 1 and edition 2 messages. ECMWF provides the ecCodes software library for handling and manipulating grib data using either command line tools and a programming library for C, Fortran and Python.

For plotting GRIB data, ECMWF provides the Metview software, which is freely available under the Apache license.

Note that the OpenIFS model outputs a mix of GRIB edition 1 & 2 messages in a single file. Unlike Metview, not all GRIB software available on the Internet can read such files and extra steps may be necessary to separate the messages before use if you want to use 3rd party software.

A. The model can be run in 'reference' mode where it computes statistical numerical norms and compares against values calculated at ECMWF. Please see 'Testing the Installation' for more details.

A. The timestep depends on the horizontal and vertical resolution, the type of grid (full, linear, cubic reduced Gaussian), whether Eulerian or semi-Lagrangian (SL) transport is used and whether the 2 or 3 time-level SL scheme is used. The recommended default is the 2 time-level SL scheme. For this configuration, timesteps that give a good combination of run-time and forecast are: T21 - T159 : 60mins;  T255: 45mins;  T399: 20mins;  T511: 15mins;  T1279 10mins.

If the 2 time-level scheme is not used, these timesteps should be halved. If Eulerian, not semi-Lagrangian advection is used, these timesteps should be a quarter. For more details, see the code in the routine sudyn.F90 which has approximate values for the timestep (n.b. the NAMDYN namelist variable TSTEP is in units of secs).

Note the model will calculate the optimum timestep for its particular configuration. This value is written to the NODE_001.01 output file. Look for the line:
    PTSTEP_OPT (optimal timestep computed in SUTSTEP) 0.36000000E+04
to see a suggested value in seconds. However, please use these values as a rough guide only. This code has been removed in later versions of IFS.

With longer timesteps, it is possible to get errors in the NODE.001.01 log file regarding trajectories going underground. Whilst some warnings may appear with the recommended timestep values above, the timestep should be reduced with excessive numbers of warnings.

A. The file 'MCICA' is a climatological input file that the model reads on the first time step. The model expects to read this file and all the climatological files from a sub-directory called 'ifsdata' in the same directory containing the binary executable (e.g. master.exe). See Downloading OpenIFS for more details on how to download and install the data files. It is recommended that the data files are kept separate from the directories where model runs are done and a link to the climatology data directory is made. e.g. if the ifsdata tar files have been unpacked in a directory $HOME/ecmwf/ifsdata then use the command ln -s $HOME/ecmwf/ifsdata/climatology ifsdata in the run directory to make sure the MCICA file is available. Contact OpenIFS support for further help.

A. This usually indicates an error in the GRIB file or GRIB_API/ecCodes configuration rather than the model. Try using the 'grib_ls' or 'grib_dump' commands on the file causing the problem to try to reproduce the error. See the GRIB FAQ on this page, or email openifs-support@ecmwf.int and include the model output file (NODE_001.01) and the job log.

A. This error means the model can't find the GRIB templates it needs to write out the model results. These are contained in a directory called 'ifs_samples' which is provided with the ecCodes/grib_api library. First check that the directory pointed to by OIFS_GRIB_DIR exists and is readable. Then check if the directory $OIFS_GRIB_DIR/share/[grib_api|eccodes]/ifs_samples exists and is readable. If it does then it's possible that the library may have been moved (or a link to it changed) and because these paths are hardcoded into the ecCodes/grib_api library at compile time, it means the model can't find it. To overcome this, set the environment variable:

GRIB_SAMPLES_PATH - Set to the folder containing the set of samples you want eccodes/grib_api to use instead of the default one.

e.g.

export GRIB_SAMPLES_PATH=$HOME/ecmwf/eccodes/share/grib_api/ifs_samples/grib1_mlgrib2

A. This could be for a number of reasons. However, a common one is that when compiled optimized the model requires more memory.  With the optimized settings OpenMP is enabled. This causes the model to require more what's known as 'stack memory'. To increase your stack memory to the maximum allowed on your system, use the command 'ulimit -s unlimited'. If that does not solve your problem, email openifs-support@ecmwf.int.

A. This normally means that the OpenMP stacksize is not large enough. The default per process stack size limit is too low and needs to be increased. Check the value of the OMP_STACKSIZE environment variable. It might also mean the model was not compiled correctly. RPASSF is a fortran77 routine and must be compiled with auto-double options to promote REAL*4 to REAL*8 to match the rest of the model. This is set by default in the compiler configuration files but check this is compiled correctly. Contact openifs-support@ecmwf.int for further help.

A. This means that the number of MPI processes (NPROC) specified in the job control language (PBS, SLURM, etc) in the job script and the value in the fort.4 namelist file are different. They must be the same.Contact openifs-support@ecmwf.int for further help.

A. This could be for a number of reasons.

One possibility is that the buffer space for the MPI communication is not big enough. If you get a NODE.01_001 output file (the model log), look for the line containing "MBX_SIZE" e.g.

 MP_TYPE=          2 MBX_SIZE=  128000000

If your value of MBX_SIZE is less than this then change it in the fort.4 namelist file:

&NAMPAR0
 MP_TYPE=2,
 MBX_SIZE=128000000,

Another possibility is that the model is configured as if running at ECMWF where model I/O is via a different method to a special database. Check that the NODE_001.01 output file and make sure that in namelist NAMCT0, the variable LFDBOP is .FALSE. and in namelist NAMPAR1, the variable NOUTTYPE=1.If not, change these in the fort.4  namelist file.

If this does not solve the problem, please contact openifs-support@ecmwf.int.

See HowTo on changing the model output or contact openifs-support@ecmwf.int.

Coding standards for IFS (and it's sister models) are provided by MeteoFrance here: http://www.umr-cnrm.fr/gmapdoc//spip.php?article213&lang=en

A. The 'boot.def' file is the GRIB definitions file. The location of the boot.def file is hardcoded into the ecCodes/grib-api library when it is compiled. Make sure that the directory where ecCodes/grib-api was installed has not moved. If it has, this hardcoded location will be wrong. You can override this location by using the environment variable: GRIB_DEFINITION_PATH and set this to the directory containing the boot.def file. e.g.
export GRIB_DEFINITION_PATH=/usr/local/apps/eccodes/1.12.0/share/definitions

This error can arise when eccodes/grib-api is compiled in HPC environments where it's compiled on the front-end but the batch system has different mount points for the frontend directories causing the path to the eccodes/grib-api installation to be different.

A. This usually indicates an error in the GRIB file or ecCodes/grib-api configuration rather than the model. Try using the 'grib_ls' or 'grib_dump' commands on the file causing the problem to try to reproduce the error. See the GRIB FAQ on this page, or email openifs-support@ecmwf.int and include the model output file (NODE_001.01) and the job log.

A. When you run the model (or using the grib_ls/grib_dump tools on the GRIB file), you see this error:

ECCODES ERROR   :  Wrong size for bitmap it contains 1373624 values
ECCODES ERROR   :  unable to get bitmap as double array (Passed array is too small)

This occurs because sometimes GRIB files are incorrectly encoded for the property 'longitudeOfLastGridPoint'. This is known to happen for the 'stl1' field (surface temperature layer 1). grib_dump shows this:

  longitudeOfLastGridPointInDegrees = 340;

To correct it, use a command similar to:

grib_set -w shortName=stl1 -s longitudeOfLastGridPoint=359824 ICMCLabcdINIT ICMCLabcdINIT.fixed

where 'stl1' is the field to be corrected, 359824 is the last longitude point of (359.824degrees) of (in this case) the correct resolution, and ICMCLabcdINIT is the file to be corrected.

Email openifs-support@ecmwf.int for additional assistance.

A. The environment variables for ecCodes are described on the ecCodes support page.

 A. Try:

export ECCODES_DEBUG=-1

This enables brief debug-level logging messages to be displayed by the library.

If set to 1, you will get very verbose output.

A. If ecCodes installed correctly, the command eccodes_info will print out the location of the sample files and definitions. If you get 'eccodes_info not found' make sure you have added the eccodes installation bin directory to your PATH environment variable.

A. Please see HowTo convert convert model output to netCDF.

A. If you see this type of error:

% grib_ls ICMGGepc8+000004
ICMGGepc8+000004
edition  centre typeOfLevel level  dataDate  stepRange dataType  shortName packingType

ECCODES ERROR   :  unable to represent the step in h
1       ecmf    surface     0      20041015 Decoding invalid stepRange

it is most likely because the model step isn't expressible as an integer number of hours (for example if the model output is every timestep with a timestep less than 1hr). Change the units to either minutes or seconds:

 grib_ls -s stepUnits=m ICMGGepc8+000004
 grib_ls -s stepUnits=s ICMGGepc8+000004

A. This error means the model can't find the GRIB templates it needs to write out the model results. These are contained in a directory called 'ifs_samples' which is provided with the eccodes library. First check that the directory pointed to by OIFS_GRIB_DIR exists and is readable. Then check if the directory $OIFS_GRIB_DIR/share/eccodes/ifs_samples exists and is readable. If it does then it's possible that the library may have been moved (or a link to it changed) and because these paths are hardcoded into the eccodes library at compile time, it means the model can't find it. To overcome this, set the environment variable:

GRIB_SAMPLES_PATH - Set to the folder containing the set of samples you want eccodes to use instead of the default one.

e.g.

export GRIB_SAMPLES_PATH=$HOME/ecmwf/eccodes/share/eccodes/ifs_samples/grib1_mlgrib2

A. This is a known issue with compiling on Mac OS X. See this page on the GRIB_API Installation FAQ for the workaround.

A. On Mac OS X, the MKL library is a dynamic (shared) library which is loaded at runtime and not when the model is compiled. To ensure the library is found you need to add the directory containing the library to your dynamic library path environment variable (DYLD_LIBRARY_PATH). The Intel compiler supplies a script that correctly sets the compile environment for you which should be run. This is normally found in the Intel compiler install directory: /opt/intel/Compiler/11.0/054/Frameworks/mkl/tools/environment/mklvars.sh.