It also describes different methods to interpolate the reduced Gaussian grid to regular grids.
What are GRIB files
GRIB files are the file format used at ECMWF. GRIB is a WMO standard and consists of GRIB Edition 1 and Edition 2.
The OpenIFS/IFS models output in GRIB format. These files are a mix of GRIB-1 & GRIB-2 messages, the multi-level fields are encoded as GRIB-2, whereas surface fields are GRIB-1. GRIB files can also contain multiple vertical coordinates: pressure levels, model levels, sub-surface levels etc. This can cause a problem with some 3rd party tools, as the same GRIB variable code is used for each axis. The instructions below show how to work around this by splitting the file to separate fields on different vertical axes.
OpenIFS model output
The OpenIFS model outputs two types of files: those beginning with ICMSH contain fields represented as spherical harmonics, those that begin ICMGG contain gridpoint fields. The ICMSH files are the spherical harmonics of the wind fields, pressure and temperature and require a spectral transform to convert to gridded data. Also see controlling the OpenIFS output for more details of output fields and options.
Retrieving data from MARS archive
Please note that if using the MARS archive (apps.ecmwf.int) (e.g. for reanalysis products such as ERA-Interim/ERA-5), it is possible to download the files in netCDF format as well as GRIB.
ecCodes tools
grib_to_netcdf
This command will convert one or more GRIB files to netCDF and is available with the ECMWF ecCodes software. For more details please see: grib_to_netcdf description.
grib_to_netcdf
only works correctly when a single level coordinate is present in the GRIB file. Often the model output files have fields on multiple level types (ie. hybrid model levels and pressure levels).
grib_copy ICMGGftkm+001440 ICMGG_[typeOfLevel].grb
In this example, the GRIB model output file ICMGGftkm+001440
contains a number of different model level types. The special square bracket "[ ]" syntax is recognised by grib_copy
(and other grib commands such as grib_filter)
and can contain any valid GRIB key.
This example will copy the original file, separating the level types into their own file: ICMGG_hybrid.grb
, ICMGG_isobaricInhPA.grb
, ICMGG_surface
...and so on.
grib_to_netcdf
can then be used on the individual files:
grib_to_netcdf ICMGG_hybrid.grb -o ICMGG_hybrid.nc
By default, grib_to_netcdf will pack the data into scaled integers with an offset to optimize space.
If you prefer data stored as floats then use:
grib_to_netcdf -D NC_FLOAT ICMGG_hybrid.grb -o ICMGG_hybrid.nc
To convert from vorticity and divergence to wind u and v, please see CDO instructions below.
Note that grib_to_netcdf
does not do any regridding. If the fields use a reduced Gaussian latitude grid, they will not be converted to a regular grid. Use CDO instead to do this as described below.
Please see 'Using Metview with OpenIFS' for more details.
NCAR command language (NCL)
NCL provides a tool to convert both GRIB-1 & GRIB-2 to netCDF called ncl_convert2nc.
NCL example scripts to convert to netCDF are also available.
CDO: Climate Data Operators
These instructions assume the use of a recent version of Climate Data Operators (CDO) (available from Max-Planck-Institut, Germany).
CDO supports GRIB-2 but needs either the ecCodes library from ECMWF to be included. This is necessary in order to work correctly with OpenIFS model output.
cdo --version Climate Data Operators version 1.9.6 (http://mpimet.mpg.de/cdo) ........... Libraries: HDF5/1.10.4 proj/5.2 xml2/2.9.4 Filetypes: srv ext ieg grb1 grb2 nc1 nc2 nc4 nc4c nc5 CDI library version : 1.9.6 cgribex library version : 1.9.2 ecCodes library version : 2.12.5 NetCDF library version : 4.6.2 of Feb 14 2019 10:13:27 $ hdf5 library version : 1.10.4 threadsafe exse library version : 1.4.1 FILE library version : 1.8.3 .........
If you find the CDO commands below do not work, you can either build CDO yourself, making sure a recent version of ecCodes is used, or you can use the workaround below.
Interpolate from reduced to regular grid
CDO can be used to interpolate from the reduced Gaussian grid to a regular Gaussian grid:
cdo -R copy <input grib> <output grib>
The -R
option only works for GRIB edition 1 (GRIB 1) data as it uses the CGRIBEX decoder.
For GRIB-2 messages it's better to use the setgridtype
operator:
cdo setgridtype,regular <input grib> <output grib> cdo -f nc setgridtype,regular <input grib> <output netcdf>
If you have files with a mix of GRIB-1 and GRIB-2, then either split the file first or compile cdo with "--disable-cgribex --with-eccodes=yes"
If variable names are lost, add the "-t ecmwf" option.
If these steps do not work, see workarounds below.
Steps to convert GRIB to netCDF
Split z axis
Before converting to netCDF, separate the different vertical axes using the generic command: cdo splitzaxis <input file> <output file pattern>.
This will ensure the following steps work correctly:
cdo splitzaxis ICMSHg4a4+000000.grb ICMSHg4a4+000000_split ls ICMSH*split* ICMSHg4a4+000000_split01.grb ICMSHg4a4+000000_split02.grb ICMSHg4a4+000000_split03.grb ICMSHg4a4+000000_split04.grb
In this example, the different files might contain: '01' - fields on pressure levels, '02' - fields on model levels, '03' - hybrid levels, '04' - surface and so on. The number of files created depends on the number of different types of levels. Use the command grib_ls
to inspect the contents of each split file.
Note that the filename ICMSH
in this example, indicates it contains spectral and not gridpoint fields.
Another way to split the file would be using the grib_copy command from the ecCodes software as shown above:
grib_copy ICMSHg4a4+000000.grb ICMSHg4a4+000000_[typeOfLevel].grb
In this case 'typeOfLevel' is a GRIB key. The square brackets is a special syntax to the grib_copy
command. This approach works with any GRIB key.
Convert spectral to gridpoint
CDO supports two options for converting spectral to gridpoint data:
- sp2gpl - converts spectral to a linear Gaussian grid, appropriate for IFS data.
- sp2gp - converts spectral to a quadratic Gaussian grid (only use this for T21/42 resolutions)
Both produce a regular Gaussian grid with equal number of longitudes on each latitude row. Remember that a Gaussian grid has irregularly spaced latitudes (but nearly regular). For more details please see the description of these operators on the CDO homepage.
cdo -f nc -sp2gpl <input spectral coefficent grib> <output gridded netcdf>
Convert the resulting output files to netcdf
After any operation converting a GRIB file to any other GRIB file, the conversion to netCDF is just:
cdo -f nc copy <input grib> <output netcdf>
Convert vorticity and divergence to wind
To convert wind components u & v instead of vorticity and divergence, use:
cdo dv2uvl <input file> <output file>
Interpolate to pressure from model levels
To interpolate to pressure levels from model levels, use:
cdo ml2pl,92500,85000,50000,20000 <input file> <output file>
Viewing contents of GRIB files
There are various commands for inspecting the contents of a GRIB file. The ecCodes software installation required for OpenIFS has useful commands like grib_ls
and grib_dump
.
grib_ls ICMSHfrq2+000000.grb grib_dump ICMSHfrq2+000000.grb
Here is another example using the cdo command:
$ cdo sinfo ICMSHfrq2+000000.grb File format: GRIB -1 : Institut Source Param Time Typ Grid Size Num Levels Num 1 : ECMWF unknown 11.3 var P16 65792 1 60 1 2 : ECMWF unknown 39.3 var P16 65792 1 60 1 3 : ECMWF unknown 43.3 var P16 65792 1 60 1 4 : ECMWF unknown 44.3 var P16 65792 1 60 1 5 : ECMWF unknown 152.128 var P16 65792 1 1 2 6 : ECMWF unknown 6.3 var P16 65792 1 1 2 Horizontal grids : 1 : spectral > size : dim = 65792 truncation = 255 spc = 32896 complexPacking = 1 Vertical grids : 1 : hybrid level : 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 2 : hybrid level : 1 Time axis : 1 step RefTime = 1999-12-24 12:00:00 Units = hours Calendar = PROLEPTIC YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss YYYY-MM-DD hh:mm:ss 1999-12-24 12:00:00 cdo sinfo: Processed 6 variables over 1 timestep. ( 0.04s )
Examples
Converting a single file
In this example a single GRIB file with model level data, on a regular Gaussian grid is converted to netCDF with vorticity & divergence replaced by u & v and model level data interpolated to pressure levels.
cdo -f nc copy file.grb file.nc cdo dv2uvl file.nc file_uv.nc cdo sp2gpl file_uv.nc file_uv_gg.nc cdo ml2pl,92500,85000,50000,20000 file_uv_gg.nc file_uv_gg_pl.nc
Converting series of files
The following script shows how to loop over several output files using the steps above to convert to netCDF. Note the spectral parameter files are also converted but the extra command using the sp2gpl operator is first used to convert the variables from spectral to gridpoint on a regular, linear, Gaussian grid.
In this example, we use the pipelining feature of CDO to do both the 'copy' and 'sp2gpl' steps in one command, keeping the entire operations in memory for efficiency.
for i in 000 024 048 072 do cdo -R copy ICMGGfpp4+000${i}.grb ICMGGfpp4+000${i}_R.grb cdo splitzaxis ICMGGfpp4+000${i}_R.grb ICMGGfpp4+000${i} cdo -f nc copy ICMGGfpp4+000${i}01.grb ICMGGfpp4+000${i}_PL_GP.nc # pressure level gridpoint fields cdo -f nc copy ICMGGfpp4+000${i}02.grb ICMGGfpp4+000${i}_ML_GP.nc # model level gridpoint fields cdo splitzaxis ICMSHfpp4+000${i}.grb ICMSHfpp4+000${i} cdo -f nc copy -sp2gpl ICMSHfpp4+000${i}01.grb ICMSHfpp4+000${i}01_PL_GP.grb # spectral fields on press levels transformed to gridpoint cdo -f nc copy -sp2gpl ICMSHfpp4+000${i}03.grb ICMSHfpp4+000${i}03_ML_GP.grb # spectral fields on model levels transformed to gridpoint cdo -f nc copy -sp2gpl ICMSHfpp4+000${i}04.grb ICMSHfpp4+000${i}04_surf_GP.grb # spectral fields on surface levels transformed to gridpoint done
cdo commands can be combined into a single command for greater efficiency.
Using EMOSLIB to interpolate to regular grid
The ECMWF interpolation software library EMOSLIB also provides the capability to interpolate spectral data to regular gaussian grids or regular lat-lon grids, and interpolate regular gaussian grids to regular lat-lon grids.
The EMOSLIB library provides a Fortran library to enable users to write their own interpolation software. Please see the EMOSLIB website for more details and examples.
EMOSLIB also provides two command-line tools for using interpolation. Assuming a recent version of the EMOS library, there are two tools that can be used.
In the 'bin'
directory there is a command 'emos_tool'
which can be used as:
emos_tool --regular=256 [--area="40/-10/31/20"] in.grib out.grib
The grid specified for the --regular
option follows the grid naming convention for EMOSLIB here: Reduced Gaussian Grids.
Another command can be found in the 'tools'
directory. This allows the interpolation function to be specified and is more flexible:
tools/int --INTOUT:gridname=F256 --INTOUT:area="40/-10/31/20" --input=1.grib --output=2.grib --intf2
Here 'F256' means 'full-grid' not reduced, and --intf2
is the EMOSLIB interpolation function to be used.
Possible problems and solutions
cdo -R option does not work with GRIB-2 fields to convert to regular grid
CDO's -R option, to convert from reduced Gaussian grid to regular Gaussian grid, only works with GRIB 1 format as the CGRIBEX decoder this uses does not work with GRIB-2 (see cdo man page). This is a problem as multi-level output from OpenIFS is encoded as GRIB-2 data.
The preferred way to deal with this is to use the setgridtype operator as described above:
cdo -f nc setgridtype,regular <input grib> <output netcdf>
However, if this does not work for any reason, a workaround is to temporarily change the edition number to 1 of the GRIB file. Use the grib_set
command (from ecCodes) to change the GRIB edition number for all messages:
grib_set -s editionNumber=1 in.grb in_1.grb cdo -R copy in_1.grb in_1_R.grb grib_set -s editionNumber=2 in_1_R.grb in_2_R.grb
This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered.
It's recommended that the editionNumber
is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands.
Workaround if using more than 128 levels
The workaround above will fail if the grib file contains field with more than 128 levels. This is because there are not enough bits available in the GRIB-1 parameter to encode the total number of model half levels (which would exceed 255).
In this case, the 'pv' GRIB parameter causes the problem (nothing to do with potential vorticity!). This is an array holding the half level values of the model's A & B coefficients that define the location of the levels.
The workaround above can be extended to delete this array while setting the edition number to 1, like this:
grib_set -s deletePV=1,editionNumber=1 in.grb in_1.grb cdo -R copy in_1.grb in_1_R.grb grib_set -s editionNumber=2 in_1_R.grb in_2_R.grb
The deletePV option is known as a 'concept' rather than a parameter contained in the GRIB file itself. It ensures the PV array is deleted correctly by grib_set. Now the cdo regridding will work correctly, but remember to reset the editionNumber back to 2.
The A & B half level coefficients can be recovered from the original file by:
cdo vct in.grb > pv.txt
Parameter names are lost for GRIB-1 fields
ECMWF GRIB-1 use keys which may not be recognised by CDO because they are locally defined (e.g. the shortName
key) and not defined in the WMO GRIB tables that CDO uses. This can cause parameter names to be lost or not recognised when using the CDO commands.
A workaround is to use the -t ecmwf
option. This will make CDO use ECMWF parameter table 128 for the definitions of variables but 128 is only for GRIB 1. This will work for the surface fields which are encoded by IFS as GRIB 1, but not for the multi-level fields which are encoded in GRIB 2. Variable names become 'unknown' if -t ecmwf is used with the multi-level fields. It is best to omit it completely, even for surface fields, and only use it for GRIB 1 data if you get missing variable names after CDO has converted the data (see CDO documentation for more details).
cdo -t ecmwf -f nc copy mygrib1.grb mygrib1.nc
Do not use this option for GRIB-2 fields. It sets the GRIB table default to be specific to the GRIB-1 ECMWF tables. If problems persist, we recommend using grib_to_netcdf to convert to netCDF.
Acknowledgements
Thanks to Paul Dando of User Support for help with the contents of this page.
19 Comments
NEA MSS
Hi. I tried to use cdo to convert model levels to pressure level but obtained the following error
"No 3D variable with hybrid sigma pressure coordinate found!"
Anyone knows why? Thanks.
Paul Dando
Were you trying to convert ECMWF data that you have downloaded or converted to NetDCF ?
If so, I recommend that you either use the GRIB data directly or else convert the GRIB to NetCDF with cdo. The grib_to_netcdf tool does not preserve the information about the model level coordinate system that cdo needs to convert to pressure levels.
Paul
AKHILESH MISHRA
Hello,
I have a grib file containing 5 pressure level variables, I want to change the unit of Geopotential Heights (Pressure level) by dividing 9.8 (gravity), How can I use cdo/grib_api, keeping ecmwf grib packing as it is?
Any help/suggestion would be highly appreciated.
Thanks
-Akhilesh
Paul Dando
It depends what is meant by "as it is"....
With grib_api you can use the scaleValuesBy key to multiply all values by a constant:
The out.grib will be packed in the same way (i.e., with gridType=grid_simple) but the specifics of the packing will change because the values will change i.e., there will be a different referenceValue and binaryScaleFactor and the packing error will be different. The bitsPerValue is unchanged.
Note also that this changes only the values. The header will still describe the parameter as "geopotential" with units of m^2/s^2. This can be changed by setting the shortName key to gh:
This can be done at the same time as setting the scaleValuesBy.
ECMWF does not provide support for CDO. Please direct questions about use of CDO to the maintainers of this package at MPI Hamburg.
Paul
Marston Ward
Hi,
Conversion of grib files from OpenIFS cycle 40r1 gets corrupted when converting to netcdf using CDO as late as version 1.7.2.
I suspect an out-of-date table, which leads to come variables being renamed to something different during conversion without warning.
A work-around using mostly grib_api and given GG files with both hybrid and surface data as well as SH files with only hybrid data is:
grib_set -s editionNumber=1 ICMGGepc8_all ICMGGepc8_all.grb
cdo -R copy ICMGGepc8_all.grb ICMGGepc8_all_R.grb
cdo splitzaxis ICMGGepc8_all_R.grb ICMGGepc8_all
grib_set -s editionNumber=1 ICMSHepc8_all ICMSHepc8_all.grb
cdo sp2gpl ICMSHepc8_all.grb ICMSHepc8_all_gp.grb
grib_to_netcdf -o ICMSHepc8_all_gp.nc ICMSHepc8_all_gp.grb
grib_to_netcdf -o ICMGGepc8_all01.nc ICMGGepc8_all01.grb
grib_to_netcdf -o ICMGGepc8_all02.nc ICMGGepc8_all02.grb
Glenn Carver
Hi Marston,
The issue is the use of the '-t ecmwf' option in the CDO command as described in the instructions above. This tells CDO to use the GRIB-1 ECMWF parameter table 128. If your GRIB file contains fields encoded in GRIB2, CDO will be unable to find the variable names and you will get unknown or missing names. The best option, as you have shown, is to omit the '-t ecmwf' option. This means CDO will use the WMO GRIB standard tables, which will be fine for ECMWF data encoded using GRIB2. For GRIB1 data this is not the case generally and it may be necessary to use the -t ecmwf option if you find names are missing.
For IFS/OpenIFS model output, the multi-level fields are encoded in GRIB2 and the surface fields in GRIB1. It's best to split the z-axis as you have done and treat them separately. If variable names are lost for the surface fields then -t ecmwf option should be used on the CDO commands.
I have updated the documentation above to make this clearer.
Final note, although -t ecmwf means CDO uses the ECMWF parameter table 128, it will actually use the 128 as the parameterCategory which is completely wrong (at time of writing).
Glenn (with thanks to Paul Dando)
Marston Ward
Hi,
There is an important update to this thread.
1.) The main problem CDO has with OpenIFS grib files lies in the CGRIBEX engine to decode grib files. CGRIBEX cannot handel mixed grib types, its grib tables need to be updated regularly by the CDO folks (per request) and OpenIFS grib tables are currently not updated.
2.) CDO can use grib_api/eccodes (if one disables cgribex) to decode OpenIFS grib files without error BUT there is a major problem. Using CDO, using only grib_api/eccodes grib engine, will result in netcdf files defined on a reduced gaussian grid. This grid is difficult to view and transform using common tools such as NCL, Panoply, ncview, etc. One could get the idea of using a CDO compiled with cgribex to transform this new netcdf but this does not work. The transformation only works on grib files. It would help if eccodes also provides a transformation routine or this. I had a talk to the eccodes guys who stated that they will only concentrate on providing a netcdf file defined on a reduced gaussian grid. I'm a bit confused by this stance since no "standard" tool can read/process such a grid. Would it be such a difficult task to put a function in eccodes to define the data on a regular lat lon grid instead?
/M
Glenn Carver
Hi Marston,
I'm going to move this discussion to the OpenIFS Forums as your post there raises similar issues, otherwise this page will become very long.
See: Grib to Netcdf conversion
Glenn
Christophe Lavaysse
When converting grib to netcdf files with 10 members and 4 time steps, CDO creates 40 time steps by concatenating all the members. Any way to avoid that ?
Thank you in advance
Chris
Glenn Carver
I think this is really a question for the CDO forums rather than here. I don't know CDO that well to know if it has operators for working with ensemble data. I found a reference to the same problem on the CDO forums: https://code.mpimet.mpg.de/boards/1/topics/625
My suggestion would be to split the GRIB file first into separate files, 1 per ensemble. The ensemble member is encoded in the GRIB file.
Glenn
Shane Elipot
Any advice on using ecCode grib_to_netcdf to convert to netcdf the GRIB1 files of ERA5 wave model 2D spectra? Each message of the GRIB file is for 1 time step, 1 spectral frequency (out of 30), and 1 direction (out of 24), providing 315258 values (I think) for the N320 (HRES) reduced Gaussian grid.
grib_to_netcdf returns the error
ECCODES ERROR : First GRIB is not on a regular lat/lon grid or on a regular Gaussian grid. Exiting.
Glenn Carver
I've not tried with ERA5 but it's generally better to directly request netcdf output rather than grib on the download request. Add the keyword:
to your request (assuming you're using the python web api). grib_to_netcdf is very limited in what it can do. I normally use the CDO commands for regridding.
More information about wave spectra and ERA5 downloads is here :
About ERA wave spectra
which has some examples of downloading wave spectra to a gridded netcdf file. HTH.
Shane Elipot
Hello, Thank you for your reply. I have tried but the CDS (Copernicus) server also tries to use the grib to netcdf ecCodes command and then fails.
Glenn Carver
Hi Shane. I'm not very familiar with wave model data. I think best you contact Copernicus support directly and ask them to help. Their support page is here: https://climate.copernicus.eu/help-support .
Shane Elipot
Thank you, I have already asked for help but no solution has been found yet. The issue may be that the messages for 2D spectra GRIB files are organized as time steps, direction (24), frequency (30), then only lat/lon.
Glenn Carver
I'd suggest using the eccodes software library, with your favourite programming language (python,fortran,c) and write your own program to read the GRIB file, manipulate the data however you need, then write it out as netCDF however you like? There are examples codes available on the eccodes website: API examples
Jean Bidlot
Issue with interpolating wave data to regular lat-lon grid:
Because cdo or grib_to_netcdf do not recognise the irregular lat-lon grid used by the wave model then
if the user has also metview installed, then using the following macro will do the interpolation for you
### write the following lines until into a file ### end of macro
#Metview Macro to interpolate an input file called input
# to a regular lat-lon grid over a specific area (see below)
# and write the result to file output
## To excecute with metview batch: metview -b this_file
## specify target grid spacings:
gridx=0.5
gridy=gridx
# north, west, south and east boundaries of the target area:
north=82
west=0
south=-78
east=360
# macro:
read_interpolate = read(
source : "input",
area : [north,west,south,east],
grid : [gridx,gridy]
)
# write result to a file
write ("output", read_interpolate)
### end of macro
then
metview -b your_file
Note: use the following to get the latitudinal grid spacing of the input that can be used in specifying gridx above
grib_get -p jDirectionIncrementInDegrees input
Glenn Carver
Metview can be downloaded from here: Metview Releases or installed as a package on most linux distributions.
Joan Saladich
Hi there,
Thanks for your guide. Yet I have a problem when using CDO to transform from reduced Gaussian to regular lat lon. I have an ERA5 file ("forecast" type) which is stored in grib format and data is distributed on a reduced Gaussian grid .
Now I want to transform the coordinates of this file into a regular Gaussian, therefore I use the following command:
cdo -setgridtype,regular input_era5_reduced output_era5_latlon.grib
So far, it seems the command does the expected job when the main variable is an "analysis" one, such as t2m, u10, etc., when the main variable of the grib file a 'forecasted' one (precipitation, wind gust, etc.), I got unexpected results.
Everytime I use cdo -setgridtype,regular ... for era5 forecasted variables I got changes in the "step" and "initialization time" dimensions. Let me be more specific:
Original file ( input_era5_reduced ):
Post CDO file ( output_era5_latlon.grib ):
I expect the output CDO file to have 743 values in the "step" dimension with an initialization value (suming 744 values, which is 24hours * 31days for the month of May), maybe in a single dimension "step" or maybe as the input file, yet I just get random "steps" suming up to 410 values. What am I doing wrong?
I am using the CDO version 1.9.10 with gcc64.
Thanks a lot for your help!