Introduction

About GRIB

GRIB edition 1 and GRIB edition 2  (GRIdded Binary or rather General Regularly-distributed Information in Binary form) - referred to as GRIB1 and GRIB2 from now on - are WMO (World Meteorological Organization) binary standard formats defined by the WMO Commission for Basic Systems under GRIB FM 92 for the international exchange of meteorological gridded data. Since the implementation of IFS cycle 37r2 on the 18 May 2011, ECMWF has produced its operational data, as well as a variety of reanalysis data, in a mixture of GRIB1 and GRIB2. Both formats are table based binary data, machine independent, and they need specific software for the data processing. The meta-data are encoded in the form of integer values whose corresponding meaning is defined in code tables published by the WMO. For GRIB2, these code tables, as well as templates which describe the layouts of GRIB2 messages, are extended, or new proposed parameters / templates are added on a biennial basis in a fast-track procedure by the WMO Task Team on Table Driven Code Forms (TT-TDCF). GRIB edition 0 was introduced by WMO in 1985, GRIB1 in 1990 and GRIB2 in 2001. Edition 1 was removed from the WMO Manual on Codes, Volume I.2 (WMO-No. 306) in 2016. Since then, GRIB2 is the official WMO format, and there is an initiative for GRIB edition 3 (GRIB3).

Structure of GRIB1 and GRIB2 messages

A GRIB1 message consists of 6 and a GRIB2 message of 9 sections. The following figure illustrates the layout of a GRIB1 and GRIB2 message.

GRIB messages are record based. This means that each message contains a single parameter on a single level, for a single time instance (point in time or processed over a time interval), a single ensemble member, etc, and includes its own complete set of meta-data. GRIB2 offers also the possibility to code multi-field messages, but this feature is not used at ECMWF, as the infrastructure is designed for the usage of single field records, and the saving by using multi-field messages is only small. Multi-field messages are not further discussed. The single messages are concatenated into a sequence of messages. Each message is sub-divided into sections with the structure shown in the figure depending whether it is edition 1 or 2.

Templates are used in GRIB2 to further adapt the layout of the messages for specific products. A template defines the keys and order of bytes. Some of these keys refer to entries in the WMO code tables represented as integer values in the binary file. Section 3 (Grid Definition), section 4 (Product Definition) and section 5 (Data representation) are all template based. This makes GRIB2 much more flexible than GRIB1. Templates are also published by WMO in the same way as the parameter code tables. New parameters and templates can be proposed to WMO on a biennial basis. Meta-data, which are centre specific can be added to the section 2, which is designated for a local use and the only section which does not have to be included in the message. To see how templates change and extend the content of a GRIB2 message, see GRIB edition 2 template example for Product Definition Section.

Advantages of GRIB2 over GRIB1

• GRIB2 is much more meta-data rich than GRIB1 – this allows for a better description of the data content
• Higher precision of some meta-data in GRIB2 – e.g., the accuracy for latitude-longitude coordinates increased from milli- to micro degrees which is essential to be able to encode higher resolved model data
• Higher accuracy of coded values in GRIB2
• More sophisticated and effective compression methods of data values, e.g., CCSDS, are available in GRIB2. This helps to reduce the size of archived data, which will increase significantly with higher model resolutions.  It is also important for the adaptation of the ECMWF ENS forecast resolution to the that of the high-resolution deterministic forecast.
• GRIB2 includes a separate section (Section 2) for encoding centre-specific information. This is used at ECMWF for example to encode keys for the archiving and retrieval in the ECMWF Meteorological Archival and Retrieval System ( MARS ), e.g., the keys mars.class and mars.stream.
• GRIB2 is more template based than GRIB1 making the format more flexible and easier to extend. Extensions can be requested on a biennial basis at the WMO. Various kinds of products are represented by different templates. In GRIB1, information which was not foreseen in the official data format can be encoded by local centre specific unofficial extensions of the file format. In GRIB2, the encoding of local “non-standard keys” can be reduced and, in the case where they are still required, these local keys are added to the dedicated local section 2, whereas in GRIB1 they impact the official data layout. This makes the data exchange more robust with GRIB2 as all sections in the GRIB2 messages (except for the local section 2) are consistent with the official WMO layout and the section 2 can be simply ignored when decoding the message as each section length is encoded at the beginning of the section.
• GRIB2 allows for both the coding of multi-field as well as single-field messages. The design of the MARS archive is optimized for the usage of single field GRIB messages. This has the advantage that a single field can be extracted from a tape without reading large files, messages can be concatenated, in the case where a single message is corrupted, the other messages can still be used, or if the file transfer is interrupted, only the missing part must be sent again.

The mapping of parameters in GRIB1 and GRIB2

A parameter in GRIB1 is specified in the WMO code table 2 by the two keys, table version and indicator of parameter. GRIB2 uses instead the three keys: discipline, parameter category and parameter number. This sub-divides the parameter list in disciplines like for example Meteorology or Oceanography, which are then further divided into different parameter categories like moisture or momentum. A parameter in GRIB1 is in principle fully described by the two keys. In GRIB2 an orthogonal structure of the parameter description is used. GRIB2 parameters are defined by WMO as basic parameters, meaning that there is for example no statistical process or level specified. This specification is then provided by defining additional keys.

An example is the Maximum temperature, which has a specific entry in the WMO GRIB1 code table, whereas in GRIB2 the basic parameter Temperature should be used, extended by the key typeOfStatisticalProcessing=2, which specifies that it is the maximum. In GRIB2, this can be even further specified by adding for example the length of the time window over which the statistical processing was performed.

The WMO codings for GRIB2 parameters are listed in WMO code table 4.2. 

At ECMWF, so-called concepts are used in ecCodes to hide the technical details and to facilitate the usage of GRIB. The concepts contain a set of GRIB keys and define for this specific set among others the high-level edition independent keys paramId, shortName, name and units. They are defined in the parameter database and in the ecCodes definitions. The concepts are defined for the parameters defined in the WMO tables as well as in centre specific versions for example to define parameters with units different to those published by the WMO.
Here's an example of the name concept for the Maximum temperature in GRIB1 and GRIB2:

GRIB1 name concept: 'Maximum temperature' = {tables2version = 3, indicatorOfParameter = 15}
GRIB2 name concept: 'Maximum temperature' = {discipline = 0, parameterCategory = 0, parameterNumber = 0, typeOfStatisticalProcessing = 2}

The following example shows how the WMO and ECMWF paramId concepts in your ecCodes installation look like:

# replace paramId.def with shortName.def, name.def or units.def to see the other concepts

# show paramId concepts for WMO parameters
cat $(codes_info -d)/grib2/paramId.def
# show paramId concepts for ECMWF local parameter definitions
cat $(codes_info -d)/grib2/localConcepts/ecmf/paramId.def

$ codes_export_resource -d grib2/paramId.def paramId.def
$ cat paramId.def

ecCodes reads the concepts from ASCII files. If a certain paramId or shortName is requested within the ecCodes API or by the ecCodes binaries, the software compares the keys in the message with the keys in the concepts. For the mapping it is necessary that for each parameter, which is produced by one of the ECMWF models, a valid set of keys is defined. As it is not possible to map all parameters to the WMO tables, the missing parameters were/are proposed to be added to the WMO tables. Some of the parameters cannot be represented by simply adding code table entries. For example, for the Extreme-Forecast-Index (EFI) parameters, templates have been proposed to allow the reference period used to calculate the M-climate to be encoded.

Changes in ecCodes

Changes in ecCodes for decoding

The user interaction with the GRIB2 files via ecCodes is in principle the same as in GRIB1 by using the paramId or shortName keys to select a message or other edition independent high-level keys like dataDate or level. There are some changes in the specification of parameters which can be classified into the following categories:

• some parameters have a different paramId in GRIB2 than in GRIB1
• a paramId in GRIB2 needs to be complemented by an additional key, e.g., a layer, whereas in GRIB1 a single paramId was sufficient
• a paramId is split into two Id's (paramId + chemId)
• some parameters have a different unit than in the official WMO GRIB2 parameter definitions.
  

Decoders other than ecCodes must be extended by the new parameters and templates proposed to the WMO before they can decode the data. ecCodes must be linked to the libaec library to be able to decode CCSDS compressed GRIB2 fields and a new ecCodes version is needed, which includes the new parameter and template definitions.

Changes in ecCodes for encoding

Besides the mapping of parameters and creating new templates, GRIB2 also impacts workflows for encoding data:

• several additional keys have to be set besides the keys defined in the concepts for encoding 
• for some parameters it is necessary to define the Product Definition Template Number before they can be encoded

There will be changes in some parameters like for example for soil, ice, and snow parameters on layers. Instead of having a paramId for each layer in GRIB1, in GRIB2 there will be only a single paramId for each soil parameter on layers, which can be distinguished by the level information encoded in the message. This means that instead of using a paramId for Volumetric soil water layer 1 and another for Volumetric soil water layer 2, only a single paramId is used for Volumetric soil water, and users have additionally to choose the layer in their workflow.

Another example is a splitting of the paramId’s for chemical parameters into a paramId and a chemId. This allows for more flexibility in adding new or additional chemical species. 

Changes in the MARS language

MARS requests for some parameters will also be affected by the migration to GRIB2.

Changes to param keyword

For GRIB1 parameters, the param keyword in the mars request can be set to param=indicatorOfParameter.table2Version. Requests with this structure will not work for GRIB2 as these two keys do not exist in this edition. The paramId must be used for GRIB2 parameters. This is the recommended way of specifying parameters in MARS requests. An example is the retrieval of the U-component of the 10 metre wind, which is in GRIB1 in the indicatorOfParameter.table2Version representation param=165.128, but in GRIB2 param=165.

retrieve,
  class=od,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=sfc,
  param=165.128,
  target="output_10u.grib1"

retrieve,
  class=od,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=sfc,
  param=165,
  target="output_10u.grib2"

Changes for parameters archived with levtype=sol

Parameters on surface levels will be archived with levtype=sol (surface other levels). 

For example, in GRIB1, a separate param is used to retrieve Volumetric soil water layer 1 and a different param o retrieve Volumetric soil water layer 2, etc with levtype=sfc .  In GRIB2, only a single paramId is used for Volumetric soil water, with the layer being selected by the levelist keyword and using levtype=sol .

Instead of specifying a single parameter Id in the request, for some parameters additional keys will need to be added, for example for snow layer parameters the layer. These parameters will be archived with the new MARS keyword levtype=sol which has to be specified in the request together with the MARS keyword levelist .

Example of a retrieval of snow depth parameters with levtype=sol .

retrieve,
  class=od,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=sfc,
  param=141.128,
  target="output_sd.grib1"

retrieve,
  class=od,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=sol,
  levelist=1/2/3/4/5,
  param=228141,
  target="output_sd.grib2"

Addition of new MARS keyword chem for retrieval of chemical species

Another example are chemical parameters. Besides using the MARS keyword param in the request to specify the parameter type, e.g., mass mixing ratio, an additional MARS keyword chem will be used to specify the chemical substance(s), e.g., nitrogen dioxide.

retrieve,
  class=mc,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=ml,
  levelist=1/to/137,
  param=210121,
  target="output_no2.grib1"

retrieve,
  class=mc,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=ml,
  levelist=1/to/137,
  param=402000,      # Mass mixing ratio
  chem=no2,          # Nitrogen dioxide
  target="output_no2.grib2"

Change of MARS packing for GRIB2 parameters retrieved on a grid

GRIB2 parameters retrieved on a grid will be compressed with CCSDS. To decode these parameter, ecCodes must be linked to libaec. Alternatively, the data can be retrieved in another packing type such as grid_simple packing, which was in use for the model level GRIB2 parameters.

retrieve,
  class=mc,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=ml,
  levelist=1/to/137,
  param=210121,
  target="output_no2.grib1"

retrieve,
  class=mc,
  stream=oper,
  expver=1,
  date=-2,
  time=00:00:00,
  type=fc,
  step=0,
  levtype=ml,
  levelist=1/to/137,
  param=402000,      # Mass mixing ratio
  chem=no2,          # Nitrogen dioxide
  packing=simple,    # Simple packing
  target="output_no2.grib2"

Outlook

The design phase for the next edition 3 of GRIB has already started. The next edition 3 will bring improvements like a finer sub-division of the message into twelve sections and an extended template-based approach giving even more flexibility than is currently available in GRIB2. An example is the representation of parameters processed over the vertical dimension via a separate Vertical Domain Section, which can also be expanded via new templates. Such “total-column” parameters have to be listed in GRIB2 instead as specific parameters in the WMO parameter code table. The migration from GRIB2 to GRIB3 will be much smoother, since GRIB3 is a de facto extension/improvement/revision of the GRIB2 format.  This next step from edition 2 to edition 3 will still take several years. The expected user impact will be considerably smaller than for the transition from edition 1 to 2. The actual migration to GRIB2 is also a good opportunity to learn from user feedback and challenges experienced during the transition to tailor user-oriented proposals to be submitted to WMO as a potential part of the next migration to GRIB3.

Further reading

• Migration from GRIB1 to GRIB2: preparing ECMWF model output for the future - ECMWF Newsletter Number 175 - Spring 2023

• GRIB edition 2 template example for Product Definition Section
• Migration to GRIB2 - new in ecCodes
• Migration to GRIB2 - new WMO parameters