Migration to the Product Generation software with MIR was completed successfully during April 2019.
ECMWF thanks all users of data generated by Product Generation for the fruitful cooperation which helped to make the migration processes as smooth as possible and contributed to its successful completion.
Production of the data files with the new Product Generation software is fully supported operationally at ECMWF. Please report any operational issues to servicedesk@ecmwf.int.
Table of contents
Background
Product Generation is the process by which direct model output is converted into user's data files according to their requirements. The interpolation library lies at its core.
The Product Generation software has undergone a complete rewrite to include the new interpolation package MIR and introduce a number of other efficiency improvements.
News
See announcements here
Important dates
- 9 April 2019: Product Generation with EMOSLIB decommissioned.
- 14th of January 2019: Feedback with the results of the testing phase to be sent to Data.Services
- End of November 2018: Product Generation with MIR is introduced into operations in parallel with the present Product Generation with EMOSLIB.
Timeline
Product Generation with MIR will be released by the end of November, giving users the opportunity to migrate as soon as they are ready after the release.
Product Generation with EMOSLIB will run in parallel with Product Generation with MIR for a period of three months. After this period, Product Generation with EMOSLIB will be decommissioned.
How to test
Users that do not have access to ECPDS
Please send an email to Data.Services@ecmwf.int to request the test files to be delivered to your systems.
Users that have access to ECPDS
You will be able to access test data files at:
https://ecpds-xmonitor.ecmwf.int/do/start
Please ensure that firewalls accept internet transfers from the following servers:
193.61.196.104 (http://ecpds-xma.ecmwf.int)
193.61.196.105 (http://ecpds-xmb.ecmwf.int)
193.61.196.124
Login with your ECMWF user ID and the password obtained from the security token.
Test files have an additional extension .pgen so they won't overwrite your operational files. If you download them, they should be delivered to the same directory/host as your operational files.
When the new product generation software goes into production, files will no longer have the ".pgen" suffix. Your current file names will remain.
Please check the configuration of the host and directory configured on the test ECPDS as this may have been configured differently in the past (e.g. if you have requested a different configuration to test previous e-suites).
If you wish to change the receiving host/directory for the test files, please send an email to: Data.Services@ecmwf.int
Differences on the data files
Do not rely on the order of the fields in the file. Files produced by Product Generation can contain fields in any order which can change from day to day.
This section is a compilation of the known differences presented by the output data when generated by the present Product Generation with EMOSLIB (PRODGEN) versus the new Product Generation with MIR (PGEN).
It does not explain the differences between the two software.
Different interpolation library
Different data values
PGEN software uses MIR interpolation library and therefore data values will not be exactly the same. Prodgen uses EMOSLIB which is no longer supported. These differences are similar to, but generally much smaller than, those arising from a change of the forecast model.
See MARS interpolation with MIR for further information about differences between EMOSLIB and MIR. The section Highlights and main differences plots a number of illustrative fields showing the differences in the data values.
Different point count
Specific cases for reduced Gaussian grids present different point count. Any users affected by this have already been informed.
Differences in the parameter Relative Humidity (paramId=157)
The way of generating the Relative Humidity product has changed in PGEN. Prodgen using EMOSLIB was post-processing this product to limit the values between 0 and 100. Now we took the decision not to post-process this parameter any more so the values will be more similar to what the model produces. The values will now be as close as possible to what MARS interpolation with MIR produces.
Note that now you may find values over 100 (supersaturation) and small values <0. See IFS documentation for relative humidity.
Differences for area encoding
Different longitude encoding in GRIB1
PGEN is encoding the west/east longitudes differently in GRIB1.
- prodgen will encode longitude -180 as 180, pgen will encode -180
- prodgen will encode longitude 359 as -1, pgen will encode it as 359
| User input | PRODGEN | PGEN |
area=90/-180/-90/179.5 grid=0.5/0.5 | area=90/180/-90/179.5 | area=90/-180/-90/179.5 |
area=90/0/-90/359 grid=1/1 | area=90/0/-90/-1 | area=90/0/-90/359 |
area=90/-60/25/60 grid=640 gaussian=reduced | area=89.892/300/25.092/60 | area=89.892/-60/25.092/60 |
Different precision
In GRIB1, the precision is millidegrees; in GRIB2, it is microdegrees. PGEN lets ecCodes decide on the best precision. PRODGEN encodes the areas in millidegrees, therefore the encoded values in GRIB2 can be different.
This case will affect mostly Gaussian grids:
| User input | PRODGEN | PGEN |
area=90/-60/25/60 grid=640 gaussian=regular form=GRIB2 | area=89.892/300.093/25.092/59.906 | area=89.8924/300.094/25.0918/59.9062 |
Different rounding
| User input | PRODGEN | PGEN |
area=90/-60/25/60 grid=640 gaussian=regular | area=89.892/-59.907/25.092/59.906 | area=89.892/-59.906/25.092/59.906 |
Different east longitude
When Gaussian grids are requested, PRODGEN sets the east latitude in the GRIB header to the user provided value (modulo 360 and modulo the GRIB1 precision) PGEN will put the longitude of the most eastern point.
| User input | PRODGEN | PGEN |
area=90/0/-90/359.9 grid=128 gaussian=reduced | area=89.463/0/-89.463/359.9 gridname=N128 | area=89.463/0/-89.463/359.297 gridname=N128 |
Constant fields
Fields that may contain constant values are packed by PGEN with bitsPerValue=0. This is a more efficient way to encode such fields and leads to smaller files.
An example of such field is CLWC on model levels high in the atmosphere where the values are zero at all grid points.
As a result, file sizes may change from one day to the next.
PRODGEN encodes these with a bitsPerValue=8 so file sizes are the same from one day to the next.
Local definitions
PRODGEN does not produce files in GRIB1 with local definitions for 00 and 12 UTC HRES fields. They are kept for 06 and 18 BC and ENS (including ENS-extended and SEAS).
The local definitions are (currently) retained by PGEN.
E.g given two files:
H1D09060000091600001.pgen
H1D09060000091600001
Please retain the order of the files when doing the comparison
grib_compare -H H1D09060000091600001.pgen H1D09060000091600001 -- GRIB #1 -- shortName=sd paramId=141 stepRange=240 levelType=sfc level=0 packingType= gridType= -- long [totalLength]: [26534] != [26510] long [section1Length]: [52] != [28] [reservedNeedNotBePresent] not found in 2nd field [localDefinitionNumber] not found in 2nd field [marsClass] not found in 2nd field [marsType] not found in 2nd field [marsStream] not found in 2nd field [experimentVersionNumber] not found in 2nd field [perturbationNumber] not found in 2nd field [numberOfForecastsInEnsemble] not found in 2nd field [padding_local1_1] not found in 2nd field
Duplicated fields
PGEN does not deliver duplicated fields configured in the requirements. e.g. In the request below the parameter 2T will be delivered twice by PRODGEN and only once by PGEN
DIS, TARGET = ECM:EC, STREAM = DA, TYPE = FC, GRID = 0.125/0.125, AREA = 55/2.5/47/15.5, TIME = 00, LEVTYPE = SFC, STEP = 0/to/168/by/3, PARAM = 2T/SSRD/2T
We have cleaned all requirements to remove any instance where a parameter is duplicated.