Good morning,
I have been working with PEXTRA and Oasis EXPOUT output so far, and turned on the regular output for the first time just now. I get error messages when trying to convert to netcdf. ECCODES ERROR : Wrong number of fields ... Try using the -T option says this is overlapping "time"+"step" specifications. Since we only have one time in each file of OIFS output I don't see how this could be it here. The solution in the FAQ didn't work. It seems I don't have permission to attach files here so I copy&pasted in the namelist below.
cdo -f nc copy ICMGGgvfc+000008 ICMGGgvfc+000008.nc
Warning (cgribexScanTimestep2) : Record 25 (id=151.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!
cdo copy: Open failed on >ICMGGgvfc+000008<
Unsupported file structure
grib_to_netcdf -T ICMGGgvfc+000008 ICMGGgvfc+000008.nc
grib_to_netcdf: Version 1.15.0
grib_to_netcdf: Processing input file 'ICMGGgvfc+000008'.
GRIB_API ERROR : Wrong number of fields
GRIB_API ERROR : File contains 30 GRIBs, 30 left in internal description, 29 in request
GRIB_API ERROR : Internal description
GRIB,
domain = g,
levtype = sfc,
date = 19900101,
time = 0.0,
step = 8,
param = stl1/tp/ci/sst/swvl1/swvl2/swvl3/swvl4/sd/stl2/stl3/stl4/lsp/cp/sshf/slhf/ssr/str/tsr/ttr/ewss/nsss/e/msl/tcc/u10/v10/t2m/d2m,
class = rd,
type = fc,
stream = oper,
expver = gvfc,
_long_name = Soil temperature level 1/Total precipitation/Sea-ice cover/Sea surface temperature/Volumetric soil water layer 1/Volumetric soil water layer 2/Volumetric soil water layer 3/Volumetric soil water layer 4/Snow depth/Soil temperature level 2/Soil temperature level 3/Soil temperature level 4/Large-scale precipitation/Convective precipitation/Surface sensible heat flux/Surface latent heat flux/Surface net solar radiation/Surface net thermal radiation/Top net solar radiation/Top net thermal radiation/Eastward turbulent surface stress/Northward turbulent surface stress/Evaporation/Mean sea level pressure/Total cloud cover/10 metre U wind component/10 metre V wind component/2 metre temperature/2 metre dewpoint temperature,
_units = K/m/(0 - 1)/m**3 m**-3/m of water equivalent/J m**-2/N m**-2 s/Pa/m s**-1,
_cf_name = surface_temperature/lwe_thickness_of_surface_snow_amount/lwe_thickness_of_large_scale_precipitation_amount/lwe_thickness_of_convective_precipitation_amount/surface_upward_sensible_heat_flux/surface_upward_latent_heat_flux/surface_net_downward_shortwave_flux/surface_net_upward_longwave_flux/toa_net_upward_shortwave_flux/toa_outgoing_longwave_flux/surface_downward_eastward_stress/surface_downward_northward_stress/lwe_thickness_of_water_evaporation_amount/air_pressure_at_sea_level/cloud_area_fraction,
_validation = 58749440.000000,
_juliandate = 2447893,
_validationtime = 788936.000000
Cheers,
Jan
5 Comments
Glenn Carver
Hi Jan,
In case the grib file is damaged in some way, can you use 'grib_ls' or 'grib_dump' to check the entire file can be read, e.g.
it will produce a lot of output but will confirm the file can be read ok. I suspect the file is fine but good to check.
When using cdo, it is best to split the file so that each level type (pressure, surface) is in a separate file. Cdo's cgribex engine doesn't tend to work well if it sees a mix of vertical coordinates. You can do the split like this using cdo or grib_api depending on your preference:
Then try again.
There are, unfortunately, a few caveats when using cdo with OpenIFS grib output, particularly when converting to a regular lat-long grid.
I recommend reading both this page on the OpenIFS website: How to convert GRIB to netCDF (including comments at the bottom) and this post on the OpenIFS forum: Grib to netcdf conversion, which goes into the caveats and workarounds in much more detail.
Regards, Glenn
Jan Streffing
Hello Glenn,
I tried grib_ls and grib_dump; the files look fine to me. I have no problems opening them in a grib file viewer like panoply. cdo
splitzaxisresulted in the same error as the cdo copy command above (and every other cdo command I've tried)The grib tools solution doesnt produce any errors but I don't think it helped either.
grib_ls ICMGGgvfc+000008 ICMGGgvfc+000008 edition centre typeOfLevel levels dataDate stepRange dataType shortName packingType gridType 1 ecmf depthBelowLandLayer 0-7 19900101 8 fc stl1 grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc tp grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc ci grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc sst grid_simple reduced_gg 1 ecmf depthBelowLandLayer 0-7 19900101 8 fc swvl1 grid_simple reduced_gg 1 ecmf depthBelowLandLayer 7-28 19900101 8 fc swvl2 grid_simple reduced_gg 1 ecmf depthBelowLandLayer 28-100 19900101 8 fc swvl3 grid_simple reduced_gg 1 ecmf depthBelowLandLayer 100-MISSING 19900101 8 fc swvl4 grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc sd grid_simple reduced_gg 1 ecmf depthBelowLandLayer 7-28 19900101 8 fc stl2 grid_simple reduced_gg 1 ecmf depthBelowLandLayer 28-100 19900101 8 fc stl3 grid_simple reduced_gg 1 ecmf depthBelowLandLayer 100-MISSING 19900101 8 fc stl4 grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc lsp grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc cp grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc sshf grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc slhf grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc ssr grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc str grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc tsr grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc ttr grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc ewss grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc nsss grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc e grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc tp grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc msl grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc tcc grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc 10u grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc 10v grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc 2t grid_simple reduced_gg 1 ecmf surface not_found 19900101 8 fc 2d grid_simple reduced_gg 30 of 30 grib messages in ICMGGgvfc+000008 30 of 30 total grib messages in 1 files grib_copy ICMGGgvfc+000008 ICMGGgvfc+000008_[surface] lr -rw-r--r-- 1 a270092 ab0246 819080 Apr 11 09:38 ICMGGgvfc+000008_undef /sw/rhel6-x64/grib_api/grib_api-1.15.0-intel14/bin/grib_to_netcdf ICMGGgvfc+000008_undef ICMGGgvfc+000008_undef.nc grib_to_netcdf: Version 1.15.0 grib_to_netcdf: Processing input file 'ICMGGgvfc+000008_undef'. GRIB_API ERROR : Wrong number of fields GRIB_API ERROR : File contains 30 GRIBs, 30 left in internal description, 29 in request GRIB_API ERROR : Internal description GRIB, domain = g, levtype = sfc, param = stl1/tp/ci/sst/swvl1/swvl2/swvl3/swvl4/sd/stl2/stl3/stl4/lsp/cp/sshf/slhf/ssr/str/tsr/ttr/ewss/nsss/e/msl/tcc/u10/v10/t2m/d2m, class = rd, type = fc, stream = oper, expver = gvfc, _long_name = Soil temperature level 1/Total precipitation/Sea-ice cover/Sea surface temperature/Volumetric soil water layer 1/Volumetric soil water layer 2/Volumetric soil water layer 3/Volumetric soil water layer 4/Snow depth/Soil temperature level 2/Soil temperature level 3/Soil temperature level 4/Large-scale precipitation/Convective precipitation/Surface sensible heat flux/Surface latent heat flux/Surface net solar radiation/Surface net thermal radiation/Top net solar radiation/Top net thermal radiation/Eastward turbulent surface stress/Northward turbulent surface stress/Evaporation/Mean sea level pressure/Total cloud cover/10 metre U wind component/10 metre V wind component/2 metre temperature/2 metre dewpoint temperature, _units = K/m/(0 - 1)/m**3 m**-3/m of water equivalent/J m**-2/N m**-2 s/Pa/m s**-1, _cf_name = surface_temperature/lwe_thickness_of_surface_snow_amount/lwe_thickness_of_large_scale_precipitation_amount/lwe_thickness_of_convective_precipitation_amount/surface_upward_sensible_heat_flux/surface_upward_latent_heat_flux/surface_net_downward_shortwave_flux/surface_net_upward_longwave_flux/toa_net_upward_shortwave_flux/toa_outgoing_longwave_flux/surface_downward_eastward_stress/surface_downward_northward_stress/lwe_thickness_of_water_evaporation_amount/air_pressure_at_sea_level/cloud_area_fraction, _validation = 58749440.000000, _juliandate = 2447893, _validationtime = 788936.000000, time = 788936.00Best regards,
Jan
Glenn Carver
Hi Jan,
You need to use the mars key 'typeOfLevel' in the grib_copy, not the level type itself. The mars key is the name you see at the top of the column in the grib_ls output. That's why you have a file with _undef on the end. e.g.
This will work for any mars key, so you could split the file on "[shortName]" for example. Be careful of the capitalization.
For cdo, I notice the error message indicates it's using the cgribex decoding engine. Cgribex doesn't understand GRIB format 2 records, only GRIB 1. So it will fail to read the upper air fields which are all encoded in GRIB 2. One workaround is to change the mars key 'editionNumber' to 1:
This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered. It's recommended that the
editionNumberis changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands. This will not work if you have more 128 vertical levels.We recommend that cdo is compiled to use ECMWF's grib-api decoder rather than the cgribex engine which is rather limited. Using grib-api will give cdo the ability to decode both GRIB 1 & 2 records.
For files with a mix of GRIB-1 and GRIB-2, either split the file first or compile cdo with "--disable-cgribex --with-grib_api".
Hope that helps. There's more information the converting grib to netcdf links I posted above.
Cheers, Glenn
Jan Streffing
Hallo Glenn,
I now separated surface and depthBelowLandLayer level types. The depthBelowLandLayer files can be processed with cdo now, but the surface files still crashes as before. All fields are grib edition 1 already.
Since I did not get the same error in the coupled output, neither for OIFS-FESOM nor for OIFS-NEMO I decided to replace the NAMFPC with the one from OIFS-NEMO. And it fixed my problem...
So, I replaced:
&NAMFPC CFPFMT ='MODEL', ! Format of output files, always 'MODEL' for IFS. DO NOT CHANGE. ! Output on model levels !NFP3DFS=15, !MFP3DFS=91,92,93,94,95,96,97,98,99,100,101,102,103,104,105, !NRFP3S=91, ! Output on pressure levels !NFP3DFP = 5, ! number of variables to output !MFP3DFP = 129,130,133,138,155, ! grib codes of variables to output on pressure levels given by RFP3P ! these pressure levels match the SPEEDY model !RFP3P = 92500,85000,70000,50000,30000,20000,10000,3000, ! Output on theta levels !NFP3DFT = 1, ! number of variables to output !MFP3DFT = 60, ! grib codes of variables to output on theta levels given by RFP3TH !RFP3TH = 315,330, ! If output on P, theta, PV, Ht, also enable spectral orography (surface geopot.), surf pres (Ps), log(Ps) !NFP2DF=3, !MFP2DF=129,134,152, ! Physics fields output; number of fields (NFPPHY) and grib-codes (MFPPHY) NFPPHY = 30, MFPPHY = 139,151,164,165,166,167,168,228,31,34,39,40,41,42,141,170,183,236,142,143,146,147,176,177,178,179,180,181,182,228, ! Spectral fitting for output spectral fields (recommended) ! Disabling this will cause model spectral fields to be output as gridpoint NFITP = 1, NFITT = 1, NFITV = 1, /with:
&NAMFPC CFPFMT ='MODEL', NFP3DFP = 5, MFP3DFP = 129,130,131,132,133, RFP3P = 100000,92500,85000,70000,50000,40000,30000,20000,10000,5000,1000, NFP2DF = 1, MFP2DF = 152, NFPPHY = 52, MFPPHY = 031,032,033,034,035,039,040,041,042,078,079,136,137,139,141,142,143,144,146,147,151,164,165,166,167,168,169,170,175,176,177,178,179,180,181,182,183,186,187,188,201,202,205,208,209,210,211,212,235,236,238,243, NFITP = 1, NFITT = 1, NFITV = 1, /I looked for the difference. Its parameter 228. I'm not sure why, but I had it twice :
MFPPHY = 139,151,164,165,166,167,168,228,31,34,39,40,41,42,141,170,183,236,142,143,146,147,176,177,178,179,180,181,182,228,
Maybe this happened at some point because there are grib id 228 fields with inidicators 001, 014, 089 etc that are sometimes of interest? Sorry for the confusion! Thank you for a number of good tips for grib-tools and cdo.
Best regards,
Jan
Glenn Carver
Ah interesting. I was not aware that can cause problems with conversion. That's useful to know.
Thanks, Glenn