Good morning,

I have been working with PEXTRA and Oasis EXPOUT output so far, and turned on the regular output for the first time just now. I get error messages when trying to convert to netcdf. ECCODES ERROR : Wrong number of fields ... Try using the -T option says this is  overlapping "time"+"step" specifications. Since we only have one time in each file of OIFS output I don't see how this could be it here. The solution in the FAQ didn't work. It seems I don't have permission to attach files here so I copy&pasted in the namelist below. 


cdo -f nc copy ICMGGgvfc+000008 ICMGGgvfc+000008.nc

Warning (cgribexScanTimestep2) : Record 25 (id=151.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!

cdo copy: Open failed on >ICMGGgvfc+000008<

Unsupported file structure


grib_to_netcdf -T ICMGGgvfc+000008 ICMGGgvfc+000008.nc
grib_to_netcdf: Version 1.15.0
grib_to_netcdf: Processing input file 'ICMGGgvfc+000008'.
GRIB_API ERROR   :  Wrong number of fields
GRIB_API ERROR   :  File contains 30 GRIBs, 30 left in internal description, 29 in request
GRIB_API ERROR   :  Internal description
GRIB,
    domain     = g,
    levtype    = sfc,
    date       = 19900101,
    time       = 0.0,
    step       = 8,
    param      = stl1/tp/ci/sst/swvl1/swvl2/swvl3/swvl4/sd/stl2/stl3/stl4/lsp/cp/sshf/slhf/ssr/str/tsr/ttr/ewss/nsss/e/msl/tcc/u10/v10/t2m/d2m,
    class      = rd,
    type       = fc,
    stream     = oper,
    expver     = gvfc,
    _long_name = Soil temperature level 1/Total precipitation/Sea-ice cover/Sea surface temperature/Volumetric soil water layer 1/Volumetric soil water layer 2/Volumetric soil water layer 3/Volumetric soil water layer 4/Snow depth/Soil temperature level 2/Soil temperature level 3/Soil temperature level 4/Large-scale precipitation/Convective precipitation/Surface sensible heat flux/Surface latent heat flux/Surface net solar radiation/Surface net thermal radiation/Top net solar radiation/Top net thermal radiation/Eastward turbulent surface stress/Northward turbulent surface stress/Evaporation/Mean sea level pressure/Total cloud cover/10 metre U wind component/10 metre V wind component/2 metre temperature/2 metre dewpoint temperature,
    _units     = K/m/(0 - 1)/m**3 m**-3/m of water equivalent/J m**-2/N m**-2 s/Pa/m s**-1,
    _cf_name   = surface_temperature/lwe_thickness_of_surface_snow_amount/lwe_thickness_of_large_scale_precipitation_amount/lwe_thickness_of_convective_precipitation_amount/surface_upward_sensible_heat_flux/surface_upward_latent_heat_flux/surface_net_downward_shortwave_flux/surface_net_upward_longwave_flux/toa_net_upward_shortwave_flux/toa_outgoing_longwave_flux/surface_downward_eastward_stress/surface_downward_northward_stress/lwe_thickness_of_water_evaporation_amount/air_pressure_at_sea_level/cloud_area_fraction,
    _validation = 58749440.000000,
    _juliandate = 2447893,
    _validationtime = 788936.000000


Cheers,

Jan

5 Comments

  1. Hi Jan,

    In case the grib file is damaged in some way, can you use 'grib_ls' or 'grib_dump' to check the entire file can be read, e.g.

    grib_dump ICMGGgvfc+000008

    it will produce a lot of output but will confirm the file can be read ok. I suspect the file is fine but good to check.

    When using cdo, it is best to split the file so that each level type (pressure, surface) is in a separate file. Cdo's cgribex engine doesn't tend to work well if it sees a mix of vertical coordinates. You can do the split like this using cdo or grib_api depending on your preference:

    # cdo solution
    cdo splitzaxis ICMGGgvfc+000008 ICMGGgvfc+000008_split
    
    # grib_api solution.
    grib_copy ICMGGgvfc+000008 ICMSHg4a4+000000_[typeOfLevel]

    Then try again.

    There are, unfortunately, a few caveats when using cdo with OpenIFS grib output, particularly when converting to a regular lat-long grid.

    I recommend reading both this page on the OpenIFS website: How to convert GRIB to netCDF (including comments at the bottom) and this post on the OpenIFS forum: Grib to netcdf conversion, which goes into the caveats and workarounds in much more detail.

    Regards,   Glenn

  2. Hello Glenn,

    I tried grib_ls and grib_dump; the files look fine to me. I have no problems opening them in a grib file viewer like panoply. cdo splitzaxis resulted in the same error as the cdo copy command above (and every other cdo command I've tried)


    cdo
    cdo splitzaxis ICMGGgvfc+000008 ICMGGgvfc+000008.nc
    Warning (cgribexScanTimestep2) : Record 25 (id=151.128 lev1=0 lev2=0) timestep 2: Parameter not defined at timestep 1!
    
    cdo splitzaxis: Open failed on >ICMGGgvfc+000008<
    Unsupported file structure
    a270092@mlogin101% 



    The grib tools solution doesnt produce any errors but I don't think it helped either.

    grib-tools
    grib_ls ICMGGgvfc+000008
    ICMGGgvfc+000008
    edition      centre       typeOfLevel  levels       dataDate     stepRange    dataType     shortName    packingType  gridType     
    1            ecmf         depthBelowLandLayer  0-7          19900101     8            fc           stl1         grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           tp           grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           ci           grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           sst          grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  0-7          19900101     8            fc           swvl1        grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  7-28         19900101     8            fc           swvl2        grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  28-100       19900101     8            fc           swvl3        grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  100-MISSING  19900101     8            fc           swvl4        grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           sd           grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  7-28         19900101     8            fc           stl2         grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  28-100       19900101     8            fc           stl3         grid_simple  reduced_gg  
    1            ecmf         depthBelowLandLayer  100-MISSING  19900101     8            fc           stl4         grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           lsp          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           cp           grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           sshf         grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           slhf         grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           ssr          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           str          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           tsr          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           ttr          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           ewss         grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           nsss         grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           e            grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           tp           grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           msl          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           tcc          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           10u          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           10v          grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           2t           grid_simple  reduced_gg  
    1            ecmf         surface      not_found    19900101     8            fc           2d           grid_simple  reduced_gg  
    30 of 30 grib messages in ICMGGgvfc+000008
    
    30 of 30 total grib messages in 1 files
    
    grib_copy ICMGGgvfc+000008 ICMGGgvfc+000008_[surface]
    lr
    -rw-r--r-- 1 a270092 ab0246   819080 Apr 11 09:38 ICMGGgvfc+000008_undef
    
    /sw/rhel6-x64/grib_api/grib_api-1.15.0-intel14/bin/grib_to_netcdf ICMGGgvfc+000008_undef ICMGGgvfc+000008_undef.nc
    grib_to_netcdf: Version 1.15.0
    grib_to_netcdf: Processing input file 'ICMGGgvfc+000008_undef'.
    GRIB_API ERROR   :  Wrong number of fields
    GRIB_API ERROR   :  File contains 30 GRIBs, 30 left in internal description, 29 in request
    GRIB_API ERROR   :  Internal description
    GRIB,
        domain     = g,
        levtype    = sfc,
        param      = stl1/tp/ci/sst/swvl1/swvl2/swvl3/swvl4/sd/stl2/stl3/stl4/lsp/cp/sshf/slhf/ssr/str/tsr/ttr/ewss/nsss/e/msl/tcc/u10/v10/t2m/d2m,
        class      = rd,
        type       = fc,
        stream     = oper,
        expver     = gvfc,
        _long_name = Soil temperature level 1/Total precipitation/Sea-ice cover/Sea surface temperature/Volumetric soil water layer 1/Volumetric soil water layer 2/Volumetric soil water layer 3/Volumetric soil water layer 4/Snow depth/Soil temperature level 2/Soil temperature level 3/Soil temperature level 4/Large-scale precipitation/Convective precipitation/Surface sensible heat flux/Surface latent heat flux/Surface net solar radiation/Surface net thermal radiation/Top net solar radiation/Top net thermal radiation/Eastward turbulent surface stress/Northward turbulent surface stress/Evaporation/Mean sea level pressure/Total cloud cover/10 metre U wind component/10 metre V wind component/2 metre temperature/2 metre dewpoint temperature,
        _units     = K/m/(0 - 1)/m**3 m**-3/m of water equivalent/J m**-2/N m**-2 s/Pa/m s**-1,
        _cf_name   = surface_temperature/lwe_thickness_of_surface_snow_amount/lwe_thickness_of_large_scale_precipitation_amount/lwe_thickness_of_convective_precipitation_amount/surface_upward_sensible_heat_flux/surface_upward_latent_heat_flux/surface_net_downward_shortwave_flux/surface_net_upward_longwave_flux/toa_net_upward_shortwave_flux/toa_outgoing_longwave_flux/surface_downward_eastward_stress/surface_downward_northward_stress/lwe_thickness_of_water_evaporation_amount/air_pressure_at_sea_level/cloud_area_fraction,
        _validation = 58749440.000000,
        _juliandate = 2447893,
        _validationtime = 788936.000000,
        time       = 788936.00
    
    
    


    Best regards,

    Jan

  3. Hi Jan,

    You need to use the mars key 'typeOfLevel' in the grib_copy, not the level type itself. The mars key is the name you see at the top of the column in the grib_ls output. That's why you have a file with _undef on the end. e.g.

    % grib_copy ICMGGgurr+000001 ICMGGgurr+000001_[typeOfLevel].grb
    % ls
    ICMGGgurr+000001                          ICMGGgurr+000001_hybrid.grb         ICMGGgurr+000001_potentialVorticity.grb  ICMGGgurr+000001_theta.grb
    ICMGGgurr+000001_depthBelowLandLayer.grb  ICMGGgurr+000001_isobaricInhPa.grb  ICMGGgurr+000001_surface.grb

    This will work for any mars key, so you could split the file on "[shortName]" for example. Be careful of the capitalization.

    For cdo, I notice the error message indicates it's using the cgribex decoding engine. Cgribex doesn't understand GRIB format 2 records, only GRIB 1. So it will fail to read the upper air fields which are all encoded in GRIB 2. One workaround is to change the mars key 'editionNumber' to 1:

    % grib_set -s editionNumber=1  in.grb  in_1.grb
    % cdo -R copy  in_1.grb  in_1_R.grb
    % grib_set -s editionNumber=2  in_1_R.grb  in_2_R.grb

    This does not actually convert the message to GRIB-1, it only changes the editionNumber. Any GRIB-2 specific elements are unaltered. It's recommended that the editionNumber is changed back to 2 to ensure that the parameter names remain correct in any subsequent cdo / grib commands. This will not work if you have more 128 vertical levels.

    We recommend that cdo is compiled to use ECMWF's grib-api decoder rather than the cgribex engine which is rather limited. Using grib-api will give cdo the ability to decode both GRIB 1 & 2 records.

    For files with a mix of GRIB-1 and GRIB-2,  either split the file first or compile cdo with "--disable-cgribex --with-grib_api".

    Hope that helps. There's more information the converting grib to netcdf links I posted above.

       Cheers,  Glenn

    1. Hallo Glenn,

      I now separated surface and depthBelowLandLayer level types. The depthBelowLandLayer files can be processed with cdo now, but the surface files still crashes as before. All fields are grib edition 1 already.

      Since I did not get the same error in the coupled output, neither for OIFS-FESOM nor for OIFS-NEMO I decided to replace the NAMFPC with the one from OIFS-NEMO. And it fixed my problem...


      So, I replaced:

      OIFS-standalone
        &NAMFPC
          CFPFMT ='MODEL',         ! Format of output files, always 'MODEL' for IFS. DO NOT CHANGE.
      
          !  Output on model levels
          !NFP3DFS=15,                   
          !MFP3DFS=91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,
          !NRFP3S=91,
      
          !  Output on pressure levels
          !NFP3DFP = 5,                        ! number of variables to output
          !MFP3DFP = 129,130,133,138,155,      ! grib codes of variables to output on pressure levels given by RFP3P
          !    these pressure levels match the SPEEDY model 
          !RFP3P   = 92500,85000,70000,50000,30000,20000,10000,3000,
      
          !  Output on theta levels
          !NFP3DFT = 1,                         ! number of variables to output
          !MFP3DFT = 60,                        ! grib codes of variables to output on theta levels given by RFP3TH
          !RFP3TH  = 315,330,
      
          !  If output on P, theta, PV, Ht, also enable spectral orography (surface geopot.), surf pres (Ps), log(Ps)
          !NFP2DF=3,
          !MFP2DF=129,134,152,
      
          !  Physics fields output; number of fields (NFPPHY) and grib-codes (MFPPHY)
          NFPPHY  = 30,
          MFPPHY  = 139,151,164,165,166,167,168,228,31,34,39,40,41,42,141,170,183,236,142,143,146,147,176,177,178,179,180,181,182,228,
      
      
          !  Spectral fitting for output spectral fields (recommended)
          !  Disabling this will cause model spectral fields to be output as gridpoint
          NFITP   = 1,
          NFITT   = 1,
          NFITV   = 1,
        /


      with:

      OIFS-NEMO
      &NAMFPC
          CFPFMT  ='MODEL',
          NFP3DFP = 5,
          MFP3DFP = 129,130,131,132,133,
          RFP3P   = 100000,92500,85000,70000,50000,40000,30000,20000,10000,5000,1000,
          NFP2DF  = 1,
          MFP2DF  = 152,
          NFPPHY  = 52,
          MFPPHY  = 031,032,033,034,035,039,040,041,042,078,079,136,137,139,141,142,143,144,146,147,151,164,165,166,167,168,169,170,175,176,177,178,179,180,181,182,183,186,187,188,201,202,205,208,209,210,211,212,235,236,238,243,
          NFITP   = 1,
          NFITT   = 1,
          NFITV   = 1,
        /


      I looked for the difference. Its parameter 228. I'm not sure why, but I had it twice :

      MFPPHY = 139,151,164,165,166,167,168,228,31,34,39,40,41,42,141,170,183,236,142,143,146,147,176,177,178,179,180,181,182,228,

      Maybe this happened at some point because there are grib id 228 fields with inidicators 001, 014, 089 etc that are sometimes of interest? Sorry for the confusion! Thank you for a number of good tips for grib-tools and cdo.


      Best regards,

      Jan

      1. Ah interesting. I was not aware that can cause problems with conversion. That's useful to know.

        Thanks,  Glenn