Good morning,


I'm running cy43r3 and I would like to process the ICMUA output files. The new files contain a mix of grib1 and grib2 fields that is not easly processable by the cdo provided our computing center:

cdo -R -t ecmwf -f nc copy -remapbil,r360x180 ICMUA4317+000000 ICMUA4317+000000.nc
cdo(2) remapbil: Process started
Segmentation fault (core dumped)

grib_ls ICMUA4317+000000
ICMUA4317+000000
edition      centre       typeOfLevel  level        dataDate     stepRange    dataType     shortName    packingType  gridType     
1            ecmf         isobaricInhPa  1000         20160925     0            fc           q            grid_simple  reduced_gg  
1            ecmf         isobaricInhPa  925          20160925     0            fc           q            grid_simple  reduced_gg  
1            ecmf         isobaricInhPa  850          20160925     0            fc           q            grid_simple  reduced_gg  
1            ecmf         isobaricInhPa  700          20160925     0            fc           q            grid_simple  reduced_gg  
.
.
. 
2            ecmf         hybrid         57           20160925     0            fc           vtpha        grid_simple  reduced_gg  
2            ecmf         hybrid         58           20160925     0            fc           vtpha        grid_simple  reduced_gg  
2            ecmf         hybrid         59           20160925     0            fc           vtpha        grid_simple  reduced_gg  
2            ecmf         hybrid         60           20160925     0            fc           vtpha        grid_simple  reduced_gg 
360 of 360 grib messages in ICMUA4317+000000_grib2

360 of 360 total grib messages in 1 files


I tried to split the file into grib1 and grib2 fields. The grib1 file looks fine and can be processed by cdo, but the grib2 file has the gridSize changed (maybe?) which seems to prevent further post processing:

grib_copy ICMUA4317+000000 -w edition=1 ICMUA4317+000000_grib1
grib_copy ICMUA4317+000000 -w edition=2 ICMUA4317+000000_grib2

grib_ls ICMUA4317+000000_grib2 
ICMUA4317+000000_grib2
edition      centre       date         dataType     gridType     stepRange    typeOfLevel  level        shortName    packingType  
2            ecmf         20160925     fc           reduced_gg   0            hybrid       1            q            grid_simple 
2            ecmf         20160925     fc           reduced_gg   0            hybrid       2            q            grid_simple 
2            ecmf         20160925     fc           reduced_gg   0            hybrid       3            q            grid_simple 
2            ecmf         20160925     fc           reduced_gg   0            hybrid       4            q            grid_simple 
.
.
.

cdo -R -t ecmwf -f nc copy -remapbil,r360x180 ICMUA4317+000000_grib2 ICMUA4317+000000.nc
cdo(2) remapbil: Process started
Error (gribapiGetGridRegular): numberOfPoints (35718) and gridSize (343597383520) differ!


I tried to convert the mixed files with grib_to_netcdf but here I get an error related to the gridType:

grib_to_netcdf -T -o ICMUA4317+000000.nc ICMUA4317+000000
grib_to_netcdf: Version 2.10.0
grib_to_netcdf: Processing input file 'ICMUA4317+000000'.
grib_to_netcdf: Found 489 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file 'ICMUA4317+000000.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
ECCODES ERROR   :  First GRIB is not on a regular lat/lon grid or on a regular Gaussian grid. Exiting.


How to I get my hands on the sweet data inside this file? (wink)

Cheers, Jan



4 Comments

  1. Jan Streffing

    Problem solved. For posterity: In the namelist that came with the input files, 3D output on both pressure and surface levels was turned on. Pressure level output is written in grib1 and surface level output written in grib2. Resulting in the unprocessable files I described. I just turned off the surface level output.

    I also deactivated the K-Index(260121) and Total totals index (260123) that were written as grib2 into the 2D output file.

    Cheers, Jan

  2. Glenn Carver

    Hi Jan,

    Sorry, I missed this page earlier.

    You are a bit ahead of the curve, but my understanding was that the ICM*UA+* output file for 43r3 was just a rename of the 40r1 ICM*GG+* file, so the contents should be the same and can be treated the same way, though some of the grib keys might have changed (for the new cubic grid for example).

    I think it's the other way around, the surface fields are GRIB1 and the pressure level output is GRIB2. Only GRIB2 can encode greater than 127 levels so it must be that way around.

    I know that cdo can choke with there are multiple level types involved. The surface fields include the sub-surface levels and mixing this with the pressure level fields in one file seems to confuse cdo. 

    The recommendation is to split the level types first (see How to convert GRIB to netCDF for much more detail. You can split the vertical axes in two ways. Either using cdo:

    cdo splitzaxis ICMSHg4a4+000000.grb ICMSHg4a4+000000_split
ls ICMSH*split*
ICMSHg4a4+000000_split01.grb
ICMSHg4a4+000000_split02.grb
ICMSHg4a4+000000_split03.grb
ICMSHg4a4+000000_split04.grb

    or by using the grib_copy command that comes with ecCodes (which personally I prefer):

    grib_copy ICMSHg4a4+000000.grb ICMSHg4a4+000000_[typeOfLevel].grb

    where 'typeOfLevel' is a grib key.

    Once the level types are split cdo should work ok?

    Was there another problem with the K-index and Total totals index?

    Cheers,  Glenn


  3. Jan Streffing

    Hello Glenn,

    that is an interesting point. I removed the lines


    NFP3DFS=10,

MFP3DFS(:)=130,135,138,155,133,162108,162110,162111,162112,162113,

    and kept

    NFP3DFP=13,

MFP3DFP(:)=129,130,135,138,155,157,3,133,246,247,248,75,76,


    This should leave me with pressure level output if I interpret the names correctly. grib_ls tells me that the ICMSH output contain fields with typeOfLevel=isobaricInhPa and edition=1. I interpreted that as grib1 pressure level data and the contents do look like pressure data. Something wrong here? K-index and Total totals index were also of edition=2 and resulted in inability to post-process the ICMGG file until I removed them from the output.


    Ultimately I don't have much use for surface level data, so I am happy not writing it out. For me it would only make the model slower and use up disc space anyway.

    Cheers, Jan

  4. Joakim Kjellsson

    Hi Jan 

    I never had this problem with 43r3 using our post processing in esm-tools. I do what Glenn suggests: split into one file per vertical level and also GG and SH. So I get output files like ending with:
    SH_isobaricInhPa (U,V,T,Z)
    GG_isobaricInhPa (Q,O3)
    SH_hybrid
    GG_hybrid
    GG_surface

    but with 43r3 I also get "UA_isobaricInhPa" and "UA_hybrid" rather than "GG_isobaricInhPa" and "GG_hybrid". 

    I do the following:

    cat ICMUA* > ICMUACAT
    grib_copy -w typeOfLevel=isobaricInhPa ICMUACAT ICMUACAT_isobaricInhPa
    cdo --eccodes -f nc4c -z zip_4 -t ecmwf -setgridtype,regular ICMUACAT_isobaricInhPa h6mv_19820101_19820105_UA_isobaricInhPa.nc 

    If CDO has been compiled with ecCodes, the above will convert the UA output to a netCDF file on a regular grid with moderate compression. 

    Cheers
    Joakim


    PS: The error you get is however exactly what I get when I try to post process the output from WAM at T511. WAM (for some reason) writes data to different grid types (regular,reduced etc) depending on the resolution. No idea why...