How to merge multiple GRIB or NetCDF files

Last modified on Oct 04, 2023 10:32

For GRIB:

On Linux you can use the grib_copy tool. This tools comes with ECMWF's ecCodes package:

grib_copy grib_file_1 grib_file_2 ... output_grib_file

On Linux you can simply concatenate the GRIB files:

cat grib_file_1 grib_file_2 ... grib_file_N > output_grib_file

For NetCDF (unsupported by ECMWF)

Care needs to be taken when merging NetCDF files, as differences in the structure of the files can cause issue. Other issues such as the total size of the merged files also needs to be considered.

If the files are of similar structure, one way to merge netcdf files is using CDO merge:

cdo merge infile1 infile2 infile3 outfile

or if a valid time'

cdo mergetime infile1 infile2 infile3 infile4 outfile

Another option could be NCO (NetCDF Operators). However, please be sure that the files are 'unpacked' before merging them. The NCO 'ncpdq' command can be used to produce an unpacked version of the files.

Here an example of merging for ERA5 NetCDF files from the CDS:

Download the data:

ERA5 CDS API request

import cdsapi
c = cdsapi.Client()
c.retrieve(    
	'reanalysis-era5-single-levels',    
	{
		'product_type': 'reanalysis',        
		'variable': ['surface_latent_heat_flux', 'surface_sensible_heat_flux',],        
		'year': ['2018'],        
		'month': ['01'],        
		'day': ['31'],        
		'time': ['19:00','20:00','21:00','22:00','23:00'],        
		'format': 'netcdf',    
},'ERA5_slhf_sshf_01.nc')
    
c.retrieve(   
	'reanalysis-era5-single-levels',    
	{
		'product_type': 'reanalysis',        
		'variable': ['surface_latent_heat_flux', 'surface_sensible_heat_flux',],        
		'year': ['2018'],        
		'month': ['02'],        
		'day': ['01'],        
		'time': ['00:00', '01:00', '02:00', '03:00', '04:00', '05:00',],        
		'format': 'netcdf',    
	},'ERA5_slhf_sshf_02.nc')

Set record dimension in each file (this is required, as it the dimension to merge the individual files along):

ncks --mk_rec_dmn time ERA5_slhf_sshf_01.nc ERA5_slhf_sshf_01_time_rec.nc 
ncks --mk_rec_dmn time ERA5_slhf_sshf_02.nc ERA5_slhf_sshf_02_time_rec.nc

Note that the data are still packed - they need to be unpacked before they can be merged:

ncpdq -U ERA5_slhf_sshf_01_time_rec.nc ERA5_slhf_sshf_01_time_rec_unpack.nc 
ncpdq -U ERA5_slhf_sshf_02_time_rec.nc ERA5_slhf_sshf_02_time_rec_unpack.nc

Then they can be successfully merged:

ncrcat ERA5_slhf_sshf_01_time_rec_unpack.nc ERA5_slhf_sshf_02_time_rec_unpack.nc  ERA5_slhf_sshf_0102_time_rec_unpack.nc

_{This document has been produced in the context of the Copernicus Atmosphere Monitoring Service (CAMS) and Copernicus Climate Change Service (C3S).}

_{The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of CAMS and C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014 and Contribution Agreement signed on 22/07/2021). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.}

_{The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author's view.}

Space shortcuts

Page tree

For GRIB:

For NetCDF (unsupported by ECMWF)

Related articles