I was attempting to extract point data from ERA5-HEAT. The request failed after about 3H 26m. There was no error message given, and the request disappeared from the "Your queue" panel. It still appears in the "Your requests" panel, which gives the status as "Failed", but does not give a reason. Mousing over the "Failed" I get a tooltip saying "Request disappeared from running requests list".


I am trying to get the hourly data at these points. The number of points is quite large, and the points are all around the world, so I cannot restrict the geographic extent. The data I am requesting spans multiple years.

If I request data for a single day at only a small number of points, then my code works, so I suspect that the problem is due to the size of the request. There are around 11,000 points for which I am trying to extract the hourly data covering a 7 year period.

I understand this is quite a large request, so it would make sense if the problem is that the process is either running out of time or running out of memory. My intuition is that it is the initial ct.catalogue.retrieve  that is too large, rather than the `ct.observation.interp_from_grid(data, lat=lats, lon=lons, method="nearest", drop=True)`, because when I try with a single day of data it works, but it still fails if the number of points is small but the retrieve is large.


What I want to know:

  • Am I correct that the request running out of time or memory?
  • How do I handle a large request like this? e.g. should I loop over many smaller retrievals?

Example code follows

import cdstoolbox as ct

@ct.application(title='Subset data online utci short')
@ct.output.download()
def download_application():
    data = ct.catalogue.retrieve(
        'derived-utci-historical',
        {
            'version': '1_0',
            'variable': 'universal_thermal_climate_index',
            'product_type': 'consolidated_dataset',
            'year': [
                '2012', '2013', '2014' , '2015', '2016', '2017',
            ],
            'month': [
                '01', '02', '03',
                '04', '05', '06',
                '07', '08', '09',
                '10', '11', '12',
            ],
            'day': [
                '01', '02', '03',
                '04', '05', '06',
                '07', '08', '09',
                '10', '11', '12',
                '13', '14', '15',
                '16', '17', '18',
                '19', '20', '21',
                '22', '23', '24',
                '25', '26', '27',
                '28', '29', '30',
                '31',
            ],
        }
    )
	v_lats = [62.622, 60.167, 61.46 , 61.676, 60.292] # or many more...
    v_lons = [26.273, 24.811, 23.651, 23.817, 25.051]

 	points = ct.observation.interp_from_grid(data, lat=lats, lon=lons, method="nearest", drop=True)
	return points