CDS API queued indefinitely

Hi, I need to download data from "ERA5 hourly data on single levels from 1979 to present", and I am using the following code in python, but it takes too long and I need data from 1979-2016, does anyone know what I can do to make the request accepted:

import numpy as np
from datetime import datetime, timedelta
import cdsapi
import certifi
import urllib3

http = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED',
ca_certs=certifi.where()
)

c = cdsapi.Client()

Date_inicio = datetime(2002, 8, 1)
Date_fin = datetime(2016, 12, 31)
array_horario = np.arange(Date_inicio, Date_fin, timedelta(hours = 1)).astype(datetime)

for i in array_horario:
      year = i.strftime('%Y')
      month = i.strftime('%m')
      day = i.strftime('%d')
      hour = i.strftime('%H:%M')

     c.retrieve(
     'reanalysis-era5-single-levels',
      {
               'product_type': 'reanalysis',
                'variable': [
                            '10m_u_component_of_wind', '10m_v_component_of_wind', 'skin_temperature', '2m_dewpoint_temperature',
                             '2m_temperature','lake_cover', 'lake_depth', 'lake_mix_layer_depth', 'lake_shape_factor',
                             'lake_mix_layer_temperature', 'lake_total_layer_temperature',
                              'mean_surface_downward_long_wave_radiation_flux_clear_sky',
                              'mean_surface_downward_short_wave_radiation_flux_clear_sky',
                              'leaf_area_index_high_vegetation', 'leaf_area_index_low_vegetation', 'low_vegetation_cover',
                              'high_vegetation_cover', 'mean_surface_downward_long_wave_radiation_flux',
                              'mean_surface_downward_short_wave_radiation_flux', 'runoff', 'soil_type', 'sub_surface_runoff',
                              'skin_reservoir_content', 'surface_pressure', 'surface_runoff', 'total_precipitation',
                ],
                'year': year,
                'month': month,
                'day': [
                        day,
                 ],
                'time': [
                       hour,
                 ],
                 'format': 'netcdf',
      },
      f'.ERA5hourly_data_on_single_levels{year}{month}{day}.{i.strftime("%H")}.nc')

In my experience you have to break into chunks like monthly.

this code returns a global hourly file, yesterday it lasted almost four hours downloading 15 hourly files. Then, I decided to work with a smaller region, and now it is downloading fast. Maybe the size of the file makes the request is queued: before it was 52739 KB; now 44 KB. this process should be automatic because I need to download around 333000 files.

I leave here the new code with the selected area (bold font) in case it works for someone! (check indentation)
sorry for my English!

import numpy as np
from datetime import datetime, timedelta
import cdsapi
import certifi
import urllib3

http = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED',
ca_certs=certifi.where()
)

c = cdsapi.Client()

Date_inicio = datetime(2002, 8, 1)
Date_fin = datetime(2016, 12, 31)
array_horario = np.arange(Date_inicio, Date_fin, timedelta(hours = 1)).astype(datetime)

for i in array_horario:
      year = i.strftime('%Y')
      month = i.strftime('%m')
      day = i.strftime('%d')
      hour = i.strftime('%H:%M')

     c.retrieve(
     'reanalysis-era5-single-levels',
      {
               'product_type': 'reanalysis',
                'variable': [
                            '10m_u_component_of_wind', '10m_v_component_of_wind', 'skin_temperature', '2m_dewpoint_temperature',
                             '2m_temperature','lake_cover', 'lake_depth', 'lake_mix_layer_depth', 'lake_shape_factor',
                             'lake_mix_layer_temperature', 'lake_total_layer_temperature',
                              'mean_surface_downward_long_wave_radiation_flux_clear_sky',
                              'mean_surface_downward_short_wave_radiation_flux_clear_sky',
                              'leaf_area_index_high_vegetation', 'leaf_area_index_low_vegetation', 'low_vegetation_cover',
                              'high_vegetation_cover', 'mean_surface_downward_long_wave_radiation_flux',
                              'mean_surface_downward_short_wave_radiation_flux', 'runoff', 'soil_type', 'sub_surface_runoff',
                              'skin_reservoir_content', 'surface_pressure', 'surface_runoff', 'total_precipitation',
                ],
                'year': year,
                'month': month,
                'day': [
                        day,
                 ],
                'time': [
                       hour,
                 ],
                 'format': 'netcdf',

                 'area': [
                       -24.08, -64.6, -30.52,
                        -58.09,
                  ],
      },
      f'.ERA5hourly_data_on_single_levels{year}{month}{day}.{i.strftime("%H")}.nc')

Yes. Now that you have shortlisted a smaller region, you could also query it for a longer time period at a time (say a month). This may be more efficient as well in terms of run-time and storage.

Thank you so much for sharing this code. But, I notice the code was only giving data for 1 hour in 1 day and in the last month of the last year I defined. How can I make it download data for all the years e.g 1979-2020?

Thank you for your help!

Hello, it could be a problem of the indentation. I would need to see how the blocks are indented in your code. I leave here a photo of my code so you can see how it is indented

Hello, thank you so much for your response. Please, see my code. I use your code and modified it a little for my task.

Please, help check what I am doing wrong.

Thank you.

I don't really understand how decorators are used in python (lines 28 and 29 of your code) or what is the purpose of defining your application function. but I understand that since lines 28 and 29 are not indented, the function (application) is executed only once with the final values of the variables (year, month, day, hour). You should try indenting lines 28 and 29, or without the application function (putting all the lines with the same indentation after the for) as I show in the following photo (although I couldn't run it because I don't know how you have defined ct)

Thank you for your reply. Please, see my code below:

import numpy as np
from datetime import datetime, timedelta
import cdstoolbox as ct
import certifi
import urllib3

http = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED',
ca_certs=certifi.where()
)

Date_ini = datetime(1979, 1, 1)
Date_fin = datetime(1979, 1, 2)
array_horary = np.arange(Date_ini, Date_fin, timedelta(hours = 1)).astype(datetime)
print(array_horary)

for i in array_horary:
     year = i.strftime('%Y')
     month = i.strftime('%m')
     day = i.strftime('%d')
     hour = i.strftime('%H:%M')
     print(year,month,day,hour,i.strftime('%H'))
     if year < '1979':
         product = 'reanalysis-era5-pressure-levels-preliminary-back-extension'
     else:
         product = 'reanalysis-era5-pressure-levels'
     @ct.application(title='Compute Effective Wind Shear')
     @ct.output.download()
     @ct.output.download()
     def application():

   

      u, v = ct.catalogue.retrieve(
      product,
      {
            'product_type': 'reanalysis',
            'variable': [
                  'u_component_of_wind', 'v_component_of_wind',
             ],
             'pressure_level': [
                 '500', '550', '600',
                 '650', '700', '750',
                 '775', '800', '825',
                 '850', '875', '900',
                 '925', '950', '975',
                 '1000',
            ],
            'year': year,
            'month': month,
            'day': day,
            'time': hour,

            'area': [
                -10, 110, -45,
                155,
            ],
     },
     )

     return u(f'{year}{month}{day}.{i.strftime("%H")}'), v(f'{year}{month}{day}.{i.strftime("%H")}')

Thank you for your reply. c.retrieve does not work when I tried it in the cds toolbox editor. But, I have now tried the code below from my linux environment and the request has been in queue since:

import numpy as np
from datetime import datetime, timedelta
#import cdstoolbox as ct
import cdsapi
import certifi
import urllib3
http = urllib3.PoolManager(
cert_reqs='CERT_REQUIRED',
ca_certs=certifi.where()
)
c = cdsapi.Client()

Date_ini = datetime(1950, 1, 1)
Date_fin = datetime(2021, 10, 31)
array_horary = np.arange(Date_ini, Date_fin, timedelta(hours = 1)).astype(datetime)

for i in array_horary:
year = i.strftime('%Y')
month = i.strftime('%m')
day = i.strftime('%d')
hour = i.strftime('%H:%M')
print(year,month,day,hour,i.strftime('%H'))
if year < '1979':
product = 'reanalysis-era5-pressure-levels-preliminary-back-extension'
else:
product = 'reanalysis-era5-pressure-levels'

# RETRIEVE WIND COMPONENTS AT 500hPa

c.retrieve(
product,
{
'product_type': 'reanalysis',
'variable': [
'u_component_of_wind', 'v_component_of_wind',
],
'pressure_level': [
'500', '550', '600',
'650', '700', '750',
'775', '800', '825',
'850', '875', '900',
'925', '950', '975',
'1000',
],
'year': year,
'month': month,
'day': [
day,
],
'time': [
hour,
],
'format': 'netcdf',
'area': [
-10, 110, -45,
155,
],
},
f'.ERA5hourly_data_on_pressure_levels{year}{month}{day}.{i.strftime("%H")}.nc')

I also tried your suggestions in the cds toolbox editor but still didn't work. So, I am now using the c.retrieve code above in my linux environment but, it's been in queue since. 

I never worked with the toolbox request, since September I am downloading the hourly data from reanalysis-era5-single-levels
and I still can't finish the download. with the same code sometimes it goes fast and sometimes it takes too long. Now I was seeing the differences in the code of the api request and the toolbox request, and I think that:
@ ct.application (title = 'Compute Effective Wind Shear')
@ ct.output.download ()
determine the name of the file and you should put an f '{year} {month} {day}. {hour}' to prevent the file from being overwritten.

Hello, thank you for your suggestions. I have tried it and still didn't work. But, the other cdsapi code ran and gave the error below:

File "/workflows/internal/code/47e2e5d178811b02d28aecc62c230038d1d23ac52d76ccbb67739c49/workflows.py", line 6, in <module>
import cdsapi
ModuleNotFoundError: No module named 'cdsapi'

Thank you so much for your help! Now, it's running in my Linux environment. I am using the cdsapi that you used.