- Created by Yu Zhu on Dec 21, 2022
Hi. I am trying to download the daily ERA5-Land dataset by calling the application of daily statistics through the following scripts. It worked fine until yesterday. The same dataset could be downloaded smoothly.
However, using the same scripts today with slight revisions in variable names, etc., the NameError of "name 'reduce' is not defined" occurred (see following NameError). I suspect that this might be because the codes were compiled by python 3.9. While not intend to use python 2, I wonder what I should do to solve this problem and batch download daily statistics.
Scripts for downloading ERA5-Land dataset Expand source
# -*- coding: utf-8 -*- import cdsapi import requests # CDS API script to use CDS service to retrieve daily ERA5* variables and iterate over # all months in the specified years. # Requires: # 1) the CDS API to be installed and working on your system # 2) You have agreed to the ERA5 Licence (via the CDS web page) # 3) Selection of required variable, daily statistic, etc # Output: # 1) separate netCDF file for chosen daily statistic/variable for each month c = cdsapi.Client(timeout=300) # Uncomment years as required years = ['2012'] # Retrieve all months for a given year. months = ['10', '11', '12'] # For valid keywords, see Table 2 of: # https://datastore.copernicus-climate.eu/documents/app-c3s-daily-era5-statistics/C3S_Application-Documentation_ERA5-daily-statistics-v2.pdf # select your variable; name must be a valid ERA5 CDS API name. var = "volumetric_soil_water_layer_2" # Select the required statistic, valid names given in link above stat = "daily_mean" # Loop over years and months for yr in years: for mn in months: result = c.service( "tool.toolbox.orchestrator.workflow", params={ "realm": "c3s", "project": "app-c3s-daily-era5-statistics", "version": "master", "kwargs": { "dataset": "reanalysis-era5-land", "product_type": "reanalysis", "variable": var, "statistic": stat, "year": yr, "month": mn, "time_zone": "UTC+00:0", "frequency": "1-hourly", "grid": "0.1/0.1", "area": {"lat": [23, 90], "lon": [-180, 180]} # # Users can change the output grid resolution and selected area # # "grid": "1.0/1.0", # "area":{"lat": [10, 60], "lon": [65, 140]} }, "workflow_name": "application" }) # set name of output file for each month (statistic, variable, year, month file_name = "D:/%s/" %(var) + "ERA5_Land_" + stat + "_" + var + "_" + yr + "_" + mn + ".nc" location=result[0]['location'] res = requests.get(location, stream = True) print("Writing data to " + file_name) with open(file_name,'wb') as fh: for r in res.iter_content(chunk_size = 1024): fh.write(r) fh.close()
NameError: name 'reduce' is not defined Expand source
D:\JUNO\UCAS\JunoOwnWork\PermafrostPhenology\Codes\Juno\DownloadClimaticData\batch_download_scripts>C:/Users/AERO/Anaconda3/envs/AcaPheno/python.exe D:/JUNO/s/ERA5-Land_daily_download_vswl2_aero_5_2005.py 2022-12-21 21:24:15,577 INFO Welcome to the CDS 2022-12-21 21:24:15,577 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/tasks/services/tool/toolbox/orchestrator/workflow/clientid-34b6c140a8134ad282f83ae52e0bdd50 2022-12-21 21:24:16,169 INFO Request is queued 2022-12-21 21:24:17,525 INFO Request is running 2022-12-21 21:33:06,196 INFO Request is failed 2022-12-21 21:36:11,011 ERROR Message: 2022-12-21 21:36:11,017 ERROR Reason: Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context) File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow results = workflow_bare_func(**kwargs) File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 774, in application [daily_stat_concat, daily_stat], dim='lon' File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 557, in concat non_common_coords = list(reduce(lambda x, y: set(x)^set(y), coords)) NameError: name 'reduce' is not defined 2022-12-21 21:36:11,018 ERROR Traceback (most recent call last): 2022-12-21 21:36:11,019 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 59, in handle_request 2022-12-21 21:36:11,019 ERROR result = cached(context.method, proc, context, context.args, context.kwargs) 2022-12-21 21:36:11,019 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached 2022-12-21 21:36:11,020 ERROR result = proc(context, *context.args, **context.kwargs) 2022-12-21 21:36:11,020 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 124, in __call__ 2022-12-21 21:36:11,021 ERROR return p(*args, **kwargs) 2022-12-21 21:36:11,021 ERROR File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 60, in __call__ 2022-12-21 21:36:11,022 ERROR return self.proc(context, *args, **kwargs) 2022-12-21 21:36:11,022 ERROR File "/home/cds/cdsservices/services/workflow.py", line 35, in execute 2022-12-21 21:36:11,023 ERROR raise exceptions.CDSException(True, True, logging + traceback, '', uri) 2022-12-21 21:36:11,024 ERROR cdsclient.exceptions.CDSException: Traceback (most recent call last): 2022-12-21 21:36:11,024 ERROR File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall 2022-12-21 21:36:11,024 ERROR resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context) 2022-12-21 21:36:11,025 ERROR File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow 2022-12-21 21:36:11,025 ERROR results = workflow_bare_func(**kwargs) 2022-12-21 21:36:11,026 ERROR File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 774, in application 2022-12-21 21:36:11,026 ERROR [daily_stat_concat, daily_stat], dim='lon' 2022-12-21 21:36:11,027 ERROR File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 557, in concat 2022-12-21 21:36:11,027 ERROR non_common_coords = list(reduce(lambda x, y: set(x)^set(y), coords)) 2022-12-21 21:36:11,027 ERROR NameError: name 'reduce' is not defined Traceback (most recent call last): File "D:\JUNO\ERA5-Land_daily_download_vswl2_aero_5_2005.py", line 47, in <module> File "C:\Users\AERO\Anaconda3\envs\AcaPheno\lib\site-packages\cdsapi\api.py", line 366, in service result = self._api( File "C:\Users\AERO\Anaconda3\envs\AcaPheno\lib\site-packages\cdsapi\api.py", line 506, in _api raise Exception( Exception: . Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/jsonrequest/requests.py", line 71, in jsonrequestcall resp = coding.encode(req.callable(*req.args, **req.kwargs), register=encoders, **context) File "/usr/local/lib/python3.6/dist-packages/cdsworkflows/submit_workflow.py", line 55, in submit_workflow results = workflow_bare_func(**kwargs) File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 774, in application [daily_stat_concat, daily_stat], dim='lon' File "/workflows/c3s/app-c3s-daily-era5-statistics/master/workflows.py", line 557, in concat non_common_coords = list(reduce(lambda x, y: set(x)^set(y), coords)) NameError: name 'reduce' is not defined
12 Comments
One W
Please, have you solved the problems?
leo Aries
I also meet the same problem. Have you solved the problems?
I just the API scripts listed in Retrieve daily ERA5/ERA5-Land data using the CDS API
It's ok when I downloaded "2m_temperature“
But when I want to download ”surface_net_solar_radiation“ , I meet the same error
as below:
" NameError: name 'reduce' is not defined "
Michela Giusti
Hi,
please be aware that not all ERA5-Land variables are available for download in the Daily statistics app.
I suggest to check on the app for the list.
Thanks
leo Aries
Thank you for the reply! I just want to download "surface_net_solar_radiation" as shown in Tab2 in
https://datastore.copernicus-climate.eu/documents/app-c3s-daily-era5-statistics/C3S_Application-Documentation_ERA5-daily-statistics-v2.pdf
Is the list in this Tab2 not the list mentioned by you ?
Yu Zhu
Hi,
I am pretty sure that this problem is not strictly related to the variable we choose, as the same error occured when downloading '2m_temperature' for me. I tested the source code for application of "daily statistics calculated from ERA5 data" in Toolbox, and met the totally same NameError: name 'reduce' is not defined.
This error can be simple resolved in Toolbox by adding 'from functools import reduce' at the start of the script. To make it possible for bulk access, I wonder if it is possible for the ECMWF supportors to revise the source code?
One W
I used the cdstoolbox on the webpage of Toolbox Editor directly and got the daily data.
First use ct.catalogue.retrieve() (this can be generated on the download page) to get the data, then use the ct.climate.daily_mean(data) to calculate.
A link with the result will be generated on the left of the page.
Yu Zhu
Thanks for your tips, and the proposed process works fine for me.
However, two drawbacks remain:
1) This can only retrieve daily data one by one, as the new selection limit allows for no more than a month at the time in a single request (Release of ERA5-Land-T data and new selection limit in place);
2) This cannot automatically rename the daily data, making it hard to distinguish each downloaded file.
It will be much appreciated if you have any further suggestions.
One W
1) In fact, I met the same problem and haven't solved;
2) I pip the cdstoolbox-remote locally and do some try but failed. I find that if I want to rename the data before downloading, I should revise the source code.
If renaming the file after downloading, I think maybe you can copy the code to your local environment with cdstoolbox-remote prepared and add a main function that can rename the file instantly after every downloading. And the loop can help do the batch processing. However, I'm not sure is it ok, because I haven't tried.
leo Aries
Thank you for the reply!
So now this problem can be resolved in cdstoolbox (I am not familiar with that), meanwhile can not be resolved in cdsapi?
I've tried to add 'from functools import reduce' in my script using cdsapi,
but it can not work.
Should I try to make another script with cdstoolbox ?
Yu Zhu
I didn't resolve this problem in my local environment (i.e. importing cdsapi). So instead, I used the cdstoolbox. Still waiting for someone to radically resolve this.
I assumed that you add 'from functools import reduce' in your own script, which doesn't call reduce function. Instead, this revision should be applied to the source code of application 'Daily statistics calculated from ERA5 data' (https://cds.climate.copernicus.eu/cdsapp#!/software/app-c3s-daily-era5-statistics?tab=appcode).
To use the cdstoolbox, you can firstly copy the above codes, and then do slight changes to the original code, including:
1) Add 'from functools import reduce';
2) Delete line 7-36;
3) Uncomment line 612 and comment/delete line 613
4) (optional) Delete line 565-610, delete line 630-656. You can keep these lines if you want to choose download file by visible interface like this (https://cds.climate.copernicus.eu/cdsapp#!/software/app-c3s-daily-era5-statistics?tab=app) does.
5) (optional) If you keep lines in step 4), then skip step 5). Otherwise state variables for your download file in line 660-668. Or delete these lines, but state variables inside the function after the bracket.
Other (might be easier) ways to use the cdstoolbox to obtain daily data see the reply of One W , or see the C3S Climate Data Store Toolbox User Forum with keyword 'daily'.
Luis Hernando Gomez
Hi ...
I'm having the same .... "NameError: name 'reduce' is not defined", ... somebody knows who to solve it ??
I added the at the top .. "from functools import reduce", but the error continues ...
Any HELP please, .... Thanks in advance.
I want to calculate daily precipitation ... below the code.
from functools import reduce
import cdsapi
c = cdsapi.Client()
MONTHS = [
"01", "02", "03", "04", "05", "06",
"07", "08", "09", "10", "11", "12"
]
for month in MONTHS:
result = c.service(
"tool.toolbox.orchestrator.workflow",
params={
"realm": "c3s",
"project": "app-c3s-daily-era5-statistics",
"version": "master",
"kwargs": {
"dataset": "reanalysis-era5-single-levels",
"product_type": "reanalysis",
"variable": "total_precipitation",
"pressure_level": "-",
"statistic": "daily_mean",
"year": "2020",
"month": month,
"time_zone": "UTC+00:0",
"frequency": "1-hourly",
"grid": "0.25/0.25",
"area": {"lat": [-20, -10], "lon": [40, 30]}
},
"workflow_name": "application"
})
c.download(result)
Yu Zhu
Hi Luis,
Please see the posting comment on 29th March, 2023 by Kevin Marsh (Retrieve daily ERA5/ERA5-Land data using the CDS API - Copernicus User Support Forum - ECMWF Confluence Wiki).
I succeeded two months ago by simply modifying the line "realm": "c3s" to "realm": "user-apps".