Hi!
I want to retrieve data from the a CDS dataset (more particularly this one: https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-agrometeorological-indicators?tab=overview) using the API, throught a jupyter notebook in python 3. I want to retrieve quite a lot of parameters, so this exceeds the 100 elements limit for each request. Thus I need to do for loops for each half month of my wanted time period, as suggested by the CDS efficiency tips.
Data gets downloaded but after some requests (honestly it seems quite random, I couldn't say how many), I get the following error :
2021-08-13 14:20:26,161 INFO Welcome to the CDS 2021-08-13 14:20:26,161 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/sis-agrometeorological-indicators 2021-08-13 14:20:26,162 DEBUG POST https://cds.climate.copernicus.eu/api/v2/resources/sis-agrometeorological-indicators {"variable": "2m_relative_humidity", "year": ["2020"], "month": ["09"], "day": ["16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30"], "time": ["18_00"], "format": "zip", "area": [-6.4, 34.3, -6.5, 34.4], "nocache": "123"} 2021-08-13 14:20:26,188 DEBUG https://cds.climate.copernicus.eu:443 "POST /api/v2/resources/sis-agrometeorological-indicators HTTP/1.1" 202 124 2021-08-13 14:20:26,190 DEBUG REPLY {'state': 'queued', 'request_id': '175cf049-55ed-4775-94df-aae46b8ed2e7', 'specific_metadata_json': {'top_request_origin': 'api'}} 2021-08-13 14:20:26,190 INFO Request is queued 2021-08-13 14:20:26,191 DEBUG Request ID is 175cf049-55ed-4775-94df-aae46b8ed2e7, sleep 1 2021-08-13 14:20:27,193 DEBUG GET https://cds.climate.copernicus.eu/api/v2/tasks/175cf049-55ed-4775-94df-aae46b8ed2e7 2021-08-13 14:20:27,216 DEBUG https://cds.climate.copernicus.eu:443 "GET /api/v2/tasks/175cf049-55ed-4775-94df-aae46b8ed2e7 HTTP/1.1" 200 None 2021-08-13 14:20:27,217 DEBUG REPLY {'state': 'failed', 'request_id': '175cf049-55ed-4775-94df-aae46b8ed2e7', 'error': {'message': 'an internal error occurred processing your request', 'url': 'http://copernicus-climate.eu/exc/exception', 'reason': "[Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'", 'context': {'traceback': 'Traceback (most recent call last):\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/adaptorrequesthandler.py", line 66, in handle_request\n return super().handle_request(cdsinf, data_request, self.config)\n File "/opt/cds/cdsinf/python/lib/cdsinf/runner/requesthandler.py", line 113, in handle_request\n return handler(cdsinf, request, config)\n File "/opt/cds/adaptor/cdshandlers/archivecombiner/handler.py", line 63, in handle_retrieve\n config)\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 103, in combine\n di)\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 189, in process_one_input_archive\n di, extractor_class)\n File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 170, in dataitem_to_extractor\n allocation = di.as_file(use_server_filename=self.use_server_filenames)\n File "/opt/cds/cdsinf/python/lib/cdsinf/brokerclient.py", line 172, in as_file\n size=self.size) as allocation:\n File "/opt/cds/cdsinf/python/lib/cdsinf/cdsinf.py", line 564, in get_cache_file\n return self.cachemanager.get_file(self, **kwargs)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 80, in get_file\n return self._allocate(cdsinf, kwargs, ReusableCacheAllocationFile)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 104, in _allocate\n cdsinf, hints, alloc_class, cache, entryname)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 136, in _allocate_in_cache\n directory=alloc_class._dir)\n File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 756, in create_entry\n os.makedirs(dirname, exist_ok=not directory or not new)\n File "/usr/lib64/python3.6/os.py", line 220, in makedirs\n mkdir(name, mode)\nPermissionError: [Errno 13] Permission denied: \'/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h\'\n'}, 'permanent': True, 'who': 'server'}, 'sent_to_rmq_at': '2021-08-13T14:20:26.533Z', 'specific_metadata_json': {'top_request_origin': 'api'}} 2021-08-13 14:20:27,218 INFO Request is failed 2021-08-13 14:20:27,218 ERROR Message: an internal error occurred processing your request 2021-08-13 14:20:27,219 ERROR Reason: [Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h' 2021-08-13 14:20:27,220 ERROR Traceback (most recent call last): 2021-08-13 14:20:27,220 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/adaptorrequesthandler.py", line 66, in handle_request 2021-08-13 14:20:27,221 ERROR return super().handle_request(cdsinf, data_request, self.config) 2021-08-13 14:20:27,222 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/runner/requesthandler.py", line 113, in handle_request 2021-08-13 14:20:27,222 ERROR return handler(cdsinf, request, config) 2021-08-13 14:20:27,223 ERROR File "/opt/cds/adaptor/cdshandlers/archivecombiner/handler.py", line 63, in handle_retrieve 2021-08-13 14:20:27,223 ERROR config) 2021-08-13 14:20:27,224 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 103, in combine 2021-08-13 14:20:27,225 ERROR di) 2021-08-13 14:20:27,225 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 189, in process_one_input_archive 2021-08-13 14:20:27,226 ERROR di, extractor_class) 2021-08-13 14:20:27,229 ERROR File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 170, in dataitem_to_extractor 2021-08-13 14:20:27,229 ERROR allocation = di.as_file(use_server_filename=self.use_server_filenames) 2021-08-13 14:20:27,230 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/brokerclient.py", line 172, in as_file 2021-08-13 14:20:27,230 ERROR size=self.size) as allocation: 2021-08-13 14:20:27,231 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cdsinf.py", line 564, in get_cache_file 2021-08-13 14:20:27,232 ERROR return self.cachemanager.get_file(self, **kwargs) 2021-08-13 14:20:27,232 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 80, in get_file 2021-08-13 14:20:27,233 ERROR return self._allocate(cdsinf, kwargs, ReusableCacheAllocationFile) 2021-08-13 14:20:27,233 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 104, in _allocate 2021-08-13 14:20:27,234 ERROR cdsinf, hints, alloc_class, cache, entryname) 2021-08-13 14:20:27,235 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 136, in _allocate_in_cache 2021-08-13 14:20:27,235 ERROR directory=alloc_class._dir) 2021-08-13 14:20:27,236 ERROR File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 756, in create_entry 2021-08-13 14:20:27,236 ERROR os.makedirs(dirname, exist_ok=not directory or not new) 2021-08-13 14:20:27,237 ERROR File "/usr/lib64/python3.6/os.py", line 220, in makedirs 2021-08-13 14:20:27,240 ERROR mkdir(name, mode) 2021-08-13 14:20:27,240 ERROR PermissionError: [Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'
I'll add a code snippet below, but know that I've already read this topic and added a
|
but I still have the error. In order to see clearer if it is a cache problem or not, I also added
|
as suggested in this topic. Since the request is failed, it unfortunately does not display the 'location' alongside the 'request_id'. Thus I can't tell if it's from the cache but it would seem so when looking at the error message (PermissionError). For previous CDS requests, the request ID doesn't match the one in the location field, which means the result is provided by a cached one. Thus, the nocache argument fails. How could this be? See below to get a code snippet.
Other than that, why would I have a PermissionDenied error for cached data when the CDS frequently give cached data for specific requests?
Thank you for reading!
9 Comments
Kevin Marsh
Hi Lea,
Are you still seeing this error message when you run the script?
Thanks,
Kevin
Léa Tresch
Hi Kevin,
Thank you for your answer.
I just ran my code and got a similar error:
This time the permission error is on dataset Relative-Humidity-2m-15h whereas when I started the topic it was on dataset Relative-Humidity-2m-18h. Other than that, it seems to be the same.
Kind regards,
Léa
Kevin Marsh
Hi Léa,
The issue is under investigation; I will let you know when I have more information,
Thanks,
Kevin
Léa Tresch
Hi Kevin,
Thanks for letting me know! I am looking forward to hearing from you.
Kind regards,
Léa
David Orme
I've also had a similar issue. I suspect I'm asking for too large a request in one go, but the error seems to be the same caching issue. This is the static WFDE5 v.2.0 dataset, so a cached version shouldn't be a problem?
Kevin Marsh
Hi Léa, David,
I think the CDS cache "permission" issue may be resolved now - can you try your requests again, please?
Thanks,
Kevin
David Orme
Seems to have fixed it - my Python cdsapi script has now run cleanly. Many thanks for the quick reply and fix.
Léa Tresch
Hi Kevin,
Same here! My code ran smoothly. Thank you very much for everything!
Léa
Kevin Marsh
Hi Léa, David,
Glad we could help!
Thanks,
Kevin