Permission denied to CDS cache folder

Hi!

I want to retrieve data from the a CDS dataset (more particularly this one: https://cds.climate.copernicus.eu/cdsapp#!/dataset/sis-agrometeorological-indicators?tab=overview) using the API, throught a jupyter notebook in python 3. I want to retrieve quite a lot of parameters, so this exceeds the 100 elements limit for each request. Thus I need to do for loops for each half month of my wanted time period, as suggested by the CDS efficiency tips.

Data gets downloaded but after some requests (honestly it seems quite random, I couldn't say how many), I get the following error :

2021-08-13 14:20:26,161 INFO Welcome to the CDS
2021-08-13 14:20:26,161 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/sis-agrometeorological-indicators
2021-08-13 14:20:26,162 DEBUG POST https://cds.climate.copernicus.eu/api/v2/resources/sis-agrometeorological-indicators {"variable": "2m_relative_humidity", "year": ["2020"], "month": ["09"], "day": ["16", "17", "18", "19", "20", "21", "22", "23", "24", "25", "26", "27", "28", "29", "30"], "time": ["18_00"], "format": "zip", "area": [-6.4, 34.3, -6.5, 34.4], "nocache": "123"}
2021-08-13 14:20:26,188 DEBUG https://cds.climate.copernicus.eu:443 "POST /api/v2/resources/sis-agrometeorological-indicators HTTP/1.1" 202 124
2021-08-13 14:20:26,190 DEBUG REPLY {'state': 'queued', 'request_id': '175cf049-55ed-4775-94df-aae46b8ed2e7', 'specific_metadata_json': {'top_request_origin': 'api'}}
2021-08-13 14:20:26,190 INFO Request is queued
2021-08-13 14:20:26,191 DEBUG Request ID is 175cf049-55ed-4775-94df-aae46b8ed2e7, sleep 1
2021-08-13 14:20:27,193 DEBUG GET https://cds.climate.copernicus.eu/api/v2/tasks/175cf049-55ed-4775-94df-aae46b8ed2e7
2021-08-13 14:20:27,216 DEBUG https://cds.climate.copernicus.eu:443 "GET /api/v2/tasks/175cf049-55ed-4775-94df-aae46b8ed2e7 HTTP/1.1" 200 None
2021-08-13 14:20:27,217 DEBUG REPLY {'state': 'failed', 'request_id': '175cf049-55ed-4775-94df-aae46b8ed2e7', 'error': {'message': 'an internal error occurred processing your request', 'url': 'http://copernicus-climate.eu/exc/exception', 'reason': "[Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'", 'context': {'traceback': 'Traceback (most recent call last):\n  File "/opt/cds/adaptor/cdshandlers/adaptorlib/adaptorrequesthandler.py", line 66, in handle_request\n    return super().handle_request(cdsinf, data_request, self.config)\n  File "/opt/cds/cdsinf/python/lib/cdsinf/runner/requesthandler.py", line 113, in handle_request\n    return handler(cdsinf, request, config)\n  File "/opt/cds/adaptor/cdshandlers/archivecombiner/handler.py", line 63, in handle_retrieve\n    config)\n  File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 103, in combine\n    di)\n  File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 189, in process_one_input_archive\n    di, extractor_class)\n  File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 170, in dataitem_to_extractor\n    allocation = di.as_file(use_server_filename=self.use_server_filenames)\n  File "/opt/cds/cdsinf/python/lib/cdsinf/brokerclient.py", line 172, in as_file\n    size=self.size) as allocation:\n  File "/opt/cds/cdsinf/python/lib/cdsinf/cdsinf.py", line 564, in get_cache_file\n    return self.cachemanager.get_file(self, **kwargs)\n  File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 80, in get_file\n    return self._allocate(cdsinf, kwargs, ReusableCacheAllocationFile)\n  File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 104, in _allocate\n    cdsinf, hints, alloc_class, cache, entryname)\n  File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 136, in _allocate_in_cache\n    directory=alloc_class._dir)\n  File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 756, in create_entry\n    os.makedirs(dirname, exist_ok=not directory or not new)\n  File "/usr/lib64/python3.6/os.py", line 220, in makedirs\n    mkdir(name, mode)\nPermissionError: [Errno 13] Permission denied: \'/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h\'\n'}, 'permanent': True, 'who': 'server'}, 'sent_to_rmq_at': '2021-08-13T14:20:26.533Z', 'specific_metadata_json': {'top_request_origin': 'api'}}
2021-08-13 14:20:27,218 INFO Request is failed
2021-08-13 14:20:27,218 ERROR Message: an internal error occurred processing your request
2021-08-13 14:20:27,219 ERROR Reason:  [Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'
2021-08-13 14:20:27,220 ERROR   Traceback (most recent call last):
2021-08-13 14:20:27,220 ERROR     File "/opt/cds/adaptor/cdshandlers/adaptorlib/adaptorrequesthandler.py", line 66, in handle_request
2021-08-13 14:20:27,221 ERROR       return super().handle_request(cdsinf, data_request, self.config)
2021-08-13 14:20:27,222 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/runner/requesthandler.py", line 113, in handle_request
2021-08-13 14:20:27,222 ERROR       return handler(cdsinf, request, config)
2021-08-13 14:20:27,223 ERROR     File "/opt/cds/adaptor/cdshandlers/archivecombiner/handler.py", line 63, in handle_retrieve
2021-08-13 14:20:27,223 ERROR       config)
2021-08-13 14:20:27,224 ERROR     File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 103, in combine
2021-08-13 14:20:27,225 ERROR       di)
2021-08-13 14:20:27,225 ERROR     File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 189, in process_one_input_archive
2021-08-13 14:20:27,226 ERROR       di, extractor_class)
2021-08-13 14:20:27,229 ERROR     File "/opt/cds/adaptor/cdshandlers/adaptorlib/url_fetch.py", line 170, in dataitem_to_extractor
2021-08-13 14:20:27,229 ERROR       allocation = di.as_file(use_server_filename=self.use_server_filenames)
2021-08-13 14:20:27,230 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/brokerclient.py", line 172, in as_file
2021-08-13 14:20:27,230 ERROR       size=self.size) as allocation:
2021-08-13 14:20:27,231 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/cdsinf.py", line 564, in get_cache_file
2021-08-13 14:20:27,232 ERROR       return self.cachemanager.get_file(self, **kwargs)
2021-08-13 14:20:27,232 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 80, in get_file
2021-08-13 14:20:27,233 ERROR       return self._allocate(cdsinf, kwargs, ReusableCacheAllocationFile)
2021-08-13 14:20:27,233 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 104, in _allocate
2021-08-13 14:20:27,234 ERROR       cdsinf, hints, alloc_class, cache, entryname)
2021-08-13 14:20:27,235 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 136, in _allocate_in_cache
2021-08-13 14:20:27,235 ERROR       directory=alloc_class._dir)
2021-08-13 14:20:27,236 ERROR     File "/opt/cds/cdsinf/python/lib/cdsinf/cachemanager.py", line 756, in create_entry
2021-08-13 14:20:27,236 ERROR       os.makedirs(dirname, exist_ok=not directory or not new)
2021-08-13 14:20:27,237 ERROR     File "/usr/lib64/python3.6/os.py", line 220, in makedirs
2021-08-13 14:20:27,240 ERROR       mkdir(name, mode)
2021-08-13 14:20:27,240 ERROR   PermissionError: [Errno 13] Permission denied: '/cache/downloads/agera5.copernicus-climate.eu/agera5/2020/Relative-Humidity-2m-18h'

I'll add a code snippet below, but know that I've already read this topic and added a

'nocache': '123'

but I still have the error. In order to see clearer if it is a cache problem or not, I also added

cdsapi.Client(quiet=False, debug=True)

as suggested in this topic. Since the request is failed, it unfortunately does not display the 'location' alongside the 'request_id'. Thus I can't tell if it's from the cache but it would seem so when looking at the error message (PermissionError). For previous CDS requests, the request ID doesn't match the one in the location field, which means the result is provided by a cached one. Thus, the nocache argument fails. How could this be? See below to get a code snippet.

Other than that, why would I have a PermissionDenied error for cached data when the CDS frequently give cached data for specific requests?

Thank you for reading!

years_input = list(df_time['year'].unique()) #this gets a list of years to get data from
cds = cdsapi.Client(quiet=False, debug=True)

for y in range(int(min(years_input)), int(max(years_input)) + 1):
    months_input = list(df_time[df_time['year'] == y]['month'].unique())
    years = [str(y)]
    for m in range(min(months_input), max(months_input) + 1):
        days_input = list(df_time[df_time['month'] == m]['day'].unique())
        months = [(lambda x: x.zfill(2))(str(m))]
        idx_days = int(len(days_input)/2)
        for days in [days_input[:idx_days], days_input[idx_days:]]:
            days = [(lambda x: x.zfill(2))(str(x)) for x in days]

            # each half of month we do a request (otherwise there is the "Request Failed: Request has too many items" error
			# here I only left the max temp request but following this one, we have : mean temp, min temp, solar radiation flux 24h mean, relative humidity for
			#	6AM, 9AM, 12AM, 3PM, 6PM, precipitation 24h mean, wind speed 24h mean
           
            # request example
            cds.retrieve('sis-agrometeorological-indicators', {
                       "variable": [
                            "2m_temperature"
                            ],
                        'statistic': [
                                '24_hour_maximum',
                            ],
                       "product_type": "reanalysis",
                        "year": years,
                        "month": months,
                        "day" : days,
                        "format": "zip",
                        "area": area,
                        'nocache': '123',
                   }, 'era5_tmp.zip')
            with ZipFile('era5_tmp.zip', 'r') as zipObj:
                zipObj.extractall(folder_era)

Page tree

9 Comments

Kevin Marsh

Léa Tresch

Kevin Marsh

Léa Tresch

David Orme

Kevin Marsh

David Orme

Léa Tresch

Kevin Marsh