1. Introduction
In a small number of cases, users may experience some issues when requesting data from the CDS API. The most common issues are documented in the "Troubleshooting" section below.
Please note that CDS API requests are only shown on the CDS "Your Requests" web page while they are running. Once they have completed, they are removed from the list.
However, as a first step we strongly advise that if you experience any of these issues you first:
- Use Python3
- Install the latest version of the CDS API
And retry your request. Only if the issues persist should you then try the solutions provided below.
Please note that a given Error Message can be triggered by an number of reasons. The most common Error Messages are shown below:
| Error Message |
| connection error |
| an internal error occurred processing your request |
| no data is available within your requested subset |
| the request you have submitted is not valid |
| Client has not agreed to the required terms and conditions. |
| Invalid 'method': cds.catalogue.retrieve |
| Invalid 'method': cds |
Examples of some of the reasons (and solutions) for these errors are given below.
2. Troubleshooting
2.1. Timeout error
Symptom: Sometimes a "Timeout Error" is seen.
WARNING Recovering from connection error [HTTPSConnectionPool(host='cds.climate.copernicus.eu', port=443): Max retries exceeded with url: /api/v2/tasks/7b4e735b-5a79-4913-bbcb-cebc0c65b420 (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x7f81056e8940>, 'Connection to cds.climate.copernicus.eu timed out. (connect timeout=60)'))], attempts 12 of 500
Diagnosis: This can be due to network issues causing delays in the communication between the user's system and the CDS.
Solution: A possible solution is to add the "timeout=600" (or longer) option for the timeout duration in seconds, to your CDS API request :
c = cdsapi.Client(timeout=600,quiet=False,debug=True)
The options "quiet=False" mean that the output is verbose, and "debug=True" will produce additional debugging information which may help when diagnosing errors.
2.2. Proxy issue
Symptom: Users may also see a 'connection error' message if there is an issue with the 'proxy' server/settings on their system.
WARNING Recovering from connection error [HTTPSConnectionPool(host='cds.climate.copernicus.eu', port=443): Max retries exceeded with url: /api/v2/resources/cems-fire-historical (Caused by ProxyError('Cannot connect to proxy.', timeout('_ssl.c:835: The handshake operation timed out',)))], attempts 0 of 500
Diagnosis: If changing the 'timeout' option in the request (as described above) does not resolve the issue, then the required ports (https, 443) may not be open on the firewall on the user's system.
Solution: Users should then check that the ports required by the CDS are open on the firewall on their system (i.e. https, 443).
2.3. CDS cache issue : Missing Data
Symptom: Incomplete data are returned from the CDS for a request (e.g. fewer timesteps/variables than expected)
Diagnosis: A user request is stored in the ADS/CDS cache, along with the returned data. If a user submits exactly the same request a short time later (e.g. within a day or so), then the same data are returned from the cache i.e. they are not re-extracted from the original dataset. If this initial request only returned some of the data requested, then subsequent identical requests will return this same data, so it will appear that data are missing from the ADS/CDS when they are in fact available.
Solutions: You can either:
a) try modifying your request so that it is different (such as by adding a variable or date, or following this example: How to avoid the CDS cache issue), or
b) wait a few days for the original requests and data to be cleared from the CDS cache.
2.4. Request Failed: Request is too large
Symptoms: A "maximum request size" error message is seen when submitting a request.
The request you have submitted is not valid Reason: Maximum retrieve size 175.00 G reached. Please split your request. - Mars server task finished in error - Double buffer error: Assertion failed: length == buffers_[i].length_ in run, line 273 of /home/cds/git/mars-server/eckit/src/eckit/io/DblBuffer.cc(RemoteException from Connector[marsmvr-0001:9701]) [mars] - Error code is -2 - Request failed - Some errors reported
Diagnosis: Each CDS dataset has a maximum limit for the volume of data which can be retrieved. See Climate Data Store (CDS) documentation
Solution: Request the data in smaller sections (e.g. less than 10GB).
2.5. Request Failed: Request has too many items
Symptoms:
% (reply["error"].get("message"), reply["error"].get("reason"))
Exception: the request you have submitted is not valid. Request too large. Requesting 368184 items, limit is 120000.
Diagnosis: Each dataset will have a maximum limit for the number of "items" which can be retrieved from a single request. See Climate Data Store (CDS) documentation
Solution: Reduce the size of your request, so that the "item" size is below the limit shown in the error message.
2.6. Request Failed: 'Not Authorised' error message
Symptoms: When a CDS API request is submitted, the following error message is seen.
raise Exception(error) Exception: <html> <head><title>401 Authorization Required</title></head> <body> <center><h1>401 Authorization Required</h1></center> <hr><center>nginx/1.16.1</center> </body> </html>
Diagnosis: If the wrong/incorrect credentials are used in the users' .cdsapirc file in their home directory, the error message may be seen.
Solution: User will need to use the correct CDS API credentials (note that these are different from their ADS credentials).
2.7. Request Failed: 'No licence agreement' error message
Symptoms: When a CDS API request is submitted, the following error message is seen.
Exception: Client has not agreed to the required terms and conditions.. To access this resource, you first need to accept the terms of 'Licence to use Copernicus Products' at https://ads.atmosphere.copernicus.eu/cdsapp/#!/terms/licence-to-use-copernicus-products
Diagnosis: Each dataset will have one or more licence agreements associated with it. Users will need to have previously agreed to the licence(s) using the relevant ADS/CDS dataset "Download Data" web form before they can download data.
Solution: User will need to agree to the relevant licence(s) on the ADS/CDS web form.
2.8. Request Failed: Conversion from GRIB to netCDF
Symptom: Conversion from GRIB to netCDF fails for one of a number of reasons.
1) combination of variables i.e. ensemble members data and reanalysis data
The request you have submitted is not valid Reason: different resolution - of lat=161, long=261
2) Size of request (can include lack of space in cache and individual netcdf variable larger than 4G for the netCDF3 format currently used)
Exception: the request you have submitted is not valid. One or more variable sizes violate format constraints.
Diagnosis: This is often seen because there is not enough space in the ADS/CDS cache to do the conversion of the data. This is currently seen for requests larger than around 10GB, although certain combinations of variables can also cause it to fail. In addition, the "experimental" grib_to_netcdf conversion currently used may also cause other errors.
Solution:
There are several possible ways to work around this:
- Reduce size: request the data in smaller sections (less than 10GB),
- Reduce size: request individual variables in 1 per request
- Convert locally: request the data in GRIB format and convert the data to netCDF on your local system.
2.9. Request Failed: Request cancelled by system admin
Symptoms: Request is unexpectedly cancelled.
Exception: the request has been cancelled by CDS admin.
Diagnosis: Sometimes in order to solve internal system issues, the CDS admin will force the cancellation of running requests.
Solution: The user must resubmit the cancelled requests.
2.10. Request Failed: Request disappeared from running requests list
Symptoms: Request has failed unexpectedly.
Request disappeared from running requests list
Diagnosis: Sometimes an internal CDS issue will cause a request to fail.
Solution: The user must resubmit the failed request.
2.11. Request Failed: Keywords ambiguous/not available
Symptom: Request fails with an error message about the keywords used, or a missing mandatory field.
Exception: the request you have submitted is not valid. Ambiguous parameter: days could be DATE or DATABASE - No request.
Diagnosis: The keywords used in a CDS API must be used exactly as specified on the ADS/CDS dataset web form or in the relevant documentation.
Solution: Create a valid request on the ADS/CDS dataset web form and click "Show API request". Read the relevant documentation.
2.12. Request Failed: MARS Connection ("MARS Tunnel") Issue
Symptom: The CDS API request fails as the connection to the MARS archive has issues.
raise exception(call, proc.returncode, output)\nhome.cds.cdsservices.services.mars_constrained.__init__.py.mars.exceptions.MarsException: Suspected MARS tunnel problem\n'}.
Diagnosis: Some ADS/CDS datasets are stored in the MARS archives. Sometimes the connection between the ADS/CDS and these archives has issues, resulting in failed requests.
Solution: Check for announcements on the Copernicus User Support forum, and the ADS/CDS home pages. Retry you requests after some time later, as the issue may have been resolved.
2.13. Data not available: Invalid request for ADS/CDS requests
Symptom: The CDS API request returns no data.
Exception: no data is available within your requested subset. Request returned no data.
Diagnosis: The CDS API currently allows users to submit incomplete/invalid requests to the ADS/CDS. In this case, no data will be returned. Currently no 'validity checking' of request syntax is performed before they are processed, so this error may not be seen until the request is submitted.
Solution: Use the ADS/CDS dataset web form to check that your requests are valid i.e. that you are using the correct dates, variable names etc.
