This page will also be used to communicate important information during the DHS Move. To get notified of any changes to this page, please log in and click on 
in the upper right.
What to expect during the DHS Move
At the start of the DHS Move all access to the DHS infrastructure located in Reading will be closed and instead essential data necessary to continue all critical activities during the move will be provided by a temporary DHS in Bologna. Archive/Store operations will not be impacted and will continue as normal. Data archived during this period will be retrievable as normal.
Five days after the start (week commencing 12th September), the HPSS, MARS and ECFS metadata will be transferred to Bologna. The services will be re-started using such metadata, which will contain pointers to all data, but not the data itself, which will gradually be moved from Reading to Bologna.
At the start of the DHS Move, all active/queued MARS retrieve requests and ECFS get operations on the DHS will fail. This is to switch MARS & ECFS to the temporary-DHS configuration. After such failure, we encourage users to re-run requests/scripts once, to experience the expected behaviour during the DHS Move:
- Requests for essential/recent data should be satisfied from the temporary DHS.
- Requests for historical data should fail with an explanatory message. Users will have to wait until the data is loaded in the DHS In Bologna.
We expect that data will start arriving in Bologna the week commencing 24th September. This will be the first servers and disk sub-systems containing popular data that is already cached in the long-term cache in Reading. That cache is being preserved.
Obviously some of the dates below are tentative. We will update users as and when we have more certainty about progress.
Timetable of System Session starting on 8 September
| Date | Time (UTC) | Event | Impact | User action |
|---|---|---|---|---|
8 Sep | 09:30 | Start of DHS Move Change configuration of all services using MARS/ECFS to point to the temporary DHS in Bologna. | Check service status | |
| 09:30 - 10:30 | Drain all MARS/ECFS activity in DHS infrastructure in Reading | All active/queued MARS retrieve requests and ECFS get operations on the DHS in Reading will fail. | Please re-run to test behaviour expected during DHS Move. | |
w/c 12 Sep | MARS/ECFS/HPSS metadata transfer to Bologna. Start DHS infrastructure in Bologna (DHS-Blank) | Short interruption. Stop writing new data into temporary-DHS and start archive/store to production MARS/ECFS in Bologna. | Check service status | |
| from 14 Sep | Physical relocation starts: tapes, tape libraries, disks, servers start being shipped to Bologna | |||
| w/c 3 Oct | First data started appearing in Bologna:
| Short interruptions | The type of data becoming available:
Try resuming work for the above. Failures may occur as not all streams/types/levtypes are available. | |
| w/c 10 Oct |
Additional 1.5 PiB disk cache of popular data | Short interruptions | The type of data becoming available:
Try resuming work for the above. Failures may occur as not all class/streams/types/levtypes are available. | |
| w/c 24 Oct | The first of five tape libraries shipped from Reading to Bologna was put in production on . This provides access to additional 4,000 tape cartridges, mostly operational data. | The type of data becoming available:
| ||
| w/c 31 Oct | Three additional tape libraries were put in production this week. | System sessions on tape libraries required | The type of data becoming available:
Users are encouraged to resume their work, noting that some RD, Re-Analysis and TIGGE data may still be unavailable. | |
| w/c 7 Nov | One tape library left to complete the DHS Move | System sessions on tape libraries required | Check service status | |
| End of DHS Move | Access to full MARS/ECFS | Check service status |
Log output samples of execution at various stages
When draining MARS activity in Reading
mars - INFO - 20220331.170650 - Server task is 809 [marsod-core] mars - ERROR - 20220331.170650 - Mars server task finished in error mars - ERROR - 20220331.170650 - AccessError: DHS infrastructure in Reading disabled. Please, re-run your request. If you see this message again for the same request, please, raise a ticket [marsod-core] mars - ERROR - 20220331.170650 - Error code is -2
When requesting MARS data available in temporary DHS in Bologna
mars - INFO - 20220331.165631 - Server task is 839 [temporary-dhs-prod] mars - INFO - 20220331.165631 - Request cost: N fields, BBBBBBBB Mbytes online, nodes: mvr007 [temporary-dhs-prod] mars - INFO - 20220331.165631 - Transfering BBBBBBBBB bytes mars - INFO - 20220331.165631 - N fields retrieved from 'temporary-dhs'
When requesting MARS data not available during the DHS Move
mars - INFO - 20220331.144749 - Server task is 614 [bologna-marsod-blank] mars - ERROR - 20220331.144749 - Mars server task finished in error mars - ERROR - 20220331.144749 - AccessError: This data exists but will be unavailable during the DHS Move. For more information see https://confluence.ecmwf.int/x/jSKADQ [bologna-marsod-blank] mars - ERROR - 20220331.144749 - Error code is -2
When listing ECFS data available in temporary DHS in Bologna
Listing will appear in a similar manner as usual.
els -l ec:/uid/0001_nrt_1d_20210124_20210124_icemod.nc.gz -rw-r----- 1 uid ma 60629940 Feb 9 2021 0001_nrt_1d_20210124_20210124_icemod.nc.gz
When listing ECFS data not available during the DHS Move
Listing starting with 'o' or 'O' indicates that data are offline, i.e. on tape in Reading only and not available during the DHS move.
-> els -l ec:/uid/training_* orw-r--r-- 1 uid us 7099904 Feb 5 2007 training_2004.ecfs Orw-r--r-- 1 uid us 3564032 Jul 11 2003 training_jrc.ecfs Orw-r--r-- 1 uid us 4575744 Nov 11 2002 training_jrc_orig.ecfs orw-r----- 1 uid us 13733376 Feb 21 2011 training_material_2009.ecfs orw-r----- 1 uid us 14510080 Feb 21 2011 training_material_2010.ecfs
When accessing ECFS data not available during the DHS Move
-> ecp ec:testing-file ./ ecp: copying from ec:/uid/testing-file: Data offline for file: /uid/testing-file
Available tools
MARS
- Data available during the DHS Move is listed in the Catalogue of the temporary DHS.
- To monitor the status of your requests during the DHS Move, please, visit the Temporary DHS server activity.
ECFS
ECFS listings done during the DHS Move will show files offline i.e. on tape in Reading only and therefore not available, with 'o' or 'O' in the first column (instead of '-' or 'b'):
-> els -l ec:/uid/training_* orw-r--r-- 1 uid us 7099904 Feb 5 2007 training_2004.ecfs Orw-r--r-- 1 uid us 3564032 Jul 11 2003 training_jrc.ecfs Orw-r--r-- 1 uid us 4575744 Nov 11 2002 training_jrc_orig.ecfs orw-r----- 1 uid us 13733376 Feb 21 2011 training_material_2009.ecfs orw-r----- 1 uid us 14510080 Feb 21 2011 training_material_2010.ecfs
How to report a problem
If your activity is critical and we have missed to identify it as such, please raise a support ticket, see below, and we will look at your specific requirements.
If your request/command does not behave as described above, please, provide as much information as possible so analysts can investigate/reproduce the problem, for example:
- Service/Tool you are using (mars client, ecfs, metview, web api, verify, etc...)
- Version of the tool
- Host, such as, ecgb, ATOS, cca, your workstation, ... and the environment under which it runs, interactive, batch, ...
- Log output produced by MARS/ECFS, including the request
Please report all issues via our Support Portal, mentioning "dhs move" in the title of your computing problem ticket.