DHS move dry-run

What to expect during a dry-run

Dry-runs are system sessions to simulate the DHS behaviour during the DHS Move. During a dry-run read access to the main DHS infrastructure located in Reading will be closed and instead essential data necessary to continue all critical activities during the move will be provided by a temporary DHS in Bologna. 

Archive/Store operations will not be impacted and will continue as normal. Data archived during this period will be retrievable as normal.

At the start of the system session, all active/queued MARS retrieve requests and ECFS get operations on the DHS will fail. This is to switch MARS & ECFS to the dry-run configuration. After such failure, we encourage users to re-run requests/scripts once, to experience the expected behaviour during the DHS Move:

• Requests for essential/recent data should be satisfied from the temporary DHS.
• Requests for historical data should fail with an explanatory message. We encourage users to wait until the system session has ended before re-running requests for data known to be unavailable.

After the system session is over, user work can resume as normal. 

Timetable of System Session on 28th June - 5th July 2022

Log output samples of execution at various stages

When draining MARS activity in Reading

mars - INFO   - 20220331.170650 - Server task is 809 [marsod-core]
mars - ERROR  - 20220331.170650 - Mars server task finished in error
mars - ERROR  - 20220331.170650 - UserError: DHS infrastructure in Reading disabled. Please, re-run your request. If you see this message again for the same request, please, raise a ticket [marsod-core]
mars - ERROR  - 20220331.170650 - Error code is -2

When requesting MARS data available in temporary DHS in Bologna

mars - INFO   - 20220331.165631 - Server task is 839 [temporary-dhs-prod]
mars - INFO   - 20220331.165631 - Request cost: N fields, BBBBBBBB Mbytes online, nodes: mvr007 [temporary-dhs-prod]
mars - INFO   - 20220331.165631 - Transfering BBBBBBBBB bytes
mars - INFO   - 20220331.165631 - N fields retrieved from 'temporary-dhs'

When requesting MARS data not available during the DHS Move

mars - INFO   - 20220331.144749 - Server task is 614 [bologna-marsod-blank]
mars - ERROR  - 20220331.144749 - Mars server task finished in error
mars - ERROR  - 20220331.144749 - AccessError: This data exists but will be unavailable during the DHS Move. For more information see https://confluence.ecmwf.int/x/jSKADQ [bologna-marsod-blank]
mars - ERROR  - 20220331.144749 - Error code is -2

When listing ECFS data available in temporary DHS in Bologna

Listing will appear in a similar manner as usual.

els -l ec:/uid/0001_nrt_1d_20210124_20210124_icemod.nc.gz
-rw-r-----   1 uid     ma           60629940 Feb  9  2021 0001_nrt_1d_20210124_20210124_icemod.nc.gz

When listing ECFS data not available during the DHS Move

Listing starting with 'o' or 'O' indicates that data are offline, i.e. on tape in Reading only and not available during the DHS move. 

-> els -l ec:/uid/training_*
orw-r--r--   1 uid      us            7099904 Feb  5  2007 training_2004.ecfs
Orw-r--r--   1 uid      us            3564032 Jul 11  2003 training_jrc.ecfs
Orw-r--r--   1 uid      us            4575744 Nov 11  2002 training_jrc_orig.ecfs
orw-r-----   1 uid      us           13733376 Feb 21  2011 training_material_2009.ecfs
orw-r-----   1 uid      us           14510080 Feb 21  2011 training_material_2010.ecfs

When accessing ECFS data not available during the DHS Move

-> ecp ec:testing-file ./
ecp: copying from ec:/uid/testing-file: Data offline for file: /uid/testing-file

Available tools  

MARS

• Data available during the DHS Move dry-run is listed in the Catalogue of the temporary DHS.
• To monitor the status of your requests during the DHS Move dry-run, please, visit the Temporary DHS server activity .  

ECFS

• ECFS listings done during a DHS Move dry-run will show files offline i.e. on tape in Reading only and therefore not available, with 'o' or 'O' in the first column (instead of '-' or 'b'):  

  -> els -l ec:/uid/training_*
  orw-r--r--   1 uid      us            7099904 Feb  5  2007 training_2004.ecfs
  Orw-r--r--   1 uid      us            3564032 Jul 11  2003 training_jrc.ecfs
  Orw-r--r--   1 uid      us            4575744 Nov 11  2002 training_jrc_orig.ecfs
  orw-r-----   1 uid      us           13733376 Feb 21  2011 training_material_2009.ecfs
  orw-r-----   1 uid      us           14510080 Feb 21  2011 training_material_2010.ecfs

How to report a problem

If your activity is critical and we have missed to identify it as such, please raise a support ticket, see below, and we will look at your specific requirements.

If your request/command does not behave as described above, please, provide as much information as possible so analysts can investigate/reproduce the problem, for example:

• Service/Tool you are using (mars client, ecfs, metview, web api, verify, etc...)
• Version of the tool
• Host, such as, ecgb, ATOS, cca, your workstation, ... and the environment under which it runs, interactive, batch, ...
• Log output produced by MARS/ECFS, including the request

Please report all issues via our Support Portal, mentioning  "dhs dry-run" in the title of your computing problem ticket.