Last modified on Nov 08, 2023 11:53

Contributors: B. Calmettes (CLS)

Issued by: B Calmettes

Date: 21/04/2021

Ref: C3S_312b_Lot4.D1.LK.3-v3.0_LWL_System_Quality_Assurance_Document_i1.0.docx

Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC2

History of modifications

List of datasets covered by this document

Related documents

Acronyms

Scope of the document

This document is the System Quality Assurance Document (SQAD). It describes all elements (e.g. teams) that contribute to the processing chain, including all interfaces to C3S, and to RD component, the hardware used in the processing chain, the procedures to implement new data cycles and to reprocess the products, and those applied in case of system failures or anticipated maintenance work, as well as information on user support.

Executive summary

The C3S Lake production system (C3S ECV LK -LWL) provides an operational lake water level service, generating lake water level climate datasets for a wide variety of users within the climate change community. A dedicated team supports and maintains the system. It is divided into a science team, a system development team and an operation team. The science team directs the scientific evolution of the service and provides dedicated scientific support to the users. The system development team is responsible for the implementation of the evolutions, the maintenance and the validation of the Technical Platform. The operational team is responsible for the Operational Framework. The responsibilities of the Operations team, with respect to system maintenance and system failures, are summarised in a list of actions to trigger in case that they are required.

1. System overview

The C3S Lake Water Level production system (C3S ECV LK - LWL) is responsible for the operational provisioning of the Lake Water Level parameter within the Lake Essential Climate Variable (ECV) for the Copernicus Climate Change Service. The mission of the system is the timely production of a consistent, long-term, multi-mission and multi-sensor global lake water level data set complying with the defined key performance indicators (KPIs). The system is semi-automated, intelligible and cost-effective to foster the interoperability of the individual system elements and teams.
Integral components of the C3S ECV LK – LWL are:

• the Organizational Framework consisting of
  • the LWL Science team
  • the LWL system development team
  • the LWL operation team
• the Technical Platform and the Operations Framework
• the C3S Processing Framework (CPF) and
• Data Management and Streams

These components, their functions and interfaces within the C3S ECV LK – LWL production system will be discussed in the following.

1.1. Organizational Framework

Three teams are cooperating within the organizational framework in order to fulfil the mission objective of the system.

The Science Team oversees the physical reliability and validity of the LWL generated by the systems. Consequently, the function of the Science Team is to design scientific algorithms capable to produce a long-term consistent, multi-mission global LWL product, in accordance with the defined data key performance indicators.

The System Development Team develops and maintains the scientific algorithms based on the design of the science team and integrates them in the processing baselines. It validates the implementation in accordance with the defined data key performance indicators and produces a new version of the systems that they deliver to the corresponding Operations Team.

The Operations Team manages the system and its individual components and takes control of the regular production respectively of the LWL variable. With respect to that, the Operations Team performs the orchestration, provisioning and configuration of the technical platform as well as the setup and maintenance of the Operations Framework. Members of the Operations Teams exclusively have access to the Technical Platform and Operations Framework. Inputs provided by the System Development Teams, such as new versions of the processing baselines, are picked up by the Operations Teams and are revised in an iterative and cooperative way before those are integrated in the operational C3S Processing Framework.

1.2. Technical Platform and Operations Framework

The Technical Platform and Operations Framework is composed of a set of hardware and software components interacting with each other.

The C3S LWL Technical Platform is realised at CLS and CNES providing two major hardware components. These hardware components are:

• The CLS cluster: nine batch servers / 216 cores (432 threads) 128GB memory / server
• The CNES cluster (HAL): 300Tflops, 380 batch servers / 8400cores, 128GB memory / server, 6,2 PB GPFS / 200TB burst buffer/ 100GBs bandwidth Infiniband, low latency network, 4 GPGPU Nvidia Volta V100

The HAL cluster is used to gather and enhance the historical altimetry data thanks to its significant storage capability and computing power. The historical altimetry databases (L2E-HR) on the server are maintained and operated with a CLS software and database proprietary format directly compatible with LWL system. The CLS cluster is used to copy and store only the relevant input data from the altimetry L2E-HR databases on the HAL cluster. This data is stored temporarily on a dedicated partition (netapp4-L2P-HYDRO, 3To) for the operation team use. The partition also stores the relevant missing ancillary data and houses the operational C3S LWL Processing Framework.
The Operations Framework is based on a source code library base on GitHub  and its web-based repository manager GitLab¹, holding all code and configuration fragments needed to run the operational service. Writing access to the GitLab² code repository is restricted to the members of the LWL System Development Team. Reading access to the GitLab code repository is extended to the members of the C3S LWL Operations Team. Specific version of code packages can be cloned or downloaded to the Technical Platform. This process is automated by making use of a CLS overlayer software to the Git tools. The documentation platform of all packages is hosted on a dedicated Microsoft SharePoint repository with restricted access to the Science Team, the System Development Team and the Operations Team.

System operations are done by members of the Operations Team. Team members have unrestricted access to the Technical Platform and the Operations Framework as a single Technical User. This Technical User, with username lake_exp, handles the entire operational service of the C3S ECV LK LWL. Secure access to the Technical Platform and Operations Framework is exclusively given via SSH using public-key cryptography for authentication.

1.3. C3S Processing Framework

The C3S Processing Framework (CPF) for Lake Water Level is based upon the scientific base of the THEIA/Hydroweb project and is named Hysope. This means that the CPF uses the Hysope processor of a specific version for producing the C3S LWL product. Pre- and post-processing needed to get the input data streams into the correct format to be handle by the Hysope processor or to re-format the data to the C3S specifications is added around the Hysope processor core. Figure 1 provides a high-level look at the components of the framework, also described in Table 1.

Figure 1: Overview of the elements and interfaces of the C3S Processing Framework

Table 1: LWL C3S Processing Framework Components

1.3.1. Interface to Hysope processor

The original Hysope processor was designed to ingest Level-2 altimetry data. The input format can either be the netcdf official formats delivered by the space agency (e.g. cophub) or the CLS proprietary format. This last functionality allows the Hysope processor to ingest the historical altimetry database L2E-HR housed by and maintained on the HAL cluster. Figure 2 shows how the input data stream interacts with the Hysope processor.

Figure 2: C3S LWL input data stream (present and future)

The Hysope processor was designed to run in different modes. In the “reprocessing” mode, the Hysope processor computes processing parameters and produces the most consistent lake water level CDR based on all available input datasets (historic and current missions). The “Near-Real-Time” mode of the Hysope processor uses these processing parameters to generate the ICDR, taking into account the latest available input data from all current missions, with less computational effort compared to the “reprocessing” mode. Processing parameters are generated during each reprocessing cycle. The parameters are the geographical extraction zones for each lake, empirical correction of the high frequency variations of the geoid and inter-mission biases. These parameters are stored for each sensor and each lake and used both in “reprocessing” and “NRT” modes of the processor. Figure 1 shows how the parameter files interact with the Science Team and the Hysope processor via the FTP component and are archived on the CLS server component. More details about the algorithm and the used parameters can be found in the ATBD [D1].

1.3.2. Post Processing

The Hysope processor can produce for each lake either a CSV or GeoJSON file in both modes. Consequently, the C3S post processing block converts the format into the required NetCDF4 format and provisions the data to the EODC infrastructure that will then provide the data to the CDS.

1.4. Data Management and Streams

Data used in the C3S ECV LK LWL can be discriminated into historic and NRT inputs. Historic data are level-2 altimetry products from already decommissioned satellite missions generated in one processing cycle. These datasets are reprocessed on a regular basis by space agencies to enhance them with the state-of-the-art algorithms. The data archive is available on the disk storage of CLS cluster and on the CTOH (Center for Topographic studies of the Ocean and Hydrosphere), datacenter of the Science Team. The data is archived and is complemented by regular backups of the data.

For each processing cycle of the CDR and Test CDR, the generated data will be placed in separate directories to assure data integrity of the different versions. Furthermore, data produced by test facilities of the system will be completely separated from the operational data stream for the same reason. The latest version of an LWL CDR and ICDR will reside on disk storage (nettap4) as long as the data stream is operational. Outdated, older versions of the LWL product with its corresponding processing parameters are archived on the same disk storage (nettap4) in a different repository.

Movement, deletion of data can only be done by the Operations Team and is closely coordinated with the Science Team.

NRT data streams required for the operational production of the ICDR are regularly monitored by the Operations Team by utilising workflow routines implemented in Nagios³. These checks are executed beforehand of each ICDR production run, raising an error if a data stream is broken.

1.4.1. Active input streams

Active input data streams are based on observations from a series of altimeters on-board historic and current international satellite missions. Algorithms to derive lake water level from altimetry have been developed and are further research by LEGOS (Crétaux et al 2006, Crétaux et al 2011, Crétaux et al 2016).

Figure 3: Altimetry constellation (source: https://www.aviso.altimetry.fr)

1.4.1.1. Jason-3

Jason-3 is an ongoing altimetry radar mission, successor of TOPEX/Poseidon, Jason-1 and Jason-2 (see Section 1.4.2.1). The Jason-3 NRT Level-2 data is operationally collected on the CLS cluster through the CNES multi-missions ground segment SSALTO data server (Segment-Sol multi-missions d'ALTimétrie, Orbitographie et localisation précise) housed by CNES.

1.4.1.2. Sentinel-3 STM

The Sentinel-3 satellites fit into the Copernicus program. Sentinel-3a was launched in 2016. It has been joined by Sentinel-3b in 2018 for a 4-month tandem phase before it drifted to its nominal orbit, an interleaved orbit with respect to Sentinel-3a.
The Sentinel-3 Short Time Critical Level-2 Land products are operated by ESA and broadcasted on the Copernicus hub. The data is operationally downloaded by the Operations Team and stored on the CLS cluster back upped disk (netapp3).

1.4.2. Passive input streams

In the individual sections for the sensors, an overview is given on what specific datasets are currently available for ingestion into the C3S processing system and in case of near-real-time processing, how this is set up.

1.4.2.1. TOPEX/Poseidon and Jason

The TOPEX/Poseidon satellite was launched in 1992 with the objective of "observing and understanding the ocean circulation". Established as a joint project between NASA, the US space agency, and CNES, the French space agency, it carries two radar altimeters and precise orbit determination systems. The original design was intended to monitor oceans and it also measured inland water levels. The TOPEX/Poseidon data is stored on the CTOH cluster for future use in the Hysope processor. The reprocessing of TOPEX/Poseidon data is in the validation phase. The dataset is scheduled for 2021.

Still developed jointly by CNES and NASA, Jason-1 was launched in 2001 and is the follow-on of TOPEX/Poseidon. Its unchanged orbit, compared to that of TOPEX/Poseidon, allows a continuous acquisition of measurements and, thus, an improved stability of the C3S LWL product. The Jason-1 data is stored on the CTOH cluster. The 2016 reprocessing is stored on the HAL cluster for future use in the Hysope processor.

Jason-2/OSTM takes over and continues TOPEX/Poseidon and Jason-1 missions in 2008, in the frame of a cooperation between CNES, EUMETSAT, NASA and NOAA. It carries the same kind of payload as its two predecessors, for a high-accuracy altimetry mission: a Poseidon-class altimeter, a radiometer and three location systems. The orbit is also the same. The Jason-2 data is stored on the HAL Cluster for future use in the Hysope processor.
nb: since 2016, Jason-3 followed on Jason-2 seamlessly on the same orbit with the same payload.

1.4.2.2. Envisat

Envisat (Environmental Satellite) is the follow-on to ERS-1 and ERS-2 (not used yet in C3S LWL product). Devoted to environmental studies, and climate change in particular, its mission is to observe Earth's atmosphere and surface. Built by ESA, the European Space Agency, Envisat is carrying ten complementary instruments for observing parameters ranging from the marine geoid to high-resolution gaseous emissions. Among these instruments are a radar altimeter, the DORIS orbitography and precise location system. A reprocessing of Envisat altimetry Level-2 product has been released in 2016, upgrading standards to the state-of-the-art. This reprocessed data has been acquired and checked in our internal databases and will be considered for a reprocessing of the Lake Water Level product to improve precision, accuracy and stability.

Additionally, in the framework of the FDR4ALT project, founded by ESA, a new reprocessing of Envisat altimetry Level-2 products is ongoing. This reprocessing includes for the first time inland water thematic data products and is expected to improve the quality of LWL products during the period 2002-2009. As per the project initial planning, it should be officially available in 2022.

1.4.2.3. Geosat-Follow-On

GFO, Geosat Follow-On was launched in February 1998. Its main mission is to provide real-time ocean topography data to the US Navy thanks to its radar altimeter. It also provided inland water measurements. The data can be accessed through NOAA and has been collected by the CTOH data center.

2. Upgrade cycle implementation procedure

System upgrade cycles mainly concern the Technical Platform and Operations Framework as well as the C3S Processing Framework. These cycles are timely planned and tested before the final implementation takes place. Implementation upgrades are rolled out to the system during maintenance windows, in order to assure smooth operations of the ICDR production without interruptions. Beforehand, system upgrades are tested by making use of the provided test facilities of the corresponding CDR production line. Notifications in the direction of users will be kept minimal as long as no direct implication of the upgrade is given.

2.1. Technical Platform and Operations Framework upgrades

Upgrades related to the Technical Platform and Operations Framework are implemented and tested by the Operations Team. The utilised tools for software provisioning and configuration are fully exploited for upgrade cycles by taking advantage of the source code driven approaches of these tools. With respect to that, software deployments are stored in GitLab under version control. This allows a parallel, automated setup of a fully operational test facility to investigate the foreseen upgrades before deploying them to the operational service. Finally, upgrades will be integrated in the operational service during a maintenance window when all tests in the test facility have been successfully passed.

2.2. C3S Processing Framework upgrades

Processing Framework upgrades will be triggered by improved retrieval algorithms and the scientific advancements in the CCI LK LWL processor. All released CCI LK LWL products generated with a specific processor version are verified, validated and relevant algorithm improvements are published in the scientific literature. After new CCI lake processor versions are found suitable, their integration into the C3S production system is started.

2.2.1. Verification and Validation of CCI LK LWL products

Verification and validation of the CCI LK LWL products generated with specific CCI processors versions will be performed in the framework of the CCI lake project. Procedures to verify and validate the products will be performed on different spatial scales and, in comparison to different LWL products available. Results of this verification and validation activities will be regularly summarised in the Product Validation and Intercomparison Report (PVIR) published on the CCI LK webportal.

2.2.2. Integration of CCI LK LWL products updates into C3S

Products and algorithm updates shall only be integrated into C3S after the verification and validation of the CCI LK LWL products is performed successfully. Implementations of the algorithm updates will be performed in parallel to the operational service at least two months before a new CDR production cycle is started. Implementation tests provided by the Science Team to assure the correct implementation of the processor have to be passed successfully in order to be considered as ready for operations.

2.2.3. Communication strategy for product updates

After a successful implementation of the upgraded C3S Processing Framework (CPF), the CDR production is performed as scheduled in the production cycle based on the new version. Once per year, a Test CDR will be produced with the updated version of the CPF. Six months later, a CDR will be generated based on the Test CDR with a sixth-month update of the product. It will then be validated in accordance to the Product Quality Assurance Document (PQAD). The previous product version is stopped, and the new version takes over the operational data stream. Communication about the release of a new product version will be started another three months before the release through the Climate Data Store (CDS).

3. Procedures for reprocessing CDR's

A CDR consists of a TCDR, which represents the archive of the C3S LWL product, and the ICDR which is a consistent extension of the TCDR. The ICDR products are generated every 6 months and extend the TCDR of the same version as the ICDR. A new version of the CDR will be produced in the following cases:

1. Algorithm updates as described in Section 2
2. Processing parameter updates
3. Addition of new sensors using an existing algorithm
4. Change in NRT input products that make a reprocessing necessary

The number of versions will be kept to a minimum by pooling the possible changes into yearly releases.

3.1. Processing parameter updates

The processing parameters are regenerated during any CDR reprocessing. An explicit processing parameter update is not necessary if another reason for production of a new version of the CDR exists. In the absence of other changes, the processing parameters will be updated yearly after the release of the most recent CDR version.

3.2. Addition of new sensors

The addition of new sensors triggers the release of a new CDR version.

3.3. Change in input products

A change in input products can happen, planned or unplanned. Planned changes include improvements of level 2 surface water temperature or water level algorithms either for historical missions or NRT products. Unplanned changes can occur in the ingested NRT products for various reasons and will be handled as described in section 4.2.5.

4. System maintenance and system failures

System maintenance and failures may arise across all system components, but most likely they affect the Technical Platform and/or the Data Management and Streams. Hereafter, procedures applied in case of system maintenance and failures are described.

4.1. System Maintenance

The timeliness of the C3S LK LWL ICDR production and delivery is 6 months. Maintenance of individual system components will be planned carefully by the Operations Team resulting in an estimate of the needed effort. Based on the delivery and production of the products, the Operations Team will have a maintenance window of 3 months to perform the required actions. Within these 3 months of maintenance, the system will be adopted or at least will be prepared for maintenance in the next open maintenance window.

4.2. System Failures

Failures may arise abruptly any time but considering the 6-month timeliness of the products, failures can be treated without any major delay in the production. System failures are likely caused by failures of the Technical Platform of the system rather than other components of the system. Nevertheless, procedures to be applied in case of individual failures will be summarised in the following. Notification to users about system components failures will only happen if they are directly affected, such as product delivery issues. This notification of the users will be done via the Climate Data Store (CDS) which holds the user database for notification. The period of notice will be latest ten days before products are expected by the users.

4.2.1. Loss of key staff

At a first response key staff can be replaced by qualified colleagues from within the partner organisation. If this is not possible then the staff will be replaced by external candidates, in the meantime tasks may be distributed among partners to ensure a continuity of project progress.

4.2.2. Non-performance or loss of consortium members

The consortium team has a long-standing successful professional relationship, having worked together over various projects for the last 10 years. As such non-performance is not expected. If a consortium member leaves, then an adequate replacement will be sought and will be proposed to ECMWF.

4.2.3. Conflict or dispute between team members

All conflicts and disputes will first be addressed, and a resolution sought by the project manager. A method for conflict resolution is in place in subcontracts between EODC and the partners.

4.2.4. Cost overruns

Regular monitoring of the LWL system will allow the project manager and Operations Team to spot and mitigate any costs overruns. If cost overruns cannot be controlled within the project, then ECMWF will be informed, and an alternate solution will be sought.

4.2.5. Input data stream failure

The product relies upon a suite of EO data sets from both active and passive. The loss of a number of input products would only result in a degradation of the quality of the overall LK products, and not critically stop the production of the CDS product. Only the loss of all input products would result in the inability to generate a product. Nevertheless, users will be informed about the failure via CDS after the first recognition of the issue to adopt for and be prepared.

4.2.6. Cluster failure

Failures related to the CLS or HAL clusters are the most critical ones at the moment. First, the CLS cluster blacks out but not the HAL cluster. In this case, the historical altimetry data is still available on the HAL cluster and on CLS backup system on band tapes. CLS has a back-up cluster based on the network of staff computers that may be used to recover altimetry data from the HAL cluster and from the system backups on band tapes. Second, the HAL cluster blacks out. In this scenario, the HAL cluster data is also backed up on band tapes at CNES and can be recovered. No action towards the users is needed as long as the product can be delivered on time. In case of a total blackout of the storage system for CLS and HAL clusters and the corresponding band tapes, currently no backup solution is in place. Accordingly, all users will be notified immediately. Missed products will be disseminated on the next possible release cycle of the ICDR.

4.2.6.1. Processing Facility Failure

Computing resources of the Technical Platform may fail because of damages in the hardware, broken cooling system, etc. In that case, the LWL Operations Team will start to deploy this part of the system on the backup CLS cluster solution (or the CNES HAL cluster solution if necessary) to run the required processing there. Users will be informed if a timely delivery of the product is critical (10 days before official publication).

4.2.6.2. Processing Failure

Processing failure may arise after upgrades in the C3S Processing Framework. Such failures are very unlikely, because updates of the framework are well tested beforehand. In case of a failure, the update will be dropped, and the previous version of the C3S Processing Framework will be re-setup. No need to notify the users, because the failure will be resolved immediately and will not affect the user.

5. User support

A dedicated service desk has been set up, the Copernicus User Support (CUS) team, which provides support to users of the CAMS and C3S services at ECMWF. All enquiries about the LWL datasets must be submitted through the service desk where it will be dealt with by appropriate persons. There is a portal (https://climate.copernicus.eu/user-support) where customers can submit enquiries using a form (split into "Data Request", "Documentation and Scientific Questions", "Events, Media and Legal" and "Report an Incident"). The information from this form is passed to the CUS. Once submitted, the user may add comments or further information to the issue, including responding to questions/requests for additional information from the support team.

The C3S 312b Lot4 service provides dedicated level 2 user support to the CUS and is fully registered with the CUS Jira Ticketing Service.

In addition to submitting enquires through the portal, a knowledge base is available to users which can be searched for information.

References

Crétaux, J. F., & Birkett, C. (2006). Lake studies from satellite radar altimetry. Comptes Rendus Geoscience, 338(14-15), 1098-1112.

Crétaux, J. F., Jelinski, W., Calmant, S., Kouraev, A., Vuglinski, V., Bergé-Nguyen, M., ... & Maisongrande, P. (2011). SOLS: A lake database to monitor in the Near Real Time water level and storage variations from remote sensing data. Advances in space research, 47(9), 1497-1507.

Crétaux, J. F., Abarca-del-Río, R., Berge-Nguyen, M., Arsen, A., Drolon, V., Clos, G., & Maisongrande, P. (2016). Lake volume monitoring from space. Surveys in Geophysics, 37(2), 269-305.

Related articles

• Page:
  ERB TSI: Product User Guide and Specification (PUGS)
• Page:
  ERB TSI: Product Quality Assurance Document (PQAD)
• Page:
  ERB TSI: Product Quality Assessment Report (PQAR)
• Page:
  ERB TSI: System Quality Assurance Document (SQAD)
• Page:
  ERB TSI: Algorithm Theoretical Basis Document (ATBD)