Last modified on Nov 08, 2023 18:03

Contributors: B. Calmettes (CLS), L. Zawadzki (CLS), J.-F. Crétaux (LEGOS), L. Carrea (University of Reading), C.J. Merchant (University of Reading)

Issued by: CLS / L Zawadzki, B Calmette

Date: 31/05/2020

Ref: C3S_312b_Lot4_D2.LK.4-v2.0_202005_Product_Quality_Assessment_Report_LWL_v1.0

Official reference number service contract: 2018/C3S_312b_Lot4_EODC/SC2

History of modifications

List of datasets covered by this document

Related documents

Acronyms

General definitions

Accuracy: The closeness between the measured value and the true quantity value.

Precision: The closeness between measured values obtained by replicate measurements on the same object under similar conditions

Bias: Estimate of a systematic error

Scope of the document

This document is the Product Quality Assessment Report (PQAR). This document is meant to summarise the results from the product assessment based on the Product Quality Assurance Document [D4] for the C3S Lake Water Level product. Although, it can be made public, it is primarily intended to be shared with the Evaluation and Quality Control (EQC) activity at C3S.

Executive summary

The C3S Lakes production system (C3S ECV LK) provides an operational service, generating lake surface water temperature and lake water level climate datasets for a wide variety of users within the climate change community. The present document covers the lake water level system.

This document presents the results of the quality assessments undertaken for the product including absolute assessment by evaluating three performance indicators: (i ) dispersion, (ii) high frequency variations and (iii) mean time step between estimation. It also includes relative assessment comparing C3S datasets to independent altimetry products as well as in situ measurements.

This document concerns the second version of the C3S LWL products(CDR v2.0, product version 2.1). It will be updated as necessary for the future TCDR and CDR. The main variation in the version 2.0 is the increase in the number of lakes.

1. Product validation methodology

1.1. Validated products

The Water Level is the measure of the absolute height of the reflecting water surface beneath the satellite with respect to a vertical datum (geoid) and expressed in metres. The C3S Lakes products comprise a long-term climate data record (CDR). The time series has been computed from multiple altimetry satellites since late 1992 to 2019 inclusive. The time periods used for each satellite/instrument are provided in Table 1 but may vary from one lake to the other, depending on the orbits of the satellites with respect to the location of the lake.

Table 1: Time periods for the satellite/instrument used to generate the lake product

A detailed description of the product generation is provided in the Algorithm Theoretical Basis Document (ATBD)[D3] with further information on the product given in the Product User Guide and Specifications (PUGS) [D5].

1.2. Validating datasets

A combination of in situ and external altimetry-based products are used to assess the quality of the C3S Lakes products. The list of datasets used is provided in Table 2.

Table 2: Datasets used in the assessment of the data product split by altimetry-based and in situ data

1.3. Description of product validation methodology

The quality assessment of the Lake Water Level product includes the comparison of the dataset to external data (in situ and altimetry-based) as well as tests to determine the long-term stability of the product at climate scale.

1.3.1. Absolute assessment

The validation exercise consists of validating the local changes of water level as measured by altimetry. The uncertainties or errors in the products are induced by two categories of errors: measurement errors and processing errors. Altimeters measure the distance between satellite and lake surface. However, numerous perturbations should be considered, and corrections need to be subtracted to take into account various physical phenomena. Some are already evaluated as the geoid, the ionospheric correction, wet and dry tropospheric correction and polar tides. For other phenomena, it is not possible to correct for these effects because the information is not currently available operationally at global scale such as wind effects or "oceanic" tides in large lakes, even though this may induce an uncertainty of a few centimetres.

Measurement errors may thus have several causes but the dominant source of measurement errors in inland waters is the land contamination of the footprint: in some configurations, nearby land may be as echogenic as water and interfere with the echo. In this case, the range measurement, hence the water level measurement, may be affected. The main challenge of the processing is thus to correctly identify the valid measurements.

The Lake Water Level products may contain altimeter data from multiple satellites tracks as well as different missions. Transects (intersections between satellite tracks and lakes) are on average longer on large lakes. Since the land contamination of the footprint is the major source of error in the measurements, transects on large lakes have both a higher number and a higher percentage of measurements that are not contaminated by this type of errors. The precision is thus better for large lakes.

Three performance indicators are chosen to assess the quality of lake products:

• Dispersion: This metric quantifies the dispersion of the individual successive measurements recorded by the altimeter when flying above the lake at a given time. It thus quantifies the precision of the estimated Lake Water Level data at each time step of the time series.
• High-frequency variations: standard deviation of the high frequency signal within each time series (computed thanks to a Lanczos high-pass filter with an arbitrary 1-month cut-off period). This indicator gives additional information on the lake water level precision for small lakes. Indeed, it primarily quantifies remaining errors due to the geoid model, as well as the shifts in the satellite orbits and the inter-mission bias.
• Mean time step: Average time between two valid measures. Since the estimation of the lake water level is based on multiple missions with different repetition cycles and different ground tracks, the time step per lake is not regular. Moreover, measurements may also be missing due to the poor quality of data that has been automatically removed during the process. This indicator provides information on the average frequency of data available per lake.

These performance indicators are estimated for each lake on two time periods: the full time series of ~25 year for most lakes and the last 10 years. These last indicators give the performance of recent quality of the products and provides insight on the future quality of the next version of the TCDR and CDR LWL products.

Since LWL products are derived from multiple missions, other interesting indicators involve the comparison of the performance between missions. Missing values and per mission are calculated for current missions: Jason-3 and Sentinel-3A

1.3.2. Relative Assessment

External products using different data processing or acquisition are useful to assess the quality of the Lake Water Level products. Two types of datasets are considered: data generated by altimetry products and data obtained by in situ measurements. These products use different datums, different dates, and for the altimetry products, different altimetry missions or standards/tracks. The comparison is not straightforward. However, it provides information on the product's precision and accuracy. It must however be thoroughly analysed to understand if the differences are within the products' uncertainties or errors in one of the two products.

For each lake, we estimate the difference of the variation, the bias and the Pearson coefficient to estimate the linear correlation between Lake Water Level time series.

2. Validation results

2.1. Absolute assessment

Three performance indicators are chosen to assess the quality of the lake water level products: dispersion, high frequency variations and mean time step. These values were estimated for each lake on two time periods: the full time series of ~25 years for most lakes and the last 10 years. Annex A contains the values of those performance indicators for each lake on the two time periods. Performance indicators of the last 10 years are indicative of recent and future quality of lake products. Figure 1 provides an overview of the performance indicators for all lakes and three lake size categories:

• Small lakes: with surface less than 3000 km²
• Medium lakes: with surface within 3000 and 10000 km²
• Big lakes: with surface greater than 10000 km²

These general results will be commented more thoroughly in the subsections. However, they illustrate the general behaviour explained above: the precision decreases with the size of the lakes, ranging on average between 9 and 5cm over the full 25-year period. The dispersion improves between the full period and the last 10 years only, thanks to the improvement of the sensors but also the orbits and other geophysical corrections. However, the high-frequency variations increase because more satellites and more tracks are used in the products (decreasing the mean time step), inducing inter-calibration issues as well as uncertainties related to the limitations of the geoid models.

Figure 1: Performance indicators for the overall period (25 years) in dark colours compared to the last 10 years (2010-2019 period) in light colour for three categories of lakes depending on their size (less than 3000km², between 3000 and 10000 km² and bigger than 10000km²). Dispersion and High-frequency variations are indicated in cm and Time step in days

2.1.1. Along-track dispersion

Overall transect dispersion for the three categories of lakes considered is less than 10cm threshold in the product requirements and generally decreases with the area of the lakes. More details are provided in Figure 2, illustrating that high transect dispersions are mostly observed on the smaller lakes (<3000 km²). Indeed, land contamination in the footprint being one of the main source of error in altimetry, there is a higher probability to have land contamination with small lakes, which increases the dispersion, whereas big lakes tend to provide results similar to what is expected on oceanic surfaces for altimetry.

Figure 2: Dispersion according to lake surface (surface in logarithmic axis)

In Figure 2, the two lakes with bigger dispersion values are Tchany (33 cm) and Hongze (34 cm). All the other lakes show a dispersion lower than 20 cm. Both are small lakes, with a surface less than 3000 km2 (2000 km2 and 1960 km2 respectively).

The water level time series for lake Tchany (Figure 3) is obtained from several missions from Topex/Poseidon, Jasons 2, Jason 3 and Envisat and the dispersion along the time series is clearly different between the different missions. The lower dispersion during the Envisat period (2002-2010) can be explained by the position of the ground tracks above the lake (Figure 4). Even if all ground tracks are close to the lakeshore, Jason's tracks being the closest, are much more contaminated by land echoes.

Figure 3: Water level and dispersion time series for lake Tchany 


Figure 4: Ground Tracks over passing lake Tchany (TOPEX and Jason in red, ENVISAT in Yellow)
The result is similar for Lake Hongze, the time series (Figure 5) shows a high variation and the tracks over the lake (Figure 6) are relatively small, less than 10 km. 

Figure 5: Water level and dispersion time series for lake Hongze 


Figure 6: Ground Tracks over passing lake Hongze (TOPEX and Jason in red, ENVISAT in Yellow)

In addition to an individual analysis by lake over the entire period, it is also important to analyse the evolution of dispersion over the past few years. This information is useful for assessing the quality of the most recent measurements, which is expected to be better, thanks to the improvement of the sensors and the ground segments. Figure 7 shows the dispersion for each lake for the full period and for the last 10 years (2010-2019) ordered by lake size. As expected, the dispersion decreases for almost all of them and the lakes Tchany and Hongze with higher dispersion because only Jason 3 mission is currently active.

Figure 7: Dispersion per lake (sorted by size) during the full period compared to the last 10 years.

In a general way, the shape of the lake and the position of the ground tracks have a significant impact on the quality of the estimation of the lake water level. If we analyse the Lagoa dos Patos, a lake with a surface area of 10000 km2, just in the limit of medium/big size lakes, the dispersion has considerably evolved with the altimetric missions (Figure 8). During the period TOPEX/Poseidon (Figure 9a), only one near land track is available. With GFO (1998), a second track, with a better location but also near land, crosses the lake (Figure 9b). Thanks to ENVISAT (2002), several well-positioned ground tracks are available (Figure 9c). In 2008, Jason-2 data improved the quality of the estimated LWL product along the same ground tracks as TOPEX/Poseidon. Finally, Sentinel-3a allowed an increase in quantity and quality of the lake water level estimation (Figure 9d) with several tracks over the lake and a globally improved system.

Figure 8: Water level and dispersion time series for the Lagoa do patos

(a) TOPEX/Poseidon ground tracks	(b) TOPEX/Poseidon + GFO ground tracks
(c) TOPEX/Poseidon + GFO + ENVISAT ground tracks	(d) TOPEX/Poseidon + GFO + ENVISAT + Sentinel3-a ground tracks

Figure 9: Ground Tracks over passing the Lagoa do Patos. a) TOPEX/Poseidon in red, GFO in green, ENVISAT in Yellow and Sentinel-3a in blue)

2.1.2. High-frequency variations

The second indicator concerns the high frequency signal (Figure 10). Mean variations are evaluated by taking the standard deviation of 1-month high-pass filtered LWL time series. They mainly contain "noise" due to measurement uncertainty as well as the geophysical signal of the high-frequency water level variations. In general, measurement noise over the last ten-year period is higher than over the full period. This is mostly because there have been more satellites, thus more individual measurements (lower time step), over the last 10 years. More measurements with their specific precision and geoid errors yield to an increased amplitude of the high-frequency signal. Measurement uncertainty estimates using this indicator are less than 10 cm on average, which are also within accuracy requirements for lake products.

Figure 10: High frequency variation per lake (sorted by size) during the whole period compared to the last 10 years.

One of the examples of the increasing of the high frequency variation is lake Kariba. Since 2016, 6 ground tracks of Jason 3 and Sentinel-3A overpass the lake (Figure 11). Thanks to that, the mean time step decreased from 12,2 days to 7,3 days but the time series during the last period is a lot noisier (Figure 12). 

Figure 11: Ground Tracks over passing lake Kariba (TOPEX and Jason in red, Sentinel-3A in blue) 
Figure 12: Water level and dispersion time series for Lake Kariba

2.1.3. Time Resolution

The mean time step between valid lake water level measurements over the full period is mostly around 13 days and decreases for all the lakes for the 10 year period compared to the overall period (Figure 13). Due to a greater number of satellite tracks sampling the lakes during the two-altimeter era, the frequency of measurements is much higher: 8.6 days on average. The other factor impacting the mean time step is also the percentage of edited or missing measurements. With the improvement of the altimetry products (sensors, geophysical correction, ground segment, orbit…etc), less measurements are identified as outliers and edited.

Figure 13: Mean time step per lake (sorted by size) during the all period compare to the last 10 years.

Another interesting indicator is the percentage of missing values. This value represents the number of lake water levels that could not be estimated due to different reasons: quality of the signal, shift of the ground trajectory, fast change in the level that activates the editing of the estimate. These percentage values were estimated for the current missions: Jason-3 and Sentinel-3A, for all the lakes and for the three categories of lakes based on the size as defined in section 2.1. The missing values of Sentinel-3A are lower than missing values of Jason-3 (Table 3). It can be notice that the percent of missing values in for the big lakes is surprisingly high and need further investigation.

Table 3: percentage of missing values depending on the mission and the size of the lake

2.2. Relative assessment

2.2.1. Altimetry products

In the context of the time series comparison, for the altimetry products, monthly time series of LWL products from external sources (G-REALM, DAHITI, Hydrosat) have been selected. The values are compared for the last 10 years where all the processing and satellite systems are more stable. For each lake, the Pearson coefficient is higher than 0.8 with either G-REALM or DAHITI (Figure 14).

Figure 14: Pearson coefficient between monthly time series from (i ) C3S and G-REALM datasets (red dots) and (ii) C3S and Dahiti datasets (blue dots). Most Pearson coefficient values are higher than 0.8 indicating a positive linear correlation.

The two lakes with the lowest Pearson coefficient are Tchany and Har with Pearson coefficients 0.161 and 0.412 respectively. The position of the tracks over passing this lake Tchany is discussed in section 2.1.1. Comparing C3S time series with G-Realm time series, we notice some outliers since 2016 that that need further analysis.

Figure 15: Monthly Water level variation for Lake Tchany (red: C3S, blue: GREALM) and difference between time series.

For Lake Har, the G-REALM time series has a pattern that repeats at non regular intervals: there is a slow decrease followed by a very fast increase.

Figure 16: Monthly Water level variation for Lake Haar (red: C3S, blue: GREALM) and difference between time series.

The value of the RMS can also provide important information on the distance of time series. This value was evaluated for each lake on a monthly basis (Figure 17). For all the lakes, this value is less than 1m. The maximum value concerns lake Mossoul compared to G-REALM and Dahiti (Figure 18 and Figure 19). In both cases, the higher RMS value is driven by two outliers, one at the beginning of 2017 and the second one at the end of 2018. However, these specific outliers are not visible when comparing the original time series, but one can notice a relevant number of missing values in C3S data for these periods affecting the monthly value estimated.

Figure 17: RMS values 

Figure 18: Lake Mossoul - Comparison to G-REALM 


Figure 19. Lake Mossoul - Comparison to Dahiti

Analysing the comparison figures for all the lakes, another effect is identifed. For some lakes, a jump in the unbiased difference was detected. This phenomenon exists with both products: G-REALM and DAHITI. For example, lakes Kara Boza Gol and Tanganyika, in separate regions, present this behaviour (Figure 20 and Figure 21). The most probable reason for these discrepancies is a different estimation of the intermission biases. More analyses are needed to identify which of the products contain an error and to correct for this anomaly.

Figure 20: Lake Kara Bogaz Gol - Comparison to G-REALM 

Figure 21: Lake Tanganyika - Comparison to Dahiti products

2.2.2. In situ products

2.2.2.1. Hydrolare

Thanks to the collaboration with the International Data Centre on Hydrology of Lakes and Reservoirs, Hydrolare, the information on four lakes was provided in a monthly time step. Three indicators were estimated: Bias, RMS and Pearson coefficient (Table 4). For all the lakes the Pearson coefficient is higher than 0.9. In some cases, the bias is high (up to 2.3m) and suggest that, even if the precision is good, the accuracy of the LWL products should be considered carefully by users. Annex B contains the figures corresponding to each lake.

Table 4: Hydroweb – Hidrolare Indicators

2.2.2.2. Hidricos Argentina

The information concerning the variation on the Water Lever for lake Argentino was obtained online from the Base de datos Hidrologica integrada (BDHI): bdhi.hidricosargentina.gob.ar. The Pearson coefficient is 0.969 showing a very high correlation. Figure 22 shows the time series for the water level variation of lake Argentino, as well as the difference with the present LWL product. This again suggests that, even though the precision is acceptable, with RMS below 10cm, the accuracy must be considered carefully as it is often dependent upon an arbitrary choice of reference height.

Figure 22: C3S - Hidricos Argentina comparison for the lake Argentino

2.2.2.3. US Army Corps of Engineer

The third source of in situ data is the US Army Corps of Engineers¹. The data for the Great Lakes in the USA is available online. Table 5 contains the indicators for the Great Lakes and the figures for each lake are in Annex C. The results show a nice agreement between the present product and this in situ dataset, with RMS below 7cm and correlations above 0.96.

Table 5: C3S – US Army Corps of Engineers Indicators

2.2.2.4. US Geological Survey

The USGS provides information on water resources data collected mainly in US. The USGS investigates the occurrence, quantity, quality, distribution, and movement of surface and underground waters and disseminates the data to the public, State and local governments, public and private utilities, and other Federal agencies involved with managing our water resources. The information concerning the monthly lake water level evolution for lakes Michigan and De bois was obtained on-line. The Pearson coefficient is 0.688 and 0.965 respectively. Figure 23 and Figure 24 show the times series for the period 1992-2019.

Figure 23: C3S - USGS comparison for the lake Des Bois

Figure 24: C3S - USGS comparison for the lake Michigan

2.2.2.5. Water office of Canada

The in situ data for Canadian lakes is freely available at the Water Office of Canada. The time series of twelve lakes monitored in the C3S project were compared to this data and the Pearson coefficient, indicating the correlation between time series is most of the time higher that 0.8 (Figure 25).

Figure 25. Pearson coefficient between monthly time series from Water Office of Canada.

Concerning lakes Ontario and Winnipeg, their leak value of the Pearson coefficient is due to some outliers in the in situ measurements. Figure 26 shows the time series for the Lake Ontario. Regarding the Lake Winnipegosis (Figure 27), the correlation of the time series is lower in the past missions (before 2010) and it becomes better since Envisat mission.

Figure 26: C3S – Water Office of Canada comparison for the Lake Ontario

Figure 27: C3S – Water Office of Canada comparison for the Lake Winnipegosis

3. Application(s) specific assessments

Currently, no application(s) specific assessments have been undertaken for the January 2020 version of the C3S lake water level dataset (CDR v2.0, product version 2.1).

4. Compliance with user requirements

The requirements for the C3S Lake water levels are described in the Target Requirements and Gap Analysis document [D1]  (Table 6).

Table 6: GAP analysis between target requirements and dataset characteristic 

The data requirements for the lakes are based on GCOS 200 recommendation. Some of these requirements have been reached such as the temporal coverage. Nevertheless, other requirements such as the temporal resolution do not depend on the algorithm used to estimate the lake water levels but on the number of satellites overpassing a given lake. New mission/altimeters could improve temporal resolution and increase spatial coverage. In the near future data from Sentinel-3B will be used.

Acknowledgements

Thanks to Prof. Valery Vuglinskiy for providing us with in situ data from Hydrolare.
Thanks to C. Schwatke and the DAHITI team for sharing their database and reviewing the results.

 References

Bercher, N. (2008). Précision de l'altimétrie satellitaire radar sur les cours d'eau: Développement d'une méthode standard de quantification de la qualité des produits alti-hydrologiques et applications (Doctoral dissertation).

Biancamaria et al., 2017, Validation of Jason-3 tracking modes over French rivers, Rem. Sens. Env., submitted

Chelton, D. B., Esbensen, S. K., Schlax, M. G., Thum, N., Freilich, M. H., Wentz, F. J., ... & Schopf, P. S. (2001). Observations of coupling between surface wind stress and sea surface temperature in the eastern tropical Pacific. Journal of Climate, 14(7), 1479-1498.

Crétaux, J. F., Abarca-del-Río, R., Berge-Nguyen, M., Arsen, A., Drolon, V., Clos, G., & Maisongrande, P. (2016). Lake volume monitoring from space. Surveys in Geophysics, 37(2), 269-305.

Jekeli, C., & Dumrongchai, P. (2003). On monitoring a vertical datum with satellite altimetry and water-level gauge data on large lakes. Journal of Geodesy, 77(7-8), 447-453.

Schwatke, C., Dettmering, D., Bosch, W., and Seitz, F.: DAHITI – an innovative approach for estimating water level time series over inland waters using multi-mission satellite altimetry, Hydrol. Earth Syst. Sci., 19, 4345-4364, https://doi.org/10.5194/hess-19-4345-2015, 2015

Annex A. Performance indicators

Annex B. Comparison to Hydrolare Data

Annex C. Comparison to the US Army Corps of Engineers Data

Related articles

• Page:
  ERB TSI: Product User Guide and Specification (PUGS)
• Page:
  ERB TSI: Product Quality Assurance Document (PQAD)
• Page:
  ERB TSI: Product Quality Assessment Report (PQAR)
• Page:
  ERB TSI: System Quality Assurance Document (SQAD)
• Page:
  ERB TSI: Algorithm Theoretical Basis Document (ATBD)