Contributors: Nabiz Rahpoe (DWD)
Issued by: DWD / Nabiz Rahpoe
Date: 26/05/2021
Ref: C3S_D312b_Lot1.2.3.9-v1.1_202105_PQAR_TCWV_SSMIS_TCDR+ICDR_v1.1.1
Official reference number service contract: 2018/C3S_312b_Lot1_DWD/SC1
History of modifications
List of datasets covered by this document
Related documents
Acronyms
List of tables
List of figures
Scope of the document
This document is the Product Quality Assessment Report (PQAR) for Total Column Water Vapour (product C3S_D312b_Lot1.3.3.14 v1.0 and v1.1) based on SSM/I & SSMIS measurements. It provides a brief guide to the data quality, and describes the validation method.
The TCDR data products are provided as a brokered service from the EUMETSAT CM SAF and the ICDRs (which are a continuation of the TCDR) are produced within the framework of C3S project. This document refers extensively to the original CM SAF validation report, “CM SAF Validation Report SSM/I and SSMIS - HOAPS 4.0” [D2]. It can found at the CM SAF web site http://www.cmsaf.eu.
Executive summary
The project C3S_312b_Lot1 includes brokering of total column water vapour gridded monthly mean and 6-hourly data from the EUMETSAT CM SAF (TCDR) and the production of an ICDR continuation. The ATBD document [D3] refers to the CM SAF documentation [D1] describing the methods and algorithms that are used by the CM SAF to generate the total column water vapour data products. The ICDRs are based on the same retrieval scheme and algorithm as TCDR, and therefore no further changes have been implemented, except for the continuation and extension of the time series processing.
The TCDR covers the time period January 1988 to December 2014, while the ICDR covers the time period from January 2015 to December 2020. The updated ICDR has the same characteristics as the TCDR, which has been generated within a CM SAF reprocessing activity.
In the scope of the validation activity, the comparison has been performed for accuracy and stability of the product and its continuation with the same method as within the CMSAF validation activity for total column water vapour monthly means (1988-2014) [D2]. In addition, the extended time series (ICDR) has been included for inter-comparison (2015-2020) to evaluate the overall bias and stability of the product toward reference sensors.
The general picture of validation presented in this document, shows an overall good performance of HOAPS 4.0 TCDR + the C3S ICDR (1988-2020). The target requirements (Table 3-1) set for the inter-comparison of bias and RMSD have been met at the optimal level. The stability has met the optimal level for RSS_SSMI and ERA-5 (see Table 3-3 for further details). The comparison toward AIRS V7 shows a significant drift. The number of ICDR data values outside the 95% interval of TCDRs, is within the expected critical range at 5% significance level, that has been performed with a statistical test for RSS_SSMI and ERA-5. For AIRS, the test rejects the null hypothesis for similarity between TCDRs and ICDRs (Table 3-5).
1. Validated products
The validation includes the CM SAF product TCWV SSM/I and SSMIS from HOAPS 4.0 retrieval, containing gridded monthly mean and 6-hourly total column water vapour data for the period 1988-2020 containing the TCDR (1988-2014) and the follow-up ICDR generated within the C3S project (2015-2020).
For the validation activity, only the monthly mean data sets have been used due to the lack of daily composite data sets as reference, and their consistent statistical representativeness for inter-comparison.
2. Reference datasets for validation
The validation of TCDR was primarily based on comparisons with ERA-Interim reanalysis, COSMIC (beta-version, ROM SAF), RSS_SSMI (SSM/I+SSMIS) V7, and TMI V7. The reference datasets are described and discussed in Sections 4.5-4.7 of CM SAF Validation Report [D2].
The validation of TCDR+ICDR is primarily based on comparisons with ERA-5, merged microwave sensors RSS_SSMI V7 (SSM/I+SSMIS), and AIRS V7:
- ERA-5 replaces the ERA-Interim reanalysis which stopped being produced on 31 August 2019.
- AIRS (Atmospheric Infrared Sounder) which have been launched on NASA’s Aqua satellite in September 2002 is an additional independent reference dataset for total column water that has been used for validation.
This is the list of the reference sensors used for the current inter-comparison activity (TCDR+ICDR) and their temporal coverage:
- RSS_SSMI V7 (1988/01-2020/12)
- ERA-5 (1988-2020/12)
- AIRS V7 (2002/09-2020/12)
3. Validation results
3.1 TCDR performance
The validation methodology is outlined in Section 6.1 [D2]. The results for total column water vapour are presented in Section 6.7.1 and further discussed in Section 6.7.2 of the CM SAF Validation Report [D2].
For TCWV, the target requirements are listed in Table 3-1.
The HOAPS 4.0 validation report [D2, Section 6.7.2] gives the following summary of quality. The HOAPS-4.0 monthly mean TCWV data show the following absolute bias and RMSD results, when compared against ERA-Interim and the satellite-based RSS_SSM and TMI products (Table 3-2):
- average (absolute) biases of <0.4 kg/m2 and
- RMSD of ≤1.1 kg/m2
Thus, the monthly product meets the optimal KPI for bias and the target KPI for RMSD.
The decadal stability of HOAPS 4.0 is 0.00±0.008 kg/m2/dec, which fulfills the requirements for ‘optimal category’ (<0.08 kg/m2/decacde) as described in Section 7 [D2].
Table 3-1: KPIs for the Water Vapour TCWV TCDR as defined by CMSAF (see Table 6-6 in [D2])
Category | Bias [kg/m2] | RMSD [kg/m2] | Stability [kg/m2/decade] |
Threshold | 3 | 5 | 0.4 |
Target | 1.4 | 2 | 0.2 |
Optimal | 1.0 | 1 | 0.08 |
The SSM/I and SSMIS 6-hourly daily composites fulfil the GCOS frequency requirement of 4-hourly observations when input data from different DMSP satellites are considered.
On the other hand, the spatial resolution of 50 km x 50 km does not fulfill the spatial resolution requirement set by GCOS (25 km), which is the only limitation of the data set.
Table 3-2: The validation results of HOAPS 4.0 TCDR for RSS_SSMI, ERA-Interim, and TMI dataset with their corresponding target requirements fulfillment as defined in Table 3-1. The numbers are taken from the CM SAF Validation Report [D2] Section 6.7.1 - 6.7.2 respectively.
Reference Dataset | Bias [kg/m2] | RMSD [kg/m2] | Stability [kg/m2/decade] |
RSS_SSMI: 1988-2014 | -0.35 optimal | 0.8 optimal | 0.00±0.008 optimal |
ERA-Interim: 1988-2014 | 0.3 optimal | 1.1 target | - |
TMI: 1998-2014 | -0.39 optimal | 1.0 optimal | - |
3.2 TCDR+ICDR performance
The key performance for TCDR+ICDR is calculated in the same manner as for TCDR. The validation metrics, i.e. bias, RMSD, stability, KPI test on ICDRS vs. TCDR with corresponding error estimates on bias and stability have been calculated for RSS-SSMI, ERA-5, and. The details on methods to derive the key validation metrics are described in Section 3 of PQAD [D4]. [D4] also contains detailed results of the TCDR+ICDR quality assessment and only a summary is provided here.
The summary of inter-comparison of HOAPS 4.0 TCDR+ICDR (1988-2020) is shown in Table 3-3 and Figures 3-1 - 3-3.
The overall bias (< 1 kg/m2) and RMSD (< 1 kg/m2) fulfil the optimal requirement for all three reference sensors. The largest bias is observed between HOAPS and ERA-5 (0.56 kg/m2). On the other hand the stability is fulfilled at the optimal level for RSS_SSMI at 100% and ERA-5 at 74%, respectively. The comparison with AIRS V7 shows significant increase of the trend of the bias (drift), where even the threshold requirement is violated. The reason for the drift is the synthetic micro-wave data that have been included in the retrieval scheme.
Table 3-3: The validation results of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) when compared with RSS_SSMI, ERA-5, and AIRS datasets with their corresponding target requirements fulfillment as defined in Table 3-1. In brackets the numerical values of probability coverage of requirements are shown if these are lower than 100%.
Reference Dataset | Bias [kg/m2] | sbias [kg/m2] | RMSD [kg/m2] | Lag-1 Autocorrelation r(1) | Stability (via s) [kg/m2/decade] | Stability (via spread) [kg/m2/decade] |
RSS_SSMI: 1988/01-2020/12 | -0.28 Optimal | 0.14 | 0.31 Optimal | 0.85 | 0.033±0.007 Optimal | 0.033±0.010 Optimal |
ERA-5: 1988/01-2020/12 | 0.56 Optimal | 0.63 | 0.85 Optimal | 0.63 | 0.067±0.020 Optimal (74%) Target | 0.067±0.028 Optimal (68%) Target |
AIRS: 2002/09-2019/12 | 0.20 Optimal | 0.41 | 0.45 Optimal | 0.81 | 0.651±0.029 Not met | 0.651±0.031 Not met |
Figure 3-1: Schematic plot of bias performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements.
Figure 3-2: Schematic plot of the RMSD performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements.
Figure 3-3: Schematic plot of stability performance of HOAPS 4.0 TCDR (CMSAF) & ICDR (C3S) minus reference sensors with corresponding target requirements.
3.1 Results
The KPI test of ICDRs vs. TCDRs leads to similar conclusions. The ICDR differences relative to the RSS_SSMI and ERA-5 validation datasets stay in the expected range for TCDRs (Table 3-4). Only 5 out of 72 ICDRs show larger differences, which is expected from the 95% confidence interval. On the other hand, due to large drift between HOAPS 4.0 SSMI(S) and AIRS V7, the number of ICDR failures (K=30) exceeds the critical number of 7. Hence the ICDR differences between HOAPS & AIRS V7 do not fulfill the null hypothesis of similarity between ICDRs and TCDR. Further details can be found in the PQAD [D4] and Report on Updated KPI's [D5].
Table 3-4: Results of the hypothesis test upon the number of ICDRs falling outside the 95% confidence interval of the TCDR.
Reference Dataset | TCDR lower threshold 2.5% [kg/m2] | TCDR upper threshold 97.5% [kg/m2] | Number of ICDRs | Critical number of failures at 5% rate | Observed K failures of ICDRs outside the 95% interval [2.5%,97.5%] | Cumulative probability P(N-K,N,95%) P >5% : Accept P < 5%: Reject |
RSS_SSMI: 1988/01-2020/12 | -0.52 | 0.00085 | 72 | K > 7 | 5 | 29% Accept |
ERA-5: 1988/01-2020/12 | -0.6 | 1.7 | 72 | K > 7 | 5 | 29% Accept |
AIRS: 2002/09-2019/12 | -0.28 | 0.62 | 72 | K > 7 | 30 | 0% Reject |
The general picture of validation summarized here and in the PQAD [D4], shows an overall good performance of HOAPS 4.0 TCDR (CMSAF) + ICDR (C3S) (1988-2020) toward the reference sensors/datasets RSS_SSMI and ERA-5.