Last modified on Nov 08, 2023 17:56

Contributors: E. Carboni (UKRI-STFC RAL Space), G.E. Thomas (UKRI-STFC RAL Space)produced in

Issued by: STFC RAL Space (UKRI-STFC) / Elisa Carboni

Date: 31/05/2023

Ref: C3S2_D312a_Lot1.2.2.4-v4.0_202305_PQAR_CCIEarthRadiationBudget_v1.2

Official reference number service contract: 2021/C3S2_312a_Lot1_DWD/SC1 

History of modifications

List of datasets covered by this document

Related documents

Acronyms

List of tables

List of figures

General definitions

The "CCI product family" Climate Data Record (CDR) consists of two parts. The ATSR2-AATSR Earth Radiation Budget CDR is formed by a TCDR brokered from the ESA Cloud_cci project and an ICDR derived from the Sea and Land Surface Temperature Radiometer (SLSTR) on board of Sentinel-3A and -B. ICDR uses the same processing and infrastructure as the TCDR. Both TCDR and ICDR data have been produced by STFC RAL Space.

These Earth Radiation Budget datasets from polar orbiting satellites consist of two main variables:

Outgoing Longwave Radiation (OLR): The outgoing longwave flux, measured at the top of the atmosphere.

Reflected Solar radiation Flux (RSF): The reflected solar flux, measured at the top of the atmosphere.

Bias (accuracy): Mean difference between TCDR/ICDR and reference data

\( b=\frac{\sum_{i=1}^N (p_i - r_i)}{N} \ \ (Eq. 1) \)

Where: p_i is the CDR product, b is the mean bias and r_i is the equivalent value from the reference dataset. N is the number of observations.

bc-RMSE (precision): Bias corrected root mean squared error to express the precision of TCDR/ICDR compared to a reference data record

\( bc- RMSE=\sqrt{\frac{\sum_{i=1}^N ((p-b)-r)^2}{N}} \ \ (Eq. 2) \)

Where: p_i is the CDR product, b is the mean bias and r_i is the equivalent value from the reference dataset. N is the number of observations.

Stability: The variation of the bias over a multi-annual time period

Table 1: Definition of processing levels

Table 2: Definition of various technical terms used in the document

Scope of the document

This document provides a description of the product validation results for the Climate Data record (CDR) of the Essential Climate Variable (ECV) Earth Radiation Budget. This CDR comprises inputs from two sources: (i) brokered products from the Cloud Climate Change Initiative (ESA's Cloud_cci), namely those coming from processing of the Advanced Along-Track Scanning Radiometer ((A)ATSR) data and (ii) those produced under this contract for the Climate Data Store, specifically those coming from processing of the Sea and Land Surface Temperature Radiometers (SLSTR).

The Thematic Climate Data Record (TCDR) is the product brokered from the European Space Agency Cloud Climate Change Initiative (ESA's Cloud_cci) ATSR2-AATSR version 3.0 (Level-3C) dataset. This is produced by the Science and Technology Facilities Council (STFC), RAL Space from the second Along-Track Scanning Radiometer (ATSR-2) on board the second European Remote Sensing Satellite (ERS-2) which spanned the period 1995-2003 and the Advanced ATSR (AATSR) on board ENVISAT which spanned the period 2002-2012.

In addition, the Interim Climate Data Record (ICDR) is the product derived from the SLSTR instrument on board of Sentinel-3A and -B and spans the period from January 2017 to present. Validation for this SLSTR derived product for the period from January 2017 to March 2022 is described in this document.

Executive summary

The ESA Climate Change Initiative (CCI) Earth Radiation Budget Data Record (TCDR) is a brokered product from the ESA Cloud_cci project, while the extension Interim CDR (ICDR) produced from the Sea and Land Surface Temperature Radiometer (SLSTR) is produced specifically for C3S. The product is generated by STFC RAL Space, using the Community Cloud for Climate (CC4CL) processor, based on the Optimal Retrieval of Aerosol and Cloud (ORAC) algorithm. Please find further information in the Algorithm Theoretical Basis Document (ATBD) [D3].

The Cloud_cci dataset comprises 17 years (1995-2012) of satellite-based measurements derived from the Along Track Scanning Radiometers (ATSR-2 and AATSR) onboard the ESA second European Research Satellite (ERS-2) and ENVISAT. This TCDR is partnered with the ICDR produced from the Sentinel-3A SLSTR, beginning in 2017, and Sentinel-3B SLSTR beginning in October 2018.

The TCDR and ICDR provide level-3 data (monthly means) on a regular global latitude-longitude grid (with a resolution of 0.5° x 0.5°) and include these products: Outgoing Longwave Radiation (OLR) and Reflected Solar radiation Flux (RSF) at Top of Atmosphere (TOA). Table 2-1 (see section 2) provides a summary of the TCDR accuracies. For the ICDR, an initial validation with CERES (using the first 5 years of SLSTR data) show biases consistent with the TCDR: with a bias of 4.9 and 3.2 W/m² for RSF (SLSTR-A and B respectively) and -1.7 W/m² for OLR.

This document is divided in different sections: 

• the first section presents a brief description of the validation methodology together with a series of references for further information;
• the second section presents the results of the validation and comparison of TCDR and ICDR data;
• the third section presents the compliance with user requirements and includes recommendation on the usage and know limitations

1. Product validation methodology

The validation methodology is described in section 2.4 of [D1]. In summary, the methodology uses the bias between the Cloud_cci product and the reference data to estimate the accuracy of the dataset. The bias corrected root mean squared error (bc-RMSE) is used to express the precision of the TCDR compared to a reference data record, this is also known as the standard deviation about the mean. For the validation, the CDR dataset is compared with Clouds and Earth Radiation Energy System (CERES) Energy Balanced and Filled (EBAF) fluxes Edition 4.1 Top of atmosphere (TOA) (Loeb et al., 2018)¹.

The stability for the TCDR dataset is defined as the variation of the bias over a multi-annual time period. It is obtained by calculating the linear trend of the bias between the TCDR and reference dataset (in this case CERES dataset).

The Product Validation and Intercomparison Report [D1] includes the validation and intercomparison of the TCDR Earth Radiation Budget versus the CERES satellite dataset. The same methodology is used for the ICDR.

 2. Validation results

The validation results for the TCDR are provided in [D1] section 3 and 5. Table 2-1 provides a summary of the resulting TCDR accuracies.

Table 2-1: Summary of the TCDR and preliminary assessment of ICDR accuracy of the Earth Radiation Budget dataset.

2.1 TCDR validation with CERES satellite data

The TCDR and reference dataset are compared by calculating multi-annual mean and standard deviation, all for a common time period (2003-2011). Global maps of monthly and multiannual Outgoing Longwave Radiation (OLR) and Reflected Solar radiation Flux (RSF) are computed for the TCDR and reference dataset. The scores (bias and bc-RMSE) are calculated by including all valid data points pairwise in the CERES and the Cloud_cci dataset.

The validation for Outgoing Longwave Radiation (OLR) and Reflected Solar radiation Flux (RSF) at TOA with the CERES dataset is described in section 3.3.1, 5.1 and 5.2 of [D1].

Validation of Cloud_cci radiation products with CERES present a bias of 5.72 W/m² and standard deviation of 1.64 W/m² for RSF, and a bias of 1.72 W/m² and standard deviation of 1.12 W/m² for OLR.

General findings:

RSF (from [D1] section 5.1)

• The CDR datasets show very similar patterns to the other Cloud_cci datasets of the global mean RSF. Highest mean RSF is found in the subtropics over land, lowest mean RSF is also found in the subtropics over the ocean.
• RSF show higher temporal variability over land areas.
• The time series plots of RSF show significant seasonal cycles in the global (60°S-60°N) mean with higher values in boreal winter and lower values in boreal summer.

OLR (from [D1] section 5.1)

• Mean global OLR are lowest over Antarctica and highest over the subtropics. Despite visible differences in the spatial resolution, all Cloud_cci datasets show very similar global patterns and comparable mean values.
• Stratocumulus regions are strongly pronounced with the highest TOA upwelling thermal radiation means. In case of the eastern Pacific, a strong gradient between the sea and land surface is noticeable. Stratocumulus regions around Africa and Australia show less differences between land and sea, which is probably due to the different topographic conditions.

• Higher temporal variability is found over land than over the ocean. Subtropical land areas show the largest temporal variabilities, e.g. Southeast Asia. The stratocumulus regions and southern hemispheric storm track region show the lowest temporal variability.

• Time series plots of OLR highlight a significant seasonal cycle in the global (60° S - 60° N) mean with maximum values in boreal summer and minimum values in boreal winter.

2.2 ICDR validation with CERES satellite data

The first 5 years of SLSTR products have been compared against CERES following the same methodology as described in [D1]. We estimate the bias, i.e. mean differences, and the monthly mean global average of C3S and the CERES data. To compute the monthly mean global average of both datasets we considered only the valid data between 60° S and 60° N latitude.

Validation with CERES shows biases consistent with the TCDR: with a bias of 3.8, 4.19 and 4.63 W/m² for RSF (SLSTR-A, B and combined A+B respectively) and -0.97, -1.32 and -1.03 W/m² for OLR.

Figure 2-1 and 2-2 show an example of the ICDR monthly products for March 2017 and the equivalent monthly product from CERES. These figures are for illustrative purposes so the user knows what to expect. Nonetheless, note that for this month, the ICDR and CERES datasets are spatially similar for both variables. However, there are some small differences observed. For example CERES RSF seems a little higher in the southern Indian ocean, and also slightly higher over Northern hemisphere land areas. For a more detailed analysis please go to the [D1].

Figure 2-1: RSF and OLR from SLSTR (ICDR dataset) for March 2017

Figure 2-2: RSF and OLR from CERES dataset for March 2017

3. Application(s) specific assessments

In addition to the extensive product validation (see chapter 2 for results and chapter 2/3 in [D5] for validation methodology) a second assessment is introduced to evaluate the Interim Climate Data Record (ICDR) against the Thematic Climate Data Record (TCDR) in terms of consistency. Since frequent ICDR deliveries make detailed validation not feasible, a consistency check against the deeply validated TCDR is used as an indication of quality. This is done by a comparison of the following two evaluations:

• TCDR against a stable, long-term and independent reference dataset
• ICDR against the same stable, long-term and independent reference dataset

The evaluation method is generated to detect differences in the ICDR performance in a quantitative, binary way with so called Key Performance Indicators. The general method is outlined in [D4] chapter 3. The same difference between TCDR/ICDR and the reference dataset would lead to the conclusion that TCDR and ICDR have the same quality (key performance is "good"). Variations or trends in the differences (TCDR/ICDR against reference) would require a further investigation to analyze the reasons. The key performance would be marked as "bad". The binary decision whether the key performance is good or bad is made in a statistical way by a hypotheses test (binomial test). Based on the TCDR/reference comparison (global means, monthly or daily means) a range is defined with 95% of the differences are within. This range (2.5 and 97.5 percentile) is used for the ICDR/reference comparison to check whether the values are in or out of the range. The results could be the following:

• All or a sufficient high number of ICDR/reference differences lies within the range defined by the TCDR/reference comparison: Key performance of the ICDR is "good"
• A smaller number of ICDR/reference differences is within the pre-defined range: Key performance of the ICDR is "bad"

3.1 Results

The results of the KPI test are summarized in Table 3-1.

Table 3-1: Summary of KPI results with 2.5 and 97.5 percentiles and number of ICDR months within the range. Colors green or red mark the results of the binomial tests as good or bad, respectively.

Percentiles were calculated based on the comparison of the TCDR using the Advanced Along Track Scanning Radiometer (AATSR) instrument against CERES as reference dataset for the variables Outgoing Longwave Radiation (OLR) and Reflected Shortwave Flux (RSF). Percentiles were based on the time from 2002-2012 with monthly means and applied to the ICDR from 01/2017 (10/2018) to 06/2022 for Sentinel-3A (Sentinel-3B and merged product Sentinel-3A+B) based on measurements of the Sea and Land Surface Temperature Radiometer (SLSTR).

A part of the ICDR months are outside the TCDR-based KPI limits and leading to “bad” KPI tests. For these, the ICDR is not stable in relation to the TCDR. This is due to multiple reasons starting with the fact of a five year gap (2012-2016) between TCDR and ICDR. In addition, TCDR and ICDR are based on different instruments with SLSTR on Sentinel-3 and (A)ATSR/ATSR-2 on Envisat/ERS-2, respectively. Differences occur due to a lower bias between ICDR and reference dataset and a subtraction of the monthly means (based on the TCDR) to remove the annual cycle leads to values outside of the KPI range (see method in [D4], chapter 3.2.2). Please note that significant changes between 01/2017 - 12/2020 and 01/2017 - 12/2021 are due to bugfixes.

4. Compliance with user requirements

There are no direct user requirements for the Earth Radiation Budget defined in the Cloud_cci project. Looking at the GCOS ECV requirements for Earth Radiation Budget [https://gcos.wmo.int/en/essential-climate-variables/earth-radiation/ecv-requirements] (Last accessed on 28/02/2023) the values for RSF and OLR are 1 W/m² uncertainty, while the TCDR dataset achieves an accuracy of 5.72 W/m² for RSF and 1.72 W/m² for OLR, therefore they currently do not meet the GCOS requirements. ICDR accuracies (estimate with the dataset up to march 2022) are consistent with TCDR accuracy for OLR and slightly lower for RSF. Please find more detailed information about the target requirements in the corresponding (Target Requirement and Gap Analysis Document) TRGAD [D2].

Table 4-1 provides an overview of the GCOS requirements for the Earth Radiation Budget and the values achieved by the products distributed in the CDS. It should be noted that GCOS requirements are targets and are often not attainable using existing or historical observing systems. The Cloud_cci products distributed via the CDS do not meet the requirement for resolving the diurnal cycle due to the nature of the satellite observations, but exceeds the spatial resolution.

Table 4-1: GCOS targets for Earth Radiation Budget ECVs and CDR values.

Known limitations [From D1 table 7.1]:

• Higher uncertainties in twilight conditions, especially in the shortwave fluxes, due to limitation in retrieving Cloud Optical Thickness (COT) and Cloud Particle Effective Radius (CER) (input to the radiation calculation) in these conditions.
• Partly sparse temporal/spatial sampling, partly compensated by introduced diurnal cycles correction
• Somewhat higher uncertainties expected for TOA shortwave fluxes (RSF) for conditions with low clouds frequencies and elevated surface albedo uncertainties.

References

Loeb, N.G., Doelling, D.R., Wang, H., Su, W., Nguyen, C., Corbett, J.G., Liang, L., Mitrescu, C., Rose, F.G., and Kato, S.: Clouds and the Earth's Radiant Energy System (CERES) Energy Balanced and Filled (EBAF) Top-of-Atmosphere (TOA) Edition 4.0 Data Product, J.Climate, 31(2), 895–918, doi:10.1175/JCLI-D-17-0208.1, 2018.

Related articles

• Page:
  ERB TSI: Product User Guide and Specification (PUGS)
• Page:
  ERB TSI: Product Quality Assurance Document (PQAD)
• Page:
  ERB TSI: Product Quality Assessment Report (PQAR)
• Page:
  ERB TSI: System Quality Assurance Document (SQAD)
• Page:
  ERB TSI: Algorithm Theoretical Basis Document (ATBD)