Selection of stations

GloFAS forecast skill assessment is conducted on all GloFAS diagnostic points.

Background on GloFAS forecasting

The forecast quality of GloFAS river discharge forecasts (out to 30 day lead time) has been evaluated in order to provide additional information to users to aid decision-making. The forecast evaluation method is described in Harrigan et al. (2020a) and summarised below with the results provided in the following page: GloFAS v3.1 forecast skill as well as summarised as a new headline score included as a "Forecast skill" layer on the GloFAS web map viewer: GloFAS forecast skill product.

Headline forecast skill score 

The headline forecast skill score is the maximum lead time (in days), up to 30-days ahead, in which the Continuous Ranked Probability Skill Score (CRPSS) is greater than a value of 0.5, when compared to a persistence or climatology benchmark forecast using GloFAS-ERA5 historical river discharge reanalysis (also known as Forced simulation (sfo) within CEMS) as proxy observations (Harrigan et al., 2020b). Forecast skill is calculated using river discharge reforecasts for a set of past dates, based on a configuration as close as possible to the operational setting. ECMWF-ENS medium and extended range reforecasts are used and are run twice per week for the past 20-years with 11 ensemble members.

Scores are shown on the GloFAS map viewer in the 'Forecast skill' layer under the 'Evaluation' menu (an example of the layer is shown here: GloFAS forecast skill product). For each GloFAS web reporting point, the maximum lead time the CRPSS is greater than 0.5 is given, with stations with purple circles having high skill for at longer lead times. The category "0", marked as light pink circles represents stations that have forecast skill lower than the 0.5 threshold for any lead time. Note: This does not mean that a station has no skill. Only when the CRPSS ≤ 0 is when the forecast has no skill when compared to a persistence or climatology benchmark forecast. 

Method

Reforecasts

ECMWF-ENS medium and extended range reforecasts are generated every Monday and Thursday, for the same date in the past 20 years for 11 ensemble members out to a lead time of 30 days. For the GloFAS forecast skill evaluation, ECMWF-ENS reforecasts run over a year long reference period (for example, January to December 2019 for GloFAS version 2.2) are used. These are then forced through the GloFAS hydrological modelling chain to produce 20 years of river discharge reforecasts, twice weekly, for 11 ensemble members. A schematic of the ECMWF-ENS reforecast configuration is given in Figure 1 for the reference period January to December 2019. In total, there are 2080 start dates in a GloFAS reforecast set (52 weeks x 2 per week x 20 years). GloFAS river discharge reforecasts are run for each river cell with an upstream area > 1000 km2 at a 24 hr time-step out to a lead time of 30 days with 11-ensemble members each.

Figure 1: ECMWF-ENS reforecast configuration schematic for the reference period January to December 2019. 

Benchmark forecasts

Following Pappenberger et al. (2015) and because GloFAS produces seamless forecasts across short, medium and extended lead times (day 1 to 30), two benchmarks are considered here, each calculated for all GloFAS diagnostic river points: persistence, typically used for short lead times where the forecast signal is dominated by serial correlation of river discharge, and climatology, typically used for longer lead times where the forecast signal is dominated by the seasonality of river discharge defined as follows:

  • persistence benchmark forecast defined as the single GloFAS-ERA5 daily river discharge of the day preceding the reforecast start date. The same river discharge value is used for all lead times.
  • climatology benchmark forecast based on a 40-year climatological sample (1979-2018) of moving 31-day windows of GloFAS-ERA5 river discharge reanalysis values, centred on the date being evaluated (+- 15 days). From each 1240-valued climatological sample (i.e. 40 years 31-day window), 11 fixed quantiles (Qn) at 10 % intervals were extracted (Q0, Q10, Q20, , Q80, Q90, Q100). The fixed quantile climate distribution used therefore varies by lead time, capturing the temporal variability in local river discharge climatology.

Proxy observations

Forecast skill is evaluated against GloFAS-ERA5 river discharge reanalysis, as a proxy to river discharge observations, for diagnostic points across the GloFAS domain. The advantages of using reanalysis instead of in situ river discharge observations is that the forecast skill can be determined independently from the hydrological model error and having a complete spatial and temporal coverage, so that forecast skill can be determined across the full GloFAS domain. Users must be aware that the key assumption with the proxy observation approach is that the GloFAS-ERA5 hydrological performance is reasonably good for the station of interest. If the hydrological model performance is poor, then particular care must be made in interpreting forecast skill scores. Full assessment of the hydrological performance of GloFAS-ERA5 against a global network of observations can be found in Harrigan et al. (2020b). 

Skill score

The ensemble forecast performance is evaluated using the Continuous Ranked Probability Score (CRPS) (Hersbach, 2000), one of the most widely used headline scores for probabilistic forecasts. The CRPS compares the continuous cumulative distribution of an ensemble forecast with the distribution of the observations . It has an optimum value of 0 and measures the error in the same units as the variable of interest (here river discharge in m3 s-1). It collapses to the mean absolute error for deterministic forecasts (as is the case here for the single-valued persistence benchmark forecast). The CRPS is expressed as a skill score to calculate forecast skill, CRPSS, which measures the improvement over a benchmark forecast and is given in:

\[ {CRPSS}={1-}\frac{{CRPS_{fc}}}{{CRPS_{bench}}} \]

A CRPSS value of 1 indicates a perfect forecast, CRPSS > 0 shows forecasts more skilful than the benchmark, CRPSS = 0 shows forecasts are only as accurate as the benchmark, and a CRPSS < 0 warns that forecasts are less skilful than  the benchmark forecast. The headline GloFAS forecast skill score uses a CRPSS threshold of 0.5 in the summary layer in the GloFAS web map viewer, this can be interpreted as the GloFAS forecast is 50% more accurate than the benchmark forecast.

The CRPSS is calculated with GloFAS reforecasts against both persistence and climatology benchmark forecasts and verified against GloFAS-ERA5 river discharge reanalysis as proxy observations. CRPSS headline scores are then mapped on the GloFAS map viewer, and CRPSS and CRPS time-series plots are produced for each fixed reporting point station.

References

Alfieri, L., Pappenberger, F., Wetterhall, F., Haiden, T., Richardson, D., Salamon, P., 2014. Evaluation of ensemble streamflow predictions in Europe. Journal of Hydrology 517, 913–922.

Harrigan, S., Zoster, E., Cloke, H., Salamon, P., and Prudhomme, C., 2020a. Daily ensemble river discharge reforecasts and real-time forecasts from the operational Global Flood Awareness System, Hydrol. Earth Syst. Sci. Discuss., https://doi.org/10.5194/hess-2020-532, in review.

Harrigan, S., Zsoter, E., Alfieri, L., Prudhomme, C., Salamon, P., Wetterhall, F., Barnard, C., Cloke, H., and Pappenberger, F., 2020b. GloFAS-ERA5 operational global river discharge reanalysis 1979–present, Earth Syst. Sci. Data, 12, 2043–2060, https://doi.org/10.5194/essd-12-2043-2020.

Hersbach, H., 2000. Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems. Wea. Forecasting 15, 559–570.

Pappenberger, F., Ramos, M.H., Cloke, H.L., Wetterhall, F., Alfieri, L., Bogner, K., Mueller, A., Salamon, P., 2015. How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction. Journal of Hydrology 522, 697–713.