ERIC flash flood forecast skill

ERIC forecast skill are calculated using a one-year ERIC reforecast time series. The reforecasts were forced with 6 hourly precipitation forecasts from COSMO-LEPS, and 6 hourly temperature and soil moisture data from the EFAS long term run. In each reforecast the standard ERIC web products were created. This includes a shapefile of all the reporting points for a given reforecast, the attribute table for this shapefile records the exceedance probability of the 2, 5 and 20 year return period thresholds and the lead time of the maximum value. This shapefile was used to perform the evaluation.

The evaluation is performed separately for each lead time (0-24 h, 24-48 h, 48-72 h, 82-96 h and 96-120 h). For each reforecast, the ERIC reporting points which fell within the lead time in question were thresholded into 'yes' or 'no' events based on the exceedance probability of the 2, 5 or 20 year return period thresholds. Different exceedance probability thresholds from 0% to 100% in 10% increments were applied to the 3 different return periods.  For reporting points which exceeded the probability criterion (i.e. 'yes' events), the ID values of the administration regions in which reporting points lay were extracted. These were compared against the ID values of the administration regions where flash floods were observed at the time in question. The comparison was used to populate a contingency table (see below), a hit was when a region ID was present in both the reforecast and the observation and so forth. This method of accounting means that if two or more reporting points occurred in the same region as an observation, only one hit will be recorded.

The contingency table was calculated for every reforecast and then summarised over the whole 1 year reforecast period for each lead time and for every exceedance probability threshold of all 3 return period levels.

To summarise the evaluation results the Hanssen-Kuipers discriminant metric (Hanssen & Kuipers 1965; see Woodcock 1976 for a review) was chosen:

Hanssen-Kuipers = (Hits / Hits+Misses) - (False Alarms / False Alarms + Correct Negatives)

It compares the ratio of event occasions with non-event occasions and shows how well the forecast can separate the 'yes' and 'no' events. It ranges from -1 to 1 with 1 being a perfect score and 0 indicating no skill. The metric uses all components of the contingency table. 

The evaluation results were also summarised using a Roebber plot (Roebber, 2009), which summarises four metrics of forecast performance:

Hit Rate (HR) = Hits / Hits + Misses

Success Rate = 1 - (False Alarms / False Alarms + Hits) = 1 - False Alarm Rate (FAR)

Threat Score (TS) = Hits / Hits + Misses + False Alarms

Bias (B) = (Hits + False Alarms) / (Hits + Misses)

A perfect forecast is in the top right corner of the Roebber plot.