It is important to have measures of the effectiveness of the EFI in capturing extreme events.  The ability of the EFI to detect extreme events has been assessed using the area under the Relative Operating Characteristic (ROC) curve.

Verification of the EFI has been done using synoptic observations over Europe.  An extreme event is taken as having occurred if the observation exceeds the 95th percentile of the observed climate for that station for the given time of year (calculated from a 15-year sample).  The observed climatology is used as a reference point because here it is the effectiveness of the forecast relative to the model climate (M-climate) that is being assessed.  Verification has been performed for three directly represented parameters: 2m mean temperature, 10m mean wind speed, and total precipitation (Fig8.1.9.8-1). 

In addition, and in contrast to these three parameters, the EFIs for CAPE and for CAPE-shear (Fig8.1.9.8-2) are verified against the observed convective hazards and not against observations of these parameters.  CAPE and  CAPE-shear are used for forecasting severe convection.  If CAPE and CAPE-shear are anomalous compared to their respective climatologies, severe convective hazards are likely to happen.


Fig8.1.9.8-1: Skill scores (2*ROCA-1) for EFI regarding 2m mean temperatures, 10m mean wind, and total precipitation.  Colours indicate forecast lead-times.  Solid curves show a four-season running mean and dashed curves show seasonal EFI skill scores.  Forecasts that have no capacity to discriminate between when extreme events are likely and when they are not (which could be termed "random forecasts") yield a score of 0.  The plot shows that there has been improvement over the years in ECMWF's capacity to predict "extreme" events.  The lower skill in summer for precipitation is indicative of the difficulties of predicting rainfall at points in convective situations.  Since 2016, EFI performance as measured by these metrics has mostly been fairly constant, although wind speed skill for days 3-5 has improved slightly.

 

Fig8.1.9.8-2: Skill scores (2*ROCA-1) for EFI regarding CAPE and CAPE-shear during April-September 2016, verified against reports of tornadoes, severe wind and large hail within USA only (Blue) and Europe only (Red), as a function of lead time (x-axis). Both show a degree of skill.  The plots show that using the CAPE and CAPE-shear EFI fields are useful in the prediction of an enhanced risk of severe weather related to convection, even beyond day7.