Section 5.1 Generation of the Ensemble

• Section 5.1.1 Ensemble of Data Assimilations - EDA
• Section 5.1.2 Singular Vectors - SV
• Section 5.1.3 Stochastic Tendency Perturbations

Generation of the Ensemble

Process of calculating ensemble perturbations

Ensemble forecasts require many alternative analyses and forecasts and each ensemble member requires its own global perturbation.  Importantly, the analysis and subsequent forecast of each ensemble member must be truly independent of all the others.  The process of deriving independent perturbed analyses for ENS members is to use:

• A 50 member Ensemble of Data Assimilations (EDA) is calculated over the globe.  The differences of each member from the EDA mean gives 50 different sets of global EDA perturbations.
• Sets of Singular Vectors (SVs) are separately calculated over the Northern and Southern Hemispheres, and over the tropics between 30°N and 30°S.  These are linearly combined (using coefficients randomly sampled from a Gaussian distribution) to give 100 different sets of global SV perturbations.

Medium range ensemble perturbations

The medium range 10 day ensemble consists of 50 members and the unperturbed control member (CTRL).

The medium range 15 day ensemble consists of 50 members and the unperturbed control member (CTRL).

• The control member Analysis = the Analysis without any perturbations added.  
• The 50 sets of EDA perturbations and 50 sets of SVs are combined together to give 50 sets of global perturbations for model initialisation:
  • ENS member 1 Analysis = Analysis + (EDA member 1 - EDA mean) + SV Perturbation 1
    
  • ENS member 2 Analysis = Analysis - (EDA member 2 - EDA mean) + SV Perturbation 2
  •         and so on until 
  • ENS member 49 Analysis = Analysis + (EDA member 49 - EDA mean) + SV Perturbation 49
  • ENS member 50 Analysis = Analysis - (EDA member 50 - EDA mean) + SV Perturbation 50

Extended range ensemble perturbations

The extended range ensemble consists of 100 perturbed members and one unperturbed member (the extended range control member, CONTROL).

• The control member Analysis = the Analysis without any perturbations added.  
• The 50 sets of EDA perturbations and 100 sets of SVs are combined together to give 100 sets of global perturbations for model initialisation:
  • ENS member 1 Analysis = Analysis + (EDA member 1 - EDA mean) + SV Perturbation 1
    
  • ENS member 2 Analysis = Analysis + (EDA member 2 - EDA mean) + SV Perturbation 2
  •         and so on until
  • ENS member 49 Analysis = Analysis + (EDA member 49 - EDA mean) + SV Perturbation 49
  • ENS member 50 Analysis = Analysis + (EDA member 50 - EDA mean) + SV Perturbation 50
  • ENS member 51 Analysis = Analysis + (EDA member 1 - EDA mean) + SV Perturbation 51
  • ENS member 52 Analysis = Analysis + (EDA member 2 - EDA mean) + SV Perturbation 52
  •         and so on until
  • ENS member 99 Analysis = Analysis + (EDA member 49 - EDA mean) + SV Perturbation 99
  • ENS member 100 Analysis = Analysis + (EDA member 50 - EDA mean) + SV Perturbation 100

Additional Information

• See more on Ensemble Perturbations, particularly slide 16

Quality of the individual perturbed analyses

An unavoidable consequence of modifying the initial conditions around the most likely estimate of the truth (i.e. the 4D-Var analyses) is that the perturbed analysis is on average slightly degraded.  The RMS distance from truth for a perturbed analysis is, in the ideal case, on average √2 times the RMS distance of the unperturbed analysis from the truth (see Fig5.1-1).

 Fig5.1-1: A schematic illustration of why the perturbed initial conditions will, on average, be further from the true state of the atmosphere than the control analysis.  For a specific grid point, the analysis state (a) is known, as well as the average error of an analysis state from the true state (a-t).  The true state (t) is therefore, on this diagram, the analysis offset by the average error and therefore can lie anywhere on the darker circle radius (a-t) centred on the analysis (a).  Any perturbed analysis state (p) can be very close to the truth, but is in a majority of the cases much further away; in the ideal case the average distance is √2 times the analysis error ((a-t)√2).

Consequently, the proportion of the perturbed analyses that are better than the control (i.e. unperturbed analysis) for a specific location and for a specific parameter (e.g. 2m temperature or MSLP at Paris) is only 35% (see Fig5.1-2).  Considering more than one grid point lowers the proportion even further.

Fig5.1-2: A schematic illustration for a specific grid point of the special case where the perturbed analyses differ on average from the control analysis (radius of red circle) by as much as the control differs from the truth (TC). 

The average error of an analysis state from the true state (distance TC) is known from statistical investigations.  On this diagram therefore, the control analysis state is shown by the true state offset by the known average error between control and true states and, for each IFS model analysis, can lie anywhere on the blue curve centred on the true state (here the control analysis shown at C).  The average of the difference of the perturbed analyses from the control analysis is shown by the red circle centred on the control analysis, and the length of arc (ATB) is proportional, on average, to the number of perturbed analyses closer to the truth than the control analysis.   For the special conditions illustrated, only 35% of the perturbed analyses are, on average, closer to the truth than the control analysis.  As the average of the difference of the perturbed analyses from the control analysis varies (i.e. the radius of the red circle) so will the number of perturbed analyses (represented by the arc ATB) be, on average, closer to the truth than the control analysis.  The number will be zero where the red circle is tangent to the blue circle (i.e. where perturbed analyses are very similar to the control, or where the average of the difference of the perturbed analyses from the control analysis is twice (or more) the average error between control and true analyses.

If an ensemble member is closer to the truth than to the control in one place (e.g. Paris) it might not be so in another (e.g. Berlin).  Indeed, the larger the area, the less likely that any of the perturbed members are better than the unperturbed control analysis. For a region the size of a small ECMWF Member State, only about 7% of the perturbed analyses are, on average, better than the control analysis. For the larger Member States this decreases to only 2%.

With respect to the forecasts, in the short range only a small number of the perturbed forecasts are, on average, more skilful than the control forecast.  However, with increasing forecast range the average proportion of perturbed forecasts that are better than the control forecast increases, eventually approaching 50% asymptotically.

Fig5.1-3: Schematic representation of the percentage of perturbed forecasts with lower root mean square error than the control forecast for regions of different sizes: Northern Hemisphere, Europe, a typical ““small”” Member State and a specific location.  With increasing forecast range, fewer and fewer perturbed members are worse than the control (from Palmer et al 2006).

Quality of the individual perturbed forecasts

Since the perturbed analyses have, ideally on average, 41% larger analysis errors than ensemble mean, this makes the individual ensemble forecasts on average less skilful than the unperturbed control forecast.   Predictive skill varies with season and geographical location, but on average:

the predictive skill of the control is:

• about 1 day better than a perturbed member throughout the 10-day forecast period.

and the predictive skill of the ensemble mean is:

• better than the predictive skill of the control; with this gap widening for longer forecast leads

Skill levels vary with parameter, time of year and geographical location, so of course Fig5.1-4 is not universally applicable. Rather it gives a general guide for predictive skill for the extra-tropical synoptic pattern.

Fig5.1-4: Schematic image of the root mean square error of the ensemble members, ensemble mean and control forecast as a function of lead-time.  The asymptotic predictability limit is defined as the average difference between two randomly chosen atmospheric states.  In a perfect ensemble system the root mean square error of an average ensemble member is √2 times the error of the ensemble mean.

However, what the perturbed forecasts may lack in individual skill, they compensate for by their large number, their ability to form good median or ensemble mean values and reliable probability estimations.  The information from all the members in the ensemble should be used.  The low proportion of perturbed forecast members “better” than the control in the short range makes the task of trying to select the ensemble member with the best subsequent forecast very difficult and, perhaps, impossible.  There are no known methods to identify beforehand the “best” ensemble member beyond the first day or so (not least due to the effects of downstream spread of errors).

Additional sources of information

(Note: In older material there may be references to issues that have subsequently been addressed)

• See more information on the 50 member EDA.

• Read more about quantifying forecast uncertainty.
• Read more about combined use of EDA- and SV-based perturbations in the EPS.
• Read more on the use of Stochastic Parameterisation Schemes in Ensemble Data Assimilation (Section3)
• Watch a comprehensive lecture on Singular Vectors and Ensemble of Data Analyses.
• Read more on Stochastic methods for representing model uncertainties in the IFS.

_{Updated/Amended 06/07/20 - The Ensemble Perturbations.}