Introduction

This section shows how to configure and run OpenIFS and XIOS together.

First of all, it is necessary that OpenIFS is compiled with XIOS (see OpenIFS: Enable XIOS during compilation) and the XIOS executable is available in the experiment directory. OpenIFS should be run as usual (see 6.3 OpenIFS Running a Forecast), but adding the argument '--xios' to oifs_run. This will use by default 1 XIOS server, but in case of needing more servers, it can be done with '--xios --xios-nproc <arg>'.

It is also necessary to enable the 'LXIOS' switch in the 'NAMCT0' namelist (fort.4 file).

Output control: XML files

It is expected that users have knowledge on the use of XIOS which includes the creation of NetCDF files, computation of diagnostics, interpolations, data compression, interpretation of performance parameters, etc. Official documentation such as the user guide or tutorials is available on the webpage: https://forge.ipsl.jussieu.fr/ioserver. However, this section will explain XIOS aspects that are particular to OpenIFS.

OpenIFS and XIOS are controlled using different XML files:

• iodef.xml: it contains basic parameters to control the working mode of XIOS (attached or server mode, 1 or 2 levels of servers, etc), performance parameters, verbose level, etc.
• context_ifs.xml: it contains FullPos parameters to tune the vertical interpolations as well as two optimizations that might improve the execution time.
• axis_def_ifs.xml: it defines the different types of vertical levels available in OpenIFS: model levels, pressure levels, theta levels and PV levels.
• domain_def_ifs.xml: it defines the different domains (or types of grids) available in OpenIFS. By default, data can be output in the native grid (reduced Gaussian) or in a regular Gaussian grid with size 512x256.
• grid_def_ifs.xml: it defines the grids (XIOS terminology) available in OpenIFS used to map fields:
  • Grids for 2D fields are made of a single domain.
  • Grids for 3D fields are made of a domain and an axis.
• field_def_ifs.xml: it defines all the available fields to be output in OpenIFS:
  • Surface fields.
  • 3D fields in model levels.
  • 3D fields in pressure levels.
  • 3D fields in theta levels.
  • 3D fields in PV levels.
  • 3D PEXTRA fields in model levels.
• file_def_ifs.xml: it defines NetCDF files to be written. It allows you to configure many different parameters such as output frequency, compression, diagnostics, output mode (one file vs. multiple files), metadata, timeseries, etc.

Let's see in more detail how to configure different features particular to OpenIFS-XIOS.

FullPos parameters

In context_ifs.xml different FullPos variables can be tuned. There is a group of variables that control the spectral fitting of FullPos and a second group that contains four different variables named 'NFPCLI', 'LFPQ', 'LTRACEFP' and 'RFPCORR'. More information about all these variables can be found at How to control OpenIFS output and in the latest FullPos user guide.

Optimizations for sending data from OpenIFS processes to XIOS servers

There are two available optimizations that might be useful to improve the execution time under some circumstances. It is not possible to predict in which conditions, as many factors have to be taken into consideration. For this reason it is necessary to test them. They are disabled by default, but can be enabled in context_ifs.xml:

• 'LOPT_SEND': it enables a mechanism to change where data is sent from XIOS clients (OpenIFS processes) to XIOS servers to truly overlap OpenIFS computations with XIOS communications. 'Changing where data is sent' means to buffer output data and delay its sending until physics tendencies computation which is free of MPI communication. If this optimization is not enabled, output data is sent to XIOS servers immediately when it is produced by FullPos, 'colliding' with OpenIFS synchronous internal communications.
• 'LSINGLE_PREC_SEND': it sends data from XIOS clients to XIOS servers in single precision (32 bits) instead of double precision (64 bits). This allows you to send half of the data to decongest the network.

Define different vertical level values

It is possible to define different vertical levels to be interpolated from model levels in axis_def_ifs.xml. In particular, it is possible to define pressure levels, theta levels and PV levels. For example, if you want to define four theta levels, you should edit 'n_glo' and 'value':

<axis id="theta_levels" long_name="vertical theta (potential temperature) levels" n_glo="4" value="(0,3)[300 320 340 360]" />

In addition, it is possible to output fields with a subset of the vertical levels. This XIOS feature is known as zoom and you have to specify the indexes (from the original axis) of the levels that you want to output, and NOT the level values themselves. From the previous example, if you are only interested in theta levels 320 and 360, you should edit 'index' as follows:

<zoom_axis index="(0,1)[1 3]" />

Note that indexes go from '0' to 'n_glo - 1'.

Define different regular Gaussian domains

If the user chooses to output data in a regular lat-lon grid, by default fields use a grid with 256 latitudes and 512 longitudes. However, it is possible to change the sizes of the regular grid or even to declare more than one regular grid. Domains for regular grids are declared in domain_def_ifs.xml. There are two key attributes:

• 'ni_glo': the number of longitudes.
• 'nj_glo': the number of latitudes.

In addition, it is necessary to specify '<generate_rectilinear_domain />' to indicate that a horizontal interpolation is needed. The attribute 'write_weight' controls whether to write interpolation weights or not.

Define different grids from new axes and domains

New axes and domains might be defined to output fields, but it is recommended for 2D fields and mandatory for 3D fields to declare new grids made up of these new axes and domains. Grids are declared in grid_def_ifs.xml, so different possible combinations of axes (axis_def_ifs.xml) and domains (domain_def_ifs.xml) should go here. It is then possible to make use of this new grid to output fields (see next section).

Enable regular lat-lon grid output

With a well-defined regular lat-lon grid it is possible to use it to output fields. file_def_ifs.xml controls the netCDF files to be output with the corresponding fields. To output a field in a regular lat-lon grid it is necessary to use the attribute 'grid_ref'. For example, to output the 3D temperature field in a 256x512 lat-lon grid and model levels, you can use this XML code:

<field field_ref="t" name="t" grid_ref="regular_ml" operation="instant" />

where 'regular_ml' is defined in grid_def_ifs.xml as follows:

<grid id="regular_ml" description="3D interpolated regular grid with hybrid model levels" >
  <domain domain_ref="regular" />
  <axis axis_ref="model_levels" />
</grid>

Understanding output frequency, sampling frequency, NFRHIS and NFRPOS

It is very important to distinguish the difference between attributes 'output_freq' (output frequency) and 'freq_op' (sampling frequency), and how 'NFRHIS' and 'NFRPOS' are intrinsically related to 'freq_op'.

While 'output_freq' and 'freq_op' are attributes that belong to XIOS, 'NFRHIS' and 'NFRPOS' variables belong to FullPos. Although it would have been desirable to automatically set up these two FullPos variables during runtime (instead of giving the 'responsibility' to the user, which might be a source of unnecessary errors), it is necessary to manually set up them in context_ifs.xml because the XIOS API does not give the possibility of doing it.

That being said, let's see how to correctly set up them. The definition and use of the two XIOS attributes is the following one:

• 'output_freq': it controls the writing frequency of a netCDF file. For example, if the 'output_freq' is 3h and you run a 1-day forecast, you should have (if 'freq_op' is set correctly) a netCDF with 8 slices of data: 3h, 6h, 9h, 12h, 15h, 18h, 21h and 24h.
• 'freq_op': it controls the sampling frequency, this is, the frequency of sending data from the model (OpenIFS) to XIOS. For example, if you have a 'freq_op' of 1h, at each hour OpenIFS will send those variables to XIOS.

Two examples to illustrate how these two variables work and how 'NFRHIS' and 'NFRPOS' must be set up.

Example 1: you only want to output instant values of 't' and 'cc' fields every 6 hours.

• You should set up 'output_freq=6h'.
• You should set up 'freq_op=6h' for both 't' and 'cc'.
• Since there is only a single value of 'freq_op', you should set up both 'NFRHIS' and 'NFRPOS' to the number of time steps that are equivalent to 'freq_op', i.e. 6 hours. For example, if the time step duration is 45 min, the equivalent is 8 time steps (6h). If there were more than one 'freq_op' values, you should use the greatest common divisor (gcd) of them (see Example 2).

Example 2: you want to output these three variables:

• 't': output every 6 hours the average of hourly data. You should set up 'output_freq=6h' and 'freq_op=1h'.
• 'u': output every 12 hours the maximum of 3 hourly data. You should set up 'output_freq=12h' and 'freq_op=3h'.
• 'cc': output every 6 hours instant data. You should set up 'output_freq=6h' and 'freq_op=6h'.
• 'NFRHIS' and 'NFRPOS' must be set up taking the gcd of 'freq_op' values across all fields ('t', 'u' and 'cc'). The gcd of the three different 'freq_op' is: gcd(1, 3, 6)=1h. If the time step duration is 900 seconds for instance, then 'NFRHIS' and 'NFRPOS' should be set up to 4 (1h).

Note that always 'NFRHIS' = 'NFRPOS'. It is also possible to specify these two FullPos variables in hours. To do so, you have to use negative values:

• Example 1: 'NFRHIS' = 'NFRPOS' = '-6'
• Example 2: 'NFRHIS' = 'NFRPOS' = '-1'

Computational performance considerations

This section reviews different aspects that can considerably affect the computational performance of OpenIFS-XIOS and consequently increase the total execution time.

One file vs. multiple file mode

When XIOS is run with more than one server and compiled with parallel netCDF support, there is the possibility to output single netCDF files, i.e. all servers write in parallel into one netCDF file. However, this 'one_file' mode has a limited computational efficiency as it does not scale well when outputting a large volume of data for high resolution configurations.

On the contrary, the 'multiple_file' mode achieves a good computational efficiency as it scales with many resources. However, each XIOS server writes its own netCDF file, so output data is splitted between all these partial files. It is required later to use a post-process script to combine all these chunked netCDF files into a single one.

Horizontal interpolations

In order to output fields in a regular lat-lon grid XIOS needs to perform a horizontal interpolation. However, this operation is very expensive depending on different parameters (number of fields, size of the fields, frequency, etc), so it can have a large impact on the total execution time. This happens because interpolations are performed on the client side of XIOS, which means that OpenIFS processes have to first compute these interpolations before resuming the time stepping. The recommendation is to only use horizontal interpolations when it is necessary and not systematically.

Not correctly setting freq_op, NFRHIS and NFRPOS

It is very important to set up 'NFRHIS' and 'NFRPOS' FullPos variables at a frequency equal to the great common divisor (gcd) of all the different values of 'freq_op' so that it produces correct data and does not waste computational resources. This means that if we set up 'NFRHIS' and 'NFRPOS' with a value that is smaller than the gcd, then we would call FullPos unnecessarily (post-processed data will not be sent to XIOS as it is not required). This increases the computational cost with no gains in the accuracy of the results.

Spectral transformations

XIOS only supports fields in grid-point space, so it is not possible to directly output OpenIFS native spectral fields. To overcome this issue, the OpenIFS-XIOS integration makes use of the internal TRANS package to transform spectral fields to grid-point fields before sending them to XIOS. This operation is applied in conjunction with the FullPos post-processing, so it is also controlled with the 'NFRHIS' and 'NFRPOS' variables. This transformation operation is not cheap, and as said previously, it is very important to correctly set up 'NFRHIS' and 'NFRPOS' to make a responsible use.

Lustre filesystem

When OpenIFS is run on a cluster that uses the Lustre filesystem, it is important to pay attention to the striping. Lustre stores a file in one or more Object Storage Target (OST) devices. By default, it uses one OST, but it is possible to divide the file into chunks that are stored in different OSTs. This is known as striping, and it is used for performance purposes, especially for really big files.

It is therefore important to set up a striping for each netCDF at least as equal as to the number of XIOS servers when using the 'one_file' mode. This allows each XIOS server to write into a different OST. If the number of OSTs per file was not changed, it could affect the performance of the whole system, including the rest of the users that are performing I/O at that moment because it could fill up the OST where the big netCDF file would be. When such OST runs out of space, it also affects other user's jobs.