The regression tests can be run automatically at regular intervals under ecFlow. This page details how to configure and create the suite.

Configuring the suite(s)

First of all clone the repo, then edit <system>/suite_config.yaml (where <system> is cams or c3s). This is the file that determines everything that is configurable about the structure of the suite.

At a minimum the file must define a suites list containing at least one dictionary with a name and a stacks key. The suite name is the top-level ecFlow suite that the families will be placed into. The stacks array lists the ADS/CDS stacks to test, e.g.

suites:
   - name: 'cams_noncrit'
     stacks:
       - camsprod

This example would create families in the cams_noncrit suite to test the camsprod stack. Other stacks could be added to the list.

If you have defined tests in your own branch of the cds-regression-testing repo then add "cds_regression_testing_branch: <your_branch>" to ensure that your branch is checked out by the suite, otherwise it will not find your tests. The default branch used is master.

There are many other possible settings that will override the defaults, such as settings that can limit the number of tests that run at once and the when_testing setting which is used to override other settings when creating a suite in test mode (see below). The full list can be seen by looking at the source of cds_regression_testing/make_suites.py

Creating and loading the suite definition

Once the config file is made the suite definition can be generated by running....

bin/make_ecflow_suites <system>

The suite definitions will, by default, be written to...

$SCRATCH/cds_regression_testing/<ads|cds>_regression_tests.<suite_name>.<host>.def

where <host> is the name of the ecFlow server to be used or "unknown_host" if not set. The ecFlow server and port can be set by adding "host" and "port" tags to suite definition (i.e. as tags at the same level as the suite "name" tag).

If a host and port are set then the make_ecflow_suites script can push the suite definition directly to the server by executing it with the "-l" option.

Running a test version

You may want to test your suite on a different server (for example, one running on your local desktop) first. To make this easier, if you run bin/make_ecflow_suites with the "-t" test-mode option then any settings under the "when_testing" tag in your suite config file will override equivalent settings at the top level, so for example...

suites:
   - name: 'cams_noncrit'
     server: 'operational_server'
     port: 1234
     stacks:
       - 'camsprod'

when_testing:
    suites:
       - name: 'cams_noncrit'
         server: 'localhost'
         port: 4321
         stacks:
           - 'camsprod'

... will mean the suite is loaded to operational_server when -t is not used and localhost when it is.

Suite structure and triggering

The suites will have the following structure...

suite/
  <ads|cds>_regression_tests/
    stack1/
      - execute_tests
      tests/
        dataset1/
          test1/
            - run_test
          test2/
            - run_test
          ...
        dataset2/
          ...
    stack2/
      - execute_tests
      tests/
        dataset1/
          ...

The suites are triggered by a cron which by default is set to daily at 5am. An alternative ecFlow cron string can be provided by setting the "cron_string" in the suite config YAML file.

At the times specified by the cron, the task <ads|cds>_regression_tests/<stack>/execute_tests will be run. This task will requeue all tasks in the stack below it, even if they are in an aborted state, and then execute the tests/setup task, which clones and pulls the required repos and sets required branches. After that the tests will run.

To manually trigger the tests for a stack, run the execute_tests task.

Error behaviour

If a test fails when running under ecFlow it will send an email to the suite owner, as defined by the %OWNER% ecFlow variable, which defaults to the user who ran bin/make_ecflow_suites. It can be overridden by setting the owner key at the top level of the suite config file. If set to a null string no email will be sent. Note that the email is sent by the trap.h ecFlow include file.

Note that the suite is self-resurrecting. When the cron next triggers, all test tasks which are in an aborted state will be requeued and re-run.