The regression tests can be run automatically at regular intervals under ecFlow. This page details how to configure and create the suite.
Configuring the suite(s)
First of all clone the repo, then edit <system>/suite_config.yaml
(where <system>
is cams
or c3s
). This is the file that determines everything that is configurable about the structure of the suite.
At a minimum the file must define a suites
list containing at least one dictionary with a name
and a stacks
key. The suite name
is the top-level ecFlow suite that the families will be placed into. The stacks
array lists the ADS/CDS stacks to test, e.g.
suites: - name: 'cams_noncrit' stacks: - camsprod
This example would create families in the cams_noncrit suite to test the camsprod stack. Other stacks could be added to the list.
If you have defined tests in your own branch of the cds-regression-testing repo then add "cds_regression_testing_branch: <your_branch>
" to ensure that your branch is checked out by the suite, otherwise it will not find your tests. The default branch used is master
.
There are many other possible settings that will override the defaults, such as settings that can limit the number of tests that run at once and the when_testing
setting which is used to override other settings when creating a suite in test mode (see below). The full list can be seen by looking at the source of cds_regression_testing/make_suites.py
Creating and loading the suite definition
Once the config file is made the suite definition can be generated by running....
bin/make_ecflow_suites <system>
The suite definitions will, by default, be written to...
$SCRATCH/cds_regression_testing/<ads|cds>_regression_tests.<suite_name>.<host>.def
where <host>
is the name of the ecFlow server to be used or "unknown_host" if not set. The ecFlow server and port can be set by adding "host
" and "port
" tags to suite definition (i.e. as tags at the same level as the suite "name
" tag).
If a host and port are set then the make_ecflow_suites
script can push the suite definition directly to the server by executing it with the "-l
" option.
Running a test version
You may want to test your suite on a different server (for example, one running on your local desktop) first. To make this easier, if you run bin/make_ecflow_suites
with the "-t
" test-mode option then any settings under the "when_testing
" tag in your suite config file will override equivalent settings at the top level, so for example...
suites: - name: 'cams_noncrit' server: 'operational_server' port: 1234 stacks: - 'camsprod' when_testing: suites: - name: 'cams_noncrit' server: 'localhost' port: 4321 stacks: - 'camsprod'
... will mean the suite is loaded to operational_server
when -t
is not used and localhost
when it is.
Suite structure and triggering
The suites will have the following structure...
suite/ <ads|cds>_regression_tests/ stack1/ - execute_tests tests/ dataset1/ test1/ - run_test test2/ - run_test ... dataset2/ ... stack2/ - execute_tests tests/ dataset1/ ...
The suites are triggered by a cron which by default is set to daily at 5am. An alternative ecFlow cron string can be provided by setting the "cron_string
" in the suite config YAML file.
At the times specified by the cron, the task <ads|cds>_regression_tests/<stack>/execute_tests
will be run. This task will requeue all tasks in the stack below it, even if they are in an aborted state, and then execute the tests/setup
task, which clones and pulls the required repos and sets required branches. After that the tests will run.
To manually trigger the tests for a stack, run the execute_tests
task.
Error behaviour
If a test fails when running under ecFlow it will send an email to the suite owner, as defined by the %OWNER%
ecFlow variable, which defaults to the user who ran bin/make_ecflow_suites
. It can be overridden by setting the owner
key at the top level of the suite config file. If set to a null string no email will be sent. Note that the email is sent by the trap.h ecFlow include file.
Note that the suite is self-resurrecting. When the cron next triggers, all test tasks which are in an aborted state will be requeued and re-run.