Explicitly configured tests

These are tests that have been explicitly written and stored in the regression testing repo under a path of the form <system>/test_<dataset_name>.py, where <system> is cams or c3s (ADS or CDS).

Tests minimally consist of one or more API request dictionaries for the dataset in the filename. They can also return a summary data structure describing aspects of the expected output data.

The output from the test requests will be compared against the output summary and any differences will be highlighted and result in a failed test.

Requests which are expected to fail can also be tested, with the resulting exception compared against the expected exception.

This is most useful when testing a dataset for the first time, when no reference stack/dataset exists to compare against.

Tests can be assigned a category which allows subsets of tests to be run easily.

Tests auto-generated from sample.json files

In order to avoid writing explicit tests, you can ask the system to auto-generate tests from the dataset sample.json files.

Run on a single stack, the system will just check that the request succeeds with little-to-no checking on the returned data. But when comparing two stacks, the system will check both stacks return the same data, using a comparison that is more intelligent that a simple binary diff.

This makes the sample.json tests quite powerful when checking that a test stack behaves the same as a production stack for a large number of published datasets.

These tests are assigned the pseudo-category, "samples".

Tests auto-generated from broker database requests

Users can do unexpected and chaotic things, so to increase confidence that the stack will handle the full range of what users will throw at it you can ask the system to take recently completed (successful) requests from the broker database and create tests from them.

Like tests from the sample.json files, little-to-no-checking will be done on the output data if only one stack is specified, but when comparing two stacks it will make sure both return the same data.

These tests are assigned the pseudo-category, "broker".