This document aims at describing the workflow used for the ~200 non-regression studies that are executed before every release. The goal of these tests is to avoid any unexpected change in the results, thus ensuring compatibility of the new versions with the old ones.
This is a work-in-progress
There are two repositories :
- Antares_Simulator_Tests_NR contains the study batches. These batches are published in releases and do not contain any result.
- SimTest contains the study batches as well as some reference results.
This repository publishes releases. Each release contains the following files :
study-batches.txt, a newline-separated list of study batches. A study batch is a group of Antares studies in the same directory. Note that some studies may also be placed in a sub-directory.
- Archives containing the study batches.
Here is an excerpt from the latest
short-tests long-tests-1 long-tests-2 long-tests-3
Here is a list of the corresponding archives:
These archives can be found at the following URL:
https://github.com/AntaresSimulatorTeam/Antares_Simulator_Tests_NR/releases/download/<release tag>/<archive name>
In each study, there is a config.json file containing the necessary information for running tests.
This is the source of releases. Each release contains the following files:
study-batches.txt. It is merely a copy of
study-batches.txt from Antares_Simulator_Tests_NR.
- For each study batch listed above, two archives containing results for Ubuntu and Windows.
As above, config.json are packaged in the archives: one per study.
Here is a list of archives corresponding to the batches above:
In the future, CentOS 7 should be included as well.
The workflow is the following:
- Create a release in Antares_Simulator_Tests_NR (only if new studies are introduced). Tag it
- Create a release in SimTest (only if a new feature is introduced) from Antares_Simulator_Tests_NR, tag
Xand Antares_Simulator, tag
- Use the release in SimTest to compare results of tag
Zin Antares_Simulator. In this step,
pytestis used as well as the
config.jsonconfiguration files to test new results against the reference results.
Note that steps 1. and 2. are rarely used, therefore there is no need to automate them. Step 3. however is to be run for every Antares_Simulator release, as such it should be automated.