5. Testing
5.1. make test
After MET has been compiled, run make test from the top-level directory to execute the scripts found in the scripts/examples directory. These scripts run a subset of the MET tools reading input data from the top-level data directory and configuration files from the scripts/config directory and write output to top-level out directory. Successful completion of these tests provides reasonable assurance that MET has been compiled well and is running properly. However, these sample scripts are not comprehensive and do not exercise all possible configuration options. So it’s possible for the make test scripts to run without error, but for users to later encounter issues when running MET with new inputs files and configuration options.
5.2. Unit Tests
The MET unit tests offer much more thorough testing coverage of the MET tools than running make test, as described above. These units tests provide the basis for the regression testing performed for each pull request. Logic exists in GitHub automation to run these unit tests and check for differences in the output. However, these unit tests can also be run locally and instructions for doing so are provided in this section.
5.2.1. Running Unit Tests
5.2.1.1. Obtain Input Data
To run the MET unit tests locally, the input data must be obtained. It can be downloaded here. Extract the data into a directory that will be used to set the MET_TEST_INPUT environment variable.
5.2.1.2. Set Up Environment
Set the required environment variables needed to run.
MET_BASE - Path to the share/met directory where MET is installed. This is used to find the MET executables and static data files.
MET_TEST_BASE - Path to the MET/internal/test_unit directory that contains the unit test files
MET_TEST_INPUT - Directory containing unit test input data
MET_TEST_OUTPUT - Directory to write test output
Example:
export MET_BASE=/path/to/install/MET/share/met
export MET_TEST_BASE=/path/to/src/MET/internal/test_unit
export MET_TEST_INPUT=/path/to/MET_unit_test
export MET_TEST_OUTPUT=/path/to/my/output_dir
Other environment variables required for some of the unit tests include:
MET_TEST_MET_PYTHON_EXE - Path to Python executable to use for Python Embedding tests
MET_TEST_RSCRIPT - Path to Rscript executable used to run R script tests
5.2.1.3. Run the tests
Navigate to the internal/test_unit directory of the MET repository:
cd ${MET_TEST_BASE}
To run all of the unit tests, call the bin/unit_test.sh script:
./bin/unit_test.sh
To run a single unit test group, call the python/unit.py script, passing it an XML test config file:
./python/unit.py ./xml/unit_pcp_combine.xml
To generate commands corresponding to a single unit test group, but not actually execute those commands, add the -cmd command line argument and redirect the output to a file:
./python/unit.py ./xml/unit_pcp_combine.xml -cmd > unit_pcp_combine.sh
Extracting individual commands to be executed in this way can be convenient during the software development process.
Note
Some unit tests depend on the output of other unit tests. For example, unit_plot_data_plane.xml requires output from unit_pcp_combine.xml. Those dependencies are generally noted in comments at the top of each unit test xml file.
5.2.2. Input Data
Input data used to run the MET unit tests in CI workflows are pulled from the DTC web server and stored on DockerHub. On the web server, data is stored for each supported version, e.g. v12.0, v12.1, etc. There is also a directory called develop that includes symbolic links to the latest version, which is the version that is currently in development. This is done so that the latest state of the input data is used for new development and the final state of the input data is preserved when an official release is created. The scripts described below can be used to easily update the input data and set up a new development cycle.
5.2.2.1. Setting up a new web server
Note
These instructions require access to run commands as the met_test user on the DTC web server.
The GitHub Actions custom action metplus-action-data-update expects a specific URL defined in update_data_volumes.py script in its repo. This directory should exist on the web server. This can be a link to another directory, but the name must match the repo name, e.g. MET. If this path must differ on a new web server, then modifications will be needed to the custom action.
The directory should also be linked from the met_test user’s home directory with the name MET_unit_test. This is done so the scripts used to update the input data and setup a new development cycle can easily find the directory without having to change the path on a new machine.
It is recommended to clone the MET repository in the met_test user’s home directory, check out the develop branch, and create a symbolic link in the home directory to the scripts used to update the input data and set up the next release directory. This is not necessarily required, but makes it convenient to find and call the script.
runas met_test
cd ~/
git clone --branch develop https://github.com/dtcenter/MET
ln -s MET/internal/scripts/unit_test_ci/setup_met_next_release_data.sh
ln -s MET/internal/scripts/unit_test_ci/update_met_unit_test_data.sh
The unit test input data directory contains directories for develop and each vX.Y version that is supported. Each directory should contain a tarfile called unit_test-all.tgz and a file called volume_mount_directories. Typically, the develop directory will contain symbolic links to the latest version’s files. The vX.Y directories will also contain a directory called unit_test that contains the input data.
If setting up a new web server, create the develop directory and add the input data tarfile and volume mount files, then run the script to setup the next development cycle (see Setup next development cycle). A sample volume mount file can be found in the MET repo under internal/scripts/unit_test_ci.
5.2.2.2. Setup next development cycle
Note
These instructions require access to run commands as the met_test user on the DTC web server.
Once the main_vX.Y branch for a release has been created, the develop branch will contain development towards the next release. At this time, a new set of test data should be created for the next release so that it can be updated while preserving the test data used for an official release. For example, if the main_v12.1 branch was created when the 12.1.0-rc1 release was created, then a data directory to store data for v13.0 (or similar) should be created.
Pull changes from develop to ensure that the latest version of script is used.
runas met_test
cd ~/MET
git checkout develop
git pull
Run the script, passing the vX.Y version of the next release as an argument. If the script is linked from the home directory, run:
~/setup_met_next_release_data.sh v13.0
This will create the v13.0 directory, copy the latest tarfile and volume mount files into v13.0, extract the tarfile contents into the v13.0, and update the symbolic links in the develop directory to point to the files in v13.0.
5.2.2.3. Adding new test files
Note
These instructions require access to run commands as the met_test user on the DTC web server.
Updates to the input data, e.g. adding new test files, are made on the DTC web server. The next time the MET CI unit tests are run, the web server will be checked and the input data will be updated automatically. Note that the unit tests are only run for develop/main branches or running via workflow dispatch. A push event to a branch will not run the full unit test suite and therefore will not update the input data.
In the MET_unit_test directory, there is a directory called unit_test. These files are the full set of fields and fields used for the unit tests. These files are used by the MET regression tests that are run locally.
First, add any new files to the unit_test directory so they will be available to the MET regression tests.
Example:
cp /path/to/my/file.ext MET_unit_test/unit_test/DIRNAME/
Next, add the new input files in the unit_test directory under the vX.Y directory that corresponds to the current development cycle.
Example:
cp /path/to/my/file.ext MET_unit_test/v23.1/unit_test/DIRNAME/
If any of the files are very large, consider creating a subset of these files. For example, GRIB2 files can be subset with wgrib2 and NetCDF files can be subset using NCO tools. After the updates have been made, run the script to update the test data tarfile.
Pull changes from develop to ensure that the latest version of script is used.
runas met_test
cd ~/MET
git checkout develop
git pull
Run the script, passing the vX.Y version of the next release as an argument. If the script is linked from the home directory, run:
~/update_met_unit_test_data.sh v13.0
This will save a copy the input data tarfile with the current date in YYYYMMDD format in case it needs to be recovered, then create the tarfile using the contents of the unit_test directory.
5.3. Regression Tests
Coming Soon!