Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test cleanup #519

Merged
merged 5 commits into from
Apr 26, 2023
Merged

Test cleanup #519

merged 5 commits into from
Apr 26, 2023

Conversation

hellkite500
Copy link
Contributor

Some long overdue cleanup of some test classes and cases. The changes here are mostly focused on implementing file path checking/substitution to allow the tests to discover required input data from different initial working directories. Ultimately, this now allows ctest to run effectively and do both test discovery and execution. So from the repo root directory, one can do the following:

Activate most all features for a build

make -B test_build -S . -DNETCDF_ACTIVE:BOOL=On -DBMI_C_LIB_ACTIVE:BOOL=On -DBMI_FORTRAN_ACTIVE:BOOL=On -DNGEN_ACTIVATE_PYTHON:BOOL=On -DNGEN_ACTIVATE_ROUTING:BOOL=On

Compile all targets (including activated tests for the various options)

cmake --build test_build --target all -j 8 

Discover and run all built tests

cmake --build test_build --target test

The builds and tests can also be done via make, e.g.

cd test_build
make test

What is important to note here, is that we have a couple standard test targets we test often, test_unit and test_all, which have 152 and 128 total possible tests, for a combined 280 tests. (using the above build flags).

When you run the entirety of the discoverable tests using ctest, there are 822 tests discovered! Even with the overlap in test_unit/test_all, there are clearly some test cases not be exercised frequently (some of where are fixed up in this PR).

This should have no regression on existing testing workflows, but should allow for more comprehensive ones, and we may even want to revisit the automated action runners and consider using the ctest interface to discover and run tests.

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Target Environment support

  • Linux
  • MacOS

Copy link
Contributor

@donaldwj donaldwj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see no problems. On the exception handling commits I don't think we are losing information so long as the caught exception puts all needed information into its e.what() method. Admittedly there is no guarantee that all child exception classes will do this.

@hellkite500 hellkite500 mentioned this pull request Apr 26, 2023
11 tasks
@hellkite500 hellkite500 merged commit 6ab044e into NOAA-OWP:master Apr 26, 2023
@mattw-nws
Copy link
Contributor

It looks like this was merged in spite of two failing tests, because the failing tests were believed to be fixed by 7550dfc (in #516) ... but after that commit, the same tests in this PR would have failed for a different reason, which is now happening in #524.

I am going to roll back some of the pathing changes in the dependent files in RoutingPyBindTest to fix the tests, but this may make some of the goals of this PR unachieved--particularly the cd test_build; make test workflow. However, since these tests rely on static YAML files with relative paths in them, there is not a better answer at the moment--I continue to doubt whether the ability to run the tests from three different locations in the hierarchy is a worthwhile goal, for this and other reasons. At a minimum (and perhaps maximum), the tests should work when run from the repo root, and making them run from anywhere else incurs cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants