From a23777bce75818577736e57c57b223bfb4d3b46c Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Thu, 8 Feb 2024 20:14:56 +0000 Subject: [PATCH 1/7] Add doc/AUTOMATED_TEST.md --- doc/AUTOMATED_TEST.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 doc/AUTOMATED_TEST.md diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md new file mode 100644 index 0000000000..50d34381f0 --- /dev/null +++ b/doc/AUTOMATED_TEST.md @@ -0,0 +1,17 @@ +# Github Automated Testing + +## What Is Automated Test and How to Start It + +In **ngen** github repo, we use github actions/workflows to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his fork and creates a `Pull Request`. The successful test is marked by a green check mark, a failed test is marked by a red cross mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail?](#what-to-do-when-a-test-fail?) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. + +## What Code Tests Are Performed + +- Unit tests: this inclodes every set of codes that serves a unique functionality. Unit test eveolves as new codes are added. +- BMI (Basical Model Interface) formulation interface tests including codes in C, C++, Fortran, and Python. +- Running **ngen** executable on example hydrofabric with various realistic modules/models, initial condition, and forcing data. + +## What to Do When a Test Fail? + +- Before getting into debugging, first thing, we recommend that you perform all the necessary tests listed above on you local computer. If all tests are successful, then push up your codes to the intended branch in your fork. +- Sometimes, some tests may fail even they have passed tests on local computer. If that happens, you have to look into details why they failed. To do that, click on the word `Details` in blue on the right for a particular test. This will open a window with detailed information for that particular test, the error information are usually at the bottom. You can scroll up and down the side bar for more information. you can also search for key workds in *Search logs* at the upper right corner. +- Othertimes, if you are lucky, the test may have failed due to time out, in that case, you may rerun the test by pointing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is recommended taht you carefully study the fail error first. From d8f30d3acf7b065a035ec7a110570da7337872a6 Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Thu, 8 Feb 2024 22:07:57 +0000 Subject: [PATCH 2/7] Revised version --- doc/AUTOMATED_TEST.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md index 50d34381f0..24ac1c1a4a 100644 --- a/doc/AUTOMATED_TEST.md +++ b/doc/AUTOMATED_TEST.md @@ -1,17 +1,20 @@ # Github Automated Testing +- [What Is Automated Test and How to Start It](#what-to-do-when-a-test-fail) +- [What Code Tests Are Performed](#what-code-tests-are-performed) +- [What to Do When a Test Fail](#what-to-do-when-a-test-fail) ## What Is Automated Test and How to Start It -In **ngen** github repo, we use github actions/workflows to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his fork and creates a `Pull Request`. The successful test is marked by a green check mark, a failed test is marked by a red cross mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail?](#what-to-do-when-a-test-fail?) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. +In **ngen** github repo, we use github actions/workflows to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. ## What Code Tests Are Performed - Unit tests: this inclodes every set of codes that serves a unique functionality. Unit test eveolves as new codes are added. -- BMI (Basical Model Interface) formulation interface tests including codes in C, C++, Fortran, and Python. +- [BMI](https://bmi.readthedocs.io/en/stable/) (Basical Model Interface) based formulations tests including codes in C, C++, Fortran, and Python. - Running **ngen** executable on example hydrofabric with various realistic modules/models, initial condition, and forcing data. -## What to Do When a Test Fail? +## What to Do When a Test Fail - Before getting into debugging, first thing, we recommend that you perform all the necessary tests listed above on you local computer. If all tests are successful, then push up your codes to the intended branch in your fork. -- Sometimes, some tests may fail even they have passed tests on local computer. If that happens, you have to look into details why they failed. To do that, click on the word `Details` in blue on the right for a particular test. This will open a window with detailed information for that particular test, the error information are usually at the bottom. You can scroll up and down the side bar for more information. you can also search for key workds in *Search logs* at the upper right corner. -- Othertimes, if you are lucky, the test may have failed due to time out, in that case, you may rerun the test by pointing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is recommended taht you carefully study the fail error first. +- Sometimes, some tests may fail even they have passed tests on local computer. If that happens, you have to look into details why they failed. To do that, click on the word `Details` in blue on the right for a particular test. This will open a window with detailed information for that particular test, the error information are usually at the bottom. You can scroll up and down the side bar for more information. you can also search by key workds in `Search logs` menu entry at the upper right corner. +- Othertimes, if you are lucky (or unlucky), the test may have failed due to time out or some mysterious reasons, in that cases, you may rerun the test by placing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is recommended that you carefully examine the fail error first. From d3d5da64b31103ca9c106dc7fc164901674f5139 Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Fri, 9 Feb 2024 16:02:10 +0000 Subject: [PATCH 3/7] Add a section on local testing --- doc/AUTOMATED_TEST.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md index 24ac1c1a4a..9ca13a40f3 100644 --- a/doc/AUTOMATED_TEST.md +++ b/doc/AUTOMATED_TEST.md @@ -1,6 +1,7 @@ # Github Automated Testing - [What Is Automated Test and How to Start It](#what-to-do-when-a-test-fail) - [What Code Tests Are Performed](#what-code-tests-are-performed) +- [Local Testing](#local-testing) - [What to Do When a Test Fail](#what-to-do-when-a-test-fail) ## What Is Automated Test and How to Start It @@ -13,6 +14,21 @@ In **ngen** github repo, we use github actions/workflows to automatically test t - [BMI](https://bmi.readthedocs.io/en/stable/) (Basical Model Interface) based formulations tests including codes in C, C++, Fortran, and Python. - Running **ngen** executable on example hydrofabric with various realistic modules/models, initial condition, and forcing data. +## Local Testing +We strongly recommend you run all the tests on your local computer before you push up the new codes to the branch in your fork. To be able to run the tests, when building the executables (for `build` with `cmake` see [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/BUILDS_AND_CMAKE.md), you need to set the test option to `ON` by setting: + + `-DNGEN_WITH_TESTS:BOOL=ON` + +After `build` complete, suppose your build directory is `cmake_build`, you can check that the `ngen` executable is in `cmake_build` directory, and all `unit test` executables are in `cmake_build/test` directory. To run a unit test, for example, run the following command while in the top project directory: + + `./cmake_build/test/test_unit` + +To run a ngen job, using the data set in the `data` directory, and run the following command: + + `./cmake_build/ngen data/catchment_data.geojson '' data/nexus_data.geojson '' data/example_bmi_multi_realization_config.json` + +To run a multi-processors job with MPI, please see a complete description in [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/DISTRIBUTED_PROCESSING.md) + ## What to Do When a Test Fail - Before getting into debugging, first thing, we recommend that you perform all the necessary tests listed above on you local computer. If all tests are successful, then push up your codes to the intended branch in your fork. From 1390c7423efe79d6f3b1bcb3180c6a14fb1bb148 Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Fri, 9 Feb 2024 16:35:10 +0000 Subject: [PATCH 4/7] Add a reference --- doc/AUTOMATED_TEST.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md index 9ca13a40f3..252e092321 100644 --- a/doc/AUTOMATED_TEST.md +++ b/doc/AUTOMATED_TEST.md @@ -6,7 +6,7 @@ ## What Is Automated Test and How to Start It -In **ngen** github repo, we use github actions/workflows to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. +In **ngen** github repo, we use github actions/workflows (for an online reference, see for [example](https://docs.github.com/en/actions/learn-github-actions) to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. ## What Code Tests Are Performed @@ -15,7 +15,8 @@ In **ngen** github repo, we use github actions/workflows to automatically test t - Running **ngen** executable on example hydrofabric with various realistic modules/models, initial condition, and forcing data. ## Local Testing -We strongly recommend you run all the tests on your local computer before you push up the new codes to the branch in your fork. To be able to run the tests, when building the executables (for `build` with `cmake` see [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/BUILDS_AND_CMAKE.md), you need to set the test option to `ON` by setting: + +We strongly recommend you run all the tests on your local computer before you push up the new codes to the branch in your fork. To be able to run the tests, when building the executables (for `build` with `cmake` see [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/BUILDS_AND_CMAKE.md), you need to set the test option to `ON` using `cmake` option: `-DNGEN_WITH_TESTS:BOOL=ON` From 97b1dc00b37e4d9aac8ff256e0b3fa60fa68bec6 Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Fri, 9 Feb 2024 17:01:09 +0000 Subject: [PATCH 5/7] Fix some text errors --- doc/AUTOMATED_TEST.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md index 252e092321..6dbbbfd507 100644 --- a/doc/AUTOMATED_TEST.md +++ b/doc/AUTOMATED_TEST.md @@ -6,25 +6,26 @@ ## What Is Automated Test and How to Start It -In **ngen** github repo, we use github actions/workflows (for an online reference, see for [example](https://docs.github.com/en/actions/learn-github-actions) to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. +In **ngen** github repo, we use github actions/workflows (for an online reference, see for [example](https://docs.github.com/en/actions/learn-github-actions)) to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. ## What Code Tests Are Performed -- Unit tests: this inclodes every set of codes that serves a unique functionality. Unit test eveolves as new codes are added. +- Unit tests: this includes every set of codes that serves a unique functionality. Unit test eveolves as new codes are added. - [BMI](https://bmi.readthedocs.io/en/stable/) (Basical Model Interface) based formulations tests including codes in C, C++, Fortran, and Python. - Running **ngen** executable on example hydrofabric with various realistic modules/models, initial condition, and forcing data. ## Local Testing -We strongly recommend you run all the tests on your local computer before you push up the new codes to the branch in your fork. To be able to run the tests, when building the executables (for `build` with `cmake` see [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/BUILDS_AND_CMAKE.md), you need to set the test option to `ON` using `cmake` option: +We strongly recommend you run all the tests on your local computer before you push up the new codes to the branch in your fork. To be able to run the tests, when building the executables (for `build` with `cmake` see [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/BUILDS_AND_CMAKE.md)), you need to set the test option to `ON` using `cmake` option: `-DNGEN_WITH_TESTS:BOOL=ON` -After `build` complete, suppose your build directory is `cmake_build`, you can check that the `ngen` executable is in `cmake_build` directory, and all `unit test` executables are in `cmake_build/test` directory. To run a unit test, for example, run the following command while in the top project directory: +After `build` completes, assuming your build directory is `cmake_build`, you can check that the `ngen` executable is in `cmake_build` directory, and all `unit test` executables are in `cmake_build/test` directory. To run a unit test, for example, run the following command while in the top project directory: `./cmake_build/test/test_unit` -To run a ngen job, using the data set in the `data` directory, and run the following command: +There are many other unit test executables you need to run for a complete test. +To run a ngen test job, for example, using the data set in the `data` directory, and run the following command: `./cmake_build/ngen data/catchment_data.geojson '' data/nexus_data.geojson '' data/example_bmi_multi_realization_config.json` @@ -33,5 +34,7 @@ To run a multi-processors job with MPI, please see a complete description in [he ## What to Do When a Test Fail - Before getting into debugging, first thing, we recommend that you perform all the necessary tests listed above on you local computer. If all tests are successful, then push up your codes to the intended branch in your fork. -- Sometimes, some tests may fail even they have passed tests on local computer. If that happens, you have to look into details why they failed. To do that, click on the word `Details` in blue on the right for a particular test. This will open a window with detailed information for that particular test, the error information are usually at the bottom. You can scroll up and down the side bar for more information. you can also search by key workds in `Search logs` menu entry at the upper right corner. -- Othertimes, if you are lucky (or unlucky), the test may have failed due to time out or some mysterious reasons, in that cases, you may rerun the test by placing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is recommended that you carefully examine the fail error first. + +- Sometimes, some tests may fail even they have passed tests on local computer. If that happens, you have to look into details why they failed. To do that, click on the word `Details` in blue on the right for a particular test. This will open a window with detailed information for that particular test, the error information are usually near the bottom. You can scroll up and down the side bar for more information. you can also search by key workds in `Search logs` menu entry at the upper right corner. + +- Othertimes, if you are lucky (or unlucky depending on your perspective), the test may have failed due to time out or some unknown reasons, in that cases, you may rerun the test by placing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is strongly recommended that you carefully examine the fail error first. From 680b14f0a501761eda8da35f291ef4f43592ccdc Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Thu, 15 Feb 2024 17:08:08 +0000 Subject: [PATCH 6/7] Revise command lines --- doc/AUTOMATED_TEST.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md index 6dbbbfd507..24968de3c1 100644 --- a/doc/AUTOMATED_TEST.md +++ b/doc/AUTOMATED_TEST.md @@ -18,16 +18,16 @@ In **ngen** github repo, we use github actions/workflows (for an online referenc We strongly recommend you run all the tests on your local computer before you push up the new codes to the branch in your fork. To be able to run the tests, when building the executables (for `build` with `cmake` see [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/BUILDS_AND_CMAKE.md)), you need to set the test option to `ON` using `cmake` option: - `-DNGEN_WITH_TESTS:BOOL=ON` + -DNGEN_WITH_TESTS:BOOL=ON After `build` completes, assuming your build directory is `cmake_build`, you can check that the `ngen` executable is in `cmake_build` directory, and all `unit test` executables are in `cmake_build/test` directory. To run a unit test, for example, run the following command while in the top project directory: - `./cmake_build/test/test_unit` + ./cmake_build/test/test_unit There are many other unit test executables you need to run for a complete test. To run a ngen test job, for example, using the data set in the `data` directory, and run the following command: - `./cmake_build/ngen data/catchment_data.geojson '' data/nexus_data.geojson '' data/example_bmi_multi_realization_config.json` + ./cmake_build/ngen data/catchment_data.geojson '' data/nexus_data.geojson '' data/example_bmi_multi_realization_config.json To run a multi-processors job with MPI, please see a complete description in [here](https://github.com/stcui007/ngen/blob/ngen_automated_test/doc/DISTRIBUTED_PROCESSING.md) From 37a217c6a3ee477166ae249062d5a15d0f7ccf43 Mon Sep 17 00:00:00 2001 From: Shengting Cui Date: Mon, 26 Feb 2024 21:51:00 +0000 Subject: [PATCH 7/7] Incorporate reviewer's comments --- doc/AUTOMATED_TEST.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/doc/AUTOMATED_TEST.md b/doc/AUTOMATED_TEST.md index 24968de3c1..6224cf56ae 100644 --- a/doc/AUTOMATED_TEST.md +++ b/doc/AUTOMATED_TEST.md @@ -1,17 +1,17 @@ # Github Automated Testing -- [What Is Automated Test and How to Start It](#what-to-do-when-a-test-fail) +- [What Are Automated Tests and How to Start Them](#what-are-automated-tests-and-how-to-start-them) - [What Code Tests Are Performed](#what-code-tests-are-performed) - [Local Testing](#local-testing) - [What to Do When a Test Fail](#what-to-do-when-a-test-fail) -## What Is Automated Test and How to Start It +## What Are Automated Tests and How to Start Them -In **ngen** github repo, we use github actions/workflows (for an online reference, see for [example](https://docs.github.com/en/actions/learn-github-actions)) to automatically test the validity of commited codes by developers. The test is triggered when a developer pushes some codes to a branch in his **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your codes (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit and push to the same branch again and the automatic testing will restart in the `Pull Request`. +In the **ngen** github repo, we use github actions/workflows (for an online reference, see for [example](https://docs.github.com/en/actions/learn-github-actions)) to automatically test validity of committed codes by developers. The test is triggered when a developer pushes some code to a branch in their **ngen** fork and creates a `Pull Request`. The successful test is marked by a `green check` mark, a failed test is marked by a `red cross` mark. If a test fails, you have to debug your code (see [What to Do When a Test Fail](#what-to-do-when-a-test-fail) below) and commit/push to the same branch again. The automatic testing will restart in the `Pull Request`. ## What Code Tests Are Performed -- Unit tests: this includes every set of codes that serves a unique functionality. Unit test eveolves as new codes are added. -- [BMI](https://bmi.readthedocs.io/en/stable/) (Basical Model Interface) based formulations tests including codes in C, C++, Fortran, and Python. +- Unit tests: this includes every set of codes that serves a unique functionality. Unit test evolve as new codes are added. +- [BMI](https://bmi.readthedocs.io/en/stable/) (Basic Model Interface) based formulation tests for C, C++, Fortran, and Python. - Running **ngen** executable on example hydrofabric with various realistic modules/models, initial condition, and forcing data. ## Local Testing @@ -25,7 +25,7 @@ After `build` completes, assuming your build directory is `cmake_build`, you can ./cmake_build/test/test_unit There are many other unit test executables you need to run for a complete test. -To run a ngen test job, for example, using the data set in the `data` directory, and run the following command: +For example, to run an **ngen** test job using the data set in the `data` directory, use the following command: ./cmake_build/ngen data/catchment_data.geojson '' data/nexus_data.geojson '' data/example_bmi_multi_realization_config.json @@ -37,4 +37,4 @@ To run a multi-processors job with MPI, please see a complete description in [he - Sometimes, some tests may fail even they have passed tests on local computer. If that happens, you have to look into details why they failed. To do that, click on the word `Details` in blue on the right for a particular test. This will open a window with detailed information for that particular test, the error information are usually near the bottom. You can scroll up and down the side bar for more information. you can also search by key workds in `Search logs` menu entry at the upper right corner. -- Othertimes, if you are lucky (or unlucky depending on your perspective), the test may have failed due to time out or some unknown reasons, in that cases, you may rerun the test by placing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is strongly recommended that you carefully examine the fail error first. +- Otherwise, if you are lucky (or unlucky depending on your perspective), the test may have failed due to time out or some unknown reasons, in those cases, you may rerun the test by placing the cursor on the test name, a cycling icon will appear and you can rerun your test by clicking on the icon. In any case, you may manually rerun any failed test by following procedure described above. But it is strongly recommended that you carefully examine the fail error first.