Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CTest Test Suite Delimiter prevents running all tests #4092

Closed
Jesseeee opened this issue Sep 23, 2024 · 6 comments · Fixed by #4154
Closed

CTest Test Suite Delimiter prevents running all tests #4092

Jesseeee opened this issue Sep 23, 2024 · 6 comments · Fixed by #4154
Labels
bug a bug in the product Feature: ctest Feature: Test Explorer
Milestone
1.20

Comments

@Jesseeee
Copy link

Jesseeee commented Sep 23, 2024 •
edited
Loading

Brief Issue Summary

When using the test explorer and enabling the test suite delimiter, I see it correctly get split up and I can run the test suites separately, however when I click on the run of the application itself in the explorer, execute "CMake: run tests" or "Test: run all tests" it only runs a single test suite (presumably the first one it finds)

One peculiarity is that when I click on the run button to run them all, it first shows "0/49" which is the correct amount of tests and then only ends up with "15/15" from that first test suite

CMake Tools Diagnostics

No response

Debug Log

No response

Additional Information

No response

Example project & gif:

ezgif-3-3c37dcf0ed

ctest-project.zip
@github-project-automation github-project-automation bot moved this to Blocked in CMake Tools Sep 23, 2024
@Yingzi1234
Copy link
Collaborator

@Jesseeee We can't reproduce the problem based on the information you provided, can you provide us with a project that can reproduce the problem? This will go a long way in helping us to investigate this problem, thank you

@Yingzi1234 Yingzi1234 added more info needed More info is needed from the community for us to properly triage and investigate. and removed triage labels Sep 24, 2024
@Jesseeee
Copy link
Author

Jesseeee commented Sep 24, 2024 •
edited
Loading

2024-09-24 11-58-02 (online-video-cutter com)

Hello @Yingzi1234 , hopefully this gif makes the issue more clear, so when just adding the delimiter nothing is seen as run, I click start and would expect all tests to run, and only one test suite is run.

@Yingzi1234
Copy link
Collaborator

Hi @Jesseeee , we can't reproduce your issue on latest CMake tools version 1.20.10(pre-release), you can get the details repro steps below, please let us know if it is wrong. Thank you!

ENV:
C/C++: V1.21.6
CMake tools: v1.20.10(pre-release)
VSCode: 1.93.1

Repro steps:

  1. Open this folder with VSCode
  2. Click "Run tests" button
  3. Open settings page and set the "CTest: Test Suite Delimiter" to "."
  4. Re-click "Run tests" button
  5. Check if all the test suites can be run successfully

Actual result:
All test suites can be run successfully
4092

@Jesseeee
Copy link
Author

Jesseeee commented Sep 25, 2024 •
edited
Loading

Hello @Yingzi1234 that's not really what I'm running into, I've made a small example project because I can't share my actual project, but it demonstrates the issue well, it's really when you have what is seen as 2 separate test suites and you try to run them all, be that through clicking that button or executing a command

ezgif-3-3c37dcf0ed

ctest-project.zip

@Yingzi1234
Copy link
Collaborator

Yingzi1234 commented Sep 26, 2024 •
edited
Loading

@Jesseeee Thanks for the update on this issue, we can reproduce the issue based on the items you provided, you can get the specific information below.
@gcampbell-msft We were able to reproduce this issue on the latest version, and we have added the bug label to this issue, you can get the exact reproduction steps and reproduction information from below, thanks!

ENV:
C/C++: V1.21.6
CMake tools: v1.20.10(pre-release)
VSCode: 1.93.1

Repro steps:

  1. Open this folder ctest-project.zip with VSCode
  2. Click "Run tests" button
  3. Open settings page and set the "CTest: Test Suite Delimiter" to "."
  4. Re-click "Run tests" button
  5. Check if all the test suites can be run successfully

Actual result:
Only one suite can be run successfully after setting the value of “CTest: Test Suite Delimiter”. When this setting is not set, only all suites can be run successfully.
Repro_issue4092

@Yingzi1234 Yingzi1234 added bug a bug in the product and removed more info needed More info is needed from the community for us to properly triage and investigate. labels Sep 26, 2024
@gcampbell-msft gcampbell-msft added this to the On Deck milestone Sep 30, 2024
@gcampbell-msft gcampbell-msft moved this from Blocked to Pending Prioritization in CMake Tools Sep 30, 2024
hippo91 added a commit to hippo91/vscode-cmake-tools that referenced this issue Nov 2, 2024
@hippo91
fix: CTest Test Suite Delimiter prevents running all tests (microsoft…
681d3d9 
…#4092)
@hippo91 hippo91 mentioned this issue Nov 2, 2024
Merged
hippo91 added a commit to hippo91/vscode-cmake-tools that referenced this issue Nov 3, 2024
@hippo91
fix: CTest Test Suite Delimiter prevents running all tests (microsoft…
afa0530 
…#4092)

* Introduces type alias (DriverMapT) to factorize type declaration
* Introduces the getProjectDriver method
* Adds comments
hippo91 added a commit to hippo91/vscode-cmake-tools that referenced this issue Nov 3, 2024
@hippo91
fix: CTest Test Suite Delimiter prevents running all tests (microsoft…
c558b2f 
…#4092)

* Introduces type alias (DriverMapT) to factorize type declaration
* Introduces the getProjectDriver method
* Adds comments
* Adds entry in the CHANGELOG
hippo91 added a commit to hippo91/vscode-cmake-tools that referenced this issue Nov 3, 2024
@hippo91
fix: CTest Test Suite Delimiter prevents running all tests (microsoft…
7c2b060 
…#4092)

* Introduces type alias (DriverMapT) to factorize type declaration
* Introduces the getProjectDriver method
* Adds comments
* Adds entry in the CHANGELOG
@gcampbell-msft gcampbell-msft modified the milestones: On Deck, 1.20 Nov 5, 2024
@gcampbell-msft gcampbell-msft linked a pull request Nov 6, 2024 that will close this issue
fix: CTest Test Suite Delimiter prevents running all tests (#4092) #4154
Merged
@gcampbell-msft gcampbell-msft moved this from Pending Prioritization to In Progress in CMake Tools Nov 6, 2024
gcampbell-msft added a commit that referenced this issue Nov 6, 2024
@hippo91 @gcampbell-msft
fix: CTest Test Suite Delimiter prevents running all tests (#4092) (#…
565d905 
…4154)

* fix: CTest Test Suite Delimiter prevents running all tests (#4092)

* Introduces type alias (DriverMapT) to factorize type declaration
* Introduces the getProjectDriver method
* Adds comments
* Adds entry in the CHANGELOG

* Fix wrong object initialization

* Takes into account code review remarks.

Do not return CMakeDriver | String but do return CMakeDriver and throw error in case of problem

* Fix lint issue

---------

Co-authored-by: Garrett Campbell <[email protected]>
@github-project-automation github-project-automation bot moved this from In Progress to Completed in CMake Tools Nov 6, 2024
@Yingzi1234
Copy link
Collaborator

@Jesseeee @gcampbell-msft This issue has been fixed on latest CMake tools version 1.20.24(pre-release), you can get the details below. Thank you!
Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug a bug in the product Feature: ctest Feature: Test Explorer
Projects
CMake Tools
Status: Completed
Milestone
1.20
Development

Successfully merging a pull request may close this issue.

fix: CTest Test Suite Delimiter prevents running all tests (#4092) hippo91/vscode-cmake-tools
3 participants
@Jesseeee @gcampbell-msft @Yingzi1234