Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable convenient unit testing using googletest on POSIX #11573

Merged
merged 12 commits into from
May 9, 2019

Conversation

MaEtUgR
Copy link
Member

@MaEtUgR MaEtUgR commented Mar 2, 2019

Describe problem solved by the proposed pull request
Part of the description I wrote in #11308

The idea is to enable unit testing for any C++ class of the project with very low overhead. For example if you want to test the class AttitudeControl you:

  • Create a file AttitudeControlTest.cpp next to the classes source files
  • Add px4_add_gtest(SRC AttitudeControlTest.cpp LINKLIBS AttitudeControl) to the folder's CMakeLists.txt
  • Start writing your test e.g. :
#include <gtest/gtest.h>
#include <AttitudeControl.hpp>

TEST(AttitudeControlTest, AllZeroCase) {
AttitudeControl attitude_control;
matrix::Vector3f rate_setpoint = attitude_control.update(matrix::Quatf(), matrix::Quatf(), 0.f);
EXPECT_EQ(rate_setpoint(0), 0.f);
}

When you run make unit_test you get the following output (runs all the unit tests):
first!UNITO-UNDERSCORE!test
And it neither needs to rebuild all the libraries you want to test if you're using px4_sitl_default anyways nor does gtest need to be loaded when you're not unit testing with it thanks to #11533

Describe your preferred solution
A clear and concise description of what you have implemented.

Additional context

Test data / coverage
Developed and tested on Cygwin Windows toolchain, I will also test on Ubuntu.

############################################################################

# Download and unpack googletest at configure time
configure_file(${CMAKE_CURRENT_LIST_DIR}/CMakeLists.txt.in googletest-download/CMakeLists.txt)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm missing why this has to be done at configure time like this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because otherwise you don't have the CMakeLists-tree of gtest at configure time and can't configure the dependencies correctly.
I see two options around it:

  • You have the source anyways because you add it as a submodule but since all submodules have to be present at anytime I prefered the download only if it's actually needed.
  • You hack around and achieve that cmake only needs to configure the dependencies to gtest on build time but I didn't achieve that because you need to wrap it and make assumptions about the structure of gtest. It's easy to screw up the include directories and other things this way.

We can still revisit, this is the first state I find acceptable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw the method for the download at configure time is the one from the docs, see last point and following description here https://github.com/google/googletest/blob/master/googletest/README.md#incorporating-into-an-existing-cmake-project

cmake/gtest/CMakeLists.txt.in Outdated Show resolved Hide resolved
CMakeLists.txt Outdated
# Testing - Automatic unit and integration testing with CTest
#

option(unit_testing "Configure unit test targets" OFF)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

@MaEtUgR MaEtUgR Mar 3, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found out about the concept and wanted to do that but they are used already for the SITL integration tests. I'd like to be able to run the tests separately since the SITL tests take so much longer. I planned to have a structured test name and use ctest -R to filter the correct ones. So you think distinguishing build time doesn't make sense? I could agree with that. Then gtest is always needed for tests in general and it all depends on BUILD_TESTING.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah and testing is always enabled for sitl... https://github.com/PX4/Firmware/blob/fc7c7ac20630951b17e8644ccc175b9c4e743409/CMakeLists.txt#L303-L305
Can I make it optional?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since the SITL tests take so much longer

Are you talking about the SITL unit tests? Each is roughly 1-3 seconds.
Example: http://ci.px4.io:8080/blue/organizations/jenkins/PX4%2FFirmware/detail/master/722/pipeline/21

Screen Shot 2019-03-09 at 3 45 14 PM

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I expected the basic tests to run two order of magnitudes faster. Is that impatient? 😇 I think it's just starting the simulator binary that takes the big part of that time, otherwise the smallest tests would not all have around the same duration.
Anyways I think we should allow to run the unit tests separately since I presume they'll finish a lot faster until we write a crazy amount of tests or huge loops.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll just add all tests and the gtest library fetching with enable_testing() and hence BUILD_TESTING but make enabling tests optional for posix. Then I can still filter the tests if I only want to run the quick unit tests e.g. with make tests unit.

@dagar dagar self-requested a review March 2, 2019 19:37
@MaEtUgR MaEtUgR changed the base branch from master to modular-attitude-control March 2, 2019 20:43
src/include/visibility.h Outdated Show resolved Hide resolved
@MaEtUgR MaEtUgR force-pushed the modular-attitude-control branch 2 times, most recently from 16cb44b to 4556184 Compare March 10, 2019 21:44
@MaEtUgR MaEtUgR force-pushed the unit-testing-gtest branch 2 times, most recently from c287a96 to a503ca1 Compare March 10, 2019 22:30
@MaEtUgR MaEtUgR force-pushed the unit-testing-gtest branch from a503ca1 to 8da3163 Compare March 27, 2019 11:32
@MaEtUgR MaEtUgR changed the base branch from modular-attitude-control to master March 27, 2019 11:32
@MaEtUgR
Copy link
Member Author

MaEtUgR commented Mar 27, 2019

rebased after merging #11308

@MaEtUgR MaEtUgR force-pushed the unit-testing-gtest branch 2 times, most recently from ee9e99e to 701d7d5 Compare April 14, 2019 18:55
@MaEtUgR
Copy link
Member Author

MaEtUgR commented Apr 14, 2019

Rebased and started to enable the unit test framework with camkes BUILD_TESTING after the comment here: #11573 (comment) Next up is filtering for the different tests such that it's possible to specify what test you want to run for local test-driven development and to exclude WIP tests in CI.

@mcsauder
Copy link
Contributor

@MaEtUgR , Its not clear to me how to build/run the tests with your latest commit, could you clue me in? :)

Your previous commit and others before it run great on Ubuntu 18.10:

image

@MaEtUgR MaEtUgR force-pushed the unit-testing-gtest branch from 82e88cd to c1f685f Compare April 23, 2019 07:41
@MaEtUgR
Copy link
Member Author

MaEtUgR commented Apr 23, 2019

@mcsauder Sorry the latest commit at the time you wrote your comment was an unfinished temporary dump of ongoing tests that's why the commit message started with TEMP:. It ran certain tests double when commanding make tests. I rebased and replaced it over the weekend to add the possibility to filter tests with make tests TESTFILTER=unit. I still need to clean up the unit_test make target because it will get obsolete with one test target and filtering.

@MaEtUgR MaEtUgR force-pushed the unit-testing-gtest branch 4 times, most recently from 581d8f1 to 105909a Compare April 25, 2019 05:59
@MaEtUgR MaEtUgR marked this pull request as ready for review April 25, 2019 06:01
@MaEtUgR
Copy link
Member Author

MaEtUgR commented Apr 25, 2019

I rebased again. Since I would now start writing useful unit tests I would say this pr is ready for review. One thing I already noticed is the license text is not in every file, I'll fix that quickly.

@MaEtUgR MaEtUgR requested review from dagar and RomanBapst April 25, 2019 06:17
@MaEtUgR
Copy link
Member Author

MaEtUgR commented Apr 30, 2019

@dagar Any chance to get the test framework in? I'm willing to continue to work on how it's embedded in cmake but would appreciate if I was able to start to write unit tests without blowing up this PR. The tests are now included in make tests, built and executed in the poisx_sitl_test folder and are hence checked in CI.

@RomanBapst
Copy link
Contributor

This is quite useful!

@mcsauder
Copy link
Contributor

mcsauder commented May 8, 2019

Nice work @MaEtUgR !

MaEtUgR added 12 commits May 8, 2019 19:24
visibility.h is included globally in PX4 via cmake compile flags.
It contains poisoning the exit() command which is used by gtest
to close the test application. Removing the flag for gtest compilation
fixes the compile error:
gtest.cc:4757:7: error: attempt to use poisoned "exit"
Example: Before when you passed "make tests TESTFILTER=Attitude"
and subsequently "make tests TESTFILTER=Atti" it found the string
"TESTFILTER=Atti" in "TESTFILTER=Attitude" and hence the check if
the configuration is already correct passed. The fix checks for
the configuration parameter including the subsequent space separator
and after that strips the space away again such that the list
VERIFIED_CMAKE_OPTIONS doesn't contain trailing spaces.
@MaEtUgR MaEtUgR force-pushed the unit-testing-gtest branch from 378ec31 to 715a9ba Compare May 8, 2019 17:25
@MaEtUgR
Copy link
Member Author

MaEtUgR commented May 8, 2019

@dagar rebased on rc1 to run CI again

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants