rosunit should allow simultaneous XML output and text output

[v-lopez]

Right now we are executing unit tests using make test in our buildfarm.

Fixing an unexpected failure or a crash on a unit test in the buildfarm is an almost impossible task since we have no logs and the XML output only reports a crash and zero useful logs.

Example of XML output on the case of a crash:

test [serial_driver_test] did not generate test results
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/opt/ros/indigo/lib/python2.7/dist-packages/rostest/runner.py", line 164, in fn
self.assert_(os.path.isfile(test_file), "test [%s] did not generate test results"%test_name)
File "/usr/lib/python2.7/unittest/case.py", line 424, in assertTrue
raise self.failureException(msg)

It would be very helpful to have alll std output as part of the XML report, if it cannot be made visible in the terminal running the tests.

[dirk-thomas]

Please consider to contribute a pull request implementing this feature.

[talhaHavadar]

is there any development going on about this issue?

[v-lopez]

Not yet from our side.

We haven't discarded doing it, but it's not on our sights yet.

[talhaHavadar]

Then I can start implementation of it if it is ok for you

[v-lopez]

Sure, if you want a tester just let me know.

[talhaHavadar]

I changed the super class of _XMLTestResult from 'unittest.TestResult' to 'unittest.TextTestResult'. And also updated the functions like 'startTest', 'stopTest', 'addError' etc. With this change we can write the output to the stdout while creating the xml. I need to do some spot checks to be sure it is working correctly then I will open a PR for this changes.

[v-lopez]

I've been looking further into this issue. I've created this repo with examples of different ways of running tests:
https://github.com/pal-robotics/ros_testing_experiments

With the kinetic-devel branch, I'm not seeing a uniform behaviour with regards to the handling of the log output of the tests:

I am testing four cases, the bold text is the name I'll use:
cpp_test: C++ gtest
cpp_rostest: C++ gtest executed within rostest
py_test: Python test
py_rostest: Python test executed within rostest

Running the tests with make run_tests I have the following behaviour:

test	stdout	stderr	log info	log warn	log error
cpp_test	terminal	terminal	terminal	terminal	terminal
py_test	-	terminal	-	-	-
cpp_rostest	~/.ros/log	terminal	~/.ros/log	~/.ros/log	terminal
py_rostest	rosunit xml	rosunit xml	rosunit xml	rosunit xml	rosunit xml
cpp_rostest --text	terminal	terminal	terminal	terminal	terminal
py_rostest --text	terminal	terminal	terminal	terminal	terminal

The - indicates that the logs could not be found.

With @talhaHavadar changes I see the same behaviour, but the generated XML are invalid (that was the cause of the test failure in #163).

@dirk-thomas Do you know if this is the actual behaviour or am I doing something wrong?
What would be the desired behaviour, and if I fixed it would it make it to kinetic?

From my understanding, at least cpp_rostest and py_rostest should have the same behaviour, and log onto the rosunit (or maybe rostest?) xml file. Regardless of what was requested in this issue.

And cpp_test and py_test should log everything to the terminal.

Finally, what was requested from this issue, it would be that cpp_rostest and py_rostest also logged to the terminal while still creating an xml with the result for parsers such as jenkins.

[v-lopez]

With some modifications I have the following behavior, on bold the behaviours that have changed.

The changes:
https://github.com/ros/ros_comm/compare/kinetic-devel...pal-robotics-forks:rostest-screen-output?expand=1
https://github.com/ros/ros/compare/kinetic-devel...pal-robotics:tee-test-output?expand=1

test	stdout	stderr	log info	log warn	log error
cpp_test	terminal	terminal	terminal	terminal	terminal
py_test	terminal1	terminal1	terminal1	terminal1	terminal1
cpp_rostest	terminal	terminal	terminal	terminal	terminal
py_rostest	terminal1 & rosunit xml	terminal1 & rosunit xml	terminal1 & rosunit xml	terminal1 & rosunit xml	terminal1 & rosunit xml
cpp_rostest --text	terminal	terminal	terminal	terminal	terminal
py_rostest --text	terminal	terminal	terminal	terminal	terminal

¹ The logs have no color, probably related to the py logging macros, haven't looked into it.

While keeping test result output to XML for jenkins.
These changes would satisfy what was requested in this issue.

I see three drawbacks to these fixes:

1. cpp rostests no longer write to log file
I would say that the previous behaviour was not consistent, and difficult to use reliably since a rostest could be writing to 3 different places.

2. Logs colors are missing for python tests
Probably this can be fixed if there's an interest into these changes.

3. Tests are now more verbose
It's true

[progtologist]

Are you planning of making a PR with those changes?

[dirk-thomas]

Closing due to inactivity.