exit code 0 when there are exceptions prior to tests

[nbogie]

This problem is demonstrated by putting the following code in spec/spec_helper.rb of a rails project.

require 'simplecov'
SimpleCov.start 'rails'
raise "some exception in the class loading before the tests start"

When rspec is invoked via rake, the above exception is logged to stderr and no tests are run, but the rake process exits 0 (i.e. success), where it should exit non-zero (failure) as it does when the SimpleCov.start is omitted.

Context:
This is a simplified version of the problem we encountered. We had a CI box that didn't receive a new config file required by a codebase change. When rake (rspec) ran, our app initialization code correctly raised an exception to protest but because our test suite was running under simplecov the rake (rspec) task exited 0 (success) indicating a passing build.

Environment:
ruby 1.9.2-p180,
simplecov gem 0.4.2
rspec gem 2.5.0
rubygems 1.6.2

Extra information:
I note that if I register an at_exit handler which does NOT call SimpleCov.result.format!, the exit code is propogated correctly (though of course no report will be produced). E.g. SimpleCov.at_exit { puts "simplecov at_exit handler - doing nothing" }

Thanks for any help, and for your work on this gem!

[colszowka]

Thanks for the report. I'll have a look into it, though I can't come up with an easy solution right now.

[nbogie]

Hi, I've made a fix for this issue but it currently lacks automated tests:
https://github.com/nbogie/simplecov/commit/ec8bd54f9507204440665f51b34ee584d16516f4

In the manual test documented in the above issue, simplecov's at_exit() will be reached with $! populated with some exception other than SystemExit (specifically a RuntimeError). Simplecov was only calling exit explicitly if it found a SystemExit exception was present. I extended that to work for other exceptions. For some reason, the presence of $! was not anyway causing a non-zero exit, at least in the situation where the registered at_exit handler had to prepare a simplecov report. I haven't found good documentation on this aspect of ruby (pointers welcome).

I'm dubious of the existing approach of doing the reporting work (and of calling exit) from within an at_exit handler. Also, what happens if the report formatter raises an exception, given that it is also generally launched from this handler? Isn't there some other point at which to prepare the report rather than using the at_exit hook? I get that such a callback would likely be framework-dependent.

I've only glanced at the rest of the code; please excuse my ignorance.

Resources:

• https://www.ruby-forum.com/topic/992363
• http://stackoverflow.com/questions/1144066/ruby-at-exit-exit-status
• http://ruby-doc.org/core/classes/Kernel.html#M001431
• simplecov BREAKS rspec #5

[colszowka]

I don't think there is a better way to capture the coverage results than with an at_exit handler - What other way could there be to let simplecov do it's thing when the test suite is finished?

Your commit makes a lot of sense to me, keeping this open so I remember to add tests and pull it in.

[colszowka]

This is included in the new 0.5.4 release, thanks a lot!