Rspec 2.14 + Ruby 2.1.0 + Simplecov 0.8.x doesn't pass along the correct exit status

[agrobbin]

Looks like Simplecov 0.8.x, Ruby 2.1.0, and Rspec 2.14 don't quite work together as you would expect. When running bundle exec rspec in that environment (with Rails 4.0.2), the exit code when tests fail is 0, when it should be 1. However, when I run them on Ruby 2.0.0(-p353) with Simplecov 0.8.x, it correctly exits 1. Even weirder, when running them on Ruby 2.1.0 and Simplecov 0.7.1, it also correctly exits 1.

[nanaya]

it seems to happen only when combined with rspec javascript test (capybara+poltergeist).

[nanaya]

I've created test case. Perhaps a bit too big though.

Travis report: https://travis-ci.org/edogawaconan/capytestruby21/builds/17829008

(pardon the repo name because initially I thought it's capybara bug)

[BanzaiMan]

Is this a dupe of #269?

[nanaya]

in my case, ruby 2.0 isn't affected though.

[BanzaiMan]

I dug a little deeper.

https://travis-ci.org/BanzaiMan/travis_production_test/jobs/18281155 tests this code base, running on Ruby 2.1.0 and using this Gemfile.lock. As you can see, the combination of Ruby 2.1.0, RSpec 2.14.1 and SimpleCov 0.8.2, is not sufficient to show the problem here.

I also added poltergeist for kicks (https://travis-ci.org/BanzaiMan/travis_production_test/jobs/18284015) but the problem does not manifest. Perhaps it is rspec-rails and/or Rails itself playing a part.

[BanzaiMan]

#269 (comment) implicates Rails 4.

[nanaya]

@BanzaiMan try enabling poltergeist by adding 'js: true' in the test case.

[BanzaiMan]

@edogawaconan I don't know how to exercise JavaScript, I guess. At any rate, it seems to me that developers from poltergeist should be involved to track this down.

[nanaya]

when I checked last time, same error also occured when using capybara-webkit driver.

[BanzaiMan]

@edogawaconan Could it be capybara, then? It's the common factor here, right?

[nanaya]

yeah, probably a combination of simplecov, capybara, and .

[colszowka]

So, I've finally also bumped into this at work. Mysteriously, it only manifests when I run the whole test suite, not for individual specs that fail (tried both with capybara acceptance spec and a model unit test). Digging in.

[colszowka]

Closed #269 in favor of this - I tried reverting the exit status changes from the 0.8 branch and also did a bit of binding.pry around the behaviour, I can not figure out what is going wrong, the behaviour is seemingly totally random :( This is happening with rspec 3 and rails 4 for me by the way. For now, I'm bailing and will downgrade to 0.7.1 myself...

[thibaudgg]

Same issue for us (Simplecov 0.8.x, Ruby 2.1.0, Rspec 2.14 and Rails 4.0.2), rolling back to 0.7.1 fixes it.

[tmikoss]

Same issue here. Downgrading simplecov to 0.7.1 fixes it.

• Ruby 2.1.0 / 2.1.1
• rspec (2.14.1)
• rspec-rails (2.14.0)
• capybara (2.1.0)
• poltergeist (1.4.1)

[phoet]

just saw a warning on circle-ci (which is pretty awesome) mentioning this issue. is there any known problem with this version and TestUnit?

[colszowka]

It's still unclear to me when exactly this is happening. I did try to debug the issue by comparing the code that changed in comparison to 0.7, but nothing obvious came up. It's possible this also happens with test unit if simplecov is the problem (which the downgrade fix suggests).

[zaburt]

Might be Rails 4 specific, "bundle exec rspec" returns exit code properly with the following;

rails 3.2.17
ruby 2.1.1
rspec  2.14.1
rspec-rails 2.14.1
capybara 2.2.1
poltergeist from git revision: 09c6570dea3804cf6ea47821081710b6311be445 

[mszyndel]

@bf4 since you are maintainer now can you take a look at this?, especially pr #302

[colszowka]

#302 is merged, though a test would really make sense here. I added a suggestion on that over at the other issue. Everyone else who was seeing this behaviour, could you please give the current master a shot against your repo via gem 'simplecov', require: false, github: 'colszowka/simplecov' and report back?

[mmell]

@colszowka with gem 'simplecov', require: 'false', github: 'colszowka/simplecov', a bundle exec rspec throws ruby-2.1.2/gems/bundler-1.6.2/lib/bundler/runtime.rb:76:in 'require': cannot load such file -- false (LoadError). Removing require: 'false' will resolve the issue.

[pcreux]

Do require: false instead of 'false'

φ
On Jun 19, 2014 8:41 AM, "Michael Mell" [email protected] wrote:

@colszowka https://github.com/colszowka with gem 'simplecov', require:
'false', github: 'colszowka/simplecov', a bundle exec rspec throws
ruby-2.1.2/gems/bundler-1.6.2/lib/bundler/runtime.rb:76:inrequire':
cannot load such file -- false (LoadError). Removingrequire: 'false, will
resolve the issue.

—
Reply to this email directly or view it on GitHub
#281 (comment).

[bf4]

@hajder @colszowka et al: sorry for not being helpful lately. It's been a busy week.

http://www.commitstrip.com/en/2014/05/07/the-truth-behind-open-source-apps/

[xaviershay]

I have seen this issue on 0.8.*, but cannot replicate reliably. I am now running master branch as advised above, and have not seen this issue yet.

[bf4]

Has anyone been able to reliably replicate this? I wonder if we should change our at_exit handling code in general. Perhaps an END block.

[mszyndel]

@bf4 maybe it's time to release this fix. I don't think it can be more broken than currently is (in 0.8 version)

[bf4]

@hajder you mean master, right? only @colszowka has gem push

[mszyndel]

yup. lot's of people waiting for this fix, I don't think my commit broke it more than it was ;)
(actually I believe it fixed it!)

[colszowka]

Hello Gentlemen, yes, this needs shipping. I just became a dad and hence am slightly busy, but I'll try to get it out over the weekend!

[BanzaiMan]

@colszowka Congrats! Enjoy these moments.

[bf4]

If this is fixed in master, can we close this issue? (And for those of you who have put more time into this than me, is there anything in #273 to consider merging?)

[mszyndel]

#273 is pretty much the same fix. I think both can be closed.

[bf4]

Confirmed fixed in master by

• case reported by @fiedl https://travis-ci.org/bf4/wingolfsplattform/builds
• case reported by @edogawaconanhttps://travis-ci.org/bf4/capytestruby21/builds

I've had trouble reproducing https://travis-ci.org/magiclabs/alchemy_cms/jobs/16516963#L403 2

Were there other reproducible failures we could/should test for fixes?

While looking into this, I came across http://www.davekonopka.com/2013/rspec-exit-code.html by @davekonopka 2, 3 which also looked interesting, in general

if defined?(RUBY_ENGINE) && RUBY_ENGINE == "ruby" && RUBY_VERSION >= "1.9"
  module Kernel
    alias :__at_exit :at_exit
    def at_exit(&block)
      __at_exit do
        exit_status = $!.status if $!.is_a?(SystemExit)
        block.call
        exit exit_status if exit_status
      end
    end
  end
end

[mszyndel]

@bf4 so is this released as a gem in 0.8 branch?

[xaviershay]

no, 0.9 is out.