Skip to content
This repository has been archived by the owner on Apr 14, 2021. It is now read-only.

Add to_spec to stub_specification #5593

Closed
wants to merge 3 commits into from

Conversation

jules2689
Copy link
Contributor

Fixes #5592

Interestingly enough, I did not hit this case in Shopify core but a smaller app due to a gem's loading of Gems.

I think this is caused by the gem loading the gem after we initialize our caches. This causes the stub to be of type Bundler::StubSpecification, which means we need to also delegate to_spec to the stub's stub.

cc @segiddins

@jules2689
Copy link
Contributor Author

Not sure how I'd write a spec for this though :/

Copy link
Member

@segiddins segiddins left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unit test this, asserting that StubSpec#to_spec returns the right thing

@@ -68,6 +68,10 @@ def raw_require_paths
stub.raw_require_paths
end

def to_spec
stub.to_spec
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This maybe should return self?

Copy link
Contributor Author

@jules2689 jules2689 Apr 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stub != self though, one is a Bundler::StubSpecification and the other is Gem::StubSpecification.

18:03:57 web.1      | ===
18:03:57 web.1      | Gem::StubSpecification
18:03:57 web.1      | Bundler::StubSpecification
18:03:57 web.1      | ===

From adding

      STDOUT.puts "==="
      STDOUT.puts stub.class.to_s
      STDOUT.puts self.class.to_s
      STDOUT.puts "==="

Copy link
Contributor Author

@jules2689 jules2689 Apr 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yea, normally we do this:

stub = Gem::StubSpecification.gemspec_stub(file, install_path.parent, install_path.parent)
stub.full_gem_path = Pathname.new(file).dirname.expand_path(root).to_s.untaint
StubSpecification.from_stub(stub)

But on second load, we have stub which is not a Gem::StubSpecification

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps another option is

return stub if stub.is_a?(Bundler::StubSpecification)
 spec = new(stub.name, stub.version, stub.platform, nil)
 spec.stub = stub
 spec

@jules2689 jules2689 force-pushed the jules2689-stub-spec-method-missing branch from 9e638d5 to 56ae54f Compare April 17, 2017 23:54
@segiddins
Copy link
Member

@jules2689
Copy link
Contributor Author

yea, I wasn't sure if it would actually evaluate it. Need a real gem to use now.

@segiddins
Copy link
Member

Just stub the stub specification?

@jules2689 jules2689 force-pushed the jules2689-stub-spec-method-missing branch from 56ae54f to c966aeb Compare April 18, 2017 03:38
@jules2689
Copy link
Contributor Author

@segiddins got it. 👍 Thanks for the suggestion, got me on the right path.

  let(:with_gem_stub_spec) do
    stub = Gem::Specification.stubs.first
    described_class.from_stub(stub)
  end

  let(:with_bundler_stub_spec) do
    described_class.from_stub(with_gem_stub_spec)
  end

# frozen_string_literal: true
require "spec_helper"

RSpec.describe Bundler::StubSpecification do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You'll probably need to restrict this spec based on RubyGems version

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why?

# frozen_string_literal: true
require "spec_helper"

RSpec.describe Bundler::StubSpecification do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because old versions of RubyGems didn't have stubs

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know which version or where to check?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2.1

@jules2689 jules2689 force-pushed the jules2689-stub-spec-method-missing branch from c966aeb to 8576355 Compare April 18, 2017 17:58
@segiddins segiddins added this to the 1.15.0.pre.2 milestone Apr 18, 2017
require "spec_helper"

if Bundler.rubygems.provides?(">= 2.1")
RSpec.describe Bundler::StubSpecification do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be RSpec.describe Bundler::StubSpecification, :rubygems => ">= 2.1" do

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Is there a doc explaining this DSL at all?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if Bundler.rubygems.provides?(">= 2.1")
RSpec.describe Bundler::StubSpecification do
let(:with_gem_stub_spec) do
stub = Gem::Specification.stubs.first
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

watch out, this is returning nil in certain cases on CI... I'd create a new spec, serialize it to disk, and then create a stub from that known spec

described_class.from_stub(stub)
end

let(:with_bundler_stub_spec) do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont understand the point of this... the Bundler::StubSpecification should only ever be passed a Gem::StubSpecification

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue is that in some cases stub ends up being Bundler::StubSpecification. When someone loads a gem with Gem::Specification.find_by_name(*args) after require 'bundler/setup', it loads it again with a cache (it seems).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in that case, we should have a test that Gem::Specification.find_by_name also works


describe "#to_spec" do
it "returns a Gem::Specification" do
bundle :install
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the install? since this is a unit test, it shouldn't be invoking anything

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was trying to address #5593 (comment)

I'm not familiar with the DSL used in the tests :/

@segiddins
Copy link
Member

diff --git a/spec/bundler/stub_specification_spec.rb b/spec/bundler/stub_specification_spec.rb
index b0fec7570..81e601405 100644
--- a/spec/bundler/stub_specification_spec.rb
+++ b/spec/bundler/stub_specification_spec.rb
@@ -1,22 +1,43 @@
 # frozen_string_literal: true
 require "spec_helper"
 
-if Bundler.rubygems.provides?(">= 2.1")
-  RSpec.describe Bundler::StubSpecification do
-    let(:with_gem_stub_spec) do
-      stub = Gem::Specification.stubs.first
-      described_class.from_stub(stub)
+RSpec.describe Bundler::StubSpecification, :rubygems => ">= 2.1" do
+  let(:spec) do
+    build_spec("foo", "1.0").first
+  end
+
+  let(:path) do
+    tmp("foo.gemspec")
+  end
+
+  before do
+    path.open("w") {|f| f << spec.to_ruby }
+  end
+
+  let(:stub) do
+    if Gem::StubSpecification.respond_to?(:gemspec_stub)
+      Gem::StubSpecification.gemspec_stub(path.to_s, path.parent.parent, path.parent.parent)
+    else
+      Gem::StubSpecification.new(path.to_s)
     end
+  end
+
+  subject do
+    described_class.from_stub(stub)
+  end
 
-    let(:with_bundler_stub_spec) do
-      described_class.from_stub(with_gem_stub_spec)
+  describe "#to_spec" do
+    it "returns the Gem::Specification" do
+      expect(subject.to_spec).to eq(spec)
     end
+  end
+
+  context "when the stub is initialized with a #{described_class}" do
+    subject { described_class.from_stub(super()) }
 
     describe "#to_spec" do
-      it "returns a Gem::Specification" do
-        bundle :install
-        expect(with_gem_stub_spec.to_spec).to be_a(Gem::Specification)
-        expect(with_bundler_stub_spec.to_spec).to be_a(Gem::Specification)
+      it "returns the Gem::Specification" do
+        expect(subject.to_spec).to eq(spec)
       end
     end
   end
diff --git a/spec/runtime/setup_spec.rb b/spec/runtime/setup_spec.rb
index 14f973dc9..7a8a9c407 100644
--- a/spec/runtime/setup_spec.rb
+++ b/spec/runtime/setup_spec.rb
@@ -1207,5 +1207,17 @@ end
       RUBY
       expect(out).to eq("rack-1.0.0")
     end
+
+    it "allows Gem::Specification.find_by_name to work" do
+      install_gemfile! <<-G
+        source "file:#{gem_repo1}"
+        gem "rack"
+      G
+      ruby! <<-RUBY
+        require "bundler/setup"
+        puts Gem::Specification.find_by_name("rack").full_name
+      RUBY
+      expect(out).to eq("rack-1.0.0")
+    end
   end
 end

But be aware, the tests seem to pass without your patch as well

@jules2689
Copy link
Contributor Author

Update: I'll continue poking at this soon, I'm going to try and get a failing test case first though. However, it seems that foreman was doing something weird with how this worked, so I will be focussing on threading and loading using that as a basis.

@segiddins
Copy link
Member

@jules2689 are you OK with me de-scoping this from 1.15.0.pre.2, given it doesn't seem to be as severe as initially suspected?

@segiddins segiddins removed this from the 1.15.0.pre.2 milestone Apr 23, 2017
@jules2689
Copy link
Contributor Author

jules2689 commented Apr 24, 2017

@segiddins yea, I need to investigate more anyways and we have a work around

@jules2689
Copy link
Contributor Author

Superseded by #5630

@jules2689 jules2689 closed this May 2, 2017
@jules2689 jules2689 deleted the jules2689-stub-spec-method-missing branch May 2, 2017 03:51
bundlerbot added a commit that referenced this pull request May 8, 2017
Return Bundler::StubSpec if stub is a Bundler::StubSpec

Supersedes #5593
Fixes #5592

Explanation
---
In some cases the `Gem::Specification.stubs` call in [this method](https://github.com/bundler/bundler/blob/master/lib/bundler/rubygems_integration.rb#L773-L778) in the rubygems integration returns a mixed bag of `Bundler::StubSpecification` and `Gem::StubSpecification` objects. We then instantiate `Bundler::StubSpecification` objects and set the `stub` to be both `Gem::StubSpecification` and `Bundler::StubSpecification` objects.

This happens after we tell rubygems to use our overrides [here](https://github.com/bundler/bundler/blob/master/lib/bundler/runtime.rb#L21-L24).

A `Bundler::StubSpecification` does not define `to_spec` where `Gem::StubSpecification` does. In `Bundler::StubSpecification` we assume the `stub` to be a `Gem::StubSpecification` rather than the `to_spec`-less `Bundler::StubSpecification`. This means that in `_remote_specification`, the call to `to_spec` [here](https://github.com/bundler/bundler/blob/master/lib/bundler/stub_specification.rb#L88) fails. This falls back to `method_missing` [here](https://github.com/bundler/bundler/blob/master/lib/bundler/remote_specification.rb#L96-L98) which, of course, calls `_remote_specification` (and thus an infinite failing loops occurs).

### Why did this happen in such a weird way?

We needed to use a combination of `foreman`, `unicorn`, and a call to `Gem::Specification.find_by_name(*args)` to replicate.

I suspect this was required because Bundler doesn't call these methods as much. The last call in a doubly nested `bundle exec` resulted in the issue being exasperated.

You can however replicate with this:

```ruby
gem_stub = Gem::Specification.stubs.first
bundler_stub = Bundler::StubSpecification.from_stub(gem_stub)
bundler_stub = Bundler::StubSpecification.from_stub(bundler_stub)
bundler_stub.to_spec
```

We basically got to a point where we tried calling a method that doesn't exist on a `Bundler::StubSpecification`, so `_remote_specification` was called, but that had a method which didn't exist since we had the weirdness going on described here.

It was just a very specific sequence of events that is hard to replicate.

Options
---
1. We implement `to_spec` on `Bundler::StubSpecification`, as is done in #5593
2. We assume that `stub` is a `Gem::Specification`. Therefore if we try to create a `Bundler::StubSpecification` with the stub being a `Bundler::StubSpecification`, we simply return that stub we already made instead.

Thoughts
---
1. This basically ends up making a linked list of `Bundler::StubSpecifications` where you can follow `stub` all the way up until it's no longer a `Bundler::StubSpecification`. This means that the implementation is an accidental fix as `to_spec` in #5593 actually just calls `stub.to_spec` - which, if the stub is a `Bundler:StubSpecification`, would call that `Bundler::StubSpecification`, following the list up until we find a `Gem::StubSpecification`.
2. This is the right solution IMO. This breaks the weird linked list we made by mistake and just returns the object as we'd expect. Then, when `stub.to_spec` is called in `_remote_specification`, we always know it is a `Gem::StubSpecification` which has it defined.

cc @segiddins
segiddins pushed a commit that referenced this pull request May 10, 2017
Return Bundler::StubSpec if stub is a Bundler::StubSpec

Supersedes #5593
Fixes #5592

Explanation
---
In some cases the `Gem::Specification.stubs` call in [this method](https://github.com/bundler/bundler/blob/master/lib/bundler/rubygems_integration.rb#L773-L778) in the rubygems integration returns a mixed bag of `Bundler::StubSpecification` and `Gem::StubSpecification` objects. We then instantiate `Bundler::StubSpecification` objects and set the `stub` to be both `Gem::StubSpecification` and `Bundler::StubSpecification` objects.

This happens after we tell rubygems to use our overrides [here](https://github.com/bundler/bundler/blob/master/lib/bundler/runtime.rb#L21-L24).

A `Bundler::StubSpecification` does not define `to_spec` where `Gem::StubSpecification` does. In `Bundler::StubSpecification` we assume the `stub` to be a `Gem::StubSpecification` rather than the `to_spec`-less `Bundler::StubSpecification`. This means that in `_remote_specification`, the call to `to_spec` [here](https://github.com/bundler/bundler/blob/master/lib/bundler/stub_specification.rb#L88) fails. This falls back to `method_missing` [here](https://github.com/bundler/bundler/blob/master/lib/bundler/remote_specification.rb#L96-L98) which, of course, calls `_remote_specification` (and thus an infinite failing loops occurs).

### Why did this happen in such a weird way?

We needed to use a combination of `foreman`, `unicorn`, and a call to `Gem::Specification.find_by_name(*args)` to replicate.

I suspect this was required because Bundler doesn't call these methods as much. The last call in a doubly nested `bundle exec` resulted in the issue being exasperated.

You can however replicate with this:

```ruby
gem_stub = Gem::Specification.stubs.first
bundler_stub = Bundler::StubSpecification.from_stub(gem_stub)
bundler_stub = Bundler::StubSpecification.from_stub(bundler_stub)
bundler_stub.to_spec
```

We basically got to a point where we tried calling a method that doesn't exist on a `Bundler::StubSpecification`, so `_remote_specification` was called, but that had a method which didn't exist since we had the weirdness going on described here.

It was just a very specific sequence of events that is hard to replicate.

Options
---
1. We implement `to_spec` on `Bundler::StubSpecification`, as is done in #5593
2. We assume that `stub` is a `Gem::Specification`. Therefore if we try to create a `Bundler::StubSpecification` with the stub being a `Bundler::StubSpecification`, we simply return that stub we already made instead.

Thoughts
---
1. This basically ends up making a linked list of `Bundler::StubSpecifications` where you can follow `stub` all the way up until it's no longer a `Bundler::StubSpecification`. This means that the implementation is an accidental fix as `to_spec` in #5593 actually just calls `stub.to_spec` - which, if the stub is a `Bundler:StubSpecification`, would call that `Bundler::StubSpecification`, following the list up until we find a `Gem::StubSpecification`.
2. This is the right solution IMO. This breaks the weird linked list we made by mistake and just returns the object as we'd expect. Then, when `stub.to_spec` is called in `_remote_specification`, we always know it is a `Gem::StubSpecification` which has it defined.

cc @segiddins

(cherry picked from commit 47e7dd0)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants