Could not verify the SSL certificate for https://rubygems.org/ #4555

gmcquistin · 2021-04-21T21:46:53Z

Describe the problem as clearly as you can

Running bundle install fails due to https://RubyGems.org returning an invalid SSL certificate.

RubyGems.org seems to be using a new SSL certificate but it doesn't cover this domain.

Post steps to reproduce the problem

Run bundle install in a project that uses source "https://rubygems.org" in its Gemfile

Which command did you run?

bundle install

What were you expecting to happen?

I expected the project's gems to be installed

What actually happened?

An error was returned:

Fetching source index from https://rubygems.org/

Retrying fetcher due to error (2/4): Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate for https://rubygems.org/.
There is a chance you are experiencing a man-in-the-middle attack, but most likely your system doesn't have the CA certificates needed for verification. For information about OpenSSL certificates, see http://bit.ly/ruby-ssl. To connect without using SSL, edit your Gemfile sources and change 'https' to 'http'.

If not included with the output of your command, run `bundle env` and paste the output below

Environment

Bundler       1.17.3
  Platforms   ruby, x86_64-darwin-19
Ruby          2.6.3p62 (2019-04-16 revision 67580) [x86_64-darwin19]
  Full Path   /Users/gmcquistin/.rbenv/versions/2.6.3/bin/ruby
  Config Dir  /Users/gmcquistin/.rbenv/versions/2.6.3/etc
RubyGems      3.0.3
  Gem Home    /Users/gmcquistin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0
  Gem Path    /Users/gmcquistin/.gem/ruby/2.6.0:/Users/gmcquistin/.rbenv/versions/2.6.3/lib/ruby/gems/2.6.0
  User Path   /Users/gmcquistin/.gem/ruby/2.6.0
  Bin Dir     /Users/gmcquistin/.rbenv/versions/2.6.3/bin
Tools         
  Git         2.29.0
  RVM         not installed
  rbenv       rbenv 1.1.2
  chruby      not installed

Bundler Build Metadata

Built At          2018-12-27
Git SHA           d7089abb6
Released Version  true

The text was updated successfully, but these errors were encountered:

seccomiro · 2021-04-22T13:41:38Z

The same has been happening here since yesterday.

Bundler       2.2.15
  Platforms   ruby, x86_64-linux
Ruby          2.7.2p137 (2020-10-01 revision 5445e0435260b449decf2ac16f9d09bae3cafe72) [x86_64-linux]
  Full Path   /home/diego/.rbenv/versions/2.7.2/bin/ruby
  Config Dir  /home/diego/.rbenv/versions/2.7.2/etc
RubyGems      3.1.4
  Gem Home    /home/diego/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0
  Gem Path    /home/diego/.gem/ruby/2.7.0:/home/diego/.rbenv/versions/2.7.2/lib/ruby/gems/2.7.0
  User Home   /home/diego
  User Path   /home/diego/.gem/ruby/2.7.0
  Bin Dir     /home/diego/.rbenv/versions/2.7.2/bin
Tools         
  Git         2.17.1
  RVM         not installed
  rbenv       rbenv 1.1.2-40-g62d7798
  chruby      not installed

sonalkr132 · 2021-04-22T16:32:02Z

any rubygems version older than 2.7.11, 3.0.9, and 3.1.5 is expected to have cert issue. Please update to the latest patch version and let us know if the issue exists.

we would also be interested in knowing more about the timeline of the issue. were these older versions working for you before yesterday? The cert change and corresponding releases were made on 9 Dec, 2020.

https://blog.rubygems.org/2020/12/09/3.0.9-released.html
https://blog.rubygems.org/2020/12/09/3.1.5-released.html

seccomiro · 2021-04-22T17:47:33Z

were these older versions working for you before yesterday? The cert change and corresponding releases were made on 9 Dec, 2020.

For me, yes. They were working as always.
I'm still trying to solve it.

Something I realized now is that I can't event open https://rubygems.org on my browser (certificate issues).
So it looks like an issue related to my Linux. But @gmcquistin is running it on a Mac.
Whats's even more weird is that the blog subdomain (https://blog.rubygems.org) is working fine here.

I've tried to reinstall OpenSSL, but nothing. 🤷‍♂️

seccomiro · 2021-04-22T18:00:42Z

As I am using 3.1.4 (which is under 3.1.5) and it's is expected for me to have the certificate issue, I cant't even update rubygems by typing gem update --system to get the newer version.

Do you have any tips for me on how I could do that?

deivid-rodriguez · 2021-04-22T18:14:49Z

We have these instructions for manually updating the certificates: https://bundler.io/v2.2/guides/rubygems_tls_ssl_troubleshooting_guide.html#updating-ca-certificates.

deivid-rodriguez · 2021-04-22T18:16:23Z

Well, it sounds like we need to update the guide because the certificate link is broken 😅.

seccomiro · 2021-04-22T19:11:02Z

We have these instructions for manually updating the certificates: https://bundler.io/v2.2/guides/rubygems_tls_ssl_troubleshooting_guide.html#updating-ca-certificates.

Yeah. I did it. I went through all the steps. But it didn't work either. 😢

Well, it sounds like we need to update the guide because the certificate link is broken 😅.

So what's the right one?

sonalkr132 · 2021-04-22T19:36:05Z

please try this https://bundler.io/v2.1/guides/rubygems_tls_ssl_troubleshooting_guide.html#updating-ca-certificates

Alternatively, you can also download rubygems-update from your browser https://rubygems.org/pages/download and install it using the downloaded gem file.

gem install rubygems-update-3.2.16.gem  --no-doc
update_rubygems

if neither of these work, please share the commands you ran and their output here or at [email protected]

PS: try installing ca-certificates package if you can't access rubygems.org in your browser either.

gmcquistin · 2021-04-22T20:13:02Z

Thank you for your help and patience guys, much appreciated!

Is there potential for this problem to originate on the RubyGems host?

I noticed for example that the SSL certificate served by some of the subdomains is also invalid. Here are a couple of SSL reports:

uptime.rubygems.org - https://www.ssllabs.com/ssltest/analyze.html?d=uptime.rubygems.org
help.rubygems.org - https://www.ssllabs.com/ssltest/analyze.html?d=help.rubygems.org

These reports indicate the same problem that I am seeing in my browser when I visit rubygems.org and when I run bundle install - the certificate only covers these domains:

X509v3 Subject Alternative Name:
DNS:l.ssl.fastly.net, DNS:*.attribution.report, DNS:*.dollarshaveclub.com, DNS:*.nfl.com, DNS:attribution.report, DNS:fl.eat24cdn.com, DNS:nymag.com, DNS:*.nymag.com

This output is from openssl s_client -connect rubygems.org:https | openssl x509 -noout -text

gmcquistin · 2021-04-22T20:19:16Z

(@sonalkr132) we would also be interested in knowing more about the timeline of the issue. were these older versions working for you before yesterday? The cert change and corresponding releases were made on 9 Dec, 2020.

I first noticed this issue yesterday, 21-APR-2021 at 21:30 GMT. The certificate I am being served says: Not Before: Apr 21 18:44:08 2021 GMT which was just a few hours before I saw the issue.

sonalkr132 · 2021-04-22T21:05:00Z

uptime.rubygems.org - https://www.ssllabs.com/ssltest/analyze.html?d=uptime.rubygems.org
help.rubygems.org - https://www.ssllabs.com/ssltest/analyze.html?d=help.rubygems.org

Thank you for pointing this out. these subdomains were pointing to legacy endpoints, I have updated them.

openssl s_client -connect rubygems.org:https | openssl x509 -noout -text

This command is not working because you need to add -servername rubygems.org to s_client.

sonalkr132 · 2021-04-22T21:22:53Z

same problem that I am seeing in my browser when I visit rubygems.org

this is a bit unexpected. generally, browsers come bundled with popular CAs. rubygems.org cert is issued by GlobalSign Root CA - R3. Do these URLs also show invalid cert in your browsers https://valid.r3.roots.globalsign.com/ https://www.lemonde.fr/ ? Can you please share your browser name, version and a screenshot (with URL)?
Also, make sure rubygems.org resolves to these ips:

dig A rubygems.org +short
151.101.66.132
151.101.194.132
151.101.130.132
151.101.2.132

ChaelCodes · 2021-04-23T15:13:53Z

I'm using Google Chrome ( 89.0.4389.128), and experiencing the same issues.

dig A rubygems.org +short
151.101.192.70

No additional servers were offered.

sonalkr132 · 2021-04-23T15:45:18Z

dig A rubygems.org +short
151.101.192.70

hmm.. this is not correct. I will double-check if this can be an issue at our end. I am guessing others are having because of this outdated DNS reply.
Can you please debug from your end why it is replying with only one IP, that too outdated. Looks like something had pinned rubygems.org to .70 IP. It is our old IP, technically not in use for quite a few months. Does dig @8.8.8.8 rubygems.org +short return the same? You can also use https://toolbox.googleapps.com/apps/dig/#A/ to cross verify.

ChaelCodes · 2021-04-23T15:48:34Z

Can you please debug from your end why it is replying with only one IP, that too outdated. Looks like something had pinned rubygems.org to .70 IP. It is our old IP, technically not in use for quite a few months. Does dig @8.8.8.8 rubygems.org +short return the same? You can also use https://toolbox.googleapps.com/apps/dig/#A/ to cross verify.

That returns this:

dig @8.8.8.8 rubygems.org +short
151.101.66.132
151.101.2.132
151.101.130.132
151.101.194.132

seccomiro · 2021-04-23T16:40:57Z

I'm using Google Chrome ( 89.0.4389.128), and experiencing the same issues.
dig A rubygems.org +short
151.101.192.70
No additional servers were offered.

I had the same result here.

Can you please debug from your end why it is replying with only one IP, that too outdated. Looks like something had pinned rubygems.org to .70 IP. It is our old IP, technically not in use for quite a few months. Does dig @8.8.8.8 rubygems.org +short return the same? You can also use https://toolbox.googleapps.com/apps/dig/#A/ to cross verify.

That returns this:
dig @8.8.8.8 rubygems.org +short
151.101.66.132
151.101.2.132
151.101.130.132
151.101.194.132

And here too.

sonalkr132 · 2021-04-23T17:04:56Z

Thank you for the info. Can you please share the output of dig rubygems.org +trace (before implementing the fix suggested below)?

since dig @8.8.8.8 rubygems.org +short is working for you, you can avoid this issue from your end by setting your nameserver to 8.8.8.8. https://superuser.com/questions/86184/change-dns-server-from-terminal-or-script-on-mac-os-x

ChaelCodes · 2021-04-23T17:29:23Z

Can you please share the output of dig rubygems.org +trace

dig rubygems.org +trace

; <<>> DiG 9.11.3-1ubuntu1.7-Ubuntu <<>> rubygems.org +trace
;; global options: +cmd
;; Received 40 bytes from 172.30.176.1#53(172.30.176.1) in 846 ms

sonalkr132 · 2021-04-23T18:27:32Z

sorry, this doesn't seem like complete output. for example, it has no answer section (151.101.192.70 part). Can you please try dig rubygems.org +trace +all?

seccomiro · 2021-04-23T19:11:43Z

dig rubygems.org +trace +all

; <<>> DiG 9.11.3-1ubuntu1.14-Ubuntu <<>> rubygems.org +trace +all
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 29374
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 65494
; OPT=5: 05 07 08 0a 0d 0e 0f (".......")
; OPT=6: 01 02 04 ("...")
; OPT=7: 01 (".")
;; QUESTION SECTION:
;.				IN	NS

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Fri Apr 23 16:10:44 -03 2021
;; MSG SIZE  rcvd: 51

gmcquistin · 2021-04-23T20:30:55Z

Thank you everyone for your help with this!

(@sonalkr132) Can you please debug from your end why it is replying with only one IP, that too outdated. Looks like something had pinned rubygems.org to .70 IP.

This hint pointed me in the right direction. It turns out that the line 151.101.64.70 rubygems.org was present in my /etc/hosts file. bundle install works as expected after removing it!

Thanks all 🙂

sonalkr132 · 2021-04-24T04:26:14Z

->>HEADER<<- opcode: QUERY, status: REFUSED, id: 29374

status REFUSED is not expected. perhaps you should try running it again.

the line 151.101.64.70 rubygems.org was present in my /etc/hosts file.

I am trying to verify the same with +trace command for the other two users here. ideally, trace will start resolving from root servers and give the correct answer to dns query. first few lines of output would look something like:

$ dig rubygems.org +trace                

; <<>> DiG 9.16.1-Ubuntu <<>> rubygems.org +trace
;; global options: +cmd
.                       7018    IN      NS      l.root-servers.net.
.                       7018    IN      NS      e.root-servers.net.
.                       7018    IN      NS      i.root-servers.net.
.                       7018    IN      NS      f.root-servers.net.
.                       7018    IN      NS      h.root-servers.net.
.                       7018    IN      NS      c.root-servers.net.
.                       7018    IN      NS      b.root-servers.net.
.                       7018    IN      NS      k.root-servers.net.
...

If you can't set your nameserver to 8.8.8.8 or any other nameserver (1.1.1.1 is from Cloudflare) which is replying with correct records, you may have to do some sleuthing on your own. It can also be possible that something in your network has cached DNS response overzealously and is not respect TTL. Maybe try to check why dig rubygems.org +trace is not working as expected or get help from your network administrator.

ChaelCodes · 2021-04-25T23:50:25Z

My issues were fixed by running dig @8.8.8.8 rubygems.org +short and then replacing the ip address in etc/hosts with the new ip address. Removing it didn't help.

seccomiro · 2021-04-26T00:32:10Z

My issues were fixed by running dig @8.8.8.8 rubygems.org +short and then replacing the ip address in etc/hosts with the new ip address. Removing it didn't help.

Wow. I hadn't realized there was an entry for rubygems.org at my /etc/hosts too.
But I don't know why it was there. It wasn't me. I swear. 😬
Anyway. Removing it solved all my problems.

Thank you all. 😄

ChaelCodes · 2021-04-26T01:42:37Z

I remember why I added it to etc/hosts now.

MacOS will automatically fall back to IPv4 when IPv6 doesn't resolve. On Windows or Linux, you need to specify the route in etc/hosts. Or at least, 2 years ago you did, otherwise gems wouldn't be downloaded. 😅 Might be fixed now.

sonalkr132 · 2021-04-26T02:48:15Z

If you are having issues with IPv6 and need IPv4 fallback, you can enable it by setting :ipv4_fallback_enabled: true in .gemrc (supported on rubygems 3.2.11 and newer, #2662).

echo ":ipv4_fallback_enabled: true" >> ~/.gemrc

Overriding rubygems.org DNS permanently using /etc/hosts will break things again sometime in future.

deivid-rodriguez · 2021-04-26T08:16:06Z

It sounds like this can be closed now since all three affected users were affected by the same issue of having a bad entry in /etc/hosts, correct?

gmcquistin · 2021-04-26T10:48:18Z

Thank you everyone!

Yes, I'll close this issue now as it seems everyone affected has found a solution

Thank you so much for your time and patience with this. Much appreciated indeed! ❤️

santoshcop · 2021-10-04T10:56:03Z

Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate

Ruby Devs,due to "root certificate expiration" issue,

you may see Bundler::Fetcher::CertificateFailureError Could not verify the SSL certificate error during bundle install.

if you run into the issue, please run following command in your terminal and then do bundle install.

bundle config ssl_verify_mode 0 && echo ":ssl_verify_mode: 0" > ~/.gemrc

tasdendu · 2023-10-31T12:49:16Z

bundle config ssl_verify_mode 0 && echo ":ssl_verify_mode: 0" > ~/.gemrc

This is the perfect answer which works for me. Thanks for saving my precious time.

friendlyantz · 2024-02-07T04:22:48Z

if you are on Mac and manage ca-certificates via brew, consider reinstalling it:

brew reinstall ca-certificates

Also, running the above mentioned command (also see below) is dangerous, since it removes ssl verification and exposes you to a man-in-the-middle attack

bundle config ssl_verify_mode 0 && echo ":ssl_verify_mode: 0" > ~/.gemrc

v-natalia · 2024-05-25T14:58:22Z

if you are on Mac and manage ca-certificates via brew, consider reinstalling it:

brew reinstall ca-certificates

This fixed it! After hours of trying every solution out there, thanks so much @friendlyantz

gmcquistin added the Bundler label Apr 21, 2021

gmcquistin closed this as completed Apr 26, 2021

deivid-rodriguez mentioned this issue Apr 27, 2021

Bundler: Network error while fetching (almost anything) #4561

Closed

tjyang mentioned this issue May 10, 2021

Your bundle is locked to mimemagic (0.3.5), but that version could not be found ManageIQ/manageiq-appliance-build#476

Closed

ElvisDot mentioned this issue Aug 24, 2023

[doc] Known setup traps and fixes ElvisDot/lewagon-setup#2

Open

48 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Could not verify the SSL certificate for https://rubygems.org/ #4555

Could not verify the SSL certificate for https://rubygems.org/ #4555

gmcquistin commented Apr 21, 2021

seccomiro commented Apr 22, 2021 •

edited

Loading

sonalkr132 commented Apr 22, 2021 •

edited

Loading

seccomiro commented Apr 22, 2021

seccomiro commented Apr 22, 2021

deivid-rodriguez commented Apr 22, 2021

deivid-rodriguez commented Apr 22, 2021

seccomiro commented Apr 22, 2021

sonalkr132 commented Apr 22, 2021 •

edited

Loading

gmcquistin commented Apr 22, 2021

gmcquistin commented Apr 22, 2021

sonalkr132 commented Apr 22, 2021

sonalkr132 commented Apr 22, 2021

ChaelCodes commented Apr 23, 2021

sonalkr132 commented Apr 23, 2021

ChaelCodes commented Apr 23, 2021

seccomiro commented Apr 23, 2021 •

edited

Loading

sonalkr132 commented Apr 23, 2021

ChaelCodes commented Apr 23, 2021

sonalkr132 commented Apr 23, 2021

seccomiro commented Apr 23, 2021

gmcquistin commented Apr 23, 2021

sonalkr132 commented Apr 24, 2021

ChaelCodes commented Apr 25, 2021

seccomiro commented Apr 26, 2021

ChaelCodes commented Apr 26, 2021

sonalkr132 commented Apr 26, 2021

deivid-rodriguez commented Apr 26, 2021

gmcquistin commented Apr 26, 2021

santoshcop commented Oct 4, 2021 •

edited

Loading

tasdendu commented Oct 31, 2023

friendlyantz commented Feb 7, 2024 •

edited

Loading

v-natalia commented May 25, 2024

Could not verify the SSL certificate for https://rubygems.org/ #4555

Could not verify the SSL certificate for https://rubygems.org/ #4555

Comments

gmcquistin commented Apr 21, 2021

Describe the problem as clearly as you can

Post steps to reproduce the problem

Which command did you run?

What were you expecting to happen?

What actually happened?

If not included with the output of your command, run bundle env and paste the output below

Environment

Bundler Build Metadata

seccomiro commented Apr 22, 2021 • edited Loading

sonalkr132 commented Apr 22, 2021 • edited Loading

seccomiro commented Apr 22, 2021

seccomiro commented Apr 22, 2021

deivid-rodriguez commented Apr 22, 2021

deivid-rodriguez commented Apr 22, 2021

seccomiro commented Apr 22, 2021

sonalkr132 commented Apr 22, 2021 • edited Loading

gmcquistin commented Apr 22, 2021

gmcquistin commented Apr 22, 2021

sonalkr132 commented Apr 22, 2021

sonalkr132 commented Apr 22, 2021

ChaelCodes commented Apr 23, 2021

sonalkr132 commented Apr 23, 2021

ChaelCodes commented Apr 23, 2021

seccomiro commented Apr 23, 2021 • edited Loading

sonalkr132 commented Apr 23, 2021

ChaelCodes commented Apr 23, 2021

sonalkr132 commented Apr 23, 2021

seccomiro commented Apr 23, 2021

gmcquistin commented Apr 23, 2021

sonalkr132 commented Apr 24, 2021

ChaelCodes commented Apr 25, 2021

seccomiro commented Apr 26, 2021

ChaelCodes commented Apr 26, 2021

sonalkr132 commented Apr 26, 2021

deivid-rodriguez commented Apr 26, 2021

gmcquistin commented Apr 26, 2021

santoshcop commented Oct 4, 2021 • edited Loading

tasdendu commented Oct 31, 2023

friendlyantz commented Feb 7, 2024 • edited Loading

v-natalia commented May 25, 2024

If not included with the output of your command, run `bundle env` and paste the output below

seccomiro commented Apr 22, 2021 •

edited

Loading

sonalkr132 commented Apr 22, 2021 •

edited

Loading

sonalkr132 commented Apr 22, 2021 •

edited

Loading

seccomiro commented Apr 23, 2021 •

edited

Loading

santoshcop commented Oct 4, 2021 •

edited

Loading

friendlyantz commented Feb 7, 2024 •

edited

Loading