Root certs from keychain not installed when using Homebrew's openssl

[mislav]

I found a regression after my commit ae6a837 that landed in the latest release.

When we built openssl for every Ruby 2.x, we also extracted root certs from system keychain and put them in the openssl directory under ruby's prefix. That enabled users to easily make HTTPS requests from Ruby with VERIFY_PEER enabled, like they could with a Ruby version that was linked with OS X's openssl.

Now, Ruby 2 versions are linked against Homebrew openssl which doesn't have a default /usr/local/etc/openssl/cert.pem and we don't create the file either, since we're not in charge of setting up openssl anymore. That means Ruby 2/Homebrew users will experience HTTPS failures from their code.

I've created my cert.pem like so:

pem_file="$(ruby -ropenssl -e 'puts OpenSSL::X509::DEFAULT_CERT_FILE')"

Then I ran the two find-certificate commands from the above link. This is a one-time operation to set up one's Homebrew-installed openssl. The question is: do we do that for people from ruby-build? It feels intrusive to their environment. However, in the current state people will think their Ruby 2 installs are broken.

[sferik]

@mislav Thanks for reporting this.

IMHO, we should either revert ae6a837 or come up with a better solution quickly and push a new release. This side-effect outweighs the benefit of using OpenSSL from homebrew.

[jacknagel]

Perhaps Homebrew should take care of bootstrapping a cert file after installation. We'd almost certainly accept a patch to the openssl formula to enable this.

[mislav]

@jacknagel Agreed Homebrew could do this. However it's not an immediate enough solution for our needs. We need to push out a ruby-build release fast to help people who are installing on OS X and already have openssl installed through Homebrew. A patch to Homebrew wouldn't help them.

@sferik Let's be crazy and reckless and actually create cert.pem in people's openssl dir if they don't have it. It's what people would want.

[docwhat]

@mislav I think that's reasonable. Based on @jacknagel's comment above, I think it is fine to create the cert.pem if it doesn't exist, homebrew will take care of keeping it up-to-date once we figure out what to do there.

[sferik]

Let's be crazy and reckless and actually create cert.pem in people's openssl dir if they don't have it. It's what people would want.

If this is the prevailing feeling then let’s make the change in homebrew, so it will benefit every project that depends on openssl, not just ruby-build.

Agreed Homebrew could do this. However it's not an immediate enough solution for our needs. We need to push out a ruby-build release fast to help people who are installing on OS X and already have openssl installed through Homebrew. A patch to Homebrew wouldn't help them.

Why wouldn’t a patch to homebrew solve this?

It seems easier to:

brew update
brew upgrade openssl

versus:

brew update
brew upgrade ruby-build
rbenv install 2.0.0-p247

What am I missing?

[mislav]

I foresee two problems for us if we fix it in Homebrew:

1. Even if we teach openssl Homebrew formula to do this, brew upgrade openssl won't actually run unless openssl version changes.
2. Even if brew update && brew upgrade openssl works, how do we communicate to people that this is how they should fix their OpenSSL exceptions in Ruby? They're more likely to update ruby-build and try recompiling Ruby (the ruby-build wiki suggests that) than to upgrade Homebrew

[sferik]

Even if we teach openssl Homebrew formula to do this, brew upgrade openssl won't actually run unless openssl version changes.

Good question. Perhaps we could bump the version number to 1.0.1e-1? Curious what @jacknagel has to say about this.

Alternately, we could instruct users to:

brew update
brew rm openssl
brew install openssl

Even if brew update && brew upgrade openssl works, how do we communicate to people that this is how they should fix their OpenSSL exceptions in Ruby? They're more likely to update ruby-build and try recompiling Ruby (the ruby-build wiki suggests that) than to upgrade Homebrew

We could update the wiki, suggesting that users make sure they have the latest version of both ruby-build and openssl. I’d be happy to make this edit.

[jeremy]

Other distros use a separate package for root certs, like Ubuntu's ca-certificates package. Homebrew's curl-ca-bundle advises you to set SSL_CERT_FILE env var rather than making it the default.

Perhaps the best option here is to have curl-ca-bundle optionally install at the default location.

[jacknagel]

Perhaps we could bump the version number to 1.0.1e-1?

I am working on enabling "revisions" for formula that will allow us to force upgrades, but it will be a while before it lands in core; in the meantime I am a moderately strong -1 on forcing a rebuild of openssl just to run a few shell commands, I think it would be just as easy for ruby-build to do this temporarily.

Perhaps the best option here is to have curl-ca-bundle optionally install at the default location.

I would advise against this; placing untrusted (Homebrew doesn't do any sort of signature verification on this package) certs into the default location is asking for trouble.

[jeremy]

Good point @jacknagel. Check out how ruby-build avoids this by extracting the certs from the keychain. Perhaps a homebrew "package" could do as much.

[jacknagel]

That is what I am advocating and have proposed in Homebrew/legacy-homebrew#21065.

To summarize, my suggestion is:

• ruby-build creates a cert.pem from the system certs, even when using Homebrew openssl, as @mislav has advocated
• Homebrew incorporates the same steps into the openssl formula
• the next time openssl is updated in Homebrew, the cert.pem will be generated automatically, and ruby-build can remove the code for doing this

[docwhat]

I would suggest ruby-build generates etc/openssl/osx-cert.pem and if etc/openssl/cert.pem doesn't exist create the symlink.

Then homebrew would (somehow) update and maintain etc/openssl/osx-cert.pem.

If the user chooses to point the etc/openssl/cert.pem symlink someplace else (or already has one from fixing this problem themselves) then it won't be broken.

Homebrew and ruby-build can both emit a warning/message about the pre-existing symlink.

We don't want to clobber an existing cert.pem -- they may have a valid reason for using their certs instead of OS X`s.

[mislav]

I have changed my mind about being "crazy and reckless". I'm less convinced now that we should be installing root certs on the behalf of the user when we link against an openssl we are not in charge of (such as Homebrew's). This screws up users that care about security.

I'm more in favor of a warning + suggested line of action for the user in the output after compiling Ruby.

[sferik]

@mislav 👍 Want to write this up?

[balexand]

It seems like a friendly default would be for ruby-build to always build it's own OpenSSL (including certs file from the Mac Keychain). Advanced users can specify that they want to use Homebrew's OpenSSL if they want.

For those looking for a quick workaround: brew uninstall openssl then uninstall and reinstall Ruby.

[jeremy]

"It seems like a friendly default would be [...]" — this is what ruby-build does currently @balexand 😁

[balexand]

@jeremy I apologize in advance if I'm being a bone head, but unfortunately, that's not what I'm experiencing. If I have openssl installed via Homebrew, then ruby-build doesn't build a bundled version of openssl. If I uninstall Homebrew's openssl and then reinstall Ruby, then ruby-build does build the bundled openssl and everything works with no fiddling. The code responsible for this is in ae6a837.

I'm proposing that ruby-build should build the bundled openssl regardless of whether Homebrew's openssl is installed. That way, we get a no-fiddling-required, Just Works^TM experience. And developers who want to use Homebrew's openssl can explicitly set the --with-openssl-dir option.

[balexand]

Basically, I'm proposing that the use_homebrew_openssl method should be deleted. If a developer wants to use an existing openssl version, then they can use --with-openssl-dir explicitly.

[jeremy]

@balexand righto - I'm out of date 👍

[jeremy]

@balexand (that's how it used to work, fwiw, but it came with its own headaches: check out old issues to review.)

[mislav]

ruby-build used to build a separate openssl for each Ruby 2.0 on OS X. We're now detecting Homebrew openssl and using that instead, unless overridden. The CA certs will be installed automatically with brew install openssl. We think this behavior is friendly to the user since the system OpenSSL is outdated and broken in subtle ways because of Apple's patches.

On other systems there is no magic behavior.

[balexand]

Thanks @mislav. I must have had an old version of openssl from Homebrew, since it didn't have certs installed.

[docwhat]

@balexand Yeah, that's the downside of this transition period. But it'll get fixed as people go forward.

Note to all

If you have issues with missing certificates and you use Homebrew, try:

$ brew update
$ brew uninstall openssl
$ brew install openssl

This will set up your certificates correctly in Homebrew's OpenSSL using Apple's CA Root certificates.

[mislav]

Closing since this is now handled in Homebrew Homebrew/legacy-homebrew@0a5b437