httpie refuses to trust a self-signed certificate

[sensibleish]

I don't entirely understand the discussion for issue #480 but I think the root cause may be the same for this issue. I need httpie to trust a self-signed certificate. Technically it is a self-signed intermediate certificate authority, but I can use one of the actual server certificates from that authority and get the same result. This certificate satisfies every other application on my Mac (OS 10.12), including curl. Httpie simply won't trust it. Sample output:

$ http --debug -j --verify=/usr/local/etc/openssl/certs/intermediate-authority.pem https://www.testdomain.com/robots.txt
HTTPie 0.9.9
Requests 2.12.3
Pygments 2.1.3
Python 3.6.1 (default, Mar 24 2017, 17:14:46)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)]
/usr/local/Cellar/httpie/0.9.9/libexec/bin/python3.6
Darwin 16.5.0

<Environment {
    "colors": 256,
    "config": {
        "__meta__": {
            "about": "HTTPie configuration file",
            "help": "https://github.com/jkbrzt/httpie#config",
            "httpie": "0.9.4"
        },
        "default_options": "[]"
    },
    "config_dir": "/Users/username/.httpie",
    "is_windows": false,
    "stderr": "<_io.TextIOWrapper name='<stderr>' mode='w' encoding='UTF-8'>",
    "stderr_isatty": true,
    "stdin": "<_io.TextIOWrapper name='<stdin>' mode='r' encoding='UTF-8'>",
    "stdin_encoding": "UTF-8",
    "stdin_isatty": true,
    "stdout": "<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>",
    "stdout_encoding": "UTF-8",
    "stdout_isatty": true
}>

>>> requests.request(**{
    "allow_redirects": false,
    "auth": "None",
    "cert": "None",
    "data": "",
    "files": {},
    "headers": {
        "Accept": "application/json, */*",
        "Content-Type": "application/json",
        "User-Agent": "HTTPie/0.9.9"
    },
    "method": "get",
    "params": {},
    "proxies": {},
    "stream": true,
    "timeout": 30,
    "url": "https://www.testdomain.com/robots.txt",
    "verify": "/usr/local/etc/openssl/certs/intermediate-authority.pem"
})


http: error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749) while doing GET request to URL: https://www.testdomain.com/robots.txt
Traceback (most recent call last):
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 588, in urlopen
    self._prepare_proxy(conn)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 801, in _prepare_proxy
    conn.connect()
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connection.py", line 323, in connect
    ssl_context=context)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/util/ssl_.py", line 324, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 401, in wrap_socket
    _context=self, _session=session)
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 808, in __init__
    self.do_handshake()
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 1061, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ssl.py", line 683, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/adapters.py", line 423, in send
    timeout=timeout
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py", line 624, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/http", line 11, in <module>
    load_entry_point('httpie==0.9.9', 'console_scripts', 'http')()
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/__main__.py", line 11, in main
    sys.exit(main())
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/core.py", line 227, in main
    log_error=log_error,
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/core.py", line 99, in program
    final_response = get_response(args, config_dir=env.config.directory)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/httpie/client.py", line 70, in get_response
    response = requests_session.request(**kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/sessions.py", line 609, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/Cellar/httpie/0.9.9/libexec/lib/python3.6/site-packages/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:749)

If I use --verify=no, the connection is accepted. I've tried converting the certificate file to DER format instead of PEM with no change.

Httpie was installed via Homebrew, which installed its own Python3 as a dependency.

[alvis]

Trusting a self-signed certificate is never encouraged. It is dangerous if a program accepts it by default. A hacker can simply inject a self-signed one to make the users feeling secure while it is not the case.

However, I would suggest httpie to response the issue of using a self-signed certificate with a clearer message rather than simply fail it. A suggestion to use --verify=no to disable the verification for testing purpose would be great.

[sigmavirus24]

@giskard22

I don't entirely understand the discussion for issue #480 but I think the root cause may be the same for this issue. I need httpie to trust a self-signed certificate. Technically it is a self-signed intermediate certificate authority, but I can use one of the actual server certificates from that authority and get the same result.

Looking at #480 this is not the same root cause. HTTPie is, in this case, using the bundle you provide it.

What I suspect is actually the cause is the following:

Your "test domain" is not returning the full certificate chain but only its leaf (which works fine for browsers which can dynamically retrieve the rest of the chain) but not for httpie and other non-browser clients. In this case, you're providing part of the chain via --verify but still not all of the intermediaries. If you place them all in one bundle and provide that instead, I suspect you'll find that this works.

Trusting a self-signed certificate is never encouraged. It is dangerous if a program accepts it by default. A hacker can simply inject a self-signed one to make the users feeling secure while it is not the case.

@alvis blanket advice is usually wrong and in this case it's quite alarmingly wrong. Self-signed certificates are a solution to a problem. I don't know @giskard22's use case for using a self-signed certificate here and neither do you. In that case your advice is not only most likely useless to @giskard22, it's also noise for the rest of the people receiving email notifications from issues on this repository.

Self-signed certificates can be dangerous to trust without verification but in this case it seems @giskard22 knows the trust chain and trusts it making this usage perfectly safe. Yes, if you happen upon a site using HTTPS that you might need to enter sensitive information into, it is not advisable to use a self-signed certificate but this is a wildly different situation as I understand it.

[sensibleish]

Yes, the use case here is using httpie through a corporate web proxy that does HTTPS man-in-the-middle so it can inspect traffic. It generates a domain-specific certificate on the fly. Blah.

I don't understand how the proxy's failure to provide a full certificate chain causes this issue. Perhaps OpenSSL's behavior (or is it Python's behavior?) is different than MacOS's built-in mechanism.

The trust chain for connections through the proxy looks like this:

1. On-the-fly certificate issued to www.testdomain.com
2. Intermediate CA certificate called proxy-name
3. Self-signed root CA certificate called company-name

I have been unable to obtain certificate 3. However, certificate 2 is available from the proxy. I imported into my Keychain and marked it trusted for SSL use. That was enough to satisfy everything on my Mac except httpie, including other command-line tools like curl. The observed behavior is the validation mechanism follows the chain until it finds a trusted certificate, then stops. The root certificate's absence is irrelevant.

In the httpie debug output I posted in my initial report, I had stored certificate 2 in intermediate-authority.pem and specified that file with --verify. Are you saying that for httpie, that's not enough and I must also provide certificate 3?

[sigmavirus24]

Perhaps OpenSSL's behavior (or is it Python's behavior?) is different than MacOS's built-in mechanism.

So OpenSSL will look in the bundle you provide it for whatever it's missing. If it cannot find it there it fails the verification. This is transitively also Python's behaviour. SecureTransport (MacOS's built-in mechanism) works slightly differently but only if the certificates are in MacOS's Keychain. Also, on MacOS the default Python has a hacked-up (for lack of a better term) version of OpenSSL that uses Keychain and thus can resolve these even if you provide it your own certificate bundle (i.e., if you tell it only to trust what's in a bundle file, it'll say "LOL, but I got Keychain access and that trusts it so YOLO" ... or YOVO? (You Only Verify Once... Idk)).

If the entire trust chain for these on-the-fly certificates is what you described, I think you should be able to get away with having a bundle that only includes links 2 and 3 in the chain.

Also, if you installed httpie from homebrew on MacOS then it was installed with the homebrew versions of OpenSSL, Python, and virtualenv which is why it cannot take advantage of a Mac's hacked-up OpenSSL.

[sensibleish]

I was finally able to identify and obtain the self-signed root certificate. Upon putting both that cert and the intermediate cert in a .pem file and using that file with --verify, I no longer got the SSL error. This is good! As suggested, OpenSSL requires the entire certificate chain. Unlike SecureTransport, it won't accept a partial chain even if manually supplied.

Any thoughts on the following additional complication? The proxy lets traffic go to certain domains without inserting its custom certificate. It looks like you cannot use multiple --verify arguments. The last one wins. Is there any way to use multiple certificate bundles?

I have a workaround that involves concatenating OpenSSL's provided root CA bundle and the custom bundle upon demand, but hoping to avoid that.

[sigmavirus24]

So OpenSSL supports providing a directory full of .pem files for verification. That said, support for that in Python is spotty. I don't honestly think anything older than Python 3.5 or 3.6 supports directories in the standard library and I don't recall if pyOpenSSL will support it. The short of it is that I don't think you can accomplish your ends without having one all-serving .pem file. The other catch here is that the HTTP transport library that HTTPie uses doesn't quite support directories either given the limited support across versions and TLS library implementations.

[PostImpatica]

ol' Postman to the rescue. You can turn off detection of self signed cert.

[squarism]

SSL is good, probably devs know that. Disabling SSL because of corner cases and language limitations or whatever stinks. I know people just want to get work done. I just wonder when we'll say SSL is good and have the same gut reaction at saying "disable SSL verification". If you disable SSL verification, SSL is off. But SSL is good. We all know that right? Is it confusing or common knowledge that SSL verification is really what the whole SSL thing is? If postman has an option to disable SSL verification is that just a fancy way of saying something that is potentially insecure? curl has that same option as a flag called --insecure. Is it just a fancy way that hides exactly what the problem is (it turns off SSL)? I think it's fine to test in Postman. TIL that Postman has that option. Am I going to turn it back on? Or did I just disable SSL for every current and future project? Am I sending API keys from a coffee shop? :|

I hope https dev machines, self signed certs and corporate CAs are supplanted by scriptable automatic letsencrypt certs one day. It's a shame that python makes http less organic at trusting the system keychain / ssl store. It's a curl replacement except in this area. I guess golang has operating system detection for all this stuff. So a modern curl replacement (like bat) that's written in go would have pretty output, a nice CLI experience and handle custom CAs or ssl termination (load balancers).

[basilmusa]

SSL is good, probably devs know that. Disabling SSL because of corner cases and language limitations or whatever stinks. I know people just want to get work done. I just wonder when we'll say SSL is good and have the same gut reaction at saying "disable SSL verification". If you disable SSL verification, SSL is off. But SSL is good. We all know that right? Is it confusing or common knowledge that SSL verification is really what the whole SSL thing is? If postman has an option to disable SSL verification is that just a fancy way of saying something that is potentially insecure? curl has that same option as a flag called --insecure. Is it just a fancy way that hides exactly what the problem is (it turns off SSL)? I think it's fine to test in Postman. TIL that Postman has that option. Am I going to turn it back on? Or did I just disable SSL for every current and future project? Am I sending API keys from a coffee shop? :|

I hope https dev machines, self signed certs and corporate CAs are supplanted by scriptable automatic letsencrypt certs one day. It's a shame that python makes http less organic at trusting the system keychain / ssl store. It's a curl replacement except in this area. I guess golang has operating system detection for all this stuff. So a modern curl replacement (like bat) that's written in go would have pretty output, a nice CLI experience and handle custom CAs or ssl termination (load balancers).

Nope, not good. Need to be able to curl even if adhoc developer certificate.

[tidzerda]

I think we all know that we should not be using self signed SSL certs, but sometimes, in a dev environment, you just want to get on with it.
It should be possible to use these certificates, if they are sgned by a CA, that is entered into your OS trust store? (even if the CA is yourself). Most browsers are ok with this.
Not trusting a Self signed certs is a pain, esp if you are running on your own network.
Back to curl, postman or insomnia.

[tomtastic]

I believe it's an anti-pattern to train users to turn off certificate verification for self-signed certs.

httpie is used widely in my current (very large) org, we have a internal trusted CA cert that isn't public and isn't used for public internet trusts.

httpie - unlike curl (when compiled with SecureTransport) - can't use this internal CA trust (in the keychain for our MacOS developer systems), which is a great shame and only enforces user/developer behaviour to turn off verification, when there is absolutely a good reason not to.

[needarun]

Hi,

What is exposed in the discussion is weird: most of the intervenants seem to think that a self-signed certificate is a "bad thing".
In my case, I'm using Vault as a PKI cert issuer and I don't understand why I could not trust my own certs as I know these are secure, on my side.
I use httpie mainly to check if my URLs and entrypoints are OK, before configuring my monitoring to do some health checks. In the monitoring I WON'T turn off SSL check, so why should I disable it on my manual test ? This won't make it as close as possible to the monitoring point of view.
So, I agree with @tomtastic : this is not cool not to be able to use internal keychain for validation.
My 50 cents...

[tgrushka]

Most people use httpie for development purposes.

Most developers use self-signed certificates in development.

This issue has been open for nearly 8 years. Why?

1. Installing my self-signed certificate, running sudo update-ca-certificates, and exporting:

export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt

does not work. httpie does not obey the env setting.

2. The httpie config file does not work:

[global]
ssl_verify = True
ssl_ca_path = /etc/ssl/certs/ca-certificates.crt

or

[global]
ssl_verify = False

3. Instead of just allowing us to trust a single self-signed certificate, the authors of httpie basically require us to use verify=no, which is far worse than trusting a single self-signed certificate.

I don't know, I wonder if some of these package authors enjoy enforcing their opinions on users? This violates universal design principles far more than the security drawbacks alleged. Do they really believe bypassing verification for all requests is safer than using a self-signed certificate? Do they even use this package themselves for development?