Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSL link check fails, while curl succeeds #376

Closed
ilg-ul opened this issue Feb 4, 2017 · 44 comments
Closed

SSL link check fails, while curl succeeds #376

ilg-ul opened this issue Feb 4, 2017 · 44 comments

Comments

@ilg-ul
Copy link

ilg-ul commented Feb 4, 2017

I'm getting the following error when running html-proofer in a Travis job:

- /home/travis/out/gnuarmeclipse/gnuarmeclipse.github.io/qemu/build-procedure/index.html
  *  External link https://developer.apple.com/xcode/downloads/ failed: 302 SSL connect error
- /home/travis/out/gnuarmeclipse/gnuarmeclipse.github.io/windows-build-tools/build-procedure/index.html
  *  External link http://developer.apple.com/xcode/downloads/ failed: 302 SSL connect error
htmlproofer 3.4.0 | Error:  HTML-Proofer found 2 failures!

However, in exactly the same environment, a curl to that address was ok:

$  curl -L --url http://developer.apple.com/xcode/downloads/ --verbose
* Hostname was NOT found in DNS cache
*   Trying 17.146.1.14...
* Connected to developer.apple.com (17.146.1.14) port 80 (#0)
> GET /xcode/downloads/ HTTP/1.1
> User-Agent: curl/7.35.0
> Host: developer.apple.com
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 04 Feb 2017 14:43:25 GMT
< Content-Type: text/html
< Content-Length: 179
< Connection: keep-alive
< Location: https://developer.apple.com/xcode/downloads/
* Server Shield is not blacklisted
...
< 
* Ignoring the response-body
* Connection #1 to host developer.apple.com left intact
* Issue another request to this URL: 'https://idmsa.apple.com/IDMSWebAuth/login?appIdKey=891bd3417a7776362562d2197f89480a8547b108fd934911bcbea0110d07f757&path=%2Fdownload%2F&rv=1'
* Hostname was NOT found in DNS cache
*   Trying 17.171.11.86...
* Connected to idmsa.apple.com (17.171.11.86) port 443 (#2)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using AES128-GCM-SHA256
* Server certificate:
* 	 subject: 1.3.6.1.4.1.311.60.2.1.3=US; 1.3.6.1.4.1.311.60.2.1.2=California; businessCategory=Private Organization; serialNumber=C0806592; C=US; postalCode=95014; ST=California; L=Cupertino; street=1 Infinite Loop; O=Apple Inc.; OU=GNCS Traffic Management; CN=idmsa.apple.com
* 	 start date: 2017-01-20 00:00:00 GMT
* 	 expire date: 2019-01-20 23:59:59 GMT
* 	 subjectAltName: idmsa.apple.com matched
* 	 issuer: C=US; O=Symantec Corporation; OU=Symantec Trust Network; CN=Symantec Class 3 EV SSL CA - G3
* 	 SSL certificate verify ok.
> GET /IDMSWebAuth/login?appIdKey=891bd3417a7776362562d2197f89480a8547b108fd934911bcbea0110d07f757&path=%2Fdownload%2F&rv=1 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: idmsa.apple.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Set-Cookie: JSESSIONID=E374FE13E2A29D92D8B47263B64ACFD7; Path=/; Secure; HttpOnly
< Set-Cookie: dslang=US-EN; Domain=.apple.com; Expires=Thu, 03-Aug-2017 14:43:26 GMT; Path=/; Secure; HttpOnly
< Set-Cookie: dslang=US-EN; Domain=.apple.com; Expires=Thu, 03-Aug-2017 14:43:26 GMT; Path=/; Secure; HttpOnly
< TOTAL_TIME: 105
< DS_TIME: default
< X-FRAME-OPTIONS: DENY
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Cache-Control: private, no-cache, no-store, must-revalidate, max-age=0, post-check=0, pre-check=0
< Pragma: no-cache
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< Content-Type: text/html;charset=UTF-8
< Content-Language: en-US
< Transfer-Encoding: chunked
< Vary: Accept-Encoding
< Date: Sat, 04 Feb 2017 14:43:26 GMT
* Server APPSRV is not blacklisted
< Server: APPSRV
< Set-Cookie: X-SESS=28d4a3da9fb70633345a49ca2d42db4197df95593ba62c62d7c0d5cc6198624c97df0a3f;Version=1;Max-Age=1800;path=/;secure;httponly
< 
<!DOCTYPE html>
...

The entire log is available at:
https://travis-ci.org/gnuarmeclipse/gnuarmeclipse.github.io-source/builds/198333357

For the moment I added the offending URLs to the --url-ignore list, but perhaps there is something you can do to accept these URLs too.

Any thoughts?

@ilg-ul
Copy link
Author

ilg-ul commented Feb 5, 2017

I have three new more URLs that fail the test, but return 200 with curl:

  *  External link https://www.element14.com/community/groups/gnu-arm-eclipse failed: 503 No error
  *  External link https://www.element14.com/community/welcome failed: 503 No error
  *  External link https://www.element14.com/community/discussion/create.jspa?containerID=2436&containerType=700 failed: 503 No error

@ilg-ul
Copy link
Author

ilg-ul commented Feb 6, 2017

Some more, reported by Travis:

  *  External link https://jekyllrb.com/ failed: response code 0 means something's wrong.
             It's possible libcurl couldn't connect to the server or perhaps the request timed out.
             Sometimes, making too many requests at once also breaks things.
             Either way, the return message (if any) from the server is: SSL connect error
...
  *  External link http://kramdown.gettalong.org/syntax.html failed: 503 No error

The workaround was to add https://jekyllrb.com,http://kramdown.gettalong.org/syntax.html to the list of exclusions.

@ilg-ul
Copy link
Author

ilg-ul commented Feb 15, 2017

anybody home?

@gjtorikian
Copy link
Owner

However, in exactly the same environment, a curl to that address was ok

Well, that's part of the rub: was that curl command run on the Travis box itself?

This issue has come up before: #141

And it seems like passing { :ssl_verifyhost => 0} to the :typhoeus options works. Could you try that?

@ilg-ul
Copy link
Author

ilg-ul commented Feb 15, 2017

was that curl command run on the Travis box itself?

yes, I provided the link to the Travis log

{ :ssl_verifyhost => 0} Could you try that?

sorry, my knowledge of :typhoeus is limited, can you provide more details on how to do this from a script shell?

@gjtorikian
Copy link
Owner

can you provide more details on how to do this from a script shell?

Sure! Generally it looks like this in Ruby:

HTMLProofer.check_directory("out/", {
  :typhoeus => {
    :ssl_verifypeer => false,
    :ssl_verifyhost => 0}
}).run

I'm embarrassed to say that there doesn't seem to be a way to set Typheous options through the command-line. 😞 I'll open an issue for that.

@ilg-ul
Copy link
Author

ilg-ul commented Feb 15, 2017

I'm currently taking a look at Ruby syntax, but I'm not sure I'll be able to update my travis scripts to replace bash commands with Ruby scripts, so I might wait for the new version that uses command line options.

@mattclegg
Copy link
Contributor

Try adding to your travis.yml;

sudo: required
before_install:
  - sudo update-ca-certificates

The default CAs shipped with Ubuntu 12.04 are somewhat out of date now. Running update-ca-certificates will get the latest certs (but you need sudo to run it).

@ilg-ul
Copy link
Author

ilg-ul commented Mar 6, 2017

as the title states, in exactly the same environment, curl was able to access the links; do you think that curl uses a newer list of certificates than the list updated by update-ca-certificates?

@mattclegg
Copy link
Contributor

This should fix the problem where you get "response code 0" ie on; https://jekyllrb.com/.

Regarding the issue you opened the ticket with; The request to http://developer.apple.com/xcode/downloads/ is being redirected and html-proofer is returning an error because it looks like it doesn't seem to handle 302 requests. All the linkWithRedirect tests are either 301 or 200.

@fulldecent
Copy link
Collaborator

fulldecent commented May 18, 2017

I see several actionable items in this issue:

ddgenome pushed a commit to atomisthq/docs that referenced this issue May 25, 2017
Installing libcurl3-dev seems to fix SSL error HTML Proofer gets when
trying to get https://mochajs.org/ .  See
gjtorikian/html-proofer#141
gjtorikian/html-proofer#194
gjtorikian/html-proofer#376
ddgenome pushed a commit to atomisthq/docs that referenced this issue May 25, 2017
Give the entire rug test doc the once-over, integrating the content on
handler testing and adding as much missing stuff as I could think of.

Install libcurl3-dev to fix HTML Proofer SSL error when it tests
https://mochajs.org/ see
gjtorikian/html-proofer#141
gjtorikian/html-proofer#194
gjtorikian/html-proofer#376

Add retry to HTML Proofer.

Address review comments.
lwasser pushed a commit to earthlab/earthlab.github.io that referenced this issue Aug 17, 2017
see if this fixes response code 0 errors

gjtorikian/html-proofer#376
@mvasin
Copy link

mvasin commented Aug 27, 2017

Here's the error I get:

  *  External link https://talk.jekyllrb.com failed: response code 0 means something's wrong.
             It's possible libcurl couldn't connect to the server or perhaps the request timed out.
             Sometimes, making too many requests at once also breaks things.
             Either way, the return message (if any) from the server is: SSL connect error

I've tried sudo update-ca-certificates, but it didn't work: https://travis-ci.org/cloud-tv/cloud-tv.github.io/builds/268911228

There is no error on CircleCI and on my local machine. Maybe the solution is just to abandon Travis in favour of CircleCI.

@Floppy
Copy link
Contributor

Floppy commented Aug 28, 2017

On the issue of getting SSL failures on travis specifically, I'm working on https://github.com/Floppy/jekyll-test which plans to automate all the required setup to make it work reliably, once I've found out how :)

@gjtorikian
Copy link
Owner

That's a really great idea @Floppy. Wonder if I can convince @parkr to pull it in as an optional gem once it's completed. 🙂

@mizzao
Copy link

mizzao commented Sep 11, 2017

Also getting this error on Travis:

  *  External link https://summit.scienceathome.org failed: response code 0 means something's wrong.
             It's possible libcurl couldn't connect to the server or perhaps the request timed out.
             Sometimes, making too many requests at once also breaks things.
             Either way, the return message (if any) from the server is: SSL connect error

Re-running the build on Travis CI consistently generates this error. It doesn't occur locally (e.g. bundle exec rake test). Any tips for how I can track down the cause? I'm using the following options at the moment (which have cleared up many other connect errors) so it's not due to certificate issues. I've also tried the sudo update-ca-certificates which doesn't do anything on Trusty.

  options = {
    :allow_hash_href => true,       # don't break on <a href="#">
    :assume_extension => false,     # (true) for extensionless paths
    :http_status_ignore => [ 999 ], # LinkedIn throttling errors
    :typhoeus => {
      # avoid strange SSL errors: https://github.com/gjtorikian/html-proofer/issues/376
      :ssl_verifypeer => false,
      :ssl_verifyhost => 0
    }
  }
  HTMLProofer.check_directory("./_site", options).run

@JasonYao
Copy link

Running into the same issue here with the error-code 0:

  • I can pass my tests fine on my local machine
  • Running curl inside the travis environment prints a correct response code (200) to the same external site
  • Running the exact same tests inside the travis environment returns a response code of 0
  • Updating the certs via sudo update-ca-certificates completes successfully and does nothing (and really should do nothing since we're talking about the same environment here)
  • Not sure why the guys upstream at typhoeus are saying that we need libcurl with openssl compiled in, since we've already shown that curl (which depends on libcurl) can perfectly hit the exact same external sites (HTTP and HTTPS) just fine.

Travis information

  • Running Ubuntu 14.04 'Trusty'
  • Link to error log here

Any other information I can provide to help make this easier to debug? Or would this be more of an issue for typhoeus to deal with?

@gjtorikian
Copy link
Owner

I wonder if anyone would try Travis' instructions for Troubleshooting Locally in a Docker Image?

@ilg-ul
Copy link
Author

ilg-ul commented Sep 27, 2017

Personally I gave up, I added --only-4xx to my Travis scripts.

From time to time I run a manual build on my machine with the full check.

@Floppy
Copy link
Contributor

Floppy commented Sep 27, 2017

@gjtorikian that's a brilliant idea, I forgot you could do that. I'm downloading the docker images now and will give it a go with https://github.com/SomethingNewUK/redirector which fails on a very regular basis :)

EDIT: yes, I've now got it replicated locally. html-proofer fails, but curl succeeds. I can investigate more but it's bedtime now. 😴

@Floppy
Copy link
Contributor

Floppy commented Sep 28, 2017

OK, in my travis trusty environment, I'm running:

HTMLProofer.check_directory("./_site",
    typhoeus: {ssl_verifyhost: 0, ssl_verifypeer: false, timeout: 30},
    check_html: true,
    assume_extension: ".html").run

I've also run sudo update-ca-certificates.

Currently I'm still failing with response code 0 in a number of cases, so those aren't the fix. Still looking... :)

@Floppy
Copy link
Contributor

Floppy commented Sep 28, 2017

Sorry if I'm a bit noisy while I debug.

I decided to drop down to typheous and try directly. So, with a simple GET:

require 'typhoeus'

request = Typhoeus::Request.new(
  ARGV[0],
  method: :get,
  headers: { Accept: "text/html" }
)

request.run

puts request.response.code

...my test link (https://somethingnew.org.uk/news/2017/05/15/ge2017-brexit.html) returns 0. So that's good.

Note that a simple CURL on that url returns a 200 successfully.

@Floppy
Copy link
Contributor

Floppy commented Sep 28, 2017

AHA! Got it!

Changing the libcurl to use openssl instead of gnutls fixes it. On the travis environment, if I do sudo apt-get install libcurl4-openssl-dev, it removes the gnutls version, then all the links work with 200 responses.

I've reported this upstream to typhoeus in the linked issue, but we can resolve it here with some instructions and suggested travis config. I'll work out how to get travis to do that itself and report back.

@ilg-ul
Copy link
Author

ilg-ul commented Sep 29, 2017

ok, thank you.

@gjtorikian
Copy link
Owner

Keeping this open just so I remember to find a way to convey this fix as in my comment above.

@gjtorikian gjtorikian reopened this Sep 29, 2017
@ilg-ul
Copy link
Author

ilg-ul commented Sep 29, 2017

Yes @gjtorikian, you could also add a section explaining how to run the proofer via a Rakefile. Assume no Ruby knowledge.

@mizzao
Copy link

mizzao commented Oct 2, 2017

libcurl4-openssl-dev fixed it for me too. For reference, here's my Rakefile for a Jekyll site, including the Typhoeus stuff:

https://github.com/mizzao/andrewmao.net/blob/master/Rakefile

@ilg-ul
Copy link
Author

ilg-ul commented Oct 2, 2017

great! something like this, together with examples how to invoke it, would be useful in the README.

@gjtorikian
Copy link
Owner

@Floppy did an amazing job documenting the fix in the wiki, so I just provided links in the README to that document.

gjtorikian added a commit that referenced this issue Oct 7, 2017
jennyd added a commit to spaconference/spa-website that referenced this issue Nov 29, 2017
After updating all the spaconference.org links to use https, we're still
getting errors in the build, for example:

*  External link https://[secure]rence.org/spa2017 failed: response code 0
*  means something's wrong.
     It's possible libcurl couldn't connect to the server or perhaps the request timed out.
     Sometimes, making too many requests at once also breaks things.
     Either way, the return message (if any) from the server is: SSL connect error

This fix is recommended on the [htmlproofer wiki][wiki], following on from
gjtorikian/html-proofer#376.

[wiki]: https://github.com/gjtorikian/html-proofer/wiki/Using-HTMLProofer-From-Ruby-and-Travis
jennyd added a commit to spaconference/spa-website that referenced this issue Nov 29, 2017
After updating all the spaconference.org links to use https, we're still
getting errors in the build, for example:

*  External link https://[secure]rence.org/spa2017 failed: response code 0
*  means something's wrong.
     It's possible libcurl couldn't connect to the server or perhaps the request timed out.
     Sometimes, making too many requests at once also breaks things.
     Either way, the return message (if any) from the server is: SSL connect error

This fix is recommended on the [htmlproofer wiki][wiki], following on from
gjtorikian/html-proofer#376.

[wiki]: https://github.com/gjtorikian/html-proofer/wiki/Using-HTMLProofer-From-Ruby-and-Travis
jennyd added a commit to spaconference/spa-website that referenced this issue Nov 29, 2017
We've moved the site to HTTPS today and now the build is failing, for example:

    *  External link http://[secure]rence.org/spa2016 failed: 301 SSL connect error

This fix is recommended on the htmlproofer wiki [1], following on from
gjtorikian/html-proofer#376.

[1]: https://github.com/gjtorikian/html-proofer/wiki/Using-HTMLProofer-From-Ruby-and-Travis
Tiryoh added a commit to tofuconf/tofuconf.club that referenced this issue Feb 1, 2018
dberzano pushed a commit to dberzano/alibuild that referenced this issue Feb 20, 2018
* Use more modern APT installation on Travis not requiring `sudo`
* Install `libcurl4-openssl-dev` to prevent HTMLProofer OpenSSL issues
  (gjtorikian/html-proofer#376)
* Use custom HTMLProofer to allow specifying a sensible `timeout` option (the
  `htmlproofer` command-line does not support options)
ktf pushed a commit to alisw/alibuild that referenced this issue Feb 20, 2018
* Use more modern APT installation on Travis not requiring `sudo`
* Install `libcurl4-openssl-dev` to prevent HTMLProofer OpenSSL issues
  (gjtorikian/html-proofer#376)
* Use custom HTMLProofer to allow specifying a sensible `timeout` option (the
  `htmlproofer` command-line does not support options)
urda added a commit to urda/website that referenced this issue May 21, 2018
joshmfrankel added a commit to joshmfrankel/joshmfrankel.github.io that referenced this issue Nov 29, 2018
* Fix SSL issues in TravisCI: gjtorikian/html-proofer#376 (comment)
* Update html-proofer version
* Update Rakefile options used for html-proofer
* Fix pagination linking failure
yous added a commit to yous/yous.be that referenced this issue May 3, 2019
ppannuto added a commit to tock/tock-www that referenced this issue Jul 12, 2019
@m-reuter
Copy link

I still struggle with this. I don't want to include a rake file, it should work directly from travis with --typhoeus_config string but I can not figure out how. Here is my line:
script: bundle exec jekyll build && bundle exec htmlproofer ./_site --check_html --typhoeus_config ''{ "timeout": 30 , "verbose": "true" }''
also tried with three single-quotes and with escaping the curly brackets etc, nothing works and travis won't even start building. Can someone post an example how to pass this in a .travis.yml file? Thanks.

@gjtorikian
Copy link
Owner

gjtorikian commented Apr 22, 2020

Are you able to provide the Travis log? I just tried locally and --typhoeus_config '{ "timeout": 30 , "verbose": "true" }' definitely works.

It may be a Travis CI problem.

@m-reuter
Copy link

Thanks, yes, it is a parsing problem of the travis file, which is why travis does not even start (so there is no log). I found out that removing the space after the ":" works:

script: bundle exec jekyll build && bundle exec htmlproofer ./_site --check_html --typhoeus_config '{ "timeout":30 }'

Then travis runs and calls the htmlproofer. I still get several:

External link <!deleted!> failed: response code 0 means something's wrong.
It's possible libcurl couldn't connect to the server or perhaps the request timed out.
Sometimes, making too many requests at once also breaks things.
Either way, the return message (if any) from the server is: Server returned nothing (no headers, no data)

And it definitely does not wait 30 sec (command finishes in 5 sec). So either it does not take the timeout , or does not wait for some other reason. Passing additionally the verbose: true did have an effect.

@leelee35
Copy link

leelee35 commented Jun 7, 2020

I have three new more URLs that fail the test, but return 200 with curl:

  *  External link https://www.element14.com/community/groups/gnu-arm-eclipse failed: 503 No error
  *  External link https://www.element14.com/community/welcome failed: 503 No error
  *  External link https://www.element14.com/community/discussion/create.jspa?containerID=2436&containerType=700 failed: 503 No error

@fulldecent
Copy link
Collaborator

I have carried this habit from using Travis.

Not sure if this is still necessary with GitHub Actions on ubuntu-latest.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

11 participants