Artstation is not working #227
Labels
Comments
|
Use the latest version and it will work again: https://github.com/mikf/gallery-dl/releases/download/v1.8.2/gallery-dl.exe The default HTTP headers in v1.8.2 are a lot more browser-like and that's all it takes, apparently. |
|
I'm currently using the latest version. It's not working. Could you check again please? |
|
This is an "interesting" bug, to say the least. It will fix itself whenever Requests starts supporting urllib3 1.25 and I release a new version of gallery-dl (this Friday?) What I've figured out so far:
The Windows executable ships with urllib3 1.24.1 and therefore doesn't work at the moment. If you are willing to install a Python interpreter and use gallery-dl that way, you need to explicitly request urllib3 1.24: |
|
I've rebuilt the v1.8.2 executable with urllib3 1.24. It should work now: |
mikf
added a commit
that referenced
this issue
Apr 26, 2019
urllib3 versions 1.24.1 and 1.24.2 cause HTTP requests to https://www.artstation.com/users/<username>/quick.json to fail with a 403: Forbidden status code (#227), and provoke a CAPTCHA response after solving a Cloudflare challenge.
mikf
added a commit
that referenced
this issue
May 3, 2019
Cloudflare now also checks the client's SSL/TLS cipher capabilities and produces a 403: Forbidden response with CAPTCHA if they are insufficient. This commit replaces the default cipher list in urllib3 < 1.25 with the one from 1.25 (1), which doesn't cause problems as long as the client platform actually supports these ciphers. On some platforms (tested with Python 3.4 on Linux and Python 3.7 on an outdated Windows 7 VM) it is necessary to install pyOpenSSL to get everything to work. Explicitly setting a minimum/maximum version for urllib3 is also no longer necessary and installing gallery-dl will therefore not pull a incompatible urllib3 version (#229) Fixes the "403: Forbidden" error on Artstation (#227) (1) urllib3/urllib3@0cedb3b
|
Could someone with an up-to-date Windows installation test if the Artstation extractor works with 35f3432? I only have access to a Windows 7 installation were I disabled automatic updates years ago and it needs pyOpenSSL to not get a "403: Forbidden" response. |
|
Running latest git master, I get this: Example URL: No pyOpenSSL With pyOpenSSL |
|
With Oh, and yeah, the contents of the directory, for completeness sake: |
|
Okay, so my config file does not seem to make any difference. Will test later on another Windows box with a slightly different version of Python, but I doubt I'll see any other result. |
|
Thank you! Windows 7 or 10 (or any other I guess) makes no difference and pyOpenSSL needs to be an optional dependency for those platforms. Good to know. The |
|
Do I understand this right? For Windows, this is not something that can be changed in I mean, as long as OpenSSL/pyOpenSSL still works it's probably not really a problem, but I just wonder about the reason for this. And given this bit here from https://github.com/pyca/pyopenssl:
|
|
It all boils down to making Cloudflare believe we are an actual web browser controlled by a human being when making HTTP requests, and lately they are using the list of supported SSL/TLS ciphers sent during a SSL handshake to try to differentiate between bot and human. This list of ciphers is influenced by two things:
The latter is limited by the SSL library your Python interpreter was built against, your OS, and maybe some other things as well ... I'm not really sure what exactly is influencing this. pyOpenSSL ships with an up-to-date OpenSSL library and thereby solves these platform limitations. And I've read the warning from pyOpenSSL, but "making a TLS connection" is exactly what it is being used for in this instance, so I wouldn't worry. |
|
Yeah, I've noticed that part about "making a TLS conneciton", I wasn't concerned.. well, I should've just left that part out of my comment. 😅 Just asking out of curiosity. Because no matter what happens on Cloudflare's side, like trying to ask for some fancy and fresh state-of-the-art cipher setting, I would have assumed that this still works as expected on Windows. Otherwise, I was thinking users would have noticed some other breakage on the web in general. Given how commonly used Cloudflare is. But I think I get it now, this isn't a normal TLS handshake, like establishing agreement on both sides involved and then simply go on with business, this is rather using the peculiarities of TLS as a deliberate detection mechanism, so thanks for clearing that up. |
|
same issue |
|
Issue appears to be recurring. With config. Without config: I have the laterst version of pyOpenSSL installed. |
|
@KarelWintersky @kayeags13
pyOpenSSL hasn't been used for more than 2 years by now (3c534d8) |
|
@mikf , confirmed, fixed. Thank you! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
$ gallery-dl.exe "https://www.artstation.com/nshade"
[artstation][error] HTTP request failed: 403: Forbidden for url: https://www.artstation.com/users/nshade/quick.json
The text was updated successfully, but these errors were encountered: