[Hentainexus] 400 Bad Request Error

[ShadowX1X]

I've noticed that there are errors downloading certain images in Hentainexus galleries. Here is the --verbose output:

Microsoft Windows [Version 10.0.18363.1379]
(c) 2019 Microsoft Corporation. All rights reserved.

C:\Users\X1X\Documents\Scripts>gallery-dl https://hentainexus.com/view/9699 --verbose
[gallery-dl][debug] Version 1.16.5
[gallery-dl][debug] Python 3.6.4 - Windows-10-10.0.18362-SP0
[gallery-dl][debug] requests 2.25.1 - urllib3 1.26.2
[gallery-dl][debug] Starting DownloadJob for 'https://hentainexus.com/view/9699'
[hentainexus][debug] Using HentainexusGalleryExtractor for 'https://hentainexus.com/view/9699'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): hentainexus.com:443
[urllib3.connectionpool][debug] https://hentainexus.com:443 "GET /view/9699 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] https://hentainexus.com:443 "GET /read/9699 HTTP/1.1" 200 None
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): i1.wp.com:443
[urllib3.connectionpool][debug] https://i1.wp.com:443 "GET /images.hentainexus.com/gallery/fc5f7eb0a13b18581669bf7198105f25/9699/001.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i1.wp.com/images.hentainexus.com/gallery/fc5f7eb0a13b18581669bf7198105f25/9699/001.png?filter=null'
[download][error] Failed to download hentainexus_9699_001.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/1181d5ab236432d6415237cce83770bc/9699/002.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/1181d5ab236432d6415237cce83770bc/9699/002.png?filter=null'
[download][error] Failed to download hentainexus_9699_002.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (2): i1.wp.com:443
[urllib3.connectionpool][debug] https://i1.wp.com:443 "GET /images.hentainexus.com/gallery/93028c2e70fc4b17a0c37ef878653c3a/9699/003.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i1.wp.com/images.hentainexus.com/gallery/93028c2e70fc4b17a0c37ef878653c3a/9699/003.png?filter=null'
[download][error] Failed to download hentainexus_9699_003.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (2): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/6731e088f4336429424e002f85060b12/9699/004.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/6731e088f4336429424e002f85060b12/9699/004.png?filter=null'
[download][error] Failed to download hentainexus_9699_004.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (3): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/8f5b745b15a8d5bd0eee773fbe177b3d/9699/005.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/8f5b745b15a8d5bd0eee773fbe177b3d/9699/005.png?filter=null'
[download][error] Failed to download hentainexus_9699_005.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/98ef0b20a58d6ed39d9c560a93fd32b5/9699/006.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/98ef0b20a58d6ed39d9c560a93fd32b5/9699/006.png?filter=null'
[download][error] Failed to download hentainexus_9699_006.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (4): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/a0337ec9a504b97837d8a568801eadaa/9699/007.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/a0337ec9a504b97837d8a568801eadaa/9699/007.png?filter=null'
[download][error] Failed to download hentainexus_9699_007.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (2): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/de78267831813ccb2c1e2996561d313d/9699/008.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/de78267831813ccb2c1e2996561d313d/9699/008.png?filter=null'
[download][error] Failed to download hentainexus_9699_008.png
# .\gallery-dl\hentainexus\9699 Sex Communication With Both of My Bosses\hentainexus_9699_009.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (5): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/b2ca57a363f9c6c2c63a54fa14d4a6e6/9699/010.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/b2ca57a363f9c6c2c63a54fa14d4a6e6/9699/010.png?filter=null'
[download][error] Failed to download hentainexus_9699_010.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (3): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/49098bd43cd726b6bd392517a3a2f1c3/9699/011.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/49098bd43cd726b6bd392517a3a2f1c3/9699/011.png?filter=null'
[download][error] Failed to download hentainexus_9699_011.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (4): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/44cfbefb3762295d3ff17ee69b0b5700/9699/012.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/44cfbefb3762295d3ff17ee69b0b5700/9699/012.png?filter=null'
[download][error] Failed to download hentainexus_9699_012.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (3): i1.wp.com:443
[urllib3.connectionpool][debug] https://i1.wp.com:443 "GET /images.hentainexus.com/gallery/424ffccbe13c6f61164586d4a87ab036/9699/013.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i1.wp.com/images.hentainexus.com/gallery/424ffccbe13c6f61164586d4a87ab036/9699/013.png?filter=null'
[download][error] Failed to download hentainexus_9699_013.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (6): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/7259422f41028d83679df7787fb09cc2/9699/014.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/7259422f41028d83679df7787fb09cc2/9699/014.png?filter=null'
[download][error] Failed to download hentainexus_9699_014.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (4): i1.wp.com:443
[urllib3.connectionpool][debug] https://i1.wp.com:443 "GET /images.hentainexus.com/gallery/eba80cf3d50ed2f18bfd86db8e0c71d3/9699/015.png?filter=null HTTP/1.1" 200 456547
* .\gallery-dl\hentainexus\9699 Sex Communication With Both of My Bosses\hentainexus_9699_015.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (7): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/7d45b86cfc6baf49f9b30a8253076066/9699/016.png?filter=null HTTP/1.1" 200 528270
* .\gallery-dl\hentainexus\9699 Sex Communication With Both of My Bosses\hentainexus_9699_016.png
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/5339a959df2a2c64efa1a73cbdcdeb74/9699/017.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i2.wp.com/images.hentainexus.com/gallery/5339a959df2a2c64efa1a73cbdcdeb74/9699/017.png?filter=null'
[download][error] Failed to download hentainexus_9699_017.png
[urllib3.connectionpool][debug] https://i1.wp.com:443 "GET /images.hentainexus.com/gallery/b0c5fb520e44b7dc168aec592b623120/9699/018.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i1.wp.com/images.hentainexus.com/gallery/b0c5fb520e44b7dc168aec592b623120/9699/018.png?filter=null'
[download][error] Failed to download hentainexus_9699_018.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (5): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/f0843275a93e7644980dedaf51e49547/9699/019.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/f0843275a93e7644980dedaf51e49547/9699/019.png?filter=null'
[download][error] Failed to download hentainexus_9699_019.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (6): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/7fbd0e7bce1f954d313eb9f692c5e7d9/9699/020.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/7fbd0e7bce1f954d313eb9f692c5e7d9/9699/020.png?filter=null'
[download][error] Failed to download hentainexus_9699_020.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (5): i1.wp.com:443
[urllib3.connectionpool][debug] https://i1.wp.com:443 "GET /images.hentainexus.com/gallery/f78ab1d9e7ff7e893724cc79ae43a654/9699/021.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i1.wp.com/images.hentainexus.com/gallery/f78ab1d9e7ff7e893724cc79ae43a654/9699/021.png?filter=null'
[download][error] Failed to download hentainexus_9699_021.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (7): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/b8a6af5f8f9835ea78aa87dc675253a5/9699/022.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/b8a6af5f8f9835ea78aa87dc675253a5/9699/022.png?filter=null'
[download][error] Failed to download hentainexus_9699_022.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (8): i2.wp.com:443
[urllib3.connectionpool][debug] https://i2.wp.com:443 "GET /images.hentainexus.com/gallery/2cbe3d3c862de773678cf99cbf4a2b78/9699/023.png?filter=null HTTP/1.1" 200 518417
* .\gallery-dl\hentainexus\9699 Sex Communication With Both of My Bosses\hentainexus_9699_023.png
[urllib3.connectionpool][debug] Starting new HTTPS connection (8): i0.wp.com:443
[urllib3.connectionpool][debug] https://i0.wp.com:443 "GET /images.hentainexus.com/gallery/90ebd8d51681c9c80b3af468334100f9/9699/024.png?filter=null HTTP/1.1" 400 None
[downloader.http][warning] '400 Bad Request' for 'https://i0.wp.com/images.hentainexus.com/gallery/90ebd8d51681c9c80b3af468334100f9/9699/024.png?filter=null'
[download][error] Failed to download hentainexus_9699_024.png

C:\Users\X1X\Documents\Scripts>

However, I've also noticed that some images aren't loading on Hentainexus at all (when I actually visit the galleries via a browser), with the same error. I was just wondering if this was an error on their side; I've heard their servers are pretty unstable and I've just found a heck of a lot of galleries that have missing images. Is this an issue on their side (in which case I can't do anything) or is this something with gallery-dl?

[ShadowX1X]

This is also happening with galleries where the images aren't missing actually. For example, https://hentainexus.com/view/8640

[mikf]

This is an issue with Hentainexus' image server(s).

When you remove the Wordpress part from those 400 URLs and directly point to images.hentainexus.com, you get the proper 522 error with Connection Timed Out as description:

# 400: Bad Request
$ wget https://i2.wp.com/images.hentainexus.com/gallery/1181d5ab236432d6415237cce83770bc/9699/002.png?filter=null
# 522
$ wget https://images.hentainexus.com/gallery/1181d5ab236432d6415237cce83770bc/9699/002.png?filter=null

You are still able to see those images in a browser because it also requests the WebP version, which is still cached by the Wordpress servers.
I could add an option to request those with gallery-dl instead of the original files, if that would be helpful.

[mo-han]

@mikf
add option for webp fallback is very helpful, since HN site often went into such trouble and couldn't recover in short time
so please release a fix for this
btw, actually i personally convert all png into webp after download to save disk space

[ShadowX1X]

Yeah, that would be great if you could add an option to request the webp version instead. I can convert those to PNG/JPG myself easily so that's totally fine. I appreciate your help and response!

[arisboch]

Isn't it possible for gallery-dl to convert the webp files to jpeg or png automatically, it's an open format, and I'm quite sure there are libraries that handle the conversion for Python3, as well.

[rautamiekka]

^ FFmpeg.

[kattjevfel]

Isn't it possible for gallery-dl to convert the webp files to jpeg or png automatically, it's an open format, and I'm quite sure there are libraries that handle the conversion for Python3, as well.

You'd want to use imagemagick for images, ffmpeg for videos. I use these two scripts to convert pngs and optimise jpgs:
https://github.com/kattjevfel/scripts/blob/master/gallery-dl_postprocessing_selector.sh
https://github.com/kattjevfel/scripts/blob/master/png2webp.sh

[mikf]

There is now a original option for hentainexus, that selects between original jpg/png and recompressed webp (6b0ecbf), as well as a more general headers option for the HTTP downloader (5602773).

(original is implemented by changing the Accept header from */* to image/webp,*/*)

so please release a fix for this

There is a GitHub Actions workflow that builds executables for the latest commit:
https://github.com/mikf/gallery-dl/actions/workflows/executables.yml
And you can install from the latest commit with pip as well.

[ShadowX1X]

Thanks a lot! And with the issues on Hentainexus' side, I can only hope the owner sees the issues with their server and fixes it in due time... For anyone not exactly sure about how to use the new option, enter gallery-dl -o original=false [link] into the command line to download the webp versions of the image, or insert this into your config.json file:

		"hentainexus":
		{
			"original": false
		},