[deviantart] Scraps Downloader not downloading all images

[sledgehammer93]

Hello again.

Wanted to start by saying that the maintenance of this program is top-notch. This has been my go-to program for downloading images in bulk from various websites. I've found nothing else on the net that can even compare to the usefulness and utility of gallerydl. Thank you.

However, I appear to be having another issue regarding the downloading of scraps in a Deviant Artist's gallery. It appears to only be downloading the non-mature downloads present in the scraps folder and skipping the rest. I have mature set to true in my gallery-dl.conf file, and everything else appears to be downloading perfectly fine. If an example is needed, I can provide one.

Any help is appreciated,
sledgehammer93

[mikf]

The Scraps extractor uses a different interface or API than the rest, and the only way to get it to "see" mature content is to be logged in with an account that has "Display Mature Content" on the DeviantArt website enabled, it seems.

Logging in with --username and --password is not implemented (or possible, since automated login attempts get blocked), so your only option is exporting your DeviantArt cookies while logged in and making them available to gallery-dl with, for example, --cookies.

[sledgehammer93]

@mikf Ran a download session yesterday with the cookies option enabled. It worked for the first test, but subsequent scraps urls (or downloads) failed to load the cookies file.

[mikf]

What kind of error message did you get?
This is also kind of weird, since gallery-dl does not change the content your cookie files; it only reads from them and that's it. If it worked once, it should work again.

[sledgehammer93]

@mikf, It actually doesn't display any sort of error. It just reverts back to not loading the cookies file, thus not downloading all of the scraps. For example, on my first test run, I downloaded a deviantart gallery with 132 scraps images, and it successfully completed this with my predefined cookies file. However, the subsequent runs did not load this file, even when configured to do so in my gallery-dl.conf. As a result, I am noticing a lot of incomplete scraps downloads due to the cookies file not loading.

UPDATE: Turns out, even if I supply a new cookies file, it refuses to download any of the mature scraps images. Just tested on the aforementioned deviantart gallery, and it only downloaded 43 out of 132 images.

[mikf]

As it turns out, DeviantArt regularly sends new _auth cookies to its clients and, like a regular browser would, expects them to store and use the new value on subsequent HTTP requests. Because gallery-dl didn't update the cookies in a given cookies.txt file, you'd get logged out after a a few hours since it wouldn't use the new cookies when downloading a new gallery.

I've now added a cookies-update option (389d2d7), that will keep the cookies in a cookies.txt file up to date, so the login session doesn't expire after some time.

To get everything working, you should visit DeviantArt in your browser, so it can update your login session cookies there, then export them to a cookies.txt file, and enable the new cookies-update option to update the session cookies in said cookies,txt file after every use.

If you want to test if everything works, use deviantart.com/c4083497/gallery/scraps. It's just a single black-to-white gradient tagged as mature. It might also be necessary to use a secondary account for all of this, since DeviantArt could get "confused" with two active sessions for the same account.

[sledgehammer93]

@mikf, Sorry for the late reply. I ran gallery-dl with the example you provided, and it worked. I also tried the download again with my test url, and it worked as well. The cookies-update option is enabled in my gallery-dl.conf file.

However, when doing a batch download using several urls, the cookies file is still having issues. In fact, I have noticed that I cannot log into my deviantart account in my browser, which uses the same cookie file. The only way I can access my account and thus download all the scraps from all my urls is to reset my home network. Must be something on deviantart's end (they are possibly blocking my ip address when running the batch of urls).

I am providing my gallery-dl.conf file just in case it may be something on my end. I might have missed something again.

Thanks for your work.

{
    "extractor":
    {
        "base-directory": "/media/william/Shared/Important/gallery-dl/",
        "postprocessors": null,
        "cookies": "/media/william/Shared/Important/gallery-dl/cookies",
	"cookies-update": true,
        "proxy": null,
        "skip": true,
        "sleep": 0,
        "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0",
        "path-restrict": "\\\\|/<>:\"?*",
       
        "deviantart":
        {
            "extra": true,
            "flat": true,
            "folders": false,
            "journals": "html",
            "mature": true,
            "metadata": true,
            "original": true,
            "quality": 100,
	    "wait-min": 0,
	    "client-id": "client id",
	    "client-secret": "client secret here",
	    "refresh-token": "token here"
	}   
    },

    "downloader":
    {
        "part": true,
        "part-directory": null,

        "http":
        {
            "adjust-extensions": true,
            "mtime": true,
            "rate": "512k",
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        },

        "ytdl":
        {
            "format": "bestvideo+bestaudio/best",
            "forward-cookies": true,
            "mtime": true,
            "rate": null,
            "retries": 4,
            "timeout": 30.0,
            "verify": true
        }
    },

    "output":
    {
        "mode": "auto",
        "progress": true,
        "shorten": true,
        "log": "[{name}][{levelname}] {message}",
        "logfile": null,
        "unsupportedfile": null
    },

    "netrc": false
}

[mikf]

What you could do, to only use your cookies when downloading scraps, is moving your cookies options under extractor.deviantart.scraps:

{
    "extractor":
    {
        "deviantart":
        {
            "scraps":
            {
                "cookies": "/media/william/Shared/Important/gallery-dl/cookies",
                "cookies-update": true
            },
            ...
	}
    }
}

That way those options get only applied to /scraps URLs, and not elsewhere where they aren't needed.

Another thing that gallery-dl should do is to not send cookies when requesting extended information for a Deviation, which surprisingly works without authentication and happens for every file when downloading scraps.