-
-
Notifications
You must be signed in to change notification settings - Fork 996
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Weasyl Extractor #977
Weasyl Extractor #977
Conversation
Decided to give this one a go and other than the supportsites.py issue the only feedback I've got is to perhaps set a different default filename format? Others extractors tend to prefix the filenames with the category. My suggestion would be something like
Other than that, seems to work great! |
Most extractors format I can get adding {category} for consistency but I do like keeping the date around since its data that isn't carried anywhere else (it isn't in the img metadata) and can be useful for sorting. Most extractors just don't have the luxury of getting it via an API. Hows |
I was mainly thinking of the imgur, gfycat and furaffinity extractor that does in fact just get the title, perhaps others too. I personally find the date really useless, you get the same thing with ID (which is why it goes after {category}, for sorting). gallery-dl/gallery_dl/extractor/imgur.py Line 58 in 2184ec5
gallery-dl/gallery_dl/extractor/gfycat.py Line 19 in 2184ec5
Though I see now that the furaffinity one is also lacking {category}, so idk how important consistency is. |
I think it is generally a good idea for sites with 1 file per post and many posts per user to try and replicate the general structure of the furaffinity module, but it might be a bit too late for that.
|
If I'm not encoding date in the filename_fmt anymore all the date stuff can be dropped. Mind if I take a stab at #374 to embed this stuff? Everything else is in the latest revision. Unrelated question but is there a way to run tests on just the one extractor? |
Ok then, thanks a lot @Korvox. Time to merge this.
The
Sure. Do you know of a good (cross-platform) library that can do this?
Running test/test_results.py with the
|
First of all thank you very much for implementing this @Korvox !
Should I open up a followup issue for this, since I think it would be worth having? |
Is there any way to login or use cookies? As it stands, without a way to login, only SFW submissions are able to be downloaded. Being able to use an API key generated in my account settings would be good too. |
You can generate an API key here: https://www.weasyl.com/control/apikeys API calls want the I'll look into hooking it into extractor.config. Tumblr does basically the same thing. |
Does gallery-dl even have a way of using custom header values other than User-Agent? |
@Korvox There's another problem I ran into. Usernames are allowed to have tildes in their name, which means two tildes in the URL. This messes up the pattern matching for extraction. |
Can you link me an example of it? Its a weird edge case if it exists because all the API requests use login_names: "A user’s username, lowercase, and omitting all non-alphanumeric, non-ASCII characters." |
Closes #813
Supports individual posts, user galleries, folders, a journal, or all of a users journals. Uses the Weasyl API for everything but journals since there is no endpoint for that.