Doesn't download all posts; Subscribed content is missing #59
Comments
|
Yeah, it seems to be downloading preview images, but not the unlocked image posts I can see on the site as a subscriber. |
It's very unlikely to be mistaken for a duplicate, there's an actual fool proof algorithm that hashes the images and decides if they're very similar to each other or not.
What creator are you encountering this issue with? Which posts is it missing (subscriber or preview)? Like theoretically, it's able to mistake a posts as duplicates, once they're drastically similar to each other, like if the performer is taking the same stance, below the same lighting in multiple images, those would most likely get flagged as duplicates. But yes I might consider making duplicates flagging a option that you can turn on and off or maybe even can even adjust its threshold for. |
I've never spent any money on NSFW content (or any at all) ever, so it's hard for me to debug issues like this. If you're having trouble with subscribed posts not being downloaded and really want it to get fixed within fansly scraper; would you for debugging purposes provide me an account, that has access to subscriber only posts, through e-mailing me credentials to Also I'm wondering if this issue started happening recently for you or if you're encountering this issue ever since the first use of fansly scraper. |
This is a brand new issue, I'd never seen this before today - I noticed it downloaded the preview image but not the subscribed images. The last time I ran the scraper December 8th everything downloaded as normal - but today it only grabbed the preview images |
Avnsx
added
bug
Avnsx
changed the title
|
You guys are sure, subscriber only content can't be downloaded, with every creator you subscribed to, right? Does this apply to subscribed content from messages only? Or does it maybe apply to subscribed content from purchased bundles only? |
|
I have the same problem, but it doesn't download content in messages |
|
Having the same issue. What it's doing for me is when a post has content for multiple tiers, it downloads the content from the lowest tier, but not the highest (which I'm subscribed to and can see). It also only downloads the censored preview version of purchased pics, even after I've purchased them and attempted to re-download. |
|
If anyone lends me their account, which has currently access to any paid bundles and messages and what not, I can fix these issues really quickly and also add that one feature y'all been requesting for a while called PPV content (#12). Send an e-mail to Because personally, I'm not planning to spend money on nsfw content anytime soon! |
|
@Avnsx sent you an email |
|
@Avnsx Do you still need an account with access to paid content? |
Yes, still |
|
Just sent you an email. |
|
First attempt to use fansly scraper v0.3.6. I got it to run by manually configuring config.ini. I like what I see except it did not download media from the messages directory. I know there is 2 pics and 1 video that should be downloadable. Is there any thing I can try to fix this? |
Unless you're subscribed to a creator on fansly and know how to program in python, no not really Gotta wait for me to fix it, didn't find the time yet though |
|
Thanks for the reply. I'll bid my time for you to get to it. |
|
I also have the same issue :) All previews have been downloaded correctly, but when I subscribe a creator, it only downloads some posts :( I noticed something else, unlocked posts with the largest size are in fact JPEG files (they have FF D8 FF DB header), but in timeline JSON, they are declared as "image/png" :) |
|
Same here. All content from messages were downloaded, but got only a couple preview videos. Must've been from the lowest tier that only gives previews, even though I subbed the the tier above it, like what was mentioned previously. |
|
Hi all, I have never tried that content didn't get downloaded (apart from when it was already downloaded), but I have encountered some content that was marked as preview. I also encountered once where the content was posted both as preview and locked, and in that case the preview gets downloaded first, which was quite confusing. |
|
I'm using 0.3.6 and it works getting subscribed videos both in messages and timeline. |
|
I'm using 0.3.5 and it's not downloading posts posted after 5/10/23. Only all the ones on that date and before that date. |
I'm aware of this and have already locally implemented a fix for this in my developer version, which I'll ship within a couple days in 0.3.6 It's caused by fansly introducing a new media streaming format called .m3u8 and this format is served a little differently, which breaks the older versions of the scraper. Relevant: #80 |
- rewrote major parts of the codebase
- rebranded to Fansly Downloader
- additonal module requirements: json, m3u8, av, time, mimetypes
- from which only: m3u8 & av need to be installed
- option naming_convention is now deprecated & gets removed automatically
- re-named all major variables, to be more descriptive
- now reading value types, properly out of the config.ini file
- config.ini input values are now actually being validated for correctness
- e.g. user-agent is automatically guessed now, if invalid
- update_recent_download has been improved drastically
- now only supports Auto or False
- its enabled by default now
- it will be way quicker, by utilising deduplication
- deduplication is done through 1. media id & 2. hashing now
- hashes are added to filenames initially
- filenames with hashes, will not get hashed again
- epoch timestamps are now converted into local systems timezone
- added alot of comments to codebase, for easier understanding
- CWD is now way more intelligently handled for all modules utilising generate_base_dir()
- added type annotations to function arguments for functions
- adapted fansly scraper for the new m3u8 video format
- required: pyav -> pip install av
- required: m3u8 -> pip intall m3u8
- added DUPLICATE_THRESHOLD; prevents unnecessary computation or requests to fansly
- raises DuplicateCountError if dynamically calculated limit is exceeded
- all "modules" (Timeline, Messages etc.) are now tunneled through sort_download()
- sort_download is a re-written version of the old namesaked function
- it handles allocating downloads way more efficiently
- adds media ids to filenames initially
- if media ids are in filenames; uses them to perform deduplication with media id
- supports all types of file extensions & mimetypes now
- imagehash performance improved, by resizing image to 10% of original resolution b4
- all "modules" will now utilise parse_media_info() to interact with the fansly api
- way more efficent; it doesn't just iterate & try / except blocks everything as b4
- all functions of fansly scraper are separated in modules / sections w/e
- improves maintenance in the future & increases human readability
- re-wrote & improved logic for timeline & messages downloads
- added module for collections (#12)
- fansly downloader will now output more descriptive texts, to the currently taken action
- improved error robustness a little, need to more in the future though
fixed issues:
- fixed a issue with not escaping apostrophes on macOS (#78)
- fixed subscriber content missing (#59)
- all media file extensions are now supported (#75)
- fixed download_mode: single
- fixed key-pair-id missing
- fixed incorrect parsing of post creator
|
The issue should now be resolved with the changes made in the following commit: 8c591cb. I would appreciate it if you could clone the repository again and test this version using Python. It would be helpful if you could provide feedback on any bugs or issues you encounter. Once I receive feedback from users, I plan to release the compiled executable for version 0.4 in a couple of days. Thank you for your support! |
|
Any update on the exe version for 0.4? I'd test it in python to give feedback, but I dont have any clue on how it works :( |
|
Closing, as the 0.4 version is finally released and does target & solve the topics mentioned within this issue. If you're on the previous 0.3.5 version you can comfortably update using the |
There are posts that aren't downloaded, it also says 218 duplicates declined! Those 218 could be mistaken for a duplicate but they aren't those are missing posts photos and I'm pretty sure.
Wouldn't it be good if there was an option in the config "Put duplicates in separate folder", but it still downloads the duplicates?
Either way it's missing posts!
The text was updated successfully, but these errors were encountered: