Downloader doesn't "see" other images and videos from posts

[AlexMariacio]

Hi 👋🙂 I have a problem (and not only me as i see from posts on Fansly Downloader from Avnsx) I'm trying to download posts, but only thing it do is downloading only 5-6 images from post (when there are at least 50 images and 2-5 videos) I suppose it's like this because of recent Fansly "update". Preveously i could download entire post with images and videos (in collections too) now i can't do even this.
Hope the solution will be found
For example these 2 screenshots from the post with 13 images and 1 video:

[Animalm4st3r]

Can confirm it stops after 5 items in any given post. Fansly changed something on their end.

[elen07zz]

same

[prof79]

Interesting, my downloads seem to work so far - did you notice the "Follow the creator to be able to scrape media!" message? This indicates that the browser token you are using does not see/have access to the creator in question.

• Log in to Fansly and make sure you're following the creator. For creators that have paid subscriptions you must also be subscribed or only a subset of free media (if any) will be returned/downloaded.
• Make sure your browser's Fansly token matches that in your config.ini/delete it and let FDNG re-configure automatically.

The authorization token may have just expired/reset after 30 days or whatever period.

[ZincStoat]

Still an issue with 0.7.6 (sorry). I don't see any "Follow the creator..." message, and I didn't with previous occurrences of the problem with Avnsx's version. I'm very definitely using the correct credentials.
For a message or post containing multiple images it downloads only 5 before moving to the next message or post. "Amount of media" for each timespan 'cursor' is equal to the number of images + videos downloaded, but not to the number of images + videos present, so it seems like the script is doing the best it can with the information available but the API isn't telling the whole truth. The first message shown below had 11 images of which only 5 were retrieved as you can see. Subsequent video downloads were one video per message.

[KasumiDev]

The accountMedia Array only get's the 6 preview elements for each post (so for single posts it will be 6 elements).
But in the accountMediaBundles each element has an accountMediaIds Array with all IDs that could then be fetched via https://apiv3.fansly.com/api/v1/account/media?ids={accountMediaIds}. Like it's been done for the collections stuff here

fansly-downloader-ng/download/collections.py

Lines 31 to 48 in 968775f

	account_media_ids = [order['accountMediaId'] for order in account_media_orders]
	# Batch size based on API's limits
	batch_size = 150
	# Splitting the list into batches and making separate API calls for each
	for batch in batch_list(account_media_ids, batch_size):
	batched_ids = ','.join(batch)
	post_object_response = config.http_session.get(
	f"https://apiv3.fansly.com/api/v1/account/media?ids={batched_ids}",
	headers=config.http_headers())
	if post_object_response.status_code == 200:
	post_object = post_object_response.json()
	process_download_accessible_media(config, state, post_object['response'])

I hope this info was usefull :)

PS. Even the fansly devs seem to missed the change, the "post info" on the preview no longer contains video durations for posts with over 5 elements. And for locked post is also wrong ^^

[prof79]

I'm trying to figure this out @ZincStoat but sorry, I don't fully get what you're saying. You say the screenshot has 11 images of which 5 were only downloaded. I see a screenshot of a messages download, where 23 are presumably scrapable. The screenshot only shows 11 lines of downloads, what happens after that? There should be 12 more (23 - 11). What is more, you say only 5 of 11 images were downloaded. On the screenshot I see 5 images and 6 videos (= 11) which should have downloaded according to the message. So I'm confused.

Are you suggesting there are "Downloading ..." lines that effectively do not download anything? Are there less "Downloading ..." lines than "scrapable :" items? If only m3u8 videos made problems a recent code change might explain that.

What is more, without having a concrete creator to analyze things are difficult. Sadly, official Fansly account has not many media but I tried to download it anyway - it say 8 pictures, which is true; downloads 5 because 2 of them are identical to another and 1 is locked behind a subscription I do not have.

The difficult part is to tell how many double posts (identical media) a creator makes and the differentiation between preview/promo and regular media. There might also be different subscription tiers and paywalled items aside subscriptions.

I'll try my last "legit" creator which doesn't do paywall tricks and I'm fully subscribed to and compare figures. But this looks more than 5 elements per timeline. I'm also missing 11 items but what the timelinenew API delivered as "scrapable" was downloaded (273 media items + 3 message items = 276). Difficult.

[Animalm4st3r]

its not 5 elements per timeline. Its 5 elements per post If a post has more then 5 elements it only downloads the first 5 and then moves on to the next post. Its easier to see when you try downloading a post with more then 5 elements in single mode.

[ZincStoat]

its not 5 elements per timeline. Its 5 elements per post If a post has more then 5 elements it only downloads the first 5 and then moves on to the next post. Its easier to see when you try downloading a post with more then 5 elements in single mode.

Exactly this, simple as that. To observe anyway, I don't know how simple it will be to fix.

[prof79]

Thanks for the clarification and sorry, I overlooked that single post downloading was the original topic, which is also a bit of a different beast in itself.

I have a HAR file of a manual browsing session with my sample creator, maybe I can spot something where numbers don't add up.

I also have to try single post mode because TBH I never use it personally so it's the least tested code path.

Problem might also be that the creators I checked do usually post with 4 media items max.

I do however, have some private things on my plate also so this may take some time.

[ZincStoat]

I also have to try single post mode because TBH I never use it personally so it's the least tested code path.

For what it's worth I don't think I've ever used single post mode, so it's not specific to that.

[prof79]

heres a snapshot of a download attempt of a post that contains 46 Videos ... It skips the first 5 cause they are already downloaded and then finishes up ignoring the rest. I can watch the videos just normal in the browser.

As a remark - since the output of "items in posts and scrapable items" are 5 already in the informational header, this shows there might be something conceptually wrong in the usage of the reverse-engineered API, as if only media items you see while scrolling the whole timeline are seen/get downloaded by the scraper! So what the program got and downloaded is 5 and thus major part of processing and downloading itself is OK but obtaining timeline/post info seems flawed, or rather, incomplete.

[prof79]

@Animalm4st3r I deleted the post with your helpful screenshot because you, I guess unintentionally, spilled the creator name ;-)

Quoted text in my comment above, this is the fixed image.

[Animalm4st3r]

woops XD yes, i also tried it with older posts that had more then 5 elements, that i know worked previously and download in normal mode, they behave the sameway skip after 5 elements in an individual post, that also happens with the original from AVNSX, so i assume it happens because Fansly changed something on their end

[prof79]

I figured it out (I hope), easier than I thought - but less easy to properly re-structure the code and avoid duplication.

The (probably new) "timeline" structure has "posts", "account media" and "account media bundles". Avnsx just grabbed the "account media". Now each "post" has "attachments" which point to a specific "account media bundle" - which can contain more media! So as I suspected the "account media" probably only contains images you also see by scrolling through. Same is true for single posts, the deviance is even more obvious there (I could spot it instantly) - structure is the same as for "timelines".

[KasumiDev]

@prof79 Have you seen my PR #6 ? For single posts it will just load all mediaBundles that are available.

[ZincStoat]

@prof79 Have you seen my PR #6 ? For single posts it will just load all mediaBundles that are available.

I grabbed a copy of your single.py and can confirm it retrieved all missing images for an affected post. If we can get that fix into the general download code that would be great.

[ZincStoat]

I figured it out (I hope), easier than I thought - but less easy to properly re-structure the code and avoid duplication.

Ah, avoiding duplication, there's the trick. If you don't plan on having to hunt down everywhere you've fixed this when they inevitable make another update and break it again, a central "download everything for this one post" function would be the way to go.
I might have a stab at it myself just for laughs, but Python's not my native dev language so don't let me stop you going for it.

[prof79]

@KasumiDev thanks a lot for your efforts and kudos but I might have to decline, as I already have my own universal solution.

[prof79]

I figured it out (I hope), easier than I thought - but less easy to properly re-structure the code and avoid duplication.

Ah, avoiding duplication, there's the trick. If you don't plan on having to hunt down everywhere you've fixed this when they inevitable make another update and break it again, a central "download everything for this one post" function would be the way to go. I might have a stab at it myself just for laughs, but Python's not my native dev language so don't let me stop you going for it.

No need, that's the power brought by refactoring into modules/single pieces - a new API call was needed, as @KasumiDev also discovered, but summarizing media infos is identical for all messages, timeline and post objects.

[KasumiDev]

@KasumiDev thanks a lot for your efforts and kudos but I might have to decline, as I already have my own universal solution.

No problem. I just wanted to share my findings and a possible solution.
If the timeline download now also works again I'm more then happy 👍

[ZincStoat]

@KasumiDev thanks a lot for your efforts and kudos but I might have to decline, as I already have my own universal solution.

No problem. I just wanted to share my findings and a possible solution. If the timeline download now also works again I'm more then happy 👍

Running my customary 'Normal' mode download now and watching all the missing media get filled in. Thanks both.

[prof79]

Please check development release (0.7.7) and give it a good try. If nothing else comes up I'll make a proper release of it later (maybe 0.8.0).

This fixes messages, timelines and single post downloading. (I have no access to/idea about collections, though)

Elegantly, the media IDs of accountMedia and accountMediaBundles are consolidated into a unique list of media IDs which then gets processed and downloaded. Thus independent of how the Fansly API and structures are supposed to work and might be empty, you always get the full deal.

https://github.com/prof79/fansly-downloader-ng/releases/tag/ondemand

[prof79]

(PS @KasumiDev: really missed your post, likely some browser refresh issue as I just suspend my PC very often - helpful indeed! Though stepping through it myself helped grasping the dependencies)

[ZincStoat]

That's much better

One thing I would note though, when I then tried a Single post mode run on another creator's content, which was already divided automatically into Messages and Timeline, it threw up the message

and created a new Pictures directory alongside Messages and Timeline, even though it only actually downloaded the images missing from Timeline\Pictures. Should I create a new Issue for that, since it's hardly a showstopper, just a mild irritation?

[prof79]

One thing I would note though, when I then tried a Single post mode run on another creator's message content, which was already divided automatically into Messages and Timeline, it threw up the message and created a new Pictures directory alongside Messages and Timeline, even though it only actually downloaded the images missing from Timeline\Pictures. Should I create a new Issue for that, since it's hardly a showstopper, just a mild irritation?

@ZincStoat Oh yes, please do - easier to track/handle. Those were some original design decisions by Avnsx, need to think that through/cross-check with single post differences.

[prof79]

Thanks all, fixed by binary release 53bd7e1 / v0.7.10.