Skip to content

Commit

Permalink
[twitter] fall back to unfiltered search (#2766)
Browse files Browse the repository at this point in the history
  • Loading branch information
@mikf
mikf committed Jul 27, 2022
1 parent 0aa8345 commit 8d0801a
Showing 1 changed file with 11 additions and 5 deletions.
16 changes: 11 additions & 5 deletions gallery_dl/extractor/twitter.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -454,7 +454,6 @@ def tweets(self):
tweet = None
for tweet in self._select_tweet_source()(self.user):
yield tweet

if tweet is None:
return

Expand All @@ -463,12 +462,19 @@ def tweets(self):
self._user["name"], tweet["rest_id"])
if self.retweets:
query += " include:retweets include:nativeretweets"

if not self.textonly:
query += (" (filter:images OR"
" filter:native_video OR"
" card_name:animated_gif)")
# try to search for media-only tweets
tweet = None
for tweet in self.api.search_adaptive(query + (
" (filter:images OR"
" filter:native_video OR"
" card_name:animated_gif)")):
yield tweet
if tweet is not None:
return

# yield search results starting from last tweet id
# yield unfiltered search results
yield from self.api.search_adaptive(query)

def _select_tweet_source(self):
Expand Down

5 comments on commit 8d0801a

@God-damnit-all
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could lead to a ton of API calls in some circumstances, could you please add a configuration option for Twitter to never use searches?

@mikf
Copy link
Owner Author

@mikf mikf commented on 8d0801a Jul 31, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main profile URLs (https://twitter.com/USER) use a search since commit 915dba8 and will continue doing so.

The previous functionality for these URLs was moved to https://twitter.com/USER/tweets, which does not use a search, as is the case for https://twitter.com/USER/media and https://twitter.com/USER/with_replies.

If you do not want gallery-dl to do a search, rename all your https://twitter.com/USER URLs to https://twitter.com/USER/tweets, or just don't use main profile URLs.

This commit also only adds a fallback. The unfiltered search does not get used if the filtered search returns at least one Tweet.

@God-damnit-all
Copy link
Contributor

@God-damnit-all God-damnit-all commented on 8d0801a Aug 1, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main profile URLs (https://twitter.com/USER) use a search since commit 915dba8 and will continue doing so.

The previous functionality for these URLs was moved to https://twitter.com/USER/tweets, which does not use a search, as is the case for https://twitter.com/USER/media and https://twitter.com/USER/with_replies.

If you do not want gallery-dl to do a search, rename all your https://twitter.com/USER URLs to https://twitter.com/USER/tweets, or just don't use main profile URLs.

This commit also only adds a fallback. The unfiltered search does not get used if the filtered search returns at least one Tweet.

I think I understand. /media and /with_replies can potentially have tweets the other one does not though, correct?

Ideally, the json objects retrieved from /media and /with_replies would be merged (without duplicates). I was hoping the recent changes did that, but I guess I was mistaken.

@afterdelight
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://twitter.com/USER)

does https://twitter.com/USER include retweets?
what links i need to paste to gallery-dl to get all medias of user profile after the update? if possible including retweets

@Hrxn
Copy link
Contributor

@Hrxn Hrxn commented on 8d0801a Sep 17, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, with the option "retweets": true (or "retweets": "original")

Keep "text-tweets" at the default (false) to use the media timeline with the new version.

And then simply

gallery-dl "https://twitter.com/USER"

That's all you need, but those pesky API limitations are still in place.
You can try to fetch more via providing an tweed ID for the search, like this:

gallery-dl "https://twitter.com/search?q=from:USER+max_id:ID_HERE"

Maybe additionally this one:

gallery-dl "https://twitter.com/USER/with_replies"

Please sign in to comment.