Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[twitter] Fix stop before real end #573

Merged
merged 2 commits into from
Jan 14, 2020
Merged

[twitter] Fix stop before real end #573

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
42fca6d
[twitter] Fix stop before real end
alice945 Jan 11, 2020
98989c7
[twitter] Fix stop before real end (update)
alice945 Jan 13, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion gallery_dl/extractor/twitter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ def _tweets_from_api(self, url, max_position=None):
data["items_html"], '<div class="tweet ', '\n</li>'):
yield tweet

if not data["has_more_items"]:
if not data["has_more_items"] and data["min_position"] == None:
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data is not guaranteed to contain a min_position key (#544 (comment)), so either check with data.get("min_position") is None or ("min_position" in data and data["min_position"] is None), depending on whether there is a difference between min_position is set and None and min_position is not set.

(is is also preferred over == when testing for None)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know that, whoops. The fix ended up being a bit more complicated then I thought. There is 6 possible responses from Twitter:

gallery-dl should continue:
{"has_more_items":True} (This happens on the first page)
{"has_more_items":True, "min_position":"123..."}
{"has_more_items":False, "min_position":"123..."}

gallery-dl should stop:
{"has_more_items":False}
{"has_more_items":False, "min_position":None}
{"has_more_items":True, "min_position":None}

Basically we want to stop whenever min_position is None or doesn't exist, except when there is more items and min_position doesn't exist.

return

if "min_position" in data:
Expand Down