Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[twitter] add support for nitter.net URLs in pattern #890

Merged
merged 1 commit into from
Jul 13, 2020
Merged

[twitter] add support for nitter.net URLs in pattern #890

merged 1 commit into from
Jul 13, 2020

Conversation

iamleot
Copy link
Contributor

@iamleot iamleot commented Jul 13, 2020

This pull request add support for nitter.net URLs.
Nitter is an alternative front-end to Twitter - free and open-source available
at: https://github.com/zedeus/nitter

nitter.net is the official public instance.

@iamleot
Copy link
Contributor Author

iamleot commented Jul 13, 2020

JFTR, despite the results test fails all corresponding Twitter tests passes both locally and on Travis CI, copypasting corresponding lines regarding the Travis CI one:

[...]
837 test_TwitterBookmarkExtractor_1 (test.test_results.TestExtractorResults) ... ok
838 test_TwitterLikesExtractor_1 (test.test_results.TestExtractorResults) ... ok
839 test_TwitterMediaExtractor_1 (test.test_results.TestExtractorResults) ... ok
840 test_TwitterMediaExtractor_2 (test.test_results.TestExtractorResults) ... ok
841 test_TwitterSearchExtractor_1 (test.test_results.TestExtractorResults) ... ok
842 test_TwitterTimelineExtractor_1 (test.test_results.TestExtractorResults) ... ok
843 test_TwitterTimelineExtractor_2 (test.test_results.TestExtractorResults) ... ok
844 test_TwitterTweetExtractor_1 (test.test_results.TestExtractorResults) ... ok
845 test_TwitterTweetExtractor_10 (test.test_results.TestExtractorResults) ... ok
846 test_TwitterTweetExtractor_2 (test.test_results.TestExtractorResults) ... ok
847 test_TwitterTweetExtractor_3 (test.test_results.TestExtractorResults) ... ok
848 test_TwitterTweetExtractor_4 (test.test_results.TestExtractorResults) ... ok
849 test_TwitterTweetExtractor_5 (test.test_results.TestExtractorResults) ... ok
850 test_TwitterTweetExtractor_6 (test.test_results.TestExtractorResults) ... ok
851 test_TwitterTweetExtractor_7 (test.test_results.TestExtractorResults) ... ok
852 test_TwitterTweetExtractor_8 (test.test_results.TestExtractorResults) ... ok
853 test_TwitterTweetExtractor_9 (test.test_results.TestExtractorResults) ... ok
[...]

So possible failures should be unrelated to this pull request.

Copy link
Owner

@mikf mikf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the first 2 lines of each regex pattern now share the same code, could you add a global BASE_PATTERN value and simplify the regexs like in, for example, DeviantArt and Imgur?

BASE_PATTERN = (
r"(?:https?://)?(?:"
r"(?:www\.)?deviantart\.com/([\w-]+)|"
r"(?!www\.)([\w-]+)\.deviantart\.com)"
)

BASE_PATTERN = r"(?:https?://)?(?:www\.|[im]\.)?imgur\.com"

@iamleot
Copy link
Contributor Author

iamleot commented Jul 13, 2020

Yes! I was unsure to do that given that there was only an extra domain but that's indeed much more possibly easily factorable if there will be more nitter instances or similar.

I will change it in the next couple of minutes and force push to the pull request!

@iamleot
Copy link
Contributor Author

iamleot commented Jul 13, 2020

Thanks @mikf, I have just updated the pull request based on your comment and I will keep an eye on the Travis CI results to spot if there is any possible regression (locally testing them seems fine).

@mikf
Copy link
Owner

mikf commented Jul 13, 2020

I think you'll have to put an r before each string after BASE_PATTERN + … to allow \d etc without flake8 complaining, i.e. BASE_PATTERN + r"/..."

Please note that only URLs are "translated", all requests are still
done always via the Twitter API.
@iamleot
Copy link
Contributor Author

iamleot commented Jul 13, 2020

@mikf Whoops, sorry! I have just added them. Thanks again!

@mikf mikf merged commit 86e5a05 into mikf:master Jul 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants