curls & retryonlimit issue

[selaseladon]

[Edit : potentially the same error than in issue#179 https://github.com/mkearney/rtweet/issues/179 ]

Hi,

I was trying to get the last 500 tweets from a list of users (2000 users) to do some text data mining. I imediatly had an issue often met with curls when operated on dataframes (cf. topics on forums dedicated to the same error as mine).
Here is what I tried :

ego_flws_scr <- as.character(ego_flw_data_users$screen_name) # probably useless line I code just to be sure that the type of data is correct
ego_flwrs_TL <- get_timelines(ego_flws_scr, n=500, retryonratelimit = TRUE)

Here is what I've got :

ego_flwrs_TL <- get_timelines(ego_flws_scr, n=500, retryonratelimit = TRUE)
Error in `$<-.data.frame`(`tmp`, "app", value = "rtweetttttttt") :
le tableau de remplacement a 1 lignes, le tableau remplacé en a 0
De plus : There were 16 warnings (use warnings() to see them)
warnings()
Messages d'avis :
1: /1.1/statuses/user_timeline.jsonNot authorized.
(...)
15: /1.1/statuses/user_timeline.jsonNot authorized.
16: Rate limit exceeded - 88`

When operated on a smaller set of users – like I run it on x, x being

x <- ego_flws_scr[1:50]

–, and when n is not to high (n=200), it works perfectly.
It once worked a bit farer (22 warnings instead of the usual 16), but the last warning was still 'rate limit exceeded'.

It looks like it fails when the operation runs again after the warning 'rate limit exceeded'.

Error in `$<-.data.frame`(`tmp`, "app", value = "rtweetttttttt") : le tableau de remplacement a 1 lignes, le tableau remplacé en a 0"
That may explain the values 1 and 0 in the error ? Or there is an issue with some interaction btw curls ? But it does not wait 15mn to crash.

Advices on forums do not fit the situation ; I don't have any idea about this, is that an issue with the function itself or if I can solve this with a simple bit of code.

I hope this is the right place to report something like that ; sorry if it's not the case.
(And thank you for this useful package ;) it is very useful for a social sciences student like me! )

[JustJacqui]

I'm having a similar issue.

I get

Warning: 89 - Invalid or expired token.
Error in $<-.data.frame(*tmp*, "app", value = "SelfdrivingCars") :
replacement has 1 row, data has 0

This is my code, the sources are just some custom code my tutor gave us, which worked fine so I don't think that that code is the issue, since I'm not the only one getting this error.

library("bit64")
library("httr")
library("igraph")
library("magrittr")
library("RJSONIO")
library("rtweet")
library("SnowballC")
library("SocialMediaLab")
library("stringr")
library("tm")
library("twitteR")

source('multiple_plots.R')
source('utils.R')
source('comm.R')
source('oauth.R')
source('df.R')

Twitter Auth

key <- " "
secret <- " "
my_access_token <- " "
my_access_token_secret <- " "

setup_twitter_oauth(key, secret, my_access_token, my_access_token_secret)

uber_tweets_01.05 <- search_tweets("selfdriving", n = 5000, type = "popular", include_rts = FALSE)

With my keys and the like of course. I can't find anything that fits the issue. It was fine last week, I only made a second app to try and fix this issue, but it seems my old app is still being called or something?

I haven't even created a data.frame yet, this is legitimately all of my code.

It's for an assessment as well so if anyone know anything it would be much appreciated!

[JustJacqui]

I found out that my issue was with my Twitter App.

I had to edit it so that I disabled Callback URL locking.

Solved the problem right away.

[selaseladon]

Oh good to know if I'm to retry, thank you for your message ! Le mar. 1 mai 2018 à 07:37, JustJacqui <[email protected]> a écrit :

…

I found out that my issue was with my Twitter App. I had to edit it so that I disabled Callback URL locking. [image: whattochange] <https://user-images.githubusercontent.com/18604066/39462518-76defc9e-4d55-11e8-8080-37013119dab9.png> Solved the problem right away. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <https://github.com/mkearney/rtweet/issues/182#issuecomment-385603473>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ai8eCDRg_64W7U9AuZ9p_qJzQeIFpyMrks5tt_SEgaJpZM4SK9jo> .

[holnburger]

I got the same error as @selaseladon. I'm trying to run get_timelines(retryonratelimit = TRUE) and get the following error message:

> mdb_tweets <- get_timelines(mdb_user$user_id, n = 100, retryonratelimit = TRUE)
Warnung: Rate limit exceeded - 88
Fehler in `$<-.data.frame`(`*tmp*`, "app", value = "TwitterData_UHH") : 
  replacement has 1 row, data has 0

I'm currently running it with 509 twitter users. It fails even after setting up a new app, also trying to avoid the problem @JustJacqui had. It also doesn't seem like it's the same error as in issue#179 because it's a "Rate limit exceeded - 88" error code, not 89.

[selaseladon]

Hi, my error was solved when I reinstalled the package after it was updated just like in issue #179, that seemed to be a good explanation for what was happening when the rate limit went exceeded. I still chose to split my 2000 users in 10 "a, b, c..." groups to avoid losing the whole data if it crashed in spite of the update (I can copy it there if you need...).

I change the title of the topic right now since it does not fix the issue for you ;)

[holnburger]

I got it working by using a loop to avoid the rate limit (described by @mkearney, used the code here: https://github.com/mkearney/rtweet/issues/136#issuecomment-344603485)

Apparently retryonratelimit isn't implemented properly in get_timelines because I always run into problems with more then 52 user-timelines (and I got 509 users to scrape tweets from).