Untiny t.co links

[progval]

Hi,

t.co links are hosted by Twitter, so we should backup the target link of tweets too. (The same holds true for other URL shorteners, by the way).

Regards,
Valentin

[progval]

This is a workaround, but here is a Python script that does it:

import re
import twitter
import MySQLdb
import requests

DB_USERNAME = 'archivetweets'
DB_PASSWORD = 'REPLACE ME'
DB_NAME = 'archivetweets'

api = twitter.Api(consumer_key='REPLACEME',
                  consumer_secret='REPLACEME',
                  access_token_key='REPLACEME',
                  access_token_secret='REPLACEME')

conn = MySQLdb.connect('localhost', DB_USERNAME, DB_PASSWORD, DB_NAME)

c = conn.cursor()


_tco_link_re = re.compile('https?://t\.co/[a-zA-Z0-9]+')
def expandLinks(tweet):
    def repl(link):
        link = link.group(0)
        data = requests.get('http://untiny.me/api/1.0/extract?url=%s&format=json' % link).json
        try:
            return data['org_url']
        except:
            print repr(data)
            return link
    return _tco_link_re.sub(repl, repr(tweet)).decode('string_escape')

sql = 'SELECT id, tweet FROM amt_tweets WHERE tweet LIKE "%t.co%";'
c.execute(sql)
for (id, tweet) in c.fetchall():
    print '%i %r' % (id, tweet)
    try:
        tweet = expandLinks(tweet)[1:-1]
        print '%i %r' % (id, tweet)
        c2 = conn.cursor()
        c2.execute('UPDATE amt_tweets SET tweet=%s WHERE id=%s', (tweet, id))
        c2.close()
    except:
        pass


c.close()

[seefood]

excellent idea. anything that can be held locally is a good idea.

also, what about local expansion of conversations, media, user avatars? I mean, as a switchable option, not by default...

Going to install this now and test, I hope it plays nice with Unicode (many of my tweets are in Hebrew).

[progval]

There is a problem with retweets, though: if the original message + "RT @user" is longer than 140 characters, the link is cut :/

[seefood]

I didn't study the API, but if clients show the damn link correctly, why shouldn't this app? I saw the chopped URLs and ugly retweets only in RSS feeds. Is this app working on RSS? if so then I can't bother even to install it.

[progval]

yes, it's possible via the API. Either @amwhalen or I have to implement it

[progval]

Actually, we should properly implement retweets, ie. showing the author profile picture, etc... and it implies changing the tables structure, and I don't know how to do it properly without a migration-handling library.