Copyright 2010 Dave Challis <[email protected]>

1 Summary
=========

archive_twitter_search.py is a short script which performs a twitter search
using twitter's search API, and saves each tweet (including all metadata) to a
SQLite database.

It's designed to require almost no configuration (apart from specifying a
search term!).

It *only* works using twitter's search API, i.e. it will not get/archive an
individual's statuses (getting these uses twitter's REST API, which is
tightly rate limited).


2 Getting Started
=================

2.1 Archiving tweets from twitter search terms
==============================================

1. Edit 'archive_twitter_search.py' and change the 'query' variable
2. Run the script!

Assuming all goes well, you should see a message about the number of tweets
added, and a SQLite database file named 'tweets.db' will have been created.

Ideally you should add a cron job to run the script every N minutes (where N
depends on how often new tweets for your search term(s) are likely to turn
up).

Only tweets added since your last search will be searched for, so queries
after the inital one should be a lot faster.


2.2 Viewing saved tweets
========================

1. Open the database in SQLite:
    sqlite3 tweets.db

2. View the database schema by entering:
    .schema

3. Get any/all tweets using an SQL query, e.g.:
    SELECT * FROM tweets LIMIT 5;

Pretty much every language has SQLite bindings these days, so you should be
able to open and access the archived tweets from there.


3 System Requirements
=====================

Software:
* python 2.6+ (hasn't been tested with earlier releases, but might work)
* sqlite3

Non-standard Python libraries:
* dateutil
* sqlalchemy


4 About twitter's search API
============================

Full details on the twitter search API at:
    http://apiwiki.twitter.com/Twitter-API-Documentation

Twitter's search API allows the last 1500 results for a search term to be
queried for.  In order to capture *all* tweets for a given search term, you'll
need to experiment with running it often enough that you get <1500 new tweets
each time (this may not be possible for trending items!).

While the search API is rate limited, limits for the search API are apparently
much more generous than the 150 requests per hour imposed on the REST API.

Rate limiting details:
    http://apiwiki.twitter.com/Rate-limiting