02_Twitter_scraping_snsscraper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Twitter Scraping [snscrape]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#pip install git+https://github.com/JustAnotherArchivist/snscrape.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>snscrape</b> is a scraper for social networking services (SNS). It scrapes things like user profiles, hashtags, or searches and returns the discovered items, e.g. the relevant posts. We used this service to analyze the last 4 months tweets about bitcoin and dogecoin.<br>\n",
    "Compared to the Twitter api with snsscraper, we were able to analyse many more tweets and over a much longer period of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## QUERY BY USERNAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setting wanted username and number of tweets\n",
    "tweet_count = 10000\n",
    "username = \"ElonMusk\"\n",
    "\n",
    "# Using OS library to call CLI commands in Python\n",
    "os.system(\"snscrape --jsonl --max-results {} twitter-search 'from:{}'> user-tweets.json\".format(tweet_count, username))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_type</th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>...</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>inReplyToTweetId</th>\n",
       "      <th>inReplyToUser</th>\n",
       "      <th>mentionedUsers</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>place</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397626694...</td>\n",
       "      <td>2021-05-26 18:52:25+00:00</td>\n",
       "      <td>@BTC_Archive 🤣🤣</td>\n",
       "      <td>@BTC_Archive 🤣🤣</td>\n",
       "      <td>1397626694609956864</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>853</td>\n",
       "      <td>212</td>\n",
       "      <td>4653</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397627e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397625923...</td>\n",
       "      <td>2021-05-26 18:49:21+00:00</td>\n",
       "      <td>If only they had spaceships, they’d still be a...</td>\n",
       "      <td>If only they had spaceships, they’d still be a...</td>\n",
       "      <td>1397625923256549376</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>3899</td>\n",
       "      <td>3174</td>\n",
       "      <td>66283</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397625e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397625459...</td>\n",
       "      <td>2021-05-26 18:47:30+00:00</td>\n",
       "      <td>Deep thots https://t.co/J5mJff7IIL</td>\n",
       "      <td>Deep thots https://t.co/J5mJff7IIL</td>\n",
       "      <td>1397625459437826049</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>9493</td>\n",
       "      <td>14862</td>\n",
       "      <td>146358</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397623469...</td>\n",
       "      <td>2021-05-26 18:39:36+00:00</td>\n",
       "      <td>@WholeMarsBlog Not sure who wrote this, but it...</td>\n",
       "      <td>@WholeMarsBlog Not sure who wrote this, but it...</td>\n",
       "      <td>1397623469739692036</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>403</td>\n",
       "      <td>526</td>\n",
       "      <td>5046</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397479e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397456432...</td>\n",
       "      <td>2021-05-26 07:35:51+00:00</td>\n",
       "      <td>@Caspar_Stanley @Erdayastronaut @SpaceX @Super...</td>\n",
       "      <td>@Caspar_Stanley @Erdayastronaut @SpaceX @Super...</td>\n",
       "      <td>1397456432681811969</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>374</td>\n",
       "      <td>150</td>\n",
       "      <td>3115</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397444e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            _type  \\\n",
       "0  snscrape.modules.twitter.Tweet   \n",
       "1  snscrape.modules.twitter.Tweet   \n",
       "2  snscrape.modules.twitter.Tweet   \n",
       "3  snscrape.modules.twitter.Tweet   \n",
       "4  snscrape.modules.twitter.Tweet   \n",
       "\n",
       "                                                 url  \\\n",
       "0  https://twitter.com/elonmusk/status/1397626694...   \n",
       "1  https://twitter.com/elonmusk/status/1397625923...   \n",
       "2  https://twitter.com/elonmusk/status/1397625459...   \n",
       "3  https://twitter.com/elonmusk/status/1397623469...   \n",
       "4  https://twitter.com/elonmusk/status/1397456432...   \n",
       "\n",
       "                       date  \\\n",
       "0 2021-05-26 18:52:25+00:00   \n",
       "1 2021-05-26 18:49:21+00:00   \n",
       "2 2021-05-26 18:47:30+00:00   \n",
       "3 2021-05-26 18:39:36+00:00   \n",
       "4 2021-05-26 07:35:51+00:00   \n",
       "\n",
       "                                             content  \\\n",
       "0                                    @BTC_Archive 🤣🤣   \n",
       "1  If only they had spaceships, they’d still be a...   \n",
       "2                 Deep thots https://t.co/J5mJff7IIL   \n",
       "3  @WholeMarsBlog Not sure who wrote this, but it...   \n",
       "4  @Caspar_Stanley @Erdayastronaut @SpaceX @Super...   \n",
       "\n",
       "                                     renderedContent                   id  \\\n",
       "0                                    @BTC_Archive 🤣🤣  1397626694609956864   \n",
       "1  If only they had spaceships, they’d still be a...  1397625923256549376   \n",
       "2                 Deep thots https://t.co/J5mJff7IIL  1397625459437826049   \n",
       "3  @WholeMarsBlog Not sure who wrote this, but it...  1397623469739692036   \n",
       "4  @Caspar_Stanley @Erdayastronaut @SpaceX @Super...  1397456432681811969   \n",
       "\n",
       "                                                user  replyCount  \\\n",
       "0  {'_type': 'snscrape.modules.twitter.User', 'us...         853   \n",
       "1  {'_type': 'snscrape.modules.twitter.User', 'us...        3899   \n",
       "2  {'_type': 'snscrape.modules.twitter.User', 'us...        9493   \n",
       "3  {'_type': 'snscrape.modules.twitter.User', 'us...         403   \n",
       "4  {'_type': 'snscrape.modules.twitter.User', 'us...         374   \n",
       "\n",
       "   retweetCount  likeCount  ...  \\\n",
       "0           212       4653  ...   \n",
       "1          3174      66283  ...   \n",
       "2         14862     146358  ...   \n",
       "3           526       5046  ...   \n",
       "4           150       3115  ...   \n",
       "\n",
       "                                               media  retweetedTweet  \\\n",
       "0                                               None             NaN   \n",
       "1                                               None             NaN   \n",
       "2  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "3  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "4                                               None             NaN   \n",
       "\n",
       "  quotedTweet inReplyToTweetId  \\\n",
       "0        None     1.397627e+18   \n",
       "1        None     1.397625e+18   \n",
       "2        None              NaN   \n",
       "3        None     1.397479e+18   \n",
       "4        None     1.397444e+18   \n",
       "\n",
       "                                       inReplyToUser  \\\n",
       "0  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "1  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "2                                               None   \n",
       "3  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "4  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "\n",
       "                                      mentionedUsers coordinates place  \\\n",
       "0  [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "1                                               None         NaN   NaN   \n",
       "2                                               None         NaN   NaN   \n",
       "3  [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "4  [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "\n",
       "  hashtags  cashtags  \n",
       "0     None       NaN  \n",
       "1     None       NaN  \n",
       "2     None       NaN  \n",
       "3     None       NaN  \n",
       "4     None       NaN  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reads the json generated from the CLI command above and creates a pandas dataframe\n",
    "tweets_df1 = pd.read_json('user-tweets.json', lines=True)\n",
    "\n",
    "# Displays first 5 entries from dataframe\n",
    "tweets_df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_type</th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>...</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>inReplyToTweetId</th>\n",
       "      <th>inReplyToUser</th>\n",
       "      <th>mentionedUsers</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>place</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397626694...</td>\n",
       "      <td>2021-05-26 18:52:25+00:00</td>\n",
       "      <td>@BTC_Archive 🤣🤣</td>\n",
       "      <td>@BTC_Archive 🤣🤣</td>\n",
       "      <td>1397626694609956864</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>853</td>\n",
       "      <td>212</td>\n",
       "      <td>4653</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397627e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397625923...</td>\n",
       "      <td>2021-05-26 18:49:21+00:00</td>\n",
       "      <td>If only they had spaceships, they’d still be a...</td>\n",
       "      <td>If only they had spaceships, they’d still be a...</td>\n",
       "      <td>1397625923256549376</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>3899</td>\n",
       "      <td>3174</td>\n",
       "      <td>66283</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397625e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397625459...</td>\n",
       "      <td>2021-05-26 18:47:30+00:00</td>\n",
       "      <td>Deep thots https://t.co/J5mJff7IIL</td>\n",
       "      <td>Deep thots https://t.co/J5mJff7IIL</td>\n",
       "      <td>1397625459437826049</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>9493</td>\n",
       "      <td>14862</td>\n",
       "      <td>146358</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397623469...</td>\n",
       "      <td>2021-05-26 18:39:36+00:00</td>\n",
       "      <td>@WholeMarsBlog Not sure who wrote this, but it...</td>\n",
       "      <td>@WholeMarsBlog Not sure who wrote this, but it...</td>\n",
       "      <td>1397623469739692036</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>403</td>\n",
       "      <td>526</td>\n",
       "      <td>5046</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397479e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/1397456432...</td>\n",
       "      <td>2021-05-26 07:35:51+00:00</td>\n",
       "      <td>@Caspar_Stanley @Erdayastronaut @SpaceX @Super...</td>\n",
       "      <td>@Caspar_Stanley @Erdayastronaut @SpaceX @Super...</td>\n",
       "      <td>1397456432681811969</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>374</td>\n",
       "      <td>150</td>\n",
       "      <td>3115</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.397444e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9995</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/8972345269...</td>\n",
       "      <td>2017-08-14 23:12:35+00:00</td>\n",
       "      <td>@Schmidtlaxer I want one too!</td>\n",
       "      <td>@Schmidtlaxer I want one too!</td>\n",
       "      <td>897234526945988608</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>52</td>\n",
       "      <td>27</td>\n",
       "      <td>752</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9996</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/8972327576...</td>\n",
       "      <td>2017-08-14 23:05:33+00:00</td>\n",
       "      <td>Tesla Powerwalls can support houses of any siz...</td>\n",
       "      <td>Tesla Powerwalls can support houses of any siz...</td>\n",
       "      <td>897232757629173760</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>1085</td>\n",
       "      <td>4096</td>\n",
       "      <td>20942</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.Tweet', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9997</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/8972230016...</td>\n",
       "      <td>2017-08-14 22:26:47+00:00</td>\n",
       "      <td>@neddesmond @SpaceX The danger is def high for...</td>\n",
       "      <td>@neddesmond @SpaceX The danger is def high for...</td>\n",
       "      <td>897223001631899648</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>23</td>\n",
       "      <td>32</td>\n",
       "      <td>358</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>8.972211e+17</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9998</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/8972211087...</td>\n",
       "      <td>2017-08-14 22:19:16+00:00</td>\n",
       "      <td>@neddesmond @SpaceX We certainly could, but, w...</td>\n",
       "      <td>@neddesmond @SpaceX We certainly could, but, w...</td>\n",
       "      <td>897221108704751616</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>59</td>\n",
       "      <td>161</td>\n",
       "      <td>2575</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>8.972096e+17</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9999</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/elonmusk/status/8971421647...</td>\n",
       "      <td>2017-08-14 17:05:34+00:00</td>\n",
       "      <td>12th mission of our Dragon robotic space freig...</td>\n",
       "      <td>12th mission of our Dragon robotic space freig...</td>\n",
       "      <td>897142164731199488</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>519</td>\n",
       "      <td>1952</td>\n",
       "      <td>13370</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.Tweet', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10000 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                               _type  \\\n",
       "0     snscrape.modules.twitter.Tweet   \n",
       "1     snscrape.modules.twitter.Tweet   \n",
       "2     snscrape.modules.twitter.Tweet   \n",
       "3     snscrape.modules.twitter.Tweet   \n",
       "4     snscrape.modules.twitter.Tweet   \n",
       "...                              ...   \n",
       "9995  snscrape.modules.twitter.Tweet   \n",
       "9996  snscrape.modules.twitter.Tweet   \n",
       "9997  snscrape.modules.twitter.Tweet   \n",
       "9998  snscrape.modules.twitter.Tweet   \n",
       "9999  snscrape.modules.twitter.Tweet   \n",
       "\n",
       "                                                    url  \\\n",
       "0     https://twitter.com/elonmusk/status/1397626694...   \n",
       "1     https://twitter.com/elonmusk/status/1397625923...   \n",
       "2     https://twitter.com/elonmusk/status/1397625459...   \n",
       "3     https://twitter.com/elonmusk/status/1397623469...   \n",
       "4     https://twitter.com/elonmusk/status/1397456432...   \n",
       "...                                                 ...   \n",
       "9995  https://twitter.com/elonmusk/status/8972345269...   \n",
       "9996  https://twitter.com/elonmusk/status/8972327576...   \n",
       "9997  https://twitter.com/elonmusk/status/8972230016...   \n",
       "9998  https://twitter.com/elonmusk/status/8972211087...   \n",
       "9999  https://twitter.com/elonmusk/status/8971421647...   \n",
       "\n",
       "                          date  \\\n",
       "0    2021-05-26 18:52:25+00:00   \n",
       "1    2021-05-26 18:49:21+00:00   \n",
       "2    2021-05-26 18:47:30+00:00   \n",
       "3    2021-05-26 18:39:36+00:00   \n",
       "4    2021-05-26 07:35:51+00:00   \n",
       "...                        ...   \n",
       "9995 2017-08-14 23:12:35+00:00   \n",
       "9996 2017-08-14 23:05:33+00:00   \n",
       "9997 2017-08-14 22:26:47+00:00   \n",
       "9998 2017-08-14 22:19:16+00:00   \n",
       "9999 2017-08-14 17:05:34+00:00   \n",
       "\n",
       "                                                content  \\\n",
       "0                                       @BTC_Archive 🤣🤣   \n",
       "1     If only they had spaceships, they’d still be a...   \n",
       "2                    Deep thots https://t.co/J5mJff7IIL   \n",
       "3     @WholeMarsBlog Not sure who wrote this, but it...   \n",
       "4     @Caspar_Stanley @Erdayastronaut @SpaceX @Super...   \n",
       "...                                                 ...   \n",
       "9995                      @Schmidtlaxer I want one too!   \n",
       "9996  Tesla Powerwalls can support houses of any siz...   \n",
       "9997  @neddesmond @SpaceX The danger is def high for...   \n",
       "9998  @neddesmond @SpaceX We certainly could, but, w...   \n",
       "9999  12th mission of our Dragon robotic space freig...   \n",
       "\n",
       "                                        renderedContent                   id  \\\n",
       "0                                       @BTC_Archive 🤣🤣  1397626694609956864   \n",
       "1     If only they had spaceships, they’d still be a...  1397625923256549376   \n",
       "2                    Deep thots https://t.co/J5mJff7IIL  1397625459437826049   \n",
       "3     @WholeMarsBlog Not sure who wrote this, but it...  1397623469739692036   \n",
       "4     @Caspar_Stanley @Erdayastronaut @SpaceX @Super...  1397456432681811969   \n",
       "...                                                 ...                  ...   \n",
       "9995                      @Schmidtlaxer I want one too!   897234526945988608   \n",
       "9996  Tesla Powerwalls can support houses of any siz...   897232757629173760   \n",
       "9997  @neddesmond @SpaceX The danger is def high for...   897223001631899648   \n",
       "9998  @neddesmond @SpaceX We certainly could, but, w...   897221108704751616   \n",
       "9999  12th mission of our Dragon robotic space freig...   897142164731199488   \n",
       "\n",
       "                                                   user  replyCount  \\\n",
       "0     {'_type': 'snscrape.modules.twitter.User', 'us...         853   \n",
       "1     {'_type': 'snscrape.modules.twitter.User', 'us...        3899   \n",
       "2     {'_type': 'snscrape.modules.twitter.User', 'us...        9493   \n",
       "3     {'_type': 'snscrape.modules.twitter.User', 'us...         403   \n",
       "4     {'_type': 'snscrape.modules.twitter.User', 'us...         374   \n",
       "...                                                 ...         ...   \n",
       "9995  {'_type': 'snscrape.modules.twitter.User', 'us...          52   \n",
       "9996  {'_type': 'snscrape.modules.twitter.User', 'us...        1085   \n",
       "9997  {'_type': 'snscrape.modules.twitter.User', 'us...          23   \n",
       "9998  {'_type': 'snscrape.modules.twitter.User', 'us...          59   \n",
       "9999  {'_type': 'snscrape.modules.twitter.User', 'us...         519   \n",
       "\n",
       "      retweetCount  likeCount  ...  \\\n",
       "0              212       4653  ...   \n",
       "1             3174      66283  ...   \n",
       "2            14862     146358  ...   \n",
       "3              526       5046  ...   \n",
       "4              150       3115  ...   \n",
       "...            ...        ...  ...   \n",
       "9995            27        752  ...   \n",
       "9996          4096      20942  ...   \n",
       "9997            32        358  ...   \n",
       "9998           161       2575  ...   \n",
       "9999          1952      13370  ...   \n",
       "\n",
       "                                                  media  retweetedTweet  \\\n",
       "0                                                  None             NaN   \n",
       "1                                                  None             NaN   \n",
       "2     [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "3     [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "4                                                  None             NaN   \n",
       "...                                                 ...             ...   \n",
       "9995                                               None             NaN   \n",
       "9996                                               None             NaN   \n",
       "9997                                               None             NaN   \n",
       "9998                                               None             NaN   \n",
       "9999                                               None             NaN   \n",
       "\n",
       "                                            quotedTweet inReplyToTweetId  \\\n",
       "0                                                  None     1.397627e+18   \n",
       "1                                                  None     1.397625e+18   \n",
       "2                                                  None              NaN   \n",
       "3                                                  None     1.397479e+18   \n",
       "4                                                  None     1.397444e+18   \n",
       "...                                                 ...              ...   \n",
       "9995                                               None              NaN   \n",
       "9996  {'_type': 'snscrape.modules.twitter.Tweet', 'u...              NaN   \n",
       "9997                                               None     8.972211e+17   \n",
       "9998                                               None     8.972096e+17   \n",
       "9999  {'_type': 'snscrape.modules.twitter.Tweet', 'u...              NaN   \n",
       "\n",
       "                                          inReplyToUser  \\\n",
       "0     {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "1     {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "2                                                  None   \n",
       "3     {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "4     {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "...                                                 ...   \n",
       "9995                                               None   \n",
       "9996                                               None   \n",
       "9997  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "9998  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "9999                                               None   \n",
       "\n",
       "                                         mentionedUsers coordinates place  \\\n",
       "0     [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "1                                                  None         NaN   NaN   \n",
       "2                                                  None         NaN   NaN   \n",
       "3     [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "4     [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "...                                                 ...         ...   ...   \n",
       "9995                                               None         NaN   NaN   \n",
       "9996                                               None         NaN   NaN   \n",
       "9997  [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "9998  [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "9999  [{'_type': 'snscrape.modules.twitter.User', 'u...         NaN   NaN   \n",
       "\n",
       "     hashtags  cashtags  \n",
       "0        None       NaN  \n",
       "1        None       NaN  \n",
       "2        None       NaN  \n",
       "3        None       NaN  \n",
       "4        None       NaN  \n",
       "...       ...       ...  \n",
       "9995     None       NaN  \n",
       "9996     None       NaN  \n",
       "9997     None       NaN  \n",
       "9998     None       NaN  \n",
       "9999     None       NaN  \n",
       "\n",
       "[10000 rows x 28 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export dataframe into a CSV\n",
    "#tweets_df1.to_csv('Elon_10k_tweet.csv', sep=',', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## QUERY BY TEXT SEARCH"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setting variables to be used in format string command below\n",
    "tweet_count = 300000\n",
    "text_query = \"#bitcoin\"\n",
    "since_date = \"2021-01-01\"\n",
    "until_date = \"2021-01-31\"\n",
    "\n",
    "# Using OS library to call CLI commands in Python\n",
    "os.system('snscrape --jsonl --max-results {} --since {} twitter-search \"{} until:{}\"> text-query-tweets.json'.format(tweet_count, since_date, text_query, until_date))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_type</th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>...</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>inReplyToTweetId</th>\n",
       "      <th>inReplyToUser</th>\n",
       "      <th>mentionedUsers</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>place</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/MoonLamboz/status/13556670...</td>\n",
       "      <td>2021-01-30 23:59:50+00:00</td>\n",
       "      <td>@cameron Protest with #bitcoin instead. Someth...</td>\n",
       "      <td>@cameron Protest with #bitcoin instead. Someth...</td>\n",
       "      <td>1355667067303841793</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.355617e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/wollf_f_Crypto/status/1355...</td>\n",
       "      <td>2021-01-30 23:59:43+00:00</td>\n",
       "      <td>#Reddit Vs. #WallStreet - The Real Story Ft. S...</td>\n",
       "      <td>#Reddit Vs. #WallStreet - The Real Story Ft. S...</td>\n",
       "      <td>1355667036681203712</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[Reddit, WallStreet, Bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/profeluisv/status/13556670...</td>\n",
       "      <td>2021-01-30 23:59:43+00:00</td>\n",
       "      <td>Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...</td>\n",
       "      <td>Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...</td>\n",
       "      <td>1355667036190466059</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[criptomonedas, Bitcoin, Ethereum, BNB, BUSD, ...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/CryptoBox2018/status/13556...</td>\n",
       "      <td>2021-01-30 23:59:32+00:00</td>\n",
       "      <td>I earned with the help of my friends! 💸 Anyone...</td>\n",
       "      <td>I earned with the help of my friends! 💸 Anyone...</td>\n",
       "      <td>1355666992708116486</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[betfurysuccess, tron, bitcoin, referral]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/FxgolgoB/status/1355666990...</td>\n",
       "      <td>2021-01-30 23:59:32+00:00</td>\n",
       "      <td>情報が多ければ散漫になる #fx,#bitcoin</td>\n",
       "      <td>情報が多ければ散漫になる #fx,#bitcoin</td>\n",
       "      <td>1355666990636167169</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[fx, bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                            _type  \\\n",
       "0  snscrape.modules.twitter.Tweet   \n",
       "1  snscrape.modules.twitter.Tweet   \n",
       "2  snscrape.modules.twitter.Tweet   \n",
       "3  snscrape.modules.twitter.Tweet   \n",
       "4  snscrape.modules.twitter.Tweet   \n",
       "\n",
       "                                                 url  \\\n",
       "0  https://twitter.com/MoonLamboz/status/13556670...   \n",
       "1  https://twitter.com/wollf_f_Crypto/status/1355...   \n",
       "2  https://twitter.com/profeluisv/status/13556670...   \n",
       "3  https://twitter.com/CryptoBox2018/status/13556...   \n",
       "4  https://twitter.com/FxgolgoB/status/1355666990...   \n",
       "\n",
       "                       date  \\\n",
       "0 2021-01-30 23:59:50+00:00   \n",
       "1 2021-01-30 23:59:43+00:00   \n",
       "2 2021-01-30 23:59:43+00:00   \n",
       "3 2021-01-30 23:59:32+00:00   \n",
       "4 2021-01-30 23:59:32+00:00   \n",
       "\n",
       "                                             content  \\\n",
       "0  @cameron Protest with #bitcoin instead. Someth...   \n",
       "1  #Reddit Vs. #WallStreet - The Real Story Ft. S...   \n",
       "2  Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...   \n",
       "3  I earned with the help of my friends! 💸 Anyone...   \n",
       "4                          情報が多ければ散漫になる #fx,#bitcoin   \n",
       "\n",
       "                                     renderedContent                   id  \\\n",
       "0  @cameron Protest with #bitcoin instead. Someth...  1355667067303841793   \n",
       "1  #Reddit Vs. #WallStreet - The Real Story Ft. S...  1355667036681203712   \n",
       "2  Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...  1355667036190466059   \n",
       "3  I earned with the help of my friends! 💸 Anyone...  1355666992708116486   \n",
       "4                          情報が多ければ散漫になる #fx,#bitcoin  1355666990636167169   \n",
       "\n",
       "                                                user  replyCount  \\\n",
       "0  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "1  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "2  {'_type': 'snscrape.modules.twitter.User', 'us...          11   \n",
       "3  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "4  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "\n",
       "   retweetCount  likeCount  ...  media  retweetedTweet quotedTweet  \\\n",
       "0             0          0  ...   None             NaN        None   \n",
       "1             0          1  ...   None             NaN        None   \n",
       "2             0          6  ...   None             NaN        None   \n",
       "3             0          0  ...   None             NaN        None   \n",
       "4             1          0  ...   None             NaN        None   \n",
       "\n",
       "  inReplyToTweetId                                      inReplyToUser  \\\n",
       "0     1.355617e+18  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "1              NaN                                               None   \n",
       "2              NaN                                               None   \n",
       "3              NaN                                               None   \n",
       "4              NaN                                               None   \n",
       "\n",
       "                                      mentionedUsers coordinates place  \\\n",
       "0  [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "1  [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "2                                               None        None  None   \n",
       "3  [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "4                                               None        None  None   \n",
       "\n",
       "                                            hashtags  cashtags  \n",
       "0                                          [bitcoin]      None  \n",
       "1                      [Reddit, WallStreet, Bitcoin]      None  \n",
       "2  [criptomonedas, Bitcoin, Ethereum, BNB, BUSD, ...      None  \n",
       "3          [betfurysuccess, tron, bitcoin, referral]      None  \n",
       "4                                      [fx, bitcoin]      None  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Reads the json generated from the CLI command above and creates a pandas dataframe\n",
    "tweets_df2 = pd.read_json('text-query-tweets.json', lines=True)\n",
    "\n",
    "# Displays first 5 entries from dataframe\n",
    "tweets_df2.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_type</th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>...</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>inReplyToTweetId</th>\n",
       "      <th>inReplyToUser</th>\n",
       "      <th>mentionedUsers</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>place</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>299995</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/CryptoStorm3/status/135215...</td>\n",
       "      <td>2021-01-21 07:25:05+00:00</td>\n",
       "      <td>$BTC \\nStory is clear. Very big move incoming!...</td>\n",
       "      <td>$BTC \\nStory is clear. Very big move incoming!...</td>\n",
       "      <td>1352155238342529024</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[Bitcoin]</td>\n",
       "      <td>[BTC]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299996</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/richie_fallclis/status/135...</td>\n",
       "      <td>2021-01-21 07:24:46+00:00</td>\n",
       "      <td>#BTC .D trendline on the daily.\\nWhat i see mo...</td>\n",
       "      <td>#BTC .D trendline on the daily.\\nWhat i see mo...</td>\n",
       "      <td>1352155160886312961</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[BTC, Bitcoin, altcoins]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299997</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/wimpie_LFC/status/13521551...</td>\n",
       "      <td>2021-01-21 07:24:38+00:00</td>\n",
       "      <td>@bitcoin_qr #BTC  #Bitcoin</td>\n",
       "      <td>@bitcoin_qr #BTC  #Bitcoin</td>\n",
       "      <td>1352155127939960835</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.352067e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[BTC, Bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299998</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/anteropaljakka1/status/135...</td>\n",
       "      <td>2021-01-21 07:24:32+00:00</td>\n",
       "      <td>Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...</td>\n",
       "      <td>Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...</td>\n",
       "      <td>1352155099271979009</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.Tweet', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[Bitcoin, cryptocurrency]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299999</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/kurcenli34/status/13521550...</td>\n",
       "      <td>2021-01-21 07:24:19+00:00</td>\n",
       "      <td>#bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...</td>\n",
       "      <td>#bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...</td>\n",
       "      <td>1352155047052910592</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>81</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[bal, bitcoin, balbtc]</td>\n",
       "      <td>[bal]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 _type  \\\n",
       "299995  snscrape.modules.twitter.Tweet   \n",
       "299996  snscrape.modules.twitter.Tweet   \n",
       "299997  snscrape.modules.twitter.Tweet   \n",
       "299998  snscrape.modules.twitter.Tweet   \n",
       "299999  snscrape.modules.twitter.Tweet   \n",
       "\n",
       "                                                      url  \\\n",
       "299995  https://twitter.com/CryptoStorm3/status/135215...   \n",
       "299996  https://twitter.com/richie_fallclis/status/135...   \n",
       "299997  https://twitter.com/wimpie_LFC/status/13521551...   \n",
       "299998  https://twitter.com/anteropaljakka1/status/135...   \n",
       "299999  https://twitter.com/kurcenli34/status/13521550...   \n",
       "\n",
       "                            date  \\\n",
       "299995 2021-01-21 07:25:05+00:00   \n",
       "299996 2021-01-21 07:24:46+00:00   \n",
       "299997 2021-01-21 07:24:38+00:00   \n",
       "299998 2021-01-21 07:24:32+00:00   \n",
       "299999 2021-01-21 07:24:19+00:00   \n",
       "\n",
       "                                                  content  \\\n",
       "299995  $BTC \\nStory is clear. Very big move incoming!...   \n",
       "299996  #BTC .D trendline on the daily.\\nWhat i see mo...   \n",
       "299997                         @bitcoin_qr #BTC  #Bitcoin   \n",
       "299998  Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...   \n",
       "299999  #bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...   \n",
       "\n",
       "                                          renderedContent  \\\n",
       "299995  $BTC \\nStory is clear. Very big move incoming!...   \n",
       "299996  #BTC .D trendline on the daily.\\nWhat i see mo...   \n",
       "299997                         @bitcoin_qr #BTC  #Bitcoin   \n",
       "299998  Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...   \n",
       "299999  #bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...   \n",
       "\n",
       "                         id  \\\n",
       "299995  1352155238342529024   \n",
       "299996  1352155160886312961   \n",
       "299997  1352155127939960835   \n",
       "299998  1352155099271979009   \n",
       "299999  1352155047052910592   \n",
       "\n",
       "                                                     user  replyCount  \\\n",
       "299995  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "299996  {'_type': 'snscrape.modules.twitter.User', 'us...           1   \n",
       "299997  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "299998  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "299999  {'_type': 'snscrape.modules.twitter.User', 'us...           5   \n",
       "\n",
       "        retweetCount  likeCount  ...  \\\n",
       "299995             0          7  ...   \n",
       "299996             1          7  ...   \n",
       "299997             0          1  ...   \n",
       "299998             0          0  ...   \n",
       "299999             7         81  ...   \n",
       "\n",
       "                                                    media  retweetedTweet  \\\n",
       "299995  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "299996  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "299997                                               None             NaN   \n",
       "299998                                               None             NaN   \n",
       "299999  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "\n",
       "                                              quotedTweet inReplyToTweetId  \\\n",
       "299995                                               None              NaN   \n",
       "299996                                               None              NaN   \n",
       "299997                                               None     1.352067e+18   \n",
       "299998  {'_type': 'snscrape.modules.twitter.Tweet', 'u...              NaN   \n",
       "299999                                               None              NaN   \n",
       "\n",
       "                                            inReplyToUser  \\\n",
       "299995                                               None   \n",
       "299996                                               None   \n",
       "299997  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "299998                                               None   \n",
       "299999                                               None   \n",
       "\n",
       "                                           mentionedUsers coordinates place  \\\n",
       "299995                                               None        None  None   \n",
       "299996                                               None        None  None   \n",
       "299997  [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "299998                                               None        None  None   \n",
       "299999                                               None        None  None   \n",
       "\n",
       "                         hashtags  cashtags  \n",
       "299995                  [Bitcoin]     [BTC]  \n",
       "299996   [BTC, Bitcoin, altcoins]      None  \n",
       "299997             [BTC, Bitcoin]      None  \n",
       "299998  [Bitcoin, cryptocurrency]      None  \n",
       "299999     [bal, bitcoin, balbtc]     [bal]  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df2.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>_type</th>\n",
       "      <th>url</th>\n",
       "      <th>date</th>\n",
       "      <th>content</th>\n",
       "      <th>renderedContent</th>\n",
       "      <th>id</th>\n",
       "      <th>user</th>\n",
       "      <th>replyCount</th>\n",
       "      <th>retweetCount</th>\n",
       "      <th>likeCount</th>\n",
       "      <th>...</th>\n",
       "      <th>media</th>\n",
       "      <th>retweetedTweet</th>\n",
       "      <th>quotedTweet</th>\n",
       "      <th>inReplyToTweetId</th>\n",
       "      <th>inReplyToUser</th>\n",
       "      <th>mentionedUsers</th>\n",
       "      <th>coordinates</th>\n",
       "      <th>place</th>\n",
       "      <th>hashtags</th>\n",
       "      <th>cashtags</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/MoonLamboz/status/13556670...</td>\n",
       "      <td>2021-01-30 23:59:50+00:00</td>\n",
       "      <td>@cameron Protest with #bitcoin instead. Someth...</td>\n",
       "      <td>@cameron Protest with #bitcoin instead. Someth...</td>\n",
       "      <td>1355667067303841793</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.355617e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/wollf_f_Crypto/status/1355...</td>\n",
       "      <td>2021-01-30 23:59:43+00:00</td>\n",
       "      <td>#Reddit Vs. #WallStreet - The Real Story Ft. S...</td>\n",
       "      <td>#Reddit Vs. #WallStreet - The Real Story Ft. S...</td>\n",
       "      <td>1355667036681203712</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[Reddit, WallStreet, Bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/profeluisv/status/13556670...</td>\n",
       "      <td>2021-01-30 23:59:43+00:00</td>\n",
       "      <td>Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...</td>\n",
       "      <td>Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...</td>\n",
       "      <td>1355667036190466059</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>6</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[criptomonedas, Bitcoin, Ethereum, BNB, BUSD, ...</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/CryptoBox2018/status/13556...</td>\n",
       "      <td>2021-01-30 23:59:32+00:00</td>\n",
       "      <td>I earned with the help of my friends! 💸 Anyone...</td>\n",
       "      <td>I earned with the help of my friends! 💸 Anyone...</td>\n",
       "      <td>1355666992708116486</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[betfurysuccess, tron, bitcoin, referral]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/FxgolgoB/status/1355666990...</td>\n",
       "      <td>2021-01-30 23:59:32+00:00</td>\n",
       "      <td>情報が多ければ散漫になる #fx,#bitcoin</td>\n",
       "      <td>情報が多ければ散漫になる #fx,#bitcoin</td>\n",
       "      <td>1355666990636167169</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[fx, bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299995</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/CryptoStorm3/status/135215...</td>\n",
       "      <td>2021-01-21 07:25:05+00:00</td>\n",
       "      <td>$BTC \\nStory is clear. Very big move incoming!...</td>\n",
       "      <td>$BTC \\nStory is clear. Very big move incoming!...</td>\n",
       "      <td>1352155238342529024</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[Bitcoin]</td>\n",
       "      <td>[BTC]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299996</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/richie_fallclis/status/135...</td>\n",
       "      <td>2021-01-21 07:24:46+00:00</td>\n",
       "      <td>#BTC .D trendline on the daily.\\nWhat i see mo...</td>\n",
       "      <td>#BTC .D trendline on the daily.\\nWhat i see mo...</td>\n",
       "      <td>1352155160886312961</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[BTC, Bitcoin, altcoins]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299997</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/wimpie_LFC/status/13521551...</td>\n",
       "      <td>2021-01-21 07:24:38+00:00</td>\n",
       "      <td>@bitcoin_qr #BTC  #Bitcoin</td>\n",
       "      <td>@bitcoin_qr #BTC  #Bitcoin</td>\n",
       "      <td>1352155127939960835</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>1.352067e+18</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.User', 'u...</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[BTC, Bitcoin]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299998</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/anteropaljakka1/status/135...</td>\n",
       "      <td>2021-01-21 07:24:32+00:00</td>\n",
       "      <td>Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...</td>\n",
       "      <td>Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...</td>\n",
       "      <td>1352155099271979009</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.Tweet', 'u...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[Bitcoin, cryptocurrency]</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>299999</th>\n",
       "      <td>snscrape.modules.twitter.Tweet</td>\n",
       "      <td>https://twitter.com/kurcenli34/status/13521550...</td>\n",
       "      <td>2021-01-21 07:24:19+00:00</td>\n",
       "      <td>#bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...</td>\n",
       "      <td>#bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...</td>\n",
       "      <td>1352155047052910592</td>\n",
       "      <td>{'_type': 'snscrape.modules.twitter.User', 'us...</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "      <td>81</td>\n",
       "      <td>...</td>\n",
       "      <td>[{'_type': 'snscrape.modules.twitter.Photo', '...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>[bal, bitcoin, balbtc]</td>\n",
       "      <td>[bal]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>300000 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                 _type  \\\n",
       "0       snscrape.modules.twitter.Tweet   \n",
       "1       snscrape.modules.twitter.Tweet   \n",
       "2       snscrape.modules.twitter.Tweet   \n",
       "3       snscrape.modules.twitter.Tweet   \n",
       "4       snscrape.modules.twitter.Tweet   \n",
       "...                                ...   \n",
       "299995  snscrape.modules.twitter.Tweet   \n",
       "299996  snscrape.modules.twitter.Tweet   \n",
       "299997  snscrape.modules.twitter.Tweet   \n",
       "299998  snscrape.modules.twitter.Tweet   \n",
       "299999  snscrape.modules.twitter.Tweet   \n",
       "\n",
       "                                                      url  \\\n",
       "0       https://twitter.com/MoonLamboz/status/13556670...   \n",
       "1       https://twitter.com/wollf_f_Crypto/status/1355...   \n",
       "2       https://twitter.com/profeluisv/status/13556670...   \n",
       "3       https://twitter.com/CryptoBox2018/status/13556...   \n",
       "4       https://twitter.com/FxgolgoB/status/1355666990...   \n",
       "...                                                   ...   \n",
       "299995  https://twitter.com/CryptoStorm3/status/135215...   \n",
       "299996  https://twitter.com/richie_fallclis/status/135...   \n",
       "299997  https://twitter.com/wimpie_LFC/status/13521551...   \n",
       "299998  https://twitter.com/anteropaljakka1/status/135...   \n",
       "299999  https://twitter.com/kurcenli34/status/13521550...   \n",
       "\n",
       "                            date  \\\n",
       "0      2021-01-30 23:59:50+00:00   \n",
       "1      2021-01-30 23:59:43+00:00   \n",
       "2      2021-01-30 23:59:43+00:00   \n",
       "3      2021-01-30 23:59:32+00:00   \n",
       "4      2021-01-30 23:59:32+00:00   \n",
       "...                          ...   \n",
       "299995 2021-01-21 07:25:05+00:00   \n",
       "299996 2021-01-21 07:24:46+00:00   \n",
       "299997 2021-01-21 07:24:38+00:00   \n",
       "299998 2021-01-21 07:24:32+00:00   \n",
       "299999 2021-01-21 07:24:19+00:00   \n",
       "\n",
       "                                                  content  \\\n",
       "0       @cameron Protest with #bitcoin instead. Someth...   \n",
       "1       #Reddit Vs. #WallStreet - The Real Story Ft. S...   \n",
       "2       Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...   \n",
       "3       I earned with the help of my friends! 💸 Anyone...   \n",
       "4                               情報が多ければ散漫になる #fx,#bitcoin   \n",
       "...                                                   ...   \n",
       "299995  $BTC \\nStory is clear. Very big move incoming!...   \n",
       "299996  #BTC .D trendline on the daily.\\nWhat i see mo...   \n",
       "299997                         @bitcoin_qr #BTC  #Bitcoin   \n",
       "299998  Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...   \n",
       "299999  #bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...   \n",
       "\n",
       "                                          renderedContent  \\\n",
       "0       @cameron Protest with #bitcoin instead. Someth...   \n",
       "1       #Reddit Vs. #WallStreet - The Real Story Ft. S...   \n",
       "2       Mis #criptomonedas preferidas:\\n\\n#Bitcoin aho...   \n",
       "3       I earned with the help of my friends! 💸 Anyone...   \n",
       "4                               情報が多ければ散漫になる #fx,#bitcoin   \n",
       "...                                                   ...   \n",
       "299995  $BTC \\nStory is clear. Very big move incoming!...   \n",
       "299996  #BTC .D trendline on the daily.\\nWhat i see mo...   \n",
       "299997                         @bitcoin_qr #BTC  #Bitcoin   \n",
       "299998  Todella isorahan ensiaskeleita kryptoihin 👏👏 \\...   \n",
       "299999  #bal $bal #bitcoin 🔥🔥🔥\\n\\n#balbtc takip edin👌 ...   \n",
       "\n",
       "                         id  \\\n",
       "0       1355667067303841793   \n",
       "1       1355667036681203712   \n",
       "2       1355667036190466059   \n",
       "3       1355666992708116486   \n",
       "4       1355666990636167169   \n",
       "...                     ...   \n",
       "299995  1352155238342529024   \n",
       "299996  1352155160886312961   \n",
       "299997  1352155127939960835   \n",
       "299998  1352155099271979009   \n",
       "299999  1352155047052910592   \n",
       "\n",
       "                                                     user  replyCount  \\\n",
       "0       {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "1       {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "2       {'_type': 'snscrape.modules.twitter.User', 'us...          11   \n",
       "3       {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "4       {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "...                                                   ...         ...   \n",
       "299995  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "299996  {'_type': 'snscrape.modules.twitter.User', 'us...           1   \n",
       "299997  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "299998  {'_type': 'snscrape.modules.twitter.User', 'us...           0   \n",
       "299999  {'_type': 'snscrape.modules.twitter.User', 'us...           5   \n",
       "\n",
       "        retweetCount  likeCount  ...  \\\n",
       "0                  0          0  ...   \n",
       "1                  0          1  ...   \n",
       "2                  0          6  ...   \n",
       "3                  0          0  ...   \n",
       "4                  1          0  ...   \n",
       "...              ...        ...  ...   \n",
       "299995             0          7  ...   \n",
       "299996             1          7  ...   \n",
       "299997             0          1  ...   \n",
       "299998             0          0  ...   \n",
       "299999             7         81  ...   \n",
       "\n",
       "                                                    media  retweetedTweet  \\\n",
       "0                                                    None             NaN   \n",
       "1                                                    None             NaN   \n",
       "2                                                    None             NaN   \n",
       "3                                                    None             NaN   \n",
       "4                                                    None             NaN   \n",
       "...                                                   ...             ...   \n",
       "299995  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "299996  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "299997                                               None             NaN   \n",
       "299998                                               None             NaN   \n",
       "299999  [{'_type': 'snscrape.modules.twitter.Photo', '...             NaN   \n",
       "\n",
       "                                              quotedTweet inReplyToTweetId  \\\n",
       "0                                                    None     1.355617e+18   \n",
       "1                                                    None              NaN   \n",
       "2                                                    None              NaN   \n",
       "3                                                    None              NaN   \n",
       "4                                                    None              NaN   \n",
       "...                                                   ...              ...   \n",
       "299995                                               None              NaN   \n",
       "299996                                               None              NaN   \n",
       "299997                                               None     1.352067e+18   \n",
       "299998  {'_type': 'snscrape.modules.twitter.Tweet', 'u...              NaN   \n",
       "299999                                               None              NaN   \n",
       "\n",
       "                                            inReplyToUser  \\\n",
       "0       {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "1                                                    None   \n",
       "2                                                    None   \n",
       "3                                                    None   \n",
       "4                                                    None   \n",
       "...                                                   ...   \n",
       "299995                                               None   \n",
       "299996                                               None   \n",
       "299997  {'_type': 'snscrape.modules.twitter.User', 'us...   \n",
       "299998                                               None   \n",
       "299999                                               None   \n",
       "\n",
       "                                           mentionedUsers coordinates place  \\\n",
       "0       [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "1       [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "2                                                    None        None  None   \n",
       "3       [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "4                                                    None        None  None   \n",
       "...                                                   ...         ...   ...   \n",
       "299995                                               None        None  None   \n",
       "299996                                               None        None  None   \n",
       "299997  [{'_type': 'snscrape.modules.twitter.User', 'u...        None  None   \n",
       "299998                                               None        None  None   \n",
       "299999                                               None        None  None   \n",
       "\n",
       "                                                 hashtags  cashtags  \n",
       "0                                               [bitcoin]      None  \n",
       "1                           [Reddit, WallStreet, Bitcoin]      None  \n",
       "2       [criptomonedas, Bitcoin, Ethereum, BNB, BUSD, ...      None  \n",
       "3               [betfurysuccess, tron, bitcoin, referral]      None  \n",
       "4                                           [fx, bitcoin]      None  \n",
       "...                                                   ...       ...  \n",
       "299995                                          [Bitcoin]     [BTC]  \n",
       "299996                           [BTC, Bitcoin, altcoins]      None  \n",
       "299997                                     [BTC, Bitcoin]      None  \n",
       "299998                          [Bitcoin, cryptocurrency]      None  \n",
       "299999                             [bal, bitcoin, balbtc]     [bal]  \n",
       "\n",
       "[300000 rows x 28 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Export dataframe into a CSV\n",
    "tweets_df2.to_csv('bitcoin-tweets_V1_january.csv', sep=',', index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we have repeated this extraction for all the months we are interested to analyze."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}