Spark_Workshop_Notebook_Instructor_JupyterHub.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Brief introduction to Spark - on jupyter.idigbio.org\n",
    "\n",
    "This is a brief introduction to Spark intended to take about 1 hour up through the word2vec material including the introduction Powerpoint.\n",
    "\n",
    "This notebook will run on the JupyterHub cluster maintained by iDigBio at jupyter.idigbio.org . To run locally, you will need to install Spark and set up your notebook server to have a sqlContect variable and Spark context running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "import os\n",
    "import sys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset comes from the Hacker News site maintained by the Y Combinator startup incubator: https://www.hackernews.com/ . Comments from several years were downloaded and 100k comments were selected and formated as JSON into a 23 MB compressed file. It is stored on the HDFS cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "comments = sqlContext.read.json(\"/tmp/hacker_news_100k.json.gz\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the comments variable?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pyspark.sql.dataframe.DataFrame"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(comments)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is a Spark data frame, specifically one created and accessed using the pyspark Python library. Notice also that the data frame is a sub object of the SQL object, recall that in Spark data frames are part of the SQL API.\n",
    "\n",
    "What can you do with such a data frame?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100000"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comments.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- _highlightResult: struct (nullable = true)\n",
      " |    |-- author: struct (nullable = true)\n",
      " |    |    |-- matchLevel: string (nullable = true)\n",
      " |    |    |-- matchedWords: array (nullable = true)\n",
      " |    |    |    |-- element: string (containsNull = true)\n",
      " |    |    |-- value: string (nullable = true)\n",
      " |    |-- comment_text: struct (nullable = true)\n",
      " |    |    |-- matchLevel: string (nullable = true)\n",
      " |    |    |-- matchedWords: array (nullable = true)\n",
      " |    |    |    |-- element: string (containsNull = true)\n",
      " |    |    |-- value: string (nullable = true)\n",
      " |    |-- story_title: struct (nullable = true)\n",
      " |    |    |-- matchLevel: string (nullable = true)\n",
      " |    |    |-- matchedWords: array (nullable = true)\n",
      " |    |    |    |-- element: string (containsNull = true)\n",
      " |    |    |-- value: string (nullable = true)\n",
      " |    |-- story_url: struct (nullable = true)\n",
      " |    |    |-- matchLevel: string (nullable = true)\n",
      " |    |    |-- matchedWords: array (nullable = true)\n",
      " |    |    |    |-- element: string (containsNull = true)\n",
      " |    |    |-- value: string (nullable = true)\n",
      " |-- _tags: array (nullable = true)\n",
      " |    |-- element: string (containsNull = true)\n",
      " |-- author: string (nullable = true)\n",
      " |-- comment_text: string (nullable = true)\n",
      " |-- created_at: string (nullable = true)\n",
      " |-- created_at_i: long (nullable = true)\n",
      " |-- num_comments: string (nullable = true)\n",
      " |-- objectID: string (nullable = true)\n",
      " |-- parent_id: long (nullable = true)\n",
      " |-- points: long (nullable = true)\n",
      " |-- story_id: long (nullable = true)\n",
      " |-- story_text: string (nullable = true)\n",
      " |-- story_title: string (nullable = true)\n",
      " |-- story_url: string (nullable = true)\n",
      " |-- title: string (nullable = true)\n",
      " |-- url: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "comments.printSchema()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+--------------------+----------+--------------------+--------------------+------------+------------+--------+---------+------+--------+----------+--------------------+--------------------+-----+----+\n",
      "|    _highlightResult|               _tags|    author|        comment_text|          created_at|created_at_i|num_comments|objectID|parent_id|points|story_id|story_text|         story_title|           story_url|title| url|\n",
      "+--------------------+--------------------+----------+--------------------+--------------------+------------+------------+--------+---------+------+--------+----------+--------------------+--------------------+-----+----+\n",
      "|[[none,WrappedArr...|[comment, author_...|       VMG|Because you don&#...|2014-05-30T08:19:34Z|  1401437974|        null| 7820767|  7820656|     1| 7820350|      null|Mark Zuckerberg a...|http://www.mercur...| null|null|\n",
      "|[[none,WrappedArr...|[comment, author_...|hackerboos|$649 in US<p>$699...|2014-05-30T08:18:47Z|  1401437927|        null| 7820764|  7820517|     1| 7820517|      null|4K for $649: Asus...|http://techreport...| null|null|\n",
      "|[[none,WrappedArr...|[comment, author_...|  unhammer|The one I use mos...|2014-05-30T08:17:54Z|  1401437874|        null| 7820763|  7820701|     1| 7817934|      null|Laverna: Self-hos...|   http://laverna.cc| null|null|\n",
      "+--------------------+--------------------+----------+--------------------+--------------------+------------+------------+--------+---------+------+--------+----------+--------------------+--------------------+-----+----+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "comments.show(3)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------+\n",
      "|        comment_text|\n",
      "+--------------------+\n",
      "|Because you don&#...|\n",
      "|$649 in US<p>$699...|\n",
      "|The one I use mos...|\n",
      "+--------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import col\n",
    "\n",
    "comments.select(col(\"comment_text\")).show(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above line is a good oportunity to look at the documentation:\n",
    "\n",
    "http://spark.apache.org/docs/latest/api/python/pyspark.sql.html\n",
    "\n",
    "show() is in the DataFrame functions. Or search for \"show(n=20\" because the show() function is used in a lot of examples.\n",
    "\n",
    "This is also a good time to introduce the \"()\" syntax which makes reading long commands easier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|comment_text                                                                                                                                                                                                                                                                                                                                                                                              |\n",
      "+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|Because you don&#x27;t have to rely on a political apparatus to spend the money wisely.<p>By the way, nobody has to wait for billionaires anywhere, if you want to help out in education, get up and do it.                                                                                                                                                                                               |\n",
      "|$649 in US<p>$699 in Canada<p>$1000 in UK &lt;- What on earth is going on here - even with 20% VAT?                                                                                                                                                                                                                                                                                                       |\n",
      "|The one I use most in addition to Owncloud is <a href=\"http://subsonic.org/\" rel=\"nofollow\">http:&#x2F;&#x2F;subsonic.org&#x2F;</a> – after I ripped my full music collection I no longer felt the need for a Spotify subscription. Subsonic has a really nice Android app, but the only iPhone app I found was quite confusing, so if you&#x27;re using an iPhone you should probably … make a new app :)|\n",
      "+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(comments\n",
    "    .select(col(\"comment_text\"))\n",
    "    .show(3, truncate=False)\n",
    " )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing\n",
    "\n",
    "\n",
    "\n",
    "**Sourcing -> Cleaning -> Tokenizing -> Stop Word Removal -> Analysis**\n",
    "\n",
    "Some example analyses include:\n",
    "\n",
    " - Part of speech tagging\n",
    " - Phrase construction\n",
    " - Word2Vec (see example below)\n",
    " - Sentiment analysis\n",
    "\n",
    "\n",
    "## Cleaning\n",
    "\n",
    "From the sample comments, we can see that if we are going to look at what words are in the comments, we will need to remove the HTML tags and we will need to convert HTML entities like \"&#x27;\" to their characters like \"'\". There are a number of libraries we could use to do that but we will work with just do a regular expression because Spark is the focus for today."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "you're a Gator!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/ipykernel/__main__.py:9: DeprecationWarning: The unescape method is deprecated and will be removed in 3.5, use html.unescape() instead.\n"
     ]
    }
   ],
   "source": [
    "import re\n",
    "expression = re.compile(r'<.*?>')\n",
    "\n",
    "from html.parser import HTMLParser\n",
    "parser = HTMLParser()\n",
    "\n",
    "def strip_tags(html):\n",
    "    return parser.unescape(\n",
    "        expression.sub('', html)\n",
    "        )\n",
    "    \n",
    "print(strip_tags(\"<b>you&#x27;re a Gator!</b>\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's a nice little function but how do we run it on a column in the dataframe? And where do we store the results? What we want to do is add a column to the dataframe with text that has no HTML in it. This is called applying a user defined function to your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "expected string or bytes-like object",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-49-054bce24e97d>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m comments_clean = comments.withColumn(\"comment_clean\", \n\u001b[0;32m----> 2\u001b[0;31m                                      \u001b[0mstrip_tags\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcol\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"comment_text\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m                                     )\n",
      "\u001b[0;32m<ipython-input-48-20e3d047ef11>\u001b[0m in \u001b[0;36mstrip_tags\u001b[0;34m(html)\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mstrip_tags\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhtml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m     return parser.unescape(\n\u001b[0;32m----> 9\u001b[0;31m         \u001b[0mexpression\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msub\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhtml\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     10\u001b[0m         )\n\u001b[1;32m     11\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: expected string or bytes-like object"
     ]
    }
   ],
   "source": [
    "comments_clean = comments.withColumn(\"comment_clean\", \n",
    "                                     strip_tags(col(\"comment_text\"))\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoops! That didn't work. We wrote a Python function that takes a string and passed it a Spark dataframe column. What we want is to get Spark to do is to run that function for every value in the column with means we have to change our function to something Spark knows how to handle. Fortunately, there's a function to do just that. Notice that the argument to udf() is a function and the return is a function. In Python, functins are first class objects, they're a thing that can be manuipulated, calculated, returned, etc. like other data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.sql.functions import udf\n",
    "\n",
    "strip_tags_udf = udf(strip_tags)\n",
    "\n",
    "comments_clean = comments.withColumn(\"comment_clean\", \n",
    "                                     strip_tags_udf(col(\"comment_text\"))\n",
    "                                    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|comment_clean                                                                                                                                                                                                                                                                                                                       |\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|Because you don't have to rely on a political apparatus to spend the money wisely.By the way, nobody has to wait for billionaires anywhere, if you want to help out in education, get up and do it.                                                                                                                                 |\n",
      "|$649 in US$699 in Canada$1000 in UK <- What on earth is going on here - even with 20% VAT?                                                                                                                                                                                                                                          |\n",
      "|The one I use most in addition to Owncloud is http://subsonic.org/ – after I ripped my full music collection I no longer felt the need for a Spotify subscription. Subsonic has a really nice Android app, but the only iPhone app I found was quite confusing, so if you're using an iPhone you should probably … make a new app :)|\n",
      "+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(comments_clean\n",
    "    .select(col(\"comment_clean\"))\n",
    "    .show(3, truncate=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenizing\n",
    "\n",
    "Today we're going to do some very simple NLP. We'll consider the comments to just be a collection of words. We'll use those words to figure out which posts are most relavent to a search term that we're interested in.\n",
    "\n",
    "First up is splitting the comments up in to lists of words. At this point we'll introduce the machine learning API:\n",
    "\n",
    "(Mention ml for dataframes vs older mllib for RRDs)\n",
    "\n",
    "http://spark.apache.org/docs/latest/api/python/pyspark.ml.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Tokenizer\n",
    "tokenizer = Tokenizer(inputCol=\"comment_clean\", outputCol=\"words\")\n",
    "\n",
    "comments_words = tokenizer.transform(comments_clean)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|words                                                                                                                                                                                                                                                                                                                                                                                             |\n",
      "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|[because, you, don't, have, to, rely, on, a, political, apparatus, to, spend, the, money, wisely.by, the, way,, nobody, has, to, wait, for, billionaires, anywhere,, if, you, want, to, help, out, in, education,, get, up, and, do, it.]                                                                                                                                                         |\n",
      "|[$649, in, us$699, in, canada$1000, in, uk, <-, what, on, earth, is, going, on, here, -, even, with, 20%, vat?]                                                                                                                                                                                                                                                                                   |\n",
      "|[the, one, i, use, most, in, addition, to, owncloud, is, http://subsonic.org/, –, after, i, ripped, my, full, music, collection, i, no, longer, felt, the, need, for, a, spotify, subscription., subsonic, has, a, really, nice, android, app,, but, the, only, iphone, app, i, found, was, quite, confusing,, so, if, you're, using, an, iphone, you, should, probably, …, make, a, new, app, :)]|\n",
      "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(comments_words\n",
    "    .select(col(\"words\"))\n",
    "    .show(3, truncate=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# What are the stop words? http://ir.dcs.gla.ac.uk/resources/linguistic_utils/stop_words\n",
    "\n",
    "from pyspark.ml.feature import StopWordsRemover\n",
    "stopWordsRemover = StopWordsRemover(inputCol=\"words\", outputCol=\"tokens\")\n",
    "\n",
    "comments_tokens = stopWordsRemover.transform(comments_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|tokens                                                                                                                                                                                                                                                                    |\n",
      "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "|[don't, rely, political, apparatus, spend, money, wisely.by, way,, nobody, wait, billionaires, anywhere,, want, help, education,, get, it.]                                                                                                                               |\n",
      "|[$649, us$699, canada$1000, uk, <-, earth, going, -, even, 20%, vat?]                                                                                                                                                                                                     |\n",
      "|[one, use, addition, owncloud, http://subsonic.org/, –, ripped, full, music, collection, longer, felt, need, spotify, subscription., subsonic, really, nice, android, app,, iphone, app, found, quite, confusing,, you're, using, iphone, probably, …, make, new, app, :)]|\n",
      "+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+\n",
      "only showing top 3 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "(comments_tokens\n",
    "    .select(col(\"tokens\"))\n",
    "    .show(3, truncate=False)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Analysis - Zipf's Law\n",
    "\n",
    "Zipf's law says that the frequency of a term is inversely proportianal to it's rank ie. the second most common term occurs 1/2 as often as the first; the third most common 1/3 as often; etc. A log-log graph of frequencey vs rank should be a straight line.\n",
    "\n",
    "Let's count our tokens and graph the counts to see what they look like. Here we're going to use a map-reduce pattern of computation which we're going to express in dataframe functions that look like a SQL statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------+-----+\n",
      "| token|count|\n",
      "+------+-----+\n",
      "|      |73208|\n",
      "|  it's|23630|\n",
      "| would|21905|\n",
      "|  like|21540|\n",
      "|people|17500|\n",
      "| don't|17306|\n",
      "|   one|14924|\n",
      "| think|13764|\n",
      "|   get|13410|\n",
      "|   use|11724|\n",
      "+------+-----+\n",
      "only showing top 10 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import explode\n",
    "\n",
    "tokens = (comments_tokens\n",
    "          .select(explode(comments_tokens[\"tokens\"]).alias(\"token\"))\n",
    "          .groupBy(\"token\")\n",
    "          .count()\n",
    "          .orderBy(\"count\", ascending=False)\n",
    "          )\n",
    "    \n",
    "tokens.show(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now how do we graph the frequency against term rank (integer sequence from 1..n)? We have a Spark dataframe but that's not something we can use with graphing tools. But, we can convert it to Pandas.\n",
    "\n",
    "We'll convert the absolute minimum information to a Pandas dataframe - storage, transfer time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokens_pdf = (tokens\n",
    "              .select(tokens[\"count\"])\n",
    "              .toPandas()\n",
    "              )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    335936\n",
       "dtype: int64"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_pdf.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>rank</th>\n",
       "      <th>log_rank</th>\n",
       "      <th>log_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>73208</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11.201060</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>23630</td>\n",
       "      <td>2</td>\n",
       "      <td>0.693147</td>\n",
       "      <td>10.070272</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>21905</td>\n",
       "      <td>3</td>\n",
       "      <td>1.098612</td>\n",
       "      <td>9.994470</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>21540</td>\n",
       "      <td>4</td>\n",
       "      <td>1.386294</td>\n",
       "      <td>9.977667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>17500</td>\n",
       "      <td>5</td>\n",
       "      <td>1.609438</td>\n",
       "      <td>9.769956</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   count  rank  log_rank  log_count\n",
       "0  73208     1  0.000000  11.201060\n",
       "1  23630     2  0.693147  10.070272\n",
       "2  21905     3  1.098612   9.994470\n",
       "3  21540     4  1.386294   9.977667\n",
       "4  17500     5  1.609438   9.769956"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "tokens_pdf[\"rank\"] = range(1, tokens_pdf.shape[0] + 1)\n",
    "tokens_pdf[\"log_rank\"] = np.log(tokens_pdf[\"rank\"])\n",
    "tokens_pdf[\"log_count\"] = np.log(tokens_pdf[\"count\"])\n",
    "tokens_pdf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# \"conda install seaborn\" at a command prompt if you don't have it already\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.5/dist-packages/matplotlib/__init__.py:898: UserWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.\n",
      "  warnings.warn(self.msg_depr % (key, alt_key))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.JointGrid at 0x7f60d3019d68>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAk0AAAJPCAYAAABhMuBTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAAPYQAAD2EBqD+naQAAIABJREFUeJzs3X2c3OO9//HX3uQ+2WRzJ0GkEXXFTVBUtWko1VYpDaVV\nBy3VVuum55QSRVXRhpaDH6otiropTrU4FFXaahyUHjdFrh5tJFEhZDfZ3EmyO/v7Y3ZjN9lkv7Oz\nszPzndfz8chD9jvfmfl8ZmTyznVd32uqWltbkSRJ0qZVF7sASZKkcmBokiRJSsDQJEmSlIChSZIk\nKQFDkyRJUgKGJkmSpAQMTZIkSQkYmiRJkhIwNEmSJCVQW+wCiqm1tbW1oWEFmUx6dkWvrq5i5Mgh\npKmvNPYE6ewrjT1BOvuyp/JRXV3FqFFDq4pdhyp8pKmqqorq6nT9f1hdXZW6vtLYE6SzrzT2BOns\ny57KR9r6KWcVHZokSZKSqujpOUlSZctkMjQ0NGz09pEjR1Jd7fiCsgxNkqSK1dDQwENPzGHo0OEb\n3LZ8+VI+vucURo8eXYTKVIoMTZKkijZ06HDqRowsdhkqA445SpIkJWBokiRJSsDQJEmSlIChSZIk\nKQFDkyRJUgKGJkmSpAQMTZIkSQkYmiRJkhIwNEmSJCVgaJIkSUrA0CRJkpSAoUmSJCkBQ5MkSVIC\ntcUuQJKkQspkMjQ0NHR5W0PDYlozrX1ckcqVoUmSlGoNDQ089MQchg4dvsFtb7w+n6HDRzGcUUWo\nTOXG0CRJSr2hQ4dTN2LkBseXNTUWoRqVK9c0SZIkJWBokiRJSsDQJEmSlIChSZIkKQFDkyRJUgKG\nJkmSpAQMTZIkSQkYmiRJkhIwNEmSJCXgjuCSpLKXyWR4++3FNDevZMmSFTQ3v/t9cn6/nHqLoUmS\nVPYaGhr4/VOR0WPGsGrVGjIdQpLfL6feYmiSJKXCkKHDGVE/iv4DVtPSITT5/XLqLa5pkiRJSsDQ\nJEmSlIChSZIkKQFDkyRJUgKGJkmSpAQMTZIkSQkYmiRJkhJwnyZJUtnIZDI0NDRscLyhYTGtre76\nrcIyNEmSykZDQwMPPTGHoUOHdzr+xuvzGV4/ukhVqVIYmiRJZWXo0OHUjRjZ6Zi7fqsvuKZJkiQp\nAUOTJElSAoYmSZKkBAxNkiRJCbgQXJJUUja2rQC0bS2QcWsBFYehSZJUUja2rQBktxYYOnwUwxlV\nhMpU6QxNkqSS09W2AuDWAiou1zRJkiQlYGiSJElKwNAkSZKUgGuaJEl9zivkVI4MTZKkPucVcipH\nhiZJUlF4hZzKjWuaJEmSEnCkSZJUEK5bUtoYmiRJBeG6JaWNoUmSVDCuW1KauKZJkiQpAUeaJEk9\n5rolVRJDkySpx1y3pEpiaJIk5cV1S6oUhiZJUrc2Ng3nFJwqiaFJktStjU3DOQWnSmJokiQB3S/q\nHjK4boNpOKfgVEkMTZJUQdqDUW1tFc3NK1myZAXNzdnptYaGxfxlzlsMq3NRt9QVQ5MkpUx3I0Z/\nmfMWw0eMYNCg/qxatYZM25qk9mDkom6pa4YmSSpDSYLRJkeMRoxk8OAB9B+wmpa20GQwkjbN0CRJ\nJWxTV611F4wcMZJ6l6FJkvrApkaGMpkMANXVG36z1cbCkcFI6nuGJklloaehozdu67hoes2alh49\nXncjQ9W1/Rg7dnyXt3UVjgxGUt8zNKXMwoUL+d3v/sry5e+sW9zZbtttpzBmzNgiVdZzXV3lkwZp\n7KuQPTU0LOaPz/yDQYOHbHDb24sWUlXTj1GjRhfkturqKgYO7M8776xh0Ruv9/jxhg4f2WVo6s7K\nFU00LekcGFcsX0Z1bb8Njie9bemSBtas7rwQvCePmW8dvXXbiuXL6NdvNUsaF3fqKZ/nWr58KbBh\nkFXlqmptTceHtSRJUiFtOIYsSZKkDRiaJEmSEjA0SZIkJWBokiRJSsDQJEmSlIChSZIkKQFDkyRJ\nUgKGJkmSpARKckfwEMJ04FvAbmS3Y50RY7yn7bZa4ELgk8DWwFLgYWBmjHFhcSqWJElpV6ojTUOA\nZ4ETgfW3LB8M7AKcB7wPOAQIwN19WaAkSaosJf81KiGEDB1GmjZyzu7Ak8DEGONrfVacJEmqGKU6\n0pSrEWRHpJYUuxBJkpROZR+aQggDgFnArTHG5cWuR5IkpVNZh6a2ReF3kh1l+nqRy5EkSSlWklfP\nJdEhME0A9u3JKFNra2trVVVVr9cmSVIv6/FfVv5dl0iiF6gsQ1OHwLQ1sE+MsbEnj1NVVUVT0ypa\nWjK9Wl8x1dRUU1c3KFV9pbEnSGdfaewJ0tmXPZWP9r56qqFhBdXVhqZNqa8fkui8kgxNIYQhwDa8\nm/y2DiHsDDQArwO/IrvtwKeAfiGEzdrOa4gxrs3luVpaMjQ3p+cPV7s09pXGniCdfaWxJ0hnX/aU\nfplMK5lMaV8pXy5KMjQBuwOPkl2r1Apc0nb8RrL7Mx3UdvzZtuNVbT/vA/ypTyuVJEkVoSRDU4zx\nj2x6kXpZL2CXJEnlx/AhSZKUgKFJkiQpAUOTJElSAoYmSZKkBAxNkiRJCRiaJEmSEjA0SZIkJWBo\nkiRJSqAkN7eUJJWXRx55mOuuu4aFCxcyYcJWnHDCSXzwg9M2eZ9f/eoOfv3rO1m48HXGjRvP0Ucf\ny/77H7ju9t/+9r/5/vfPo6qqitbW7NeA9O/fn9//fnZBe+ktPXlNVNoMTZJUQM3NzdTW9v1HbV8+\n79/+9jznnXcWX/vayXzoQx/md797kG9/+zSuv/4WJk3ausv7/PrX/8XPfnY1Z5xxNlOmbM9LL/2N\niy66kLq64XzoQx9ed96QIUO57ba7yH5TFiT8Mvqi68lrotJnaJJUEU4++atsvfVkAB588H5qa2uZ\nMeMwjj/+hHXnrF27lp/85Cp+//uHWL58GVtvvQ0nnHAS73//+wFYunQpP/zhLJ5//lmampayxRZb\ncvTRx7Lffp/Y4Hlqamp46KHfMnnye7n88h9z3XU/4f7776WxsYHhw0fwkY98lG9841QAli1bxmWX\n/ZDHH/8za9euYZddduXf//1bbLnlBCA74nL55Zfwve/9gCuuuIQ333yTnXbahbPOOpeRI0cB8P3v\nn8eyZcvYbrvtueuuO+nfvz933HF3n7y2d955G3vu+SGOOOIoAL70pa/y1FNP8Ktf3cFpp83s8j4P\nPXQ/Bx98KPvssx8A48dvzssvv8Qtt9zYKTRVVVVRX1+fV32HH34wBx54MK++OpfZs//E0KHDOPro\nYzn00MPzetxN6clrotJnaJJUMR544D4+9alP87Of3cScOS9x8cUXMm7cOD71qRkAXHrpRcyb9yrf\n+94PGDVqNH/606Ocdtop3HLLHdTXT2HNmjVMmbIdRx99LIMHD+bxx//MBRecyxZbbMl22+3Q6Xlm\nzDiMH//4egD+8Iffc+edt/G9783iPe+ZREPDYl555e/rzr/wwnP5179e4+KL/5PBg4dw9dVXcNpp\n3+CWW+6kpqYGgNWr3+GXv7yZ73znfKqqqjjvvHO46qrLOOec89c9zjPP/IWhQ4dy2WVXb/Q1eO65\nZznttFOo6jBg09r67u+rqqr41rfO5GMf2z/x6/q3v72wLhy022OPPfnzn/+40fusWbOWAQMGdDrW\nv39/Xn75RVpaWtb1vWrVSg477CAymQwhTOErXzmx00jNddf9hAceuI8777xnkzX+8pc3c8wxx3H8\n8Sfw5JOPc/nlP2KrrSay++57dHn+Qw89wA9/+P2NPl5VVRU/+tHl7LTTLl3e3pPXRKXP0CSpYowd\nuxknn/xNACZM2Ip//OMVbr/9Vj71qRm88cYb3H//vdx1132MGjUagCOOOIonnnic//7ve5g6dQpj\nxozp9BfhZz7zWZ588n949NGHO4WmLbaYwNe+dvK6nx9//DFGjRrNbru9n5qaGsaO3YwpU7YH4LXX\nFjB79mNcc83P2WGHHQE499zzOfTQA3nssT/wkY98FICWlha+9a1vM3785uue+4Ybru3U36BBgzjj\njLM3OS233Xbbc8MNt1JbW83w4YNZunQlzc2ZTue0j14l1dCwmJEjR27wGA0Nizd6nz322JN77/0N\nH/7w3oQwhTlzXuK+++6mubmZpUuXMHLkKCZMmMiZZ36HyZPfy4oVy7n11pv42teO4xe/uIMxY8YC\nMGJEPVtssWW3NU6dujNHHnkMAFtu+Tmef/457rjj1o2GpunT9173fmxMew1d6clrotJnaJJUMXbY\nYWqnn3fccSq3334Lra2tzJ37CplMhs9//jPrFh0DNDevXTc9lMlkuOGGa3n00Yd56623WLt2Lc3N\naxk0aGCnx50yZbtOP++zz37cccdtHH74wXzgAx/igx+cxrRp06mpqeHVV+dSW1vL9tu/G7rq6oaz\n1VYTefXVueuODRgwcF1gAhg1ajRLljR2ep7Jk7fpdh1T//792WKLLamtraa+fghDh67YIDRtzJtv\nvsFRR30WyI60HH30sRx99Be7PDf7Gm58/dEXv3g8jY0NnHDCsWQyGUaNGs0nP3kQt956E9XV2VGm\nHXecyo47vvue7bDDVI466nDuuefXfOlLXwWy4fEzn/lst7Vv+N7vxJ133rbR8wcNGpQojOWiu9dE\npc/QJEnAypWrqKmp4frrb6aqqvNfbMOGDQXg5ptv5L/+63a+8Y1T2XrrbRg4cCCXX34Ja9c2dzp/\n0KBBnX4eO3YzbrvtLv7ylyd5+umnuOSSWdx22y+48sqf8u4C585aW1s71bF+GKqqolO4Axg4sPPz\ndiWf6bnRo8dwww23rvu5rm440D6C0tDp3MbGhg1GWjoaMGAAM2eew7e+9W0aGxsYNWo0d9/9KwYP\nHsyIESO6vE9tbS3vfW/gtdcWdNtnEuu/zx3lOz3Xk9dEpc/QJKlivPjiC51+/tvfXmDLLSdQVVXF\nttsGMpkMDQ2LN/iLsLY2u6Xd888/x/Tpe68LFK2trbz22nze857ur4bq378/06ZNZ9q06RxyyGH8\n278dxj//+Qrvec/WtLS08OKLf1s3qrJ06RIWLEj2uLnKZ3qupqamy9GXHXecyjPPPMXhhx+x7tjT\nTz/JDjvs1G09NTU1jB49BoCHH36IadOmb/TcTCbD3Ln/YM89c79s/6WX/tbp5xdffIGJE9+z0fPz\nnZ7L5zVR6TI0SaoYixa9yZVXXsbBBx9CjC/zq1/dwSmnvLvG6WMf+wQXXHAuJ57472y7baCxsYFn\nnnmaEAIHHvhxJkzYikcf/T1/+9vzDB06jNtvv5WGhsXdhpvf/va/aWlpYfvtd2TgwIE8+OD9DBw4\nkM02G09dXR3Tpu3FxRdfwGmnncmgQYO55porGTt2Mz784b16/TXIZ3puYw4//POcdNJX+OUvb153\neX2Mczj99LPXnfOTn1zFW28t4uyzzwNgwYL5vPzyi2y//Y40NS3l9ttv4dVX/8nZZ39v3X1uuOFa\ndthhR7bYYgLLly/jlltu4o03FnLQQTNyrvH555/j1lt/wfTpe/PUU0/whz/8nh/+8PKNnp/v9FyS\n10Tlx9CUInMXNvHbJ+excnULgwfU8MkPTGTS+LpilyWVjP33P5DVq1fzla98gZqaGj73uSM7/QX8\n7W9/lxtvvI6rrrqMt99+i7q64ey441T23ntvAI499nj+9a9/ceqpJzNw4EAOPvhQ9tprH5YvX77u\nMbqa8hk6dCg333wjV155GZlMhsmTJ3PRRf9JXV32z+dZZ32Xyy//EWec8U2am9eyyy678sMfXr7u\nCrJSt+OOO/Hd717IT396NT/96Y+ZMGECP/jBJZ2uclu8+G0WLXpz3c+ZTIZf/vJmFiyYT01NLbvu\nujs//vH1jBs3bt05y5Y1cfHF32fx4sUMGzaMELbjmmt+3mmEKOnVc0cc8W/E+BI///lPGTJkKCef\n/E3e//4P9N6LsJ4kr4nKT9X6c+IVprWxMf9/ZZWCuQubuOquF2hYtnrdsZHDBnDioVPLPji1/4s4\nLe9VuzT2Vco9nXzyV9l227Du6rlclHJfPZWWni688LtUV1dz5pnf2WhPhx9+MJ/97JGdpsrKSVtf\nPV5B/tZbyyr6L/okxowZluj19bvnUuL+J+Z1CkwADctWc/8T84pUkSQV3rPP/pUvf/lrxS5DFcLp\nuZRoWrEmp+NSpdnUlVIqX91Ny2VV9nv/9ttvM3r06GKXkQqGppSoG9I/p+NSpbniimuKXYKK5M47\n++brZJR+Ts+lxAF7TmTksM5fSTBy2AAO2HNikSqSJCldHGlKiUnj6zjx0Kk88OR8VqxuZsiAGvb3\n6jlJknqNoSlFJo2v4+TDdkrFFTGSJJUap+ckSZISMDRJkiQlYGiSJElKwNAkSZKUgKFJkiQpAUOT\nJElSAoYmSZKkBAxNkiRJCRiaJEmSEjA0SZIkJWBokiRJSsDQJEmSlIChSZIkKQFDkyRJUgKGJkmS\npAQMTZIkSQkYmiRJkhIwNEmSJCVgaJIkSUrA0CRJkpSAoUmSJCkBQ5MkSVIChiZJkqQEDE2SJEkJ\nGJokSZISMDRJkiQlUFvsAroSQpgOfAvYDRgPzIgx3rPeOd8DjgdGALOBr8UYX+nrWiVJUmUoydAE\nDAGeBa4HfrX+jSGEM4CTgC8Ac4ELgAdDCNvFGNf0ZaFpNHdhE/c/MY+mFWuoG9KfA/acyKTxdcUu\nS5KkoirJ0BRjfAB4ACCEUNXFKd8Azo8x3tt2zjHAm8AM4I6+qjON5i5s4qq7XqBh2ep3j73exImH\nTjU4SZIqWkmGpk0JIUwCxgG/bz8WY2wKITwJfBBDU17uf2Jep8AE0LBsNfc/MY8TD5m6wfndjUo5\naiVJSouyC01kA1Mr2ZGljt5su015aFrR9exmV8e7G5Vy1EqSlCblGJo2popsmMpJTU26LiBs76en\nfQ0f2r/L4yOG9qe2tvNj/vbJrkelHnhyHicftnO3tyeVb0+lKo19pbEnSGdf9lQ+8u2nupoNPr/V\nM+UYmt4gG5A2o/No01jgf3N9sLq6Qb1UVmnpaV9H7r8d895YzltLVq07NmbEID6//3bU1w/pdO7K\n1S1dPsaK1S3U1w/p9vZc+V6VjzT2BOnsy57Sr75+SI8+c7WhsgtNMca5IYQ3gI8CzwOEEOqADwBX\n5fp4TU2raGnJ9G6RRVRTU01d3aAe9zV6aH9OPHQq9z3+KktXrGH4kH4c+KFJjB7an8bGFZ3OHTyg\npsvHGDKghsbGFd3enlS+PZWqNPaVxp4gnX3ZU/lo76unGhtX0K9f8s/cSpQ0VJZkaAohDAG2ITui\nBLB1CGFnoCHGuAC4DDg7hPAK8CpwPvAacHeuz9XSkqG5OT1/uNrl09dWY4fytRk7djrW1WN98gMT\n+ee/mjpNwY0cNoD9PzCR5uZMt7fnyveqfKSxJ0hnX/aUfplM15/hyl1JhiZgd+BRsmuUWoFL2o7f\nCBwXY7w4hDAY+AnZzS0fAz7pHk19a9L4Ok48dCq/fWIeS7u4Oq672yVJKidVra05r51Ok9bGxhWp\nSuC1tdXU1w+h1PvqaisCoMvtCcqlp1ylsa809gTp7MueykdbX13tWZjIyy/PbR09enRvlpQ6Y8YM\nS/T6lupIk1Ksq60I/m9+I1RV07Ty3cHC9u0J3jthRDHKlCSpE69BVJ/ragPNplXNnQITvLuppiRJ\npcDQpD63sQ008z1XkqRCMjSpz9UN6XoDzXzPlSSpkFzTpD53wJ4Tmft6560I6gbVbrCmqW5QLStW\nNXPBjX9hdP1gPr77BLYaO7QYJUuSZGhS39vYVgTAumM11VW8/vZK5sxvBODvC5YSX23g64f4vXWS\npOIwNKkoJo2v4+uHTN3gePuxq379wgYLwxc3rebS259lysR693uSJPU5Q5NK0sYWgK94p5ln4ls8\n/8pixo0azNj6QQYoSVKfcCG4SlJ3C8DXtmRYsGg5z8S3uOquF5i7sKmPKpMkVSpDk0rSAXtOZOSw\nAYnOdT8nSVJfMDSpJLUvFt89jGHooO5nkd3PSZJUaIYmlaz2xeKnfX5XxowYtMlzX397BVf92mk6\nSVLhuBBcJW/rzes484vv57YH5vDG4hW82biKNet9GacLxCVJhWZoUll474R6Tj5sJ5qbM8xd2MRv\nn5jHnHmNLH+nudN57QvEFyxaznP/9zbbbDmCw/eZbHiSJOXN6TmVnfZpu/Gjh2zyvOZMK3PmN3p1\nnSSpVzjSpLKV9HvpGpat5gc3P8P4UUOctpMk9ZihSWWrq++w25jmltZ103b/G99i8KB+1A8bYIiS\nJCXm9JzKVsdtCSaMGUL/2mT/O2eA5avWujmmJCknjjSprHX8Dru5C5u489FXeOVfS2luaU38GA3L\nVvOj2/6X7SeNdNRJkrRRjjQpNSaNr+P0I3flzKN2o35osvVO7VataXHUSZK0SYYmpc6k8XWc9Jmd\nEn8NS0cNy1Zz56OvFKAqSVK5MzQpldZf7zR0UC011VWJ7jtn/hIuvvV/HXGSJHXimialVsf1TsC6\nTTFfnNvAqjUtm7zvnPmNzLr5rxyz/7ZMm7p5oUuVJJUBR5pUMd79Lrv3JZq6W9uS4RcP/t0RJ0kS\n4EiTKlD71F2SUac1zRnOv/FpqoCqKhjQr4aJ4+r8ahZJqkCONKki5Trq1ApkWrNX2c2Z38j5Nz7N\n9254ylEoSaoghiZVtPZRpylbjSDZMvF3vfrGcn7wi2eY/cLrBalNklRaDE2qeO37Ox134JScg1Nz\nppXr7pvDbx77R0FqkySVDkOT1Gba1M0JW9X36L73zJ7niJMkpZyhSerg8H0m92hTTIDr7pvj/k6S\nlGKGJqmDrjbF7F+bfNJuzvxGLr/zeYOTJKWQWw5I61l/U8x2cxc2cdMDc5j35vJN3r9p5RrufPQV\nTj9y10KVKEkqAkeapIQmja/j3GP34OBpE7s9d878JW5JIEkpY2iScjRj+mTeM25ot+e9+sZyp+ok\nKUUMTVIPHP2JKdQN6n52u2nlGm56YE4fVCRJKjRDk9QDk8bX8Y3P7sKUrUZ0e+68N5e7j5MkpYCh\nSeqh9k0xpyTY2+me2fMMTpJU5rx6TsrT4ftM5pJfPsvK1c2bPO+e2fO4Z/Y8qquq2GqzIXzxgO3Z\nrX5IH1UpScqXI01SniaNr+PUI5JN1QFkWlt59Y3lXHTz0/zfgsYCVydJ6i2GJqkXtE/VfenAKYnv\ns2pNhm9f/Wf++bpX10lSOXB6TupF06ZuzltLVnHP7HmJzn9nTYbvXv/Uup8HD6jh8/u9l2lTNy9U\niZKkHnKkSeplM6ZPTrQBZldWrm7huvvm+OW/klSCDE1SAcyYPpnpO43r8f1/8aB7O0lSqTE0SQVy\n7AHb86UDpzAswSaY61vTDF/54aOOOElSCXFNk1RA06Zuvm590m8e+0fitU4AzS2tXHffHN5asooZ\n0ycXqkRJUkKONEl9pKdrne6ZPc/vr5OkEmBokvrQjOmTOecLuzNxs+6/8Lejy+98tkAVSZKScnpO\n6mOTxtdx7rF7UFtbTX39EK77zfP85rG5m7xP08pmTrnsT/zH53Zh0vi6PqpUktSRI01SkR2692R2\n2WZUt+ctf6eZH/ziaafqJKlIDE1SCTjlsJ0TrXdqzjhVJ0nFYmiSSsSM6ZMTfQ1L08pmZt38dB9U\nJEnqqGxDUwihOoRwfgjhnyGElSGEV0IIZxe7Likf06Zunmiq7u+vNfHli93HSZL6UtmGJmAm8FXg\n68AU4HTg9BDCSUWtSsrTKYftzLZbdr/YuyWT3cfpiv96rg+qkiSVc2j6IHB3jPGBGOP8GONdwEPA\nHkWuS8rbzKN2TzTiBPDsK4s58dI/uEBckgqsnEPT48BHQwjvBQgh7AxMA+4valVSLznlsJ0TB6dV\nazKcf+PTTtdJUgGVc2iaBdwOzAkhrAGeAS6LMf6yuGVJvSfpVF276+6b43fWSVKBlPPmlp8DjgSO\nAF4CdgEuDyG8HmP8RdIHqakp59y4ofZ+0tRXGnuC5H2d/cU9uPDGp4gLkk2/tX9n3Sv/WsqXPrVD\n3nXmotLfq3JiT+Uj336qq6G2Nl2vSbFUtba2FruGHgkhzAe+H2O8psOxs4B/izFun/BhyrN5VaQL\nrnuCJ196M6f7fHyPrTj5c+8rUEWS+lBVT++4aNGi1rFjx/ZmLWmU6PUt55GmwWwYejLkOOXY1LSK\nlpZMrxVVbDU11dTVDUpVX2nsCXLv68RDp7LTNqP42T0vJX6Oh56az7yFSzjrC31zfYTvVfmwp/LR\n3ldPNTauoF+/Fb1YUfrU1w9JdF45h6Z7gbNCCAuAF4Fdgf8Ars3lQVpaMjQ3p+cPV7s09pXGniC3\nvj64/Tg+uP04zr3uCRa8tTLRfeKCJk678jFmnTAtnzJz4ntVPuwp/TIZfD16STlPcp4E/BdwFdk1\nTRcDPwa+U8yipL5w3pf2zGmB+KIlqzn1yj8VsCJJSr+yHWmKMa4Avtn2S6o4M4/and889g/umT0v\n0fmNy5s5/qJHOOuY3Zk0PnngkiRllfNIk1TxZkyfzPUz9028n1OmFc6/8Wl3EZekHjA0SSmQy0aY\nkN1F3Ok6ScqNoUlKiVMO25npO41LfH7j8maOm/UIv3nsHwWsSpLSw9AkpcixB2zPlw6cktN97pk9\nj1k3P12giiQpPQxNUspMm7o553xh95zu8/fXmph5zewCVSRJ6WBoklJo0vg6rp+5L8MGJf8jvmjJ\nak740SMFrEqSypuhSUqxy7/xkZz2c1rTDMfNesSr6ySpC4YmKeVmHrV7TgvEwavrJKkrhiapAvRk\ngXjj8maDkyR1YGiSKsS0qZtz/cx9mTBmcOL7NC5v5sRLXeckSWBokipOrt9bt2pNdp2TJFU6Q5NU\ngWYetTsHT5uY032Om/WI+zlJqmiGJqlCzZg+uUf7ObktgaRKZWiSKlj7fk65rHNa0wzHO10nqQIZ\nmiRx3pf2zCk4ZXCdk6TKY2iSBOS+QBwMTpIqi6FJ0jo92QjT4CSpUhiaJHVy7AHb57xA3OAkqRIY\nmiRtoH3KeDGOAAAgAElEQVSB+NgRAxLf57hZj3Ddf79YwKokqbgMTZI2atYJ03JaIP7HZxdyyLfu\nLmBFklQ8hiZJm3Tel/bMaZ1TcwaOueDhAlYkScVhaJLUrWMP2L5HO4hLUpoYmiQlMmP6ZK+sk1TR\nDE2SEjv2gO3ZZZtROd3H4CQpLQxNknJyymE7O1UnqSIZmiTlbMb0yVw/c9+c7mNwklTuDE2Seuz6\nmftSP7Q28fkGJ0nlzNAkKS+XnLQXwwYl/ygxOEkqV4YmSXm7/BsfccRJUuoZmiT1iktO2osZ0ycl\nPt/gJKncGJok9ZpD957Mx/fYKvH5BidJ5cTQJKlXnfy591HnGidJKWRoktTrrjx1Xwb1T37+iZca\nnCSVPkOTpIK46pv7Jv6AWbUGZl4zu6D1SFK+DE2SCubamftSm/BTZtGS1U7VSSpphiZJBfXT05OP\nOIFrnCSVLkOTpIK71q9ckZQChiZJfcLvqpNU7gxNkvqMwUlSOTM0SepTBidJ5crQJKnPGZwklSND\nk6SiMDhJKjeGJklFY3CSVE4MTZKK6vocNsAEg5Ok4jE0SSq6n57uiJOk0mdoklQSnKqTVOoMTZJK\nhsFJUikzNEkqKQYnSaXK0CSp5OQanM697okCVSJJ7zI0SSpJuQSnBW+tLGAlkpRlaJJUsnIJTk7T\nSSq0sg5NIYTNQwi/CCG8HUJYGUJ4LoSwa7HrktR7DE6SSkXZhqYQwghgNrAa+ASwHXAq0FjMuiT1\nPoOTpFJQW+wC8jATmB9jPL7DsXnFKkZSYV0/c9/Egei4WY/kvJhckrrT45GmEMIjIYQpG7lt2xBC\nof+5dxDwdAjhjhDCmyGEv4YQju/2XpLK1sHTJiY+93hHnCT1snym5z4C1G3ktjpgrzweO4mtga8B\nEfg4cA1wRQjhqAI/r6QimTF9MsMGJfvYygA/v/+lwhYkqaLkOz3XupHjHwIW5fnY3akGnooxntP2\n83MhhB3IBqmbkz5ITU3ZLuvqUns/aeorjT1BOvvqi56uOnVfjrng4UTnPvb8G3z54B3zfk7fq/KQ\nxp4g/36qq6E2l2/F1kZVtbZuLPdsKIRwJnBm249DgFVk/0HX0QCyYezqGOPJvVHkRmp5FXgoxviV\nDsdOAM6KMU5I+DDJm5dUUg469e7E5957yacLWInUJ6p6esdFixa1jh07tjdrSaNEr2+uI02PA5e0\nPfh3gNuA19Y7Zw3wMnBvjo+dq9lAWO9YIMfF4E1Nq2hpWT/3la+ammrq6galqq809gTp7Ksve7rp\n7P0SjzgddOrd3HT2fj1+Lt+r8pDGnuDdvnqqsXEF/fqt6MWK0qe+fkii83IKTTHGPwJ/BAghtAI/\nizG+nnN1veM/gdlto193AB8Ajge+nMuDtLRkaG5Ozx+udmnsK409QTr76quecrmi7pgLHs77ijrf\nq/KQxp7ykcng69FLejzJGWM8r4iBiRjj08AhwOeBF4CzgG/EGH9ZrJok9b1cgtAJP/KKOkk91+OF\n4CGEarIjO4cBWwID1zulNcY4OY/auhVjvB+4v5DPIan0bbtlHX9/ranb89Y090ExklIrn+X0F5G9\nzL8f8Chw93q/7sm7OklKYOZRuyc+1x3DJfVUPlsO/Btwbozx/N4qRpJ6yh3DJRVaPiNNA8leTSdJ\nJcHvqJNUSPmEplvIfpWJJJWMXILTiZcanCQll8/03BPABSGEzYDfAUvWPyHGeFcejy9JPXLOF3bn\n/Buf7va8VWv6oBhJqZFPaPpF238nAp/r4vZWoCaPx5ekHpk0vo5dthnFs68s7vZc1zdJSiqf6blJ\n3fzaOu/qJKmHTjls58Tnur5JUhI9HmmKMeb0dSWS1Ne8ok5Sb8pnc8utujsnxji/p48vSb0hl+B0\n/KxHuNbgJGkj8lnT9CrZdUub4pomSUWXdH2T384laVPyWdN0CHDoer+OJ/vlufOBI/OuTpJ6geub\nJPWGfNY03b2Rm34eQrgU2Bu4vaePL0m9yfVNkvKVz0jTptwPHFGgx5akHnHHcEn5KFRo+hDwToEe\nW5J6zOAkqafyuXruii4O9we2Az4M/Kinjy1JhTR9p3E89vwbxS5DUpnJZ6TpoC5+7Q2sAb4OfDvv\n6iSpAI49YPvEH36ONklql89C8Em9WYgk9aVrc1gYfswFD3PvJZ8ucEWSSl2h1jRJUsnLZX3TQadu\n7IJhSZUin80tCSG8j+w03IeBkUAD8Bjw/Rjjs/mXJ0mFlctWBJIqW49HmkII04H/Ad4P3AZ8p+2/\n7wf+J4Tw4V6pUJJKxDEXPFzsEiQVUT7Tc7OAPwDbxBi/GWO8KMb4TWAb4I9tt0tSyXMbAklJ5BOa\n3gdcEWNs7ngwxtgCXAHsmk9hktSXcglOs194vYCVSCpV+YSmFcDYjdy2WdvtklQ2kgan6+6bU+BK\nJJWifELTvcBFIYT9Oh5s+/kHwD35FCZJxVCb8FPRaTqp8uQTmk4FXgUeDCE0hhBiCKEReLDt+Gn5\nlydJfeunp7u+SVLXehyaYoyNwAeBQ4CfAX8CfgrMAD7UdrsklZ1c1jfNvGZ2ASuRVEry2qcpxpgh\nOw3nVJykVEm6f9OiJav7oBpJpSCffZo+GkI4diO3fTGEsE/Py5Kk8uE0nVQZ8lnTdAHZq+S6Mqbt\ndkkqW+7fJKmjfELTDsDTG7ntr223S1JZMzhJapdPaGoFhm/ktnqgJo/HlqSScdPZ+3V/kqTUyyc0\nPQmcGEKo6niw7eevt90uSRXF0SYpvfIJTecCHwKeDyGcGkI4MoRwGvBc2/Gze6NASSoF917y6cTn\nGpykdMpnn6b/AT4KNAEXATeT/ZLepcBHY4xP9EqFklQinKaTKls+I03EGGfHGKcBw4AtgboY4/QY\n4wa7vYUQtgoh5LUvlCQVW9KF4Y42SemTV2hqF2NcFWN8Pca4sqvbQwg1wFxgp954PkkqBwYnKV16\nJTQlVNX9KZJU+nLZhkBSevRlaJKk1HCaTqo8hiZJ6qFdthmV6DyDk5QOhiZJ6qFTDts58bkGJ6n8\nGZokKQ+ub5Iqh6FJkvJ08LSJic5ztEkqb30VmlqBPwLL+uj5JKnPzJg+OfG5BiepfPVJaIoxZmKM\n+8QY/68vnk+S+lou03SzX3i9gJVIKpQe79AdQsiQHUHqSivZr1N5FrgsxnhvT59HksrF9TP3TTSS\ndN19c5g2dfM+qEhSb8pnpOlbwGvAK8B/AmcClwH/AF4HrgL6Ab8JIRyRZ52SlCpO00nlJ5/QNBJ4\nGpgSYzwtxnhxjPFUILQdHxRj3Au4Ezgj/1IlqfR5NZ2UXvmEpi8BP4sxdpqia/v5p8AX2g7dCkzJ\n43kkqay4W7iUTvmEpsHAVhu5bSIwsO33y4E1eTyPJJUdg5OUPj1eCA7cA8wKISwH7o0xLgshDAM+\nDcwCftN23k5k1z1JkiSVrXxGmr4O/AG4GVgSQngHWALcBDwCnNR23nyyi8QlqaI42iSlS49HmmKM\nTcChIYTtgD2AccBC4C8xxpc7nHdX3lV2I4RwJnAh2e0Nvlno55OkpA6eNpF7Zs/r9rzjZj3iInKp\nxOUzPQdAW0B6udsTCySE8H7gy8BzxapBkjZmxvTJiUKTpNKXV2gKIQwBvgh8mOwWBA3AY8CNMcYV\neVfX/fMPJTs9eDxwTqGfT5J6Iumml442SaWtx2uaQggTgOeBK8juzZRp++8VwHNttxfaVWQXobsg\nQJIkFVQ+C8Evbfvv9jHGXWOMn4wx7grsQPZrVC7Ju7pNaNtlfBdcZC6pDLgoXCp/+UzPfQz4aowx\ndjwYY4whhHOAa/KqbBNCCFuS/cqWj8UY1+bzWDU1ffKdxX2mvZ809ZXGniCdfaWxJ+i9vm46ez+O\nueDhbs87btYj3HT2fnk9V3fS+F6lsSfIv5/qaqitTddrUiz5hKZaYNVGblsF1OTx2N3ZDRgDPBNC\nqGo7VgPsFUI4CRiw/k7lG1NXN6hAJRZXGvtKY0+Qzr7S2BP0bV83P/R3Tv7c+wr+PGl8r9LYUz7q\n64dQXz+k2GWkQlVra6JssYEQwoNAPfDxGOOSDseHA78DGmKM+/dKlRs+9xCyu453dAPZq/hmddzy\noButTU2raGnJ9GZ5RVVTU01d3SDS1Fcae4J09pXGnqD3+0oy2gQUdLQpje9VGnuCdX1VdX9m12Kc\n2zp27NjeLCl16uuHJHp98xlpOhX4EzA/hPAI8CYwFvgosBY4Lo/H3qS2K/Ne6ngshLACWJxDYAKg\npSVDc3N6/nC1S2NfaewJ0tlXGnuC3usr6dV0x1zwcMGvpkvje5XGnvKRyeDr0Ut6PMkZY/wbsDNw\nLbA5sG/bf38G7Nx2e1/q2ZCZJJUwF4ZLpSOvfZpijAuAktiBO8bo5iaSykbS0SZJpSOn0BRCeIHk\nIzqtMcadcy9JkiqDm15K5SXXkaZncBpMkvrcbx77BzOmTy52GVJFyyk0xRi/WKA6JKkiJR1tumf2\nPEOTVGTudiVJReZu4VJ5MDRJUhkxOEnFY2iSpBLgQm+p9BmaJKlEOE0nlTZDkyRJUgKGJkkqIY42\nSaXL0CRJJcb1TVJpMjRJUplytEnqW4YmSSpBjjZJpcfQJEllzNEmqe8YmiSpRLkoXCothiZJkqQE\nDE2SVMIcbZJKh6FJkkpc/dDaROcZnKTCMjRJUom75KS9Ep9rcJIKx9AkSWUgly0IDE5SYRiaJEmS\nEjA0SVKZcMNLqbgMTZJURryaTioeQ5MklRlHnKTiMDRJUhlKEpwcbZJ6l6FJklLM4CT1HkOTJElS\nAoYmSSpTLgqX+pahSZIqgMFJyp+hSZLKmFfSqTsNDYvJZDLFLiMVDE2SVOacptOm/PGZf9DQ0FDs\nMlLB0CRJUooNGjyk2CWkhqFJklLA0Sap8AxNkpQSrm+SCsvQJEkVxtEmqWcMTZKUIklHm4654OEC\nVyKlj6FJkirUQafeXewS1AcWL37bLQd6iaFJklLGtU3qqLVlbbFLSA1DkyRVMKfp0m/02PFUV/vX\nfW/wVZSkFHK0Sep9hiZJqnBeTSclY2iSpJTKZbTJ4CR1z9AkSSnmNJ3UewxNkpRyfsVKZXPLgd5j\naJIkrWNwSh+3HOg9hiZJklLMLQd6j6+iJFUA1zZJ+TM0SVKFcG2TlB9DkyRJUgKGJkmqILmMNjni\nJHVmaJKkCuOml5XFLQd6j6FJkqQUc8uB3lNb7AJ6KoRwJnAIMAVYBTwOnBFj/HtRC5OkMnDT2ftx\nzAUPF7sM9QG3HOg95fwqTgf+H/ABYD+gH/BQCGFQUauSpJRxik7KKtuRphjjAR1/DiF8EVgE7Ab8\nuRg1SVJaHTfrEfd6UsUr55Gm9Y0AWoGGYhciSeXg3ks+XewSpLKSitAUQqgCLgP+HGN8qdj1SFK5\nuOns/RxBkhIq2+m59VwNbA9My/WONTWpyI3rtPeTpr7S2BOks6809gTp7KsnPR036xFuOnu/QpWU\ntzS+T5B/P40Nb1NdDbW16XpdiqHsQ1MI4UrgAGB6jHFhrvevq0vnuvE09pXGniCdfaWxJ0hnX7n2\ndMwFD5f8tF4a36d89OsHI0YMpr5+SLFLKXtlHZraAtOngb1jjPN78hhNTatoaUnPpl81NdXU1Q1K\nVV9p7AnS2Vcae4J09rV+T7lsQdDYuKLA1fVMGt8neLevnhpWN4amplX071+a71spSBooyzY0hRCu\nBj4PHAysCCFs1nbT0hjjO0kfp6UlQ3Nzev5wtUtjX2nsCdLZVxp7gnT21bGn62fum2h7gVJ/DdL4\nPuWjtbWV5uZWX5NeULahCTiB7NVyf1jv+LHATX1ejSRViI7BykXkqiRlG5pijK5ok6Qic/8mVRKD\nhyRpHQNQ+ixe/DYNDYv90t5eYGiSJHVicEqXQQMH8tScN2locO/nfJXt9JwkSereuM23KnYJqeFI\nkyRJUgKGJknSBnKZojtu1iOJtiqQyp2hSZLUpetn7rvuVxIGJ6WdoUmSJCkBF4JLkpRiixZlv5Y1\nk9msmzPVHUOTJEkplmleW+wSUsPQJElSirVvOVBd7YqcfPkKSpK6lWQxuJtiKu0caZIkJdIxFHV1\npVxXxwxSShNHmiRJOcllawG3IVCaGJokSZIScHpOkqQUW758advvxhe1jjQwNEmSlGK7bzuGkSNH\nMXLkyGKXUvYMTZIkpdjIkaMYPXp0sctIBdc0SZJykssVcV49pzRxpEmSlDPDkCqRI02SJEkJONIk\nSeoVue7J5GiVyo0jTZKkvPVkE0s3vuwbDQ2Lefvtt9f9ymQyxS6pbDnSJElSij3997cYOnQNkN2z\n6eN7TvFquh4yNEmSlGJDhw6nboR7NPUGp+ckSZISMDRJkiQlYGiSJOWtJ1fCefWcyo1rmiRJvcIQ\npLQzNEmSlGKLFi1k+YoVAKxc0URDQ/8uzxs5ciTV1U5AbYqhSZKkFMs0r6W1JbvlwKCBA3nptXeo\nfn1hp3PciiAZQ5MkSSk2bvOt3HKglzgOJ0mSlIChSZIkKQGn5yRJBZfv98x5ZZ5KgSNNkqSC6o0v\n5vXLfVUKHGmSJCnFli9fmvCc8YUvpswZmiRJSrHdtx3DyJGjujlrPCNHeoVddwxNkiSl2MiRo9x/\nqZe4pkmSJCkBQ5MkqaB648o3r55TKXB6TpJUcIYepYEjTZIkSQkYmiRJkhIwNEmSJCVgaJIkSUrA\n0CRJkpSAoUmSJCkBQ5MkSVIChiZJkqQE3NxSkqQUe+edVaxatarYZZSU2tpa+vXrl/v9ClCLJEkq\nEU+/vBBYWOwySkrdgBb23euDOd+v7ENTCOFE4DRgHPAccHKM8S/FrUqSpNIwYswWxS6h5FSterNH\n9yvrNU0hhM8BlwDnAu8jG5oeDCGMLmphkiQpdco6NAH/AfwkxnhTjHEOcAKwEjiuuGVJkqS0Kdvp\nuRBCP2A34Pvtx2KMrSGEh4HcJyolSSXhuFmPFLuELl0/c99il6AiK+eRptFADbD+xOSbZNc3SZLK\nTKkGJijt2tQ3ynakaROqgNakJ9fUlHNu3FB7P2nqK409QTr7SmNPkM6+0thTX6it7fvXK9/36J2V\nK6iq6qViUmJAa0uP3styDk1vAy3AZusdH8uGo08bVVc3qDdrKhlp7CuNPUE6+0pjT5DOvtLYUyHV\n1w8pdgk5m7bTKKpMTZ0MGzapR+9l2YamGOPaEMIzwEeBewBCCFVtP1+R9HGamlbR0pIpTJFFUFNT\nTV3doFT1lcaeIJ19pbEnSGdfaeypLzQ2rujz52x/r3pqeN1oqqsNTevr+F4mDVBlG5raXArc2Bae\nniJ7Nd1g4IakD9DSkqG5OX0fGGnsK409QTr7SmNPkM6+0thTIZXja5XJtJLJJF61ok0o68nsGOMd\nwKnA94D/BXYCPhFjfKuohUmSeqSUr1Ar5drUN8p9pIkY49XA1cWuQ5LUO3oaTmprq6mvH0Jj44qy\nHBFS6SvrkSZJkqS+YmiSJElKwNAkSZKUgKFJkiQpAUOTJElSAoYmSZKkBAxNkiRJCRiaJEmSEjA0\nSZIkJWBokiRJSsDQJEmSlIChSZIkKQFDkyRJUgKGJkmSpAQMTZIkSQkYmiRJkhIwNEmSJCVgaJIk\nSUrA0CRJkpSAoUmSJCkBQ5MkSVIChiZJkqQEDE2SJEkJGJokSZISMDRJkiQlYGiSJElKwNAkSZKU\ngKFJkiQpAUOTJElSAoYmSZKkBAxNkiRJCRiaJEmSEjA0SZIkJWBokiRJSsDQJEmSlIChSZIkKQFD\nkyRJUgKGJkmSpAQMTZIkSQkYmiRJkhIwNEmSJCVgaJIkSUrA0CRJkpSAoUmSJCkBQ5MkSVIChiZJ\nkqQEDE2SJEkJGJokSZISMDRJkiQlYGiSJElKwNAkSZKUQG2xC+iJEMJE4BxgX2Ac8C/gFuDCGOPa\nYtYmSZLSqSxDEzAFqAK+DPwD2BG4FhgMnF7EuiRJUkqVZWiKMT4IPNjh0KshhB8BJ2BokiRJBZCm\nNU0jgIZiFyFJktIpFaEphLANcBJwTbFrkSRJ6VRS03MhhB8AZ2zilFZguxjj3zvcZwvgt8DtMcbr\nc33OmppU5MZ12vtJU19p7AnS2Vcae4J09mVP5SPffqqrq6iuruqlaipbVWtra7FrWCeEMAoY1c1p\n/4wxNredvznwKPB4jPHYQtcnSZIqV0mFply0jTA9AvwFODrGWJ6NSJKkslCWoSmEMB74E/Aq8AWg\npf22GOObRSpLkiSlWEmtacrBx4Gt234taDtWRXbNU02xipIkSelVliNNkiRJfS1dlxhIkiQViKFJ\nkiQpAUOTJElSAoYmSZKkBAxNkiRJCRiaJEmSEijXfZp6VQhhInAOsC8wDvgXcAtwYYxxbTFry0UI\n4UTgNLI9PAecHGP8S3Gr6rkQwpnAIcAUYBXwOHBGx+8eLHdtPV4IXBZj/Gax68lH29caXQR8EhgM\n/B9wbIzxr0UtrIdCCNXAecC/kf0z9TpwQ4zxgqIWlqMQwnTgW8BuwHhgRozxnvXO+R5wPDACmA18\nLcb4Sl/XmtSmegoh1JL9M/VJsnv5LQUeBmbGGBcWp+JkkrxXHc79CfBl4N9jjFf0XZWVzZGmrClk\nN8f8MrA98B/ACWT/4JWFEMLngEuAc4H3kQ1ND4YQRhe1sPxMB/4f8AFgP6Af8FAIYVBRq+olIYT3\nk/1/7rli15KvEEL7X7argU8A2wGnAo3FrCtPM4GvAl8n+xlxOnB6COGkolaVuyHAs8CJZDcA7iSE\ncAZwEtle9wBWkP3s6N+XReZoUz0NBnYhG3jfR/YfXgG4uy8L7KFNvlftQggzyL5X/+qjutTGkSYg\nxvgg8GCHQ6+GEH5ENjidXpyqcvYfwE9ijDcBhBBOAA4EjgMuLmZhPRVjPKDjzyGELwKLyP4r7M/F\nqKm3hBCGAjeT/df9OUUupzfMBObHGI/vcGxesYrpJR8E7o4xPtD28/wQwpFk/7IqG231PwAQQujq\nq+6/AZwfY7y37ZxjgDeBGcAdfVVnLjbVU4yxiWxwX6ct6D4ZQtgyxvhanxWaowTvVfv3rl5Btsf7\n+646gSNNmzICaCh2EUmEEPqRDRK/bz/W9gXGD5P94E+LEWT/9VUW70s3rgLujTE+UuxCeslBwNMh\nhDtCCG+GEP4aQji+23uVtseBj4YQ3gsQQtgZmEaK/qIKIUwiO/XY8bOjCXiSdH52LCl2IfloC1I3\nARfHGF8udj2VyNDUhRDCNmSHq68pdi0JjSb7nXvrf1nxm2Q/EMte24fFZcCfY4wvFbuefIQQjiA7\nfXBmsWvpRVsDXwMi2e+GvAa4IoRwVFGrys8s4HZgTghhDfAM2bVnvyxuWb1qHNkwkebPjgFk38tb\nY4zLi11PnmYCa2KMVxa7kEqV6um5EMIPgDM2cUorsF3HhcVtQ5+/BW6PMV5f4BILrf1LjNPgarLr\nzaYVu5B8hBC2JBv+PlZOFxkkUA08FWNsn2p8LoSwA9kgdXPxysrL54AjgSOAl8gG3ctDCK/HGH9R\n1MoKLxWfHW2Lwu8k28vXi1xOXkIIuwGnkF2npSJJdWgCfgT8vJtz/tn+m7arfx4hO5rx1UIW1sve\nBlqAzdY7PpYN/wVZdkIIVwIHANNL/eqXBHYDxgDPdFizUAPs1bbuYkDb1Gq5WQisP13wMnBoEWrp\nLRcD348x3tn284shhPeQHSFMS2h6g2xA2ozOnxVjgf8tSkW9pENgmgDsm4JRpg+T/exYEEJoP1YD\nXBpC+PcY49ZFq6yCpDo0xRgXA4uTnNs2wvQI8Beyi6fLRoxxbQjhGeCjQPtlt1VtP5f1pahtgenT\nwN4xxvnFrqcXPAxMXe/YDWQDxqwyDUyQvXIurHcsUN6LwQez4WhLhhQta4gxzg0hvEH2s+J5gBBC\nHdkrVq8qZm356BCYtgb2iTGW81Wc7W4CfrfesYfajnc3OKBekurQlFQIYTzwB+BVslfLjW1P8jHG\nchmpuRS4sS08PUX2arrBZP9CLkshhKuBzwMHAytCCO0jaUtjjO8Ur7KeizGuIDvVs04IYQWwuMwX\ndv4nMLtt36k7yP6lezzZLRXK1b3AWSGEBcCLwK5k/1xdW9SqchRCGAJsQ3ZECWDrtkXtDTHGBWSn\ni88OIbxC9jPwfOA1SvgS/U31RHY/rV+RnU79FNCvw2dHQylPiyd4rxrXO38t8EaM8f/6ttLKZWjK\n+jjZf5FsDSxoO9Y+p19TrKJyEWO8o21Ppu+RHWp/FvhEjPGt4laWlxPIvgd/WO/4sWT/dZUW5Tq6\ntE6M8ekQwiFkF9yeA8wFvlHmi6ZPIhsgriI7XfU68OO2Y+Vkd+BRsv+ftZLdzw3gRuC4GOPFIYTB\nwE/IXmX2GPDJGOOaYhSb0KZ6Oo/s1ZytZD8H4d3P832AP/VppbnZ5HvVxfll/9lRbqpaW33NJUmS\nupOauXlJkqRCMjRJkiQlYGiSJElKwNAkSZKUgKFJkiQpAUOTJElSAoYmSZKkBAxNkiRJCRiaJEmS\nEjA0SSkUQrghhPBCsesohBDC3iGETAhh12LXIqmyGJqkdGr/7qq0SnNvkkqUoUlSQYUQ+he7Bknq\nDbXFLkBS4YUQdiT7jenTgGbgd8A3Y4wLOpxTB1wNHAysBK4DGoAfxhgT/QMrhHADsBtwOvADYApw\nZAjhAeAi4GPABGAR8ABwRoyxqcP95wL/Dbzc9hgjyH7r+/ExxsWbeN79gV8BF8cYz0tSqyTlypEm\nKeVCCBOAPwH1wJHAV4FdgT+EEIZ0OPUG4ADgNOCLZAPPKeQ2FdYKbA5cDlwK7A88Cwwm+3lzZtux\ns4C9gLu6eIyD2n59ve359wb+3yb6OxT4DXC2gUlSITnSJKXff5D9s/7xGOMSgBDCs8BLZMPRVSGE\n7YAZwFExxlvbznkQmNOD5xsBfCLG+PR6x09s/00IoQZ4FXgshLBNjPGVDudVAQfFGJvbzp1ENmxt\nIP31eVsAAAK3SURBVIRwNHAtcFKM8Wc9qFWSEnOkSUq/DwOPtAcmgBhjBJ5ruw1gD7KjRPd2OKfT\nzzlY3EVgIoRwdAjhryGEZcBa4M9tN2273ql/bA9MbV4C+oUQxnY4VkV2xOxa4FgDk6S+YGiS0q8e\neLOL428CI9t+Pw5YG2Nctt45i3rwfBs8VwjhEOBG4AngcOD/t3e/LlJFUQDHv2MQF1ZsGgRBEI7J\najDpRIMoFvEHWDbo32ASk8GgQaNiFcOCLG4QVCwbFrbICYYRWbGsChoUljHcmXGYfeKbH87A+P2U\n4d25770bzzvv3POOA2cpwc+egelfBo5/dn4H550DWsCzEdYoSUMzaJLm3xawv2L8QOc/gI+UbM7e\nijnDqqqBOg+sZ+a1zFzJzDXg8wjX7r/HFWAbeB4Ri2NcS5JqMWiS5t9roBkR+7oDERHAMeBVZ2iN\nkvU50zenQSnInoQFfmeMui4xXr+lT0CTki1biYiFMa4lSX9lIbg0/+5QCr5XI+IWJYC5SSnEfgiQ\nmW8j4ilwt7OjrgUsUV6JTaKR5CpwLyJuAG8ou/RODXF+o+o4MzcjoknZHbgcEacz88cE1itJO5hp\nkuZXGyAzP1C27W8Bj4H7wDpwMjO/982/Sin8vg08At5R2hB8HeW+Ax5Q+kRdp/RTOghc+MO5VecP\njvWOM7NFCcCOAk8iwodBSf9Eo932awSSqkXES0qBeHPWa5GkWfOJTBLQaxJ5CNgAFimNME9Q+jdJ\n0n/PoElS1zfgMnAE2E1pbHkxM5cBImIXO2uLejJzexqLlKRZ8fWcpFoi4gWlNqpKGzicme+nuCRJ\nmiozTZLqWgIG+zj125zWQiRpFsw0SZIk1WDLAUmSpBoMmiRJkmowaJIkSarBoEmSJKkGgyZJkqQa\nDJokSZJqMGiSJEmqwaBJkiSphl/NZ2QU3u9r6gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f60d3019dd8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.jointplot(x=\"log_rank\", y=\"log_count\", data=tokens_pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Word2Vec Synonyms\n",
    "\n",
    "What words are related?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from pyspark.ml.feature import Word2Vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "w2v = Word2Vec(vectorSize=100, \n",
    "               minCount=5, \n",
    "               numPartitions=4,\n",
    "               seed=20,\n",
    "               inputCol=\"tokens\",\n",
    "               outputCol=\"result\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "w2v_model = w2v.fit(comments_tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42229"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_vectors = w2v_model.getVectors()\n",
    "w2v_vectors.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>science</td>\n",
       "      <td>0.645702</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>science,</td>\n",
       "      <td>0.579765</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>msc</td>\n",
       "      <td>0.570287</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>programmer)</td>\n",
       "      <td>0.552826</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>engineering</td>\n",
       "      <td>0.545989</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>80s.</td>\n",
       "      <td>0.516814</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>calculus</td>\n",
       "      <td>0.497151</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>science.</td>\n",
       "      <td>0.494038</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>math</td>\n",
       "      <td>0.493884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>designing</td>\n",
       "      <td>0.486894</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          word  similarity\n",
       "0      science    0.645702\n",
       "1     science,    0.579765\n",
       "2          msc    0.570287\n",
       "3  programmer)    0.552826\n",
       "4  engineering    0.545989\n",
       "5         80s.    0.516814\n",
       "6     calculus    0.497151\n",
       "7     science.    0.494038\n",
       "8         math    0.493884\n",
       "9    designing    0.486894"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_model.findSynonyms('computer', 10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's some clean up to be done still. \"science\", \"science,\", \"science?\", and \"science.\" really should all be the same token.\n",
    "\n",
    "But blindly removing punctuation may not be right for your dataset. What would have happened to \".net\", \"node.js\", and \"c#\" when we look for synonyms for \"java\" below? How can you clean up \"c++\" and \"c++,\"?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>compiled</td>\n",
       "      <td>0.846565</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>c++.</td>\n",
       "      <td>0.819268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>c++</td>\n",
       "      <td>0.817617</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>c#</td>\n",
       "      <td>0.813154</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>.net</td>\n",
       "      <td>0.808381</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>node.js</td>\n",
       "      <td>0.780349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>java,</td>\n",
       "      <td>0.756420</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>python,</td>\n",
       "      <td>0.755731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>jvm</td>\n",
       "      <td>0.755234</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>rails</td>\n",
       "      <td>0.748189</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       word  similarity\n",
       "0  compiled    0.846565\n",
       "1      c++.    0.819268\n",
       "2       c++    0.817617\n",
       "3        c#    0.813154\n",
       "4      .net    0.808381\n",
       "5   node.js    0.780349\n",
       "6     java,    0.756420\n",
       "7   python,    0.755731\n",
       "8       jvm    0.755234\n",
       "9     rails    0.748189"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_model.findSynonyms('java', 10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Hacker News is a site that discusses startups and stories related to technology. Similar to \"java\" above, the synonyms you get back depend on your training data. Sometimes what you get is intuitive but sometimes you get interesting things like \"mouth\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>word</th>\n",
       "      <th>similarity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>start-up</td>\n",
       "      <td>0.741585</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>novelty</td>\n",
       "      <td>0.728111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>clique</td>\n",
       "      <td>0.726317</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>sf,</td>\n",
       "      <td>0.714907</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>high-paying</td>\n",
       "      <td>0.706608</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>clueless</td>\n",
       "      <td>0.706566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>interview</td>\n",
       "      <td>0.706357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>bunny</td>\n",
       "      <td>0.706212</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>career,</td>\n",
       "      <td>0.705742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>dude</td>\n",
       "      <td>0.705565</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          word  similarity\n",
       "0     start-up    0.741585\n",
       "1      novelty    0.728111\n",
       "2       clique    0.726317\n",
       "3          sf,    0.714907\n",
       "4  high-paying    0.706608\n",
       "5     clueless    0.706566\n",
       "6    interview    0.706357\n",
       "7        bunny    0.706212\n",
       "8      career,    0.705742\n",
       "9         dude    0.705565"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "w2v_model.findSynonyms('founder', 10).toPandas()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So what comments have \"founder\" and \"bunny\" in them?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+------------+\n",
      "|comment_text|\n",
      "+------------+\n",
      "+------------+\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from pyspark.sql.functions import array_contains\n",
    "\n",
    "comments_tokens\\\n",
    "    .filter(array_contains(comments_tokens[\"tokens\"], \"founder\"))\\\n",
    "    .filter(array_contains(comments_tokens[\"tokens\"], \"bunny\"))\\\n",
    "    .select(comments_tokens[\"comment_text\"])\\\n",
    "    .show(3, truncate=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are none! This is an important lesson. Word2vec is not a measure of co-occurence like TF-IDF and other techniques. We've reduced the high-dimension co-occurence space down to only 100 dimensions and things that don't occurr together may project down to a lower space in a similar way ie have a similar vecor."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "0.125 - PySpark Python3 Nano",
   "language": "python",
   "name": "pyspark3-nano"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}