NLP Sentiment Instructor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is Natural Language Processing?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plug NLTK"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "nltk.download()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It'll bring up another window. From here, download: \n",
    "corpora -> movie_reviews\n",
    "\n",
    "corpora -> stopwords\n",
    "\n",
    "all packages -> punkt\n",
    "\n",
    "corpora -> wordnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import everything we need, explain as we use it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk.classify.util\n",
    "from nltk.corpus import movie_reviews\n",
    "from nltk.corpus import stopwords\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.corpus import wordnet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Break apart a sentence(tokenize)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "sentence = \"This is a test sentence. It will break everything apart!\"\n",
    "sentence.split(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, it misses punctuation. Luckily, NLTK has a solution!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"This is a test sentence. It will break everything apart!\"\n",
    "word_tokenize(sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also have NLTK determine the part of speech (POS) for each token in our sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"This is a test sentence. It will tag everything!\"\n",
    "w = word_tokenize(sentence)\n",
    "nltk.pos_tag(w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NN means singular noun, JJ means adjective etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use NTLK to find the definition of words, synonyms, antonyms, etc.\n",
    "\n",
    "Some words have multiple definitions and multiple parts of speech depending on the usage. This can get very complicated very fast, so for this purpose, we're just going to assume that NLTK knows what its doing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step we have are removing stopwords. Stop words are words that are used for grammaical purposes but carry little meaning (the, a, I, is, etc). \n",
    "\n",
    "So, let's remove them. Issue is, there are over 100 English words that are considered stopwords. So unless we want to create a list of stopwords and iterate every word over the list by hand everytime we write a program, we need a new solution.\n",
    "\n",
    "So, lets just let NLTK remove them for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "para = \"Our symposium will also include two rounds of workshops with several choices in each round- so you can brush up on your Python, learn about data visualization, or deepen your knowledge of machine learning. \"\n",
    "words = word_tokenize(para)\n",
    "print(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "words = word_tokenize(para)\n",
    "useful_words = [word for word in words if word not in stopwords.words('english')]\n",
    "print(useful_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that \"Our\" was not removed even though it is a stopword. Thats because NLTK's list is only in lowercase. So let's move our paragraph to lowercase first so it doesn't miss any."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "para = para.lower()\n",
    "words = word_tokenize(para)\n",
    "useful_words = [word for word in words if word not in stopwords.words('english')]\n",
    "print(useful_words)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This new shorter list means that we don't need to process over as many words, saving us time and making things more efficient without getting rid of any meaning.\n",
    "\n",
    "Now, lets start with creating our tool.\n",
    "\n",
    "Machine learning works by learning from a set of data and then applying what its learned to a new set of data.\n",
    "\n",
    "So, the first thing we need is some data. This data can be tweets, reviews, books, anything really. We're going to be using movie reviews today.\n",
    "\n",
    "Lets explore the data a little bit:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import movie_reviews\n",
    "movie_reviews.words()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_reviews.fileids()[:4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see what the most common words are in our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_words = movie_reviews.words()\n",
    "freq_dist = nltk.FreqDist(all_words)\n",
    "freq_dist.most_common(20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how a lot of these are stopwords. Its a good thing we know how to remove those!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have most of the tools we need. Let's get started.\n",
    "\n",
    "What is sentiment analysis? \n",
    "\n",
    "Sentiment analysis is the process of determining the mood of a certain piece of text. For this example, we're going to be looking at movie reviews to determine if the review is positive or negative.\n",
    "\n",
    "There are many ways of going about doing this. We're going to be using a Naive Bayes algorithm, a fairly simple but effective machine learning algorithm.\n",
    "\n",
    "Bayesian classifiers use whats called Bag-of-Words models. That means that the words are not looked at in context, only their frequency. It also uses Bayes' Theorum which states that \"the probability of A given that B is true equals the probability of B given that A is true times the probability of A being true, divided by the probability of B being true\".\n",
    "\n",
    "Basically, it uses statistics to find the probablility that something is true given something else, a concept called conditional probability. With this, we can go from P(Evidence| Known Outcome) to P(New Outcome|Known Evidence), which is called Bayes Rule. \n",
    "\n",
    "Example: \n",
    "\n",
    "Probability of Disease D given Test-positive = \n",
    "\n",
    "               Prob(Test is positive|Disease) * P(Disease)\n",
    "     _______________________________________________________________\n",
    "     (scaled by) Prob(Testing Positive, with or without the disease)\n",
    "     \n",
    "     \n",
    "Now, to Naive Bayes. What we've done until now assumes that we only have one piece of evidence for an outcome. In the real world, we have multiple pieces of evidence for an outcome. This leads to very complicated math. One way to get around this is to treat things independently, looking at data without knowing anything about the other pieces (ahhhh, now I get the name). \n",
    "\n",
    "                      P(Likelihood of Evidence) * Prior prob of outcome\n",
    "P(outcome|evidence) = _________________________________________________\n",
    "                                         P(Evidence)\n",
    "                                         \n",
    "So, an example. Lets say we have 1000 pieces of fruit. We know if its long or short, sweet or not, and yellow or not yellow. We also know what fruit it actually is (banana, orange or something else). "
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Type           Long | Not Long || Sweet | Not Sweet || Yellow |Not Yellow|Total\n",
    "             ___________________________________________________________________\n",
    "Banana      |  400  |    100   || 350   |    150    ||  450   |  50      |  500\n",
    "Orange      |    0  |    300   || 150   |    150    ||  300   |   0      |  300\n",
    "Other Fruit |  100  |    100   || 150   |     50    ||   50   | 150      |  200\n",
    "            ____________________________________________________________________\n",
    "Total       |  500  |    500   || 650   |    350    ||  800   | 200      | 1000\n",
    "             ___________________________________________________________________"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets say we now have a new piece of fruit. How will we know if its a banana, an orange or something else?\n",
    "\n",
    "Lets say our new fruit is long, sweet and yellow. What is it going to be?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "P(Banana|Long, Sweet and Yellow) \n",
    "      P(Long|Banana) * P(Sweet|Banana) * P(Yellow|Banana) * P(banana)\n",
    "    = _______________________________________________________________\n",
    "                      P(Long) * P(Sweet) * P(Yellow)\n",
    "\n",
    "    = 0.8 * 0.7 * 0.9 * 0.5 / P(evidence)\n",
    "\n",
    "    = 0.252 / P(evidence)\n",
    "\n",
    "\n",
    "P(Orange|Long, Sweet and Yellow) = 0\n",
    "\n",
    "\n",
    "P(Other Fruit|Long, Sweet and Yellow)\n",
    "      P(Long|Other fruit) * P(Sweet|Other fruit) * P(Yellow|Other fruit) * P(Other Fruit)\n",
    "    = ____________________________________________________________________________________\n",
    "                                          P(evidence)\n",
    "\n",
    "    = (100/200 * 150/200 * 50/200 * 200/1000) / P(evidence)\n",
    "\n",
    "    = 0.01875 / P(evidence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can assume that our new piece of fruit is an banana by a large margin, 0.252 >> 0.01875. Our new fruit is a banana. We can now do this with any other piece of fruit we come across. Since we can precompute most of these values once and use them over and over, this classifier is simple and effective. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lets get started on our tool!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need to start looking at some reviews."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "neg_reviews = []\n",
    "for fileid in movie_reviews.fileids('neg'):\n",
    "    words = movie_reviews.words(fileid)\n",
    "    neg_reviews.append((words, \"neg\"))\n",
    "print(neg_reviews[0])\n",
    "print(len(neg_reviews))\n",
    "\n",
    "\n",
    "pos_reviews = []\n",
    "for fileid in movie_reviews.fileids('pos'):\n",
    "    words = movie_reviews.words(fileid)\n",
    "    pos_reviews.append((words, \"pos\"))\n",
    "print(len(pos_reviews))\n",
    "    \n",
    "#print(len(pos_reviews))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have 1000 positive reviews and 1000 negative reviews in the format we want. We still need to break these into training and test sets:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets start looking at the reviews themselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_set = neg_reviews[:300] + pos_reviews[:300]\n",
    "test_set =  neg_reviews[950:] + pos_reviews[950:]\n",
    "print(len(train_set),  len(test_set))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets put this into our classifier through TextBlob's implimentation of Naive Bayes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import textblob\n",
    "from textblob.classifiers import NaiveBayesClassifier\n",
    "cl = NaiveBayesClassifier(train_set)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This might take a second. But when it's done, we have our algorithm trained!\n",
    "\n",
    "In the meantime, let's go over what training and testing sets are."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cl.accuracy(test_set)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}