notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "42c1a698",
   "metadata": {},
   "source": [
    "# Importing and cleaning the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb50ea97",
   "metadata": {},
   "source": [
    "I downloaded the data from the [Large Movie Review Dataset](http://ai.stanford.edu/~amaas/data/sentiment/) from ai.stanford.edu.\n",
    "\n",
    "Citation:\n",
    "Maas, A., Daly, R., Pham, P., Huang, D., Ng, A., & Potts, C. (2011). Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (pp. 142–150). Association for Computational Linguistics.\n",
    "\n",
    "The data is in the form of .txt files, with one review per file. These files are in the following directories:\n",
    "```\n",
    "data/aclImdb/{train or test}/{pos or neg}\n",
    "```\n",
    "### Data cleaning\n",
    "The name of each file gives the index of the file, together with the rating, from 1 to 10, as follows:\n",
    "```\n",
    "{index}_{rating}.txt\n",
    "```\n",
    "First I want to add leading zeros to the filenames, so that the alphabetical order is the same as the index number order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9cba45f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "dirs = ['data/aclImdb/train/pos', 'data/aclImdb/train/neg', 'data/aclImdb/test/pos', 'data/aclImdb/test/neg']\n",
    "for dir in dirs:\n",
    "    for dir_file in os.listdir(dir):\n",
    "            file_number, file_end = dir_file.split('_')\n",
    "            num = file_number.zfill(5)  # num is 5 characters long with leading 0\n",
    "\n",
    "            \n",
    "            new_file = \"{}_{}\".format(num, file_end)\n",
    "            os.rename(dir + '/' + dir_file, dir + '/' + new_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "988c7946",
   "metadata": {},
   "source": [
    "When importing the files I found that some had a unicode character that caused a line break. This would cause these files to be imported as two (or more) separate data rows, when I wanted the whole file to be imported as one row. So I removed this unicode character from all files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8916daac",
   "metadata": {},
   "outputs": [],
   "source": [
    "dirs = ['data/aclImdb/train/pos', 'data/aclImdb/train/neg', 'data/aclImdb/test/pos', 'data/aclImdb/test/neg']\n",
    "for dir in dirs:\n",
    "    for dir_file in os.listdir(dir):\n",
    "        # Read in the file\n",
    "        with open(dir + '/' + dir_file, 'r') as file:\n",
    "            filedata = file.read()\n",
    "        # Replace the target string\n",
    "        filedata = filedata.replace('\\u0085', ' ')\n",
    "\n",
    "        # Write the file out again\n",
    "        with open(dir + '/' + dir_file, 'w') as file:\n",
    "            file.write(filedata)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "023cd07c",
   "metadata": {},
   "source": [
    "I then wanted to extract the rating from each filename into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "21cb9282",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Put the files into ordered lists, one for train and one for test\n",
    "dirs = ['train','test']\n",
    "files = {}\n",
    "ratings = {}\n",
    "for dir in dirs:\n",
    "    files[dir] = []\n",
    "    ratings[dir] = []\n",
    "    subdirs = ['pos', 'neg']\n",
    "    for subdir in subdirs:\n",
    "        files_tmp = []\n",
    "        for dir_file in os.listdir('data/aclImdb/' + dir + '/' + subdir):\n",
    "            files_tmp += [dir_file]\n",
    "        files_tmp = sorted(files_tmp)\n",
    "        files[dir].extend(files_tmp)\n",
    "    for file in files[dir]:\n",
    "        rating = int(file.split('_')[1].split('.')[0])\n",
    "        ratings[dir].append(rating)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b5c4b7c",
   "metadata": {},
   "source": [
    "I now have two lists:\n",
    "```ratings['train']```\n",
    "and\n",
    "```ratings['test']```\n",
    "which are each of length 25,000, containing the ratings for the train and test reviews, in index order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "b8559d83",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_ratings = np.array(ratings['train'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "007936b1",
   "metadata": {},
   "source": [
    "# Data Analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "689dce14",
   "metadata": {},
   "source": [
    "### Distribution of Ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "23c2c756",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAPfklEQVR4nO3cf6zddX3H8edr1N9utkjXsLauJDYzdYlCGqhjWRzdoICx/KEEs2lDmvSfbsPFxBX/aaaSYLKImihJI53VOZGghkaJ2BSM2R8grTgEqukdgm0HtNqCbkZc9b0/7qfmiPdyz6X3nnPp5/lIbs73+/5+zve8P7k3r/O93/M931QVkqQ+/N64G5AkjY6hL0kdMfQlqSOGviR1xNCXpI4sGncDz+ecc86pVatWjbsNSXpR2b9//4+raulU2xZ06K9atYp9+/aNuw1JelFJ8vh02zy9I0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHVnQ38g9Xau2fW0sr/vYjVeO5XUlaSZDHekneSzJ95J8N8m+Vjs7yZ4kB9vjklZPkk8kmUjyYJILBvazqY0/mGTT/ExJkjSd2Zze+cuqenNVrW3r24C9VbUa2NvWAS4HVrefLcDNMPkmAWwHLgIuBLafeqOQJI3G6ZzT3wjsasu7gKsG6p+tSfcCi5OcC1wG7Kmq41V1AtgDbDiN15ckzdKwoV/AN5LsT7Kl1ZZV1RNt+UlgWVteDhwaeO7hVpuu/luSbEmyL8m+Y8eODdmeJGkYw36Q++dVdSTJHwJ7knx/cGNVVZKai4aqagewA2Dt2rVzsk9J0qShjvSr6kh7PAp8hclz8k+10za0x6Nt+BFg5cDTV7TadHVJ0ojMGPpJXpXk908tA5cCDwG7gVNX4GwC7mjLu4H3tKt41gHPtNNAdwGXJlnSPsC9tNUkSSMyzOmdZcBXkpwa/+9V9fUk9wO3JdkMPA5c3cbfCVwBTAA/B64FqKrjST4E3N/GfbCqjs/ZTCRJM5ox9KvqUeBNU9R/Aqyfol7A1mn2tRPYOfs2JUlzwdswSFJHDH1J6oihL0kdMfQlqSOGviR15Iy+tbIknY5x3Z4d5u8W7R7pS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHhg79JGcleSDJV9v6eUnuSzKR5ItJXtrqL2vrE237qoF9XN/qP0hy2ZzPRpL0vGZzpH8dcGBg/SPATVX1euAEsLnVNwMnWv2mNo4ka4BrgDcCG4BPJTnr9NqXJM3GUKGfZAVwJfDpth7gEuD2NmQXcFVb3tjWadvXt/EbgVur6tmq+iEwAVw4B3OQJA1p2CP9jwHvB37d1l8LPF1VJ9v6YWB5W14OHAJo259p439Tn+I5v5FkS5J9SfYdO3Zs+JlIkmY0Y+gneRtwtKr2j6AfqmpHVa2tqrVLly4dxUtKUjcWDTHmYuDtSa4AXg78AfBxYHGSRe1ofgVwpI0/AqwEDidZBLwG+MlA/ZTB50iSRmDGI/2qur6qVlTVKiY/iL27qv4GuAd4Rxu2CbijLe9u67Ttd1dVtfo17eqe84DVwLfnbCaSpBkNc6Q/nX8Cbk3yYeAB4JZWvwX4XJIJ4DiTbxRU1cNJbgMeAU4CW6vqV6fx+pKkWZpV6FfVN4FvtuVHmeLqm6r6BfDOaZ5/A3DDbJuUJM0Nv5ErSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6siMoZ/k5Um+neQ/kzyc5J9b/bwk9yWZSPLFJC9t9Ze19Ym2fdXAvq5v9R8kuWzeZiVJmtIwR/rPApdU1ZuANwMbkqwDPgLcVFWvB04Am9v4zcCJVr+pjSPJGuAa4I3ABuBTSc6aw7lIkmYwY+jXpP9pqy9pPwVcAtze6ruAq9ryxrZO274+SVr91qp6tqp+CEwAF87FJCRJw1k0zKB2RL4feD3wSeC/gKer6mQbchhY3paXA4cAqupkkmeA17b6vQO7HXzO4GttAbYAvO51r5vldCTNp1XbvjaW133sxivH8rpnoqE+yK2qX1XVm4EVTB6dv2G+GqqqHVW1tqrWLl26dL5eRpK6NKurd6rqaeAe4C3A4iSn/lNYARxpy0eAlQBt+2uAnwzWp3iOJGkEhrl6Z2mSxW35FcBfAweYDP93tGGbgDva8u62Ttt+d1VVq1/Tru45D1gNfHuO5iFJGsIw5/TPBXa18/q/B9xWVV9N8ghwa5IPAw8At7TxtwCfSzIBHGfyih2q6uEktwGPACeBrVX1q7mdjiTp+cwY+lX1IHD+FPVHmeLqm6r6BfDOafZ1A3DD7NuUJM0Fv5ErSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjLUvXekmXhPFunFwSN9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I64nX60ovMuL4ToTODR/qS1BFDX5I6YuhLUkc8p38G8VyvpJl4pC9JHTH0Jakjhr4kdcTQl6SO+EHuPPADVUkLlaEvacHzQGrueHpHkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdWTG0E+yMsk9SR5J8nCS61r97CR7khxsj0taPUk+kWQiyYNJLhjY16Y2/mCSTfM3LUnSVIY50j8JvK+q1gDrgK1J1gDbgL1VtRrY29YBLgdWt58twM0w+SYBbAcuAi4Etp96o5AkjcaMoV9VT1TVd9ryz4ADwHJgI7CrDdsFXNWWNwKfrUn3AouTnAtcBuypquNVdQLYA2yYy8lIkp7frM7pJ1kFnA/cByyrqifapieBZW15OXBo4GmHW226uiRpRIYO/SSvBr4EvLeqfjq4raoKqLloKMmWJPuS7Dt27Nhc7FKS1AwV+klewmTgf76qvtzKT7XTNrTHo61+BFg58PQVrTZd/bdU1Y6qWltVa5cuXTqbuUiSZjDM1TsBbgEOVNVHBzbtBk5dgbMJuGOg/p52Fc864Jl2Gugu4NIkS9oHuJe2miRpRIa5y+bFwLuB7yX5bqt9ALgRuC3JZuBx4Oq27U7gCmAC+DlwLUBVHU/yIeD+Nu6DVXV8LiYhSRrOjKFfVf8BZJrN66cYX8DWafa1E9g5mwYlSXPHb+RKUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOjJj6CfZmeRokocGamcn2ZPkYHtc0upJ8okkE0keTHLBwHM2tfEHk2yan+lIkp7PMEf6nwE2PKe2DdhbVauBvW0d4HJgdfvZAtwMk28SwHbgIuBCYPupNwpJ0ujMGPpV9S3g+HPKG4FdbXkXcNVA/bM16V5gcZJzgcuAPVV1vKpOAHv43TcSSdI8e6Hn9JdV1RNt+UlgWVteDhwaGHe41aar/44kW5LsS7Lv2LFjL7A9SdJUTvuD3KoqoOagl1P721FVa6tq7dKlS+dqt5IkXnjoP9VO29Aej7b6EWDlwLgVrTZdXZI0Qi809HcDp67A2QTcMVB/T7uKZx3wTDsNdBdwaZIl7QPcS1tNkjRCi2YakOQLwFuBc5IcZvIqnBuB25JsBh4Hrm7D7wSuACaAnwPXAlTV8SQfAu5v4z5YVc/9cFiSNM9mDP2qetc0m9ZPMbaArdPsZyewc1bdSZLmlN/IlaSOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHXE0Jekjhj6ktQRQ1+SOmLoS1JHDH1J6oihL0kdMfQlqSOGviR1xNCXpI4Y+pLUEUNfkjpi6EtSRwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1BFDX5I6YuhLUkcMfUnqiKEvSR0x9CWpI4a+JHVk5KGfZEOSHySZSLJt1K8vST0baegnOQv4JHA5sAZ4V5I1o+xBkno26iP9C4GJqnq0qn4J3ApsHHEPktStRSN+veXAoYH1w8BFgwOSbAG2tNVnkzw0ot4WinOAH4+7iRF7wXPOR+a4k9Hx99yHcf1t//F0G0Yd+jOqqh3ADoAk+6pq7ZhbGinn3Afn3IeFOOdRn945AqwcWF/RapKkERh16N8PrE5yXpKXAtcAu0fcgyR1a6Snd6rqZJK/A+4CzgJ2VtXDz/OUHaPpbEFxzn1wzn1YcHNOVY27B0nSiPiNXEnqiKEvSR1ZsKHf2+0akqxMck+SR5I8nOS6cfc0CknOSvJAkq+Ou5dRSbI4ye1Jvp/kQJK3jLun+ZTkH9vf9ENJvpDk5ePuaT4k2Znk6OB3i5KcnWRPkoPtcck4e4QFGvqd3q7hJPC+qloDrAO2djBngOuAA+NuYsQ+Dny9qt4AvIkzeP5JlgP/AKytqj9l8gKOa8bb1bz5DLDhObVtwN6qWg3sbetjtSBDnw5v11BVT1TVd9ryz5gMguXj7Wp+JVkBXAl8ety9jEqS1wB/AdwCUFW/rKqnx9rU/FsEvCLJIuCVwH+PuZ95UVXfAo4/p7wR2NWWdwFXjbKnqSzU0J/qdg1ndAAOSrIKOB+4b8ytzLePAe8Hfj3mPkbpPOAY8K/ttNank7xq3E3Nl6o6AvwL8CPgCeCZqvrGeLsaqWVV9URbfhJYNs5mYOGGfreSvBr4EvDeqvrpuPuZL0neBhytqv3j7mXEFgEXADdX1fnA/7IA/uWfL+0c9kYm3+z+CHhVkr8db1fjUZPXx4/9GvmFGvpd3q4hyUuYDPzPV9WXx93PPLsYeHuSx5g8fXdJkn8bb0sjcRg4XFWn/ou7nck3gTPVXwE/rKpjVfV/wJeBPxtzT6P0VJJzAdrj0TH3s2BDv7vbNSQJk+d5D1TVR8fdz3yrquurakVVrWLy93t3VZ3xR4BV9SRwKMmftNJ64JExtjTffgSsS/LK9je+njP4g+sp7AY2teVNwB1j7AVYgHfZhBd0u4YzwcXAu4HvJfluq32gqu4cX0uaJ38PfL4d0DwKXDvmfuZNVd2X5HbgO0xeofYAC/DWBHMhyReAtwLnJDkMbAduBG5Lshl4HLh6fB1O8jYMktSRhXp6R5I0Dwx9SeqIoS9JHTH0Jakjhr4kdcTQl6SOGPqS1JH/B8SeC2JrOpa0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "data = train_ratings\n",
    "d = np.diff(np.unique(data)).min()\n",
    "left_of_first_bin = data.min() - float(d)/2\n",
    "right_of_last_bin = data.max() + float(d)/2\n",
    "plt.hist(data, np.arange(left_of_first_bin, right_of_last_bin + d, d))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1eb0d8c",
   "metadata": {},
   "source": [
    "Note that there are no 5 or 6 ratings. This is because the dataset was prepared for the intention of sentiment analysis, and split into positive (ratings 7 to 10) and negative (ratings 1 to 4). Ideally our data would include a full range of ratings, but I think the model will still be able to train well without the mid-range ratings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14db81e5",
   "metadata": {},
   "source": [
    "### Baseline MSE\n",
    "Our metric for the model will be Mean Squared Error, which will give us a measure of how far each prediction is from the actual rating. It returns the sum of the squares of the differences between the predictions and the actual ratings.\n",
    "\n",
    "It will be helpful for us to have a baseline for what this MSE would be with a very naive model. One very simple baseline is to take the mean of all ratings, and predict this mean each time. This would give the following MSE:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "d383cbad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12.0159836016"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import statistics\n",
    "\n",
    "mean = statistics.fmean(train_ratings)\n",
    "mse = statistics.mean((train_ratings - mean)**2)\n",
    "mse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cfc5d8c",
   "metadata": {},
   "source": [
    "If our model is finding any signal in the data, it should be going lower than this baseline MSE."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1443c0a1",
   "metadata": {},
   "source": [
    "### Optimal MSE\n",
    "I imagine that predicting ratings from reviews will have an optimal MSE that is significantly higher than 0. Even a highly experienced group of human experts will not be able to approach perfect predicting, because the ratings given by the reviewers have a certain level of inherent unpredictability. Someone may review a movie as \"This was awesome\" and give it a rating of 10, while someone else could give exactly the same review with a rating of 7.\n",
    "\n",
    "So I am expecting that the [Bayes error](https://en.wikipedia.org/wiki/Bayes_error_rate) is significantly above zero. My guess is it might be somewhere in the region of an MSE of 4, meaning that the average prediction is wrong by two ratings points. But that is only a guess based on my intution of how much unpredictability there is in the ratings that reviewers assign."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e98c32a",
   "metadata": {},
   "source": [
    "### Length of reviews"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "5924902d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dirs = ['data/aclImdb/train/pos', 'data/aclImdb/train/neg', 'data/aclImdb/test/pos', 'data/aclImdb/test/neg']\n",
    "lengths_chars = []\n",
    "lengths_words = []\n",
    "for dir in dirs:\n",
    "    for dir_file in os.listdir(dir):\n",
    "        # Read in the file\n",
    "        with open(dir + '/' + dir_file, 'r') as file:\n",
    "            filedata = file.read()\n",
    "            length_chars = len(filedata)\n",
    "            length_words = len(filedata.split())\n",
    "            lengths_chars.append(length_chars)\n",
    "            lengths_words.append(length_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "60147a78",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEICAYAAABYoZ8gAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAX70lEQVR4nO3debhkdX3n8ffHbnbU7oYebJbQGB0T4kTFHsHBUQImopDgPOMYHMe0iCExatxmpNVJjFlmcHniMiYqcWs3FHEBtziIonEc0cYFWaWFhm5kuRgQ0WSE+J0/zu9C9eXe7ntv1d36vF/PU0+fc35n+dav6n7q1O9UVaeqkCT1x/0WugBJ0vwy+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfu1QkrVJKsnyOT7O0UmuTnJnkqfO8bHenuRP5vIYOzj255KsH/E+n53kq6Pcp3ZtBv8ilmRLkifu6sds/hx4a1XtW1WfnMsDVdUfVtVfzOUxdnDsJ1fVxoU49lxLckySbQtdh3bO4NdicShw2XRWnOt3H322kH3r4zp/DP4lKMn9kmxI8oMkP0pydpJVrW18aGZ9kuuT3JrkVQPb7pVkY5LbklyR5OXjZ2lJ3g/8EvCpNuTy8oHDPnOK/T0myaYkdyS5Oclf76Du30+yOck/JjkvyYFt+Q+ABw8cd49Jtt2S5PQklwA/TbI8yVFJvpbk9iTfTXJMW/d3k2yasP1LkpzXpt+b5C8H2k5M8p22n68l+fW2/JQknxpY7+okHx2Y35rkkem8McktrR++l+ThU/TBhUme26afneSrSd7QHo9rkzx5B/13SJKPJxlrj/tbJ7RPup92P65I8pMk1yT5g4G2Y5Jsa317E/CeJCuTfLod57Y2ffDANquSvCfJD1v7J5PsA3wOOLA9hncmOXCaz9VTk1wPfDHJnkk+0Na9Pck3kxwwVZ9olqrK2yK9AVuAJ06y/EXA14GDgT2AdwBntba1QAF/B+wFPAL4f8CvtvYzgC8DK9v2lwDbpjrmNPb3f4Fntel9gaOmuC/HArcCR7Sa/xfwlZ3d1wnt3wEOaXUcBPwIeArdCcxvtvnVwN7AT4CHDmz/TeDkNv1e4C/b9KOAW4AjgWXA+nasPehejG5v+z8QuG68r1rbba3tScDFwAogwK8Ca6a4HxcCz23TzwbuAn6/Hft5wA+BTLLdMuC7wBuBfYA9gcdNZz/ACcAvt9qeAPwMOKK1HQPcDby23ee9gP2A/9j68f7AR4FPDtTyGeAjdM+h3YAnDOxr24S6p/NcfV+7T3sBfwB8qh17GfBo4AEL/be4q90WvABvO3hwpg7+K4DjBubXtD/85QN/TAcPtH9jIPSuAZ400PZcphf8U+3vK8BrgP13cl/eBbxuYH7fVvPaHd3XCXU9Z2D+dOD9E9b5PLC+TX8A+NM2/VC6F4K92/x7uTf43wb8xYT9XDUQZlvpXqxOBs5s9/1XgFOA89o6xwLfB44C7reTfriQ7YN/80Db3q2vHzTJdo8FxoDlk7RNez+t/ZPAi9r0McDPgT13UPMjgdsGnmu/AFZOst4x3Df4p/NcffBA+3OArwG/vtB/f7vyzaGepelQ4BPtrfDtdH9c/wIMviW+aWD6Z3RBC92Z69aBtsHpHZlqf6cC/xq4sr0tP3GK7cfPmAGoqjvpztAPmubxJ9Z6KPCfxvug9cPj6IIF4EPAM9r0f6Y7Y/3ZJPs8FHjZhP0c0uqF7t3RMcDj2/SFdGfNT2jzVNUXgbcCfwPckuTMJA+Y5n26p18H6tt3kvUOAa6rqrtnup8kT07y9TbEdjvdu6T9B7Ydq6p/Hp9JsneSdyS5LskddC/uK5Isa3X8Y1XdNs37N53n6uDj+n66F/APt6Gk1yXZbZrH0jQZ/EvTVuDJVbVi4LZnVd0wjW1vpHvbPe6QCe0z+rnWqrq6qp4B/Cu64YJz2njvRD+kCwEA2jr7AdOpebLattKd8Q/2wT5VdUZrPx9YneSRdC8AH5pin1uBv5qwn72r6qzWPh78/75Nf5kJwQ9QVW+pqkcDh9O9EP63Gdyv6dgK/FJmeAE03fWSjwFvAA6oqhXAZ+mGfcZNfMxfBjwMOLKqHkD3okfbZiuwKsmKSQ432XNnOs/Ve7arqruq6jVVdTjw74ATgd+b3r3VdBn8i99u7YLX+G058Hbgr5IcCpBkdZKTprm/s4FXtAt4BwEvmNB+M9349bQk+S9JVlfVL+jGw6EbCpjoLOCUdjF0D+B/ABdV1ZbpHmuCDwC/neRJSZa1vjlm/CJkVd1FNzb9emAV3QvBZP4O+MMkR7aLtPskOSHJ/Vv7l4HfAPaqqm3APwDH071ofbv1wb9t2+8G/BT45yn6YBjfoHvRPqPVuGeSo6ex3e50Y+tjwN3tou9v7WSb+wP/BNzeLsS+eryhqm6ku4j7t+05tFuS8ReGm4H9kjxwYF8zeq4m+Y0k/6a9u7iDblho1H3Zewb/4vdZuj/C8dufAW8GzgP+d5Kf0F08O3Ka+/tzYBtwLfAF4By6i7Xj/ifw39tb8/86jf0dD1yW5M5W18lV9U8TV6qqLwB/Qnf2eSPdxcaTp1nzfVTVVuAk4JV0obaV7ix78Dn9IeCJwEenGiKpqk10F0XfSnexdjPdmPl4+/eBO+kCn6q6g+46yf+pqn9pqz2A7gXkNrrhrB/RveCMTDvWbwMPAa6newx/dxrb/QT4Y7oX/Nvohr3O28lmb6K70Hor3XPr7ye0P4sukK+kuzD+4nasK+le4K9pz58Dmflz9UF0z8k76IaFvkw3/KMRGr/qr55K8jy6sH7CQtciaX54xt8zSdak+3mE+yV5GN147icWui5J88dvyvXP7nSfpT6Mbkz+w8DfLmRBkuaXQz2S1DM7HepJ8u50X0W/dGDZqiTnp/sK+/lJVrblSfKWdF/LvyTJEXNZvCRp5nZ6xt8+qnUn8L6qenhb9jq6L3GckWQD3bf4Tk/yFOCFdF8QORJ4c1Xt9NMm+++/f61du3a4eyJJPXPxxRffWlWrZ7rdTsf4q+orSdZOWHwS3ZdaADbSfZvx9Lb8fdW9mnw9yYoka9pnf6e0du1aNm3atKNVJEkTJLlu52vd12w/1XPAQJjfxL1fvz6I7b9+vY0pvpKf5LR0v+q4aWxsbJZlSJJmauiPc7az+xlfIa6qM6tqXVWtW716xu9UJEmzNNvgvznJGug+F0737T3ofndl8LdfDmZmv8UiSZpjsw3+8+h+t5z277kDy3+vfbrnKODHOxvflyTNr51e3E1yFt2F3P3T/U9Nr6b7zzzOTnIq3W+TPL2t/lm6T/Rspvvp3lPmoGZJ0hCm86meZ0zRdNwk6xbw/GGLkiTNHX+rR5J6xuCXpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWd6G/xrN3xmoUuQpAXR2+CXpL4y+CWpZ4YK/iQvSXJZkkuTnJVkzySHJbkoyeYkH0my+6iKlSQNb9bBn+Qg4I+BdVX1cGAZcDLwWuCNVfUQ4Dbg1FEUKkkajWGHepYDeyVZDuwN3AgcC5zT2jcCTx3yGJKkEZp18FfVDcAbgOvpAv/HwMXA7VV1d1ttG3DQZNsnOS3JpiSbxsbGZluGJGmGhhnqWQmcBBwGHAjsAxw/3e2r6syqWldV61avXj3bMiRJMzTMUM8TgWuraqyq7gI+DhwNrGhDPwAHAzcMWaMkaYSGCf7rgaOS7J0kwHHA5cCXgKe1ddYD5w5XoiRplIYZ47+I7iLut4DvtX2dCZwOvDTJZmA/4F0jqFOSNCLLd77K1Krq1cCrJyy+BnjMMPuVJM0dv7krST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs/0PvjXbvjMQpcgSfOq98EvSX1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwS1LPGPyS1DMGvyT1jMEvST1j8EtSz/Q6+P1JZkl9NFTwJ1mR5JwkVya5Isljk6xKcn6Sq9u/K0dVrCRpeMOe8b8Z+Puq+hXgEcAVwAbggqp6KHBBm5ckLRKzDv4kDwQeD7wLoKp+XlW3AycBG9tqG4GnDleiJGmUhjnjPwwYA96T5NtJ3plkH+CAqrqxrXMTcMBkGyc5LcmmJJvGxsaGKEOSNBPDBP9y4AjgbVX1KOCnTBjWqaoCarKNq+rMqlpXVetWr149RBmSpJkYJvi3Aduq6qI2fw7dC8HNSdYAtH9vGa5ESdIozTr4q+omYGuSh7VFxwGXA+cB69uy9cC5Q1UoSRqp5UNu/0Lgg0l2B64BTqF7MTk7yanAdcDThzyGJGmEhgr+qvoOsG6SpuOG2a8kae70+pu7ktRHBj/+dIOkfjH4JalnDH5J6hmDX5J6xuCXpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCXpJ4x+Bv/MxZJfWHwS1LPGPyS1DMGvyT1jMEvST1j8EtSzxj8ktQzBr8k9YzBL0k9Y/BLUs8Y/JLUMwa/JPWMwS9JPWPwD/CH2iT1gcEvST1j8EtSzwwd/EmWJfl2kk+3+cOSXJRkc5KPJNl9+DIlSaMyijP+FwFXDMy/FnhjVT0EuA04dQTHGCnH8iX12VDBn+Rg4ATgnW0+wLHAOW2VjcBThzmGJGm0hj3jfxPwcuAXbX4/4PaqurvNbwMOGvIYkqQRmnXwJzkRuKWqLp7l9qcl2ZRk09jY2GzLkCTN0DBn/EcDv5NkC/BhuiGeNwMrkixv6xwM3DDZxlV1ZlWtq6p1q1evHqIMSdJMzDr4q+oVVXVwVa0FTga+WFXPBL4EPK2tth44d+gqJUkjMxef4z8deGmSzXRj/u+ag2NIkmZp+c5X2bmquhC4sE1fAzxmFPuVJI2e39yVpJ4x+CWpZwx+SeoZg1+Sesbgl6SeMfglqWcMfknqGYNfknrG4JeknjH4JalnDH5J6hmDX5J6xuCfJv+fXkm7CoNfknrG4JeknjH4JalnDP5JOJ4vaVdm8EtSzxj8ktQzBr8k9YzBP4Hj+5J2dQa/JPWMwS9JPdO74J/uUM7geg7/SNqV9C74JanvDH5J6hmDX5J6xuCXpJ4x+CWpZ5YvdAGLmZ/mkbQr8oxfknrG4JeknjH4JalnDH5J6plZB3+SQ5J8KcnlSS5L8qK2fFWS85Nc3f5dObpyJUnDGuaM/27gZVV1OHAU8PwkhwMbgAuq6qHABW1ekrRIzDr4q+rGqvpWm/4JcAVwEHASsLGtthF46pA1Lhp+vFPSrmAkY/xJ1gKPAi4CDqiqG1vTTcABoziGJGk0hg7+JPsCHwNeXFV3DLZVVQE1xXanJdmUZNPY2NiwZUzJn1eWpO0NFfxJdqML/Q9W1cfb4puTrGnta4BbJtu2qs6sqnVVtW716tXDlCFJmoFhPtUT4F3AFVX11wNN5wHr2/R64NzZlzc6nu1LUmeY3+o5GngW8L0k32nLXgmcAZyd5FTgOuDpQ1UoSRqpWQd/VX0VyBTNx812v6OwdsNn2HLGCQtZgiQtWn5zV5J6ZpcN/rke0x/fv9cOJC01u2zwS5ImZ/DPkN8LkLTUGfyS1DO7dPDP5xn52g2f8R2ApCVhlw5+SdJ9GfyS1DO9CH6HYCTpXr0IfknSvQz+eea7D0kLzeCXpJ7Z5YJ/Ic6o/VKXpKVklwt+SdKOGfyzMJ2zes/8JS1WBr8k9YzBPwc825e0mBn8ktQzBv8c8tM+khYjg1+Sesbgl6SeMfgXgMM+khaSwS9JPWPwz7HZnN37jkDSXDL4JalnDP55NNmZ/PgyfwZC0nwx+CWpZwz+BTLV2f9U7wA825c0Kga/JPXMLhP8i/mMeGe17ah9Jmf+0+2DUfTVYu5vSTu2ywS/JGl6DP4FNMwZ+uD1gMF1dvTJodkcW9Kux+CXpJ7ZpYK/L2exE8/0J7sOMNny6Ww7cZ0dLRtc7vcQpKVjlwp+SdLOzUnwJzk+yVVJNifZMBfHkCTNzvJR7zDJMuBvgN8EtgHfTHJeVV0+6mNBN3yw5YwT5mLXC26UH7uczUdKZ7vNljNOmPTfidsNLt/RYzi47cT1prO9pO3NxRn/Y4DNVXVNVf0c+DBw0hwcR5I0C6mq0e4weRpwfFU9t80/Cziyql4wYb3TgNPa7MOAq2Z5yP2BW2e57UJYavXC0qt5qdULS6/mpVYvLL2ap1PvoVW1eqY7HvlQz3RV1ZnAmcPuJ8mmqlo3gpLmxVKrF5ZezUutXlh6NS+1emHp1TyX9c7FUM8NwCED8we3ZZKkRWAugv+bwEOTHJZkd+Bk4Lw5OI4kaRZGPtRTVXcneQHweWAZ8O6qumzUxxkw9HDRPFtq9cLSq3mp1QtLr+alVi8svZrnrN6RX9yVJC1ufnNXknrG4Jeknlmywb9YfhYiySFJvpTk8iSXJXlRW74qyflJrm7/rmzLk+Qtre5LkhwxsK/1bf2rk6yfh9qXJfl2kk+3+cOSXNRq+0i7OE+SPdr85ta+dmAfr2jLr0rypDmsdUWSc5JcmeSKJI9d7H2c5CXtOXFpkrOS7LnY+jjJu5PckuTSgWUj69ckj07yvbbNW5JkDup9fXteXJLkE0lWDLRN2ndT5cdUj8+oax5oe1mSSrJ/m5+fPq6qJXeju2j8A+DBwO7Ad4HDF6iWNcARbfr+wPeBw4HXARva8g3Aa9v0U4DPAQGOAi5qy1cB17R/V7bplXNc+0uBDwGfbvNnAye36bcDz2vTfwS8vU2fDHykTR/e+n4P4LD2mCybo1o3As9t07sDKxZzHwMHAdcCew307bMXWx8DjweOAC4dWDayfgW+0dZN2/bJc1DvbwHL2/RrB+qdtO/YQX5M9fiMuua2/BC6D8FcB+w/n308Z6EylzfgscDnB+ZfAbxioetqtZxL9ztFVwFr2rI1wFVt+h3AMwbWv6q1PwN4x8Dy7dabgzoPBi4AjgU+3Z40tw78Ad3Tx+3J+dg2vbytl4n9PrjeiGt9IF2IZsLyRdvHdMG/tf2hLm99/KTF2MfAWrYP0pH0a2u7cmD5duuNqt4Jbf8B+GCbnrTvmCI/dvQ3MBc1A+cAjwC2cG/wz0sfL9WhnvE/qnHb2rIF1d6ePwq4CDigqm5sTTcBB7TpqWqf7/v0JuDlwC/a/H7A7VV19yTHv6e21v7jtv581XwYMAa8J93Q1DuT7MMi7uOqugF4A3A9cCNdn13M4u3jQaPq14Pa9MTlc+k5dGe97KSuyZbv6G9gpJKcBNxQVd+d0DQvfbxUg3/RSbIv8DHgxVV1x2BbdS/Fi+Zzs0lOBG6pqosXupZpWk73VvltVfUo4Kd0QxD3WIR9vJLuxwkPAw4E9gGOX9CiZmGx9euOJHkVcDfwwYWuZUeS7A28EvjThaphqQb/ovpZiCS70YX+B6vq423xzUnWtPY1wC1t+VS1z+d9Ohr4nSRb6H499VjgzcCKJONf6hs8/j21tfYHAj+ax5q3Aduq6qI2fw7dC8Fi7uMnAtdW1VhV3QV8nK7fF2sfDxpVv97QpicuH7kkzwZOBJ7ZXqxmU++PmPrxGaVfpjsh+G77GzwY+FaSB82i5tn18SjHCufrRncGeE3rvPGLM7+2QLUEeB/wpgnLX8/2F8he16ZPYPuLN99oy1fRjWOvbLdrgVXzUP8x3Htx96Nsf2Hrj9r089n+wuPZbfrX2P7i2TXM3cXdfwAe1qb/rPXvou1j4EjgMmDvVsdG4IWLsY+57xj/yPqV+154fMoc1Hs8cDmwesJ6k/YdO8iPqR6fUdc8oW0L947xz0sfz2mozOWN7ur39+muzr9qAet4HN1b4UuA77TbU+jGCy8Arga+MPAghe4/qvkB8D1g3cC+ngNsbrdT5qn+Y7g3+B/cnkSb2x/AHm35nm1+c2t/8MD2r2r35SqG/MTGTup8JLCp9fMn25N/Ufcx8BrgSuBS4P0tgBZVHwNn0V2DuIvundWpo+xXYF27/z8A3sqEC/Qjqncz3fj3+N/f23fWd0yRH1M9PqOueUL7Fu4N/nnpY3+yQZJ6ZqmO8UuSZsngl6SeMfglqWcMfknqGYNfknrG4JeknjH4Jaln/j8TGHtLm8NliAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = lengths_chars\n",
    "data = np.array(data)\n",
    "\n",
    "d = np.diff(np.unique(data)).min()\n",
    "left_of_first_bin = data.min() - float(d)/2\n",
    "right_of_last_bin = data.max() + float(d)/2\n",
    "plt.hist(data, np.arange(left_of_first_bin, right_of_last_bin + d, d))\n",
    "plt.title('Lengths of reviews in characters')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "2bc06734",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXgAAAEICAYAAABVv+9nAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAZeElEQVR4nO3df5xcdX3v8dfbJAYK2CSwzQ1JyqJEbezjGugKaeUqBZEQaYP34Y9Qi6lFoy08LrZWDXL7KPYhLbZVWmqlDYUSqAUR9ZJKvBoRsdgSXGgIhECzQGwSQ7JA+FWVkvDpH+e75WSZ2T2zM7O785338/GYx5z5nu855/ud2XnP2e85c0YRgZmZ5edlE90AMzNrDwe8mVmmHPBmZplywJuZZcoBb2aWKQe8mVmmHPDWNpJ6JYWkqW3ezhslbZX0rKQz27ytv5b0++3cxgjb/rqkFROx7aokfUfS+ye6HVZo6xvPJo6kbcD7I+JbOW8z+UPgcxHxF+3eUER8qN3bGGHbp0/Utq0zeQ/ecnAUsLlKxXb/N9EtVHB+THJ+gbqMpJdJWiXpIUmPS7pB0qw0b2hIZYWkf5f0mKQLS8seLGmNpL2Stkj6mKQdad61wM8C/5iGSj5W2ux76qzveEn9kp6WtFvSZ0do9wckDUh6QtJaSUem8oeAV5a2O73GstskfVzSJuA/JE2VtFjSP0t6UtI9kk5Kdd8tqX/Y8r8jaW2avlrSp0rzzpC0Ma3nnyX9z1T+Pkn/WKq3VdKXSo+3S1qUgvJSSXvS83CvpJ+v8xz89/CHpN+QdLukP0uvxyOSau7hV21Lmv4lSd+X9FS6/6Vh279Y0veAHwGvlHSqpAdS/c8BKtU/RtJtad5jkr5Yq33WRhHhW4Y3YBvwlhrl5wN3APOA6cDfANeleb1AAFcABwOvB54Dfi7NvwS4DZiZlt8E7Ki3zQrr+xfg7DR9KLC4Tl9OBh4Djktt/kvgu6P1ddj8jcD81I65wOPAUoqdnFPT4x7gp4BngAWl5b8PLE/TVwOfStPHAnuAE4ApwIq0rekUHzpPpvUfCfxg6LlK8/ameacBdwEzKMLx54A5dfrxHYohMIDfAJ4HPpC2/VvADwHVWK5qW2al6bMphm/PSo8PL23/34HXpfk96bl6BzAN+B1gX6mN1wEXpnUfBJw40e+Lbrt5D777fAi4MCJ2RMRzwEXAO4YNXXwyIn4cEfcA91AEM8C7gD+KiL0RsQO4rOI2663veeAYSUdExLMRcUed5d8DXBURd6c2XwD8oqTeitsHuCwitkfEj4FfB9ZFxLqIeCEi1gP9wNKI+BFwE0W4IWkB8FpgbY11rgT+JiI2RMT+iFhD8QG2OCIepgi/RcCbgG8AP5T0WuDNwD9FxAvpOTgsbUMRsSUidlXs0w8i4oqI2A+sAeYAs4dXaqAtbwO2RsS1EbEvIq4DHgB+pbS6qyNic0TsA04HNkfEjRHxPPDnwKOlus9TDJ8dGRE/iYjbK/bLWsQB332OAr6ahhSeBLYA+zkwGMpv0h9R7F1Dsfe3vTSvPD2Seus7B3g18EAaDjijzvJDe50ARMSzFHvccytuf3hbjwLeOfQcpOfhRIqABPgHUsADvwb8vxT8wx0FfGTYeuan9kLx385JFKF6G8Ue8JvT7bbUl28DnwP+CtgjabWkV1Ts038/r6X2HVqn7qhtYdjznPyAA5/n8vN4wN9DRMSw+R+j+K/kTkmbJf3maB2y1nLAd5/twOkRMaN0OygidlZYdhfF0MyQ+cPmN3Rp0ojYGhFnAT8DfBq4UdIhNar+kCJMAUh1DgeqtLlW27YD1w57Dg6JiEvS/PVATxqXPosi8GvZDlw8bD0/lfZ84cVQ/V9p+jZeGqpExGUR8QvAQooPvI820K+qqrTlgOc5+VkOfJ7Lz+MuSn8DklR+HBGPRsQHIuJI4IPA5yUd04rOWDUO+LxNk3RQ6TYV+GvgYklHAUjqkbSs4vpuAC6QNFPSXOC8YfN3U4zpViLp1yX1pOGBJ1PxCzWqXge8Lx2UnA78EbAhIrZV3dYwfw/8iqTTJE1Jz81JkuYBpOGGLwF/SjEuvb7Oeq4APiTphHSw9BBJb5N0WJp/G/DLwMFpSOufgCUUH07/mp6DN6TlpwH/AfykznPQrFHbAqwDXi3p11QciH43xYfO1+qs82bgdZL+d/rb+j/A/xiaKemdQ88pxVh+0J6+WR0O+LytA35cul0E/AXFePI3JT1DccD1hIrr+0NgB/AI8C3gRoox5yF/DPzfNFzxexXWtwTYLOnZ1K7laYz8AFGcV//7wJcp9hpfBSyv2OaXiIjtwDLgE8AgxZ74Rznw/fAPwFuAL6Xx5lrr6ac4yPk5igAboDj4OTT/34BnKcKUiHgaeBj4Xho3B3gFxQfFXorhkMcpPlhaqkpbIuJx4AzgI6kdHwPOiIjH6qzzMeCdFAffHwcWAN8rVXkDsCG9vmuB89PxABsnKobNzBon6bcoQvnNE90WM3sp78FbZZLmqLgswMskvYZiT++rE90uM6vN3+qzRryc4rz5oynGzK8HPj+RDTKz+jxEY2aWKQ/RmJllalIM0RxxxBHR29s70c0wM+sod91112MR0VNvfuWAlzSF4uvcOyPiDElHU4zBHk5xLY2zI+I/03nK1wC/QHHq1LtHO1+5t7eX/v7+kaqYmdkwkoZ/8/gAjQzRnE/xtfYhnwYujYhjKM7hPSeVnwPsTeWXpnpmZjbOKgV8+jba24C/TY9FcYW/G1OVNcCZaXpZekyaf0qqb2Zm46jqHvyfU3yrbehrxocDT5a+4beDFy9INJd0waE0/6lU/wCSVqq4Fnj/4ODg2FpvZmZ1jRrw6Qp/eyLirlZuOCJWR0RfRPT19NQ9RmBmZmNU5SDrG4FflbSU4qL9r6C4bsgMSVPTXvo8Xrzi3E6KK8rtSBcg+mmKg61mZjaORt2Dj4gLImJeRPRSXODp2xHxHuBWil9ygeKXbG5K02vTY9L8b4e/TWVmNu6a+aLTx4HflTRAMcZ+ZSq/Ejg8lf8usKq5JpqZ2Vg09EWniPgOxS/BDP0M2PE16vyE4hKiZmY2gXypAjOzTDngzcwy5YA3M8uUA97MLFMOeDOzTDngzcwy5YA3M8uUA97MLFMOeDOzTDngzcwy5YA3M8uUA97MLFMOeDOzTDngzcwy5YA3M8uUA97MLFMOeDOzTI0a8JIOknSnpHskbZb0yVR+taRHJG1Mt0WpXJIukzQgaZOk49rcBzMzq6HKT/Y9B5wcEc9KmgbcLunrad5HI+LGYfVPBxak2wnA5enezMzG0ah78FF4Nj2clm4xwiLLgGvScncAMyTNab6pZmbWiEpj8JKmSNoI7AHWR8SGNOviNAxzqaTpqWwusL20+I5UZmZm46hSwEfE/ohYBMwDjpf088AFwGuBNwCzgI83smFJKyX1S+ofHBxsrNVmZjaqhs6iiYgngVuBJRGxKw3DPAf8HXB8qrYTmF9abF4qG76u1RHRFxF9PT09Y2q8mZnVV+Usmh5JM9L0wcCpwAND4+qSBJwJ3JcWWQu8N51Nsxh4KiJ2taHtZmY2gip78HOAWyVtAr5PMQb/NeALku4F7gWOAD6V6q8DHgYGgCuA3255q1ukd9XNE90EM7O2GfU0yYjYBBxbo/zkOvUDOLf5ppmZWTP8TVYzs0w54M3MMuWANzPLlAPezCxTDngzs0w54M3MMuWANzPLlAPezCxTDngzs0w54M3MMuWAx9ekMbM8OeDNzDLlgDczy5QD3swsUw54M7NMOeDNzDLlgDczy5QD3swsU1V+dPsgSXdKukfSZkmfTOVHS9ogaUDSFyW9PJVPT48H0vzeNvehKT4H3sxyVWUP/jng5Ih4PbAIWCJpMfBp4NKIOAbYC5yT6p8D7E3ll6Z6ZmY2zkYN+Cg8mx5OS7cATgZuTOVrgDPT9LL0mDT/FElqVYPNzKyaSmPwkqZI2gjsAdYDDwFPRsS+VGUHMDdNzwW2A6T5TwGH11jnSkn9kvoHBweb6oSZmb1UpYCPiP0RsQiYBxwPvLbZDUfE6ojoi4i+np6eZldnZmbDNHQWTUQ8CdwK/CIwQ9LUNGsesDNN7wTmA6T5Pw083orGmplZdVXOoumRNCNNHwycCmyhCPp3pGorgJvS9Nr0mDT/2xERLWyzmZlVMHX0KswB1kiaQvGBcENEfE3S/cD1kj4F/CtwZap/JXCtpAHgCWB5G9ptZmajGDXgI2ITcGyN8ocpxuOHl/8EeGdLWmdmZmPmb7KamWXKAW9mlikHvJlZphzwZmaZ6tqA90XGzCx3XRvwZma5c8CbmWXKAZ94yMbMcuOANzPLlAPezCxTDngzs0w54M3MMuWANzPLlAPezCxTDngzs0w54M3MMuWANzPLlAPezCxTVX50e76kWyXdL2mzpPNT+UWSdkramG5LS8tcIGlA0oOSTmtnB8zMrLYqP7q9D/hIRNwt6TDgLknr07xLI+LPypUlLaT4oe3XAUcC35L06ojY38qGm5nZyEbdg4+IXRFxd5p+BtgCzB1hkWXA9RHxXEQ8AgxQ48e5zcysvRoag5fUCxwLbEhF50naJOkqSTNT2Vxge2mxHdT4QJC0UlK/pP7BwcHGW25mZiOqHPCSDgW+DHw4Ip4GLgdeBSwCdgGfaWTDEbE6Ivoioq+np6eRRc3MrIJKAS9pGkW4fyEivgIQEbsjYn9EvABcwYvDMDuB+aXF56UyMzMbR1XOohFwJbAlIj5bKp9TqvZ24L40vRZYLmm6pKOBBcCdrWuymZlVUeUsmjcCZwP3StqYyj4BnCVpERDANuCDABGxWdINwP0UZ+Cc6zNozMzG36gBHxG3A6oxa90Iy1wMXNxEu8zMrEn+JquZWaYc8GZmmXLAm5llygFvZpYpB3xJ76qbJ7oJZmYt44A3M8uUA97MLFMOeDOzTDngzcwy5YA3M8uUA97MLFNdGfA+HdLMukFXBryZWTdwwA/jvXszy4UD3swsUw54M7NMOeDNzDLlgDczy1SVH92eL+lWSfdL2izp/FQ+S9J6SVvT/cxULkmXSRqQtEnSce3uhJmZvVSVPfh9wEciYiGwGDhX0kJgFXBLRCwAbkmPAU4HFqTbSuDylrfazMxGNWrAR8SuiLg7TT8DbAHmAsuANanaGuDMNL0MuCYKdwAzJM1pdcPNzGxkDY3BS+oFjgU2ALMjYlea9SgwO03PBbaXFtuRyoava6Wkfkn9g4ODjbbbzMxGUTngJR0KfBn4cEQ8XZ4XEQFEIxuOiNUR0RcRfT09PY0samZmFVQKeEnTKML9CxHxlVS8e2joJd3vSeU7gfmlxeelso7hb7OaWQ6qnEUj4EpgS0R8tjRrLbAiTa8AbiqVvzedTbMYeKo0lGNmZuNkaoU6bwTOBu6VtDGVfQK4BLhB0jnAD4B3pXnrgKXAAPAj4H2tbLCZmVUzasBHxO2A6sw+pUb9AM5tsl1mZtYkf5PVzCxTDngzs0w54M3MMuWANzPLlAPezCxTDngzs0w54M3MMuWANzPLlAPezCxTDngzs0w54M3MMuWAr8OXDDazTueANzPLlAPezCxTDngzs0w54M3MMuWANzPLVJXfZL1K0h5J95XKLpK0U9LGdFtamneBpAFJD0o6rV0NHy8+m8bMOlWVPfirgSU1yi+NiEXptg5A0kJgOfC6tMznJU1pVWPNzKy6UQM+Ir4LPFFxfcuA6yPiuYh4hOKHt49von1mZjZGzYzBnydpUxrCmZnK5gLbS3V2pDIzMxtnYw34y4FXAYuAXcBnGl2BpJWS+iX1Dw4OjrEZZmZWz5gCPiJ2R8T+iHgBuIIXh2F2AvNLVeelslrrWB0RfRHR19PTM5ZmmJnZCMYU8JLmlB6+HRg6w2YtsFzSdElHAwuAO5tropmZjcXU0SpIug44CThC0g7gD4CTJC0CAtgGfBAgIjZLugG4H9gHnBsR+9vScjMzG9GoAR8RZ9UovnKE+hcDFzfTqMnC58CbWSfzN1nNzDLlgK/Ie/Nm1mkc8GZmmXLAm5llygFvZpYpB7yZWaYc8GZmmeq6gPfZMGbWLbou4MfCHwpm1okc8GZmmXLAm5llygFvZpYpB7yZWaYc8GZmmXLAm5llygHfAJ8uaWadxAFvZpYpB7yZWaZGDXhJV0naI+m+UtksSeslbU33M1O5JF0maUDSJknHtbPxE8VDNWbWCarswV8NLBlWtgq4JSIWALekxwCnAwvSbSVweWuaaWZmjRo14CPiu8ATw4qXAWvS9BrgzFL5NVG4A5ghaU6L2toQ72WbWbcb6xj87IjYlaYfBWan6bnA9lK9HansJSStlNQvqX9wcHCMzajGYW9m3ajpg6wREUCMYbnVEdEXEX09PT3NNqOuVof70Pr8oWFmk91YA3730NBLut+TyncC80v15qUyMzMbZ2MN+LXAijS9AripVP7edDbNYuCp0lCOmZmNoyqnSV4H/AvwGkk7JJ0DXAKcKmkr8Jb0GGAd8DAwAFwB/HZbWj0GHlIxs24zdbQKEXFWnVmn1KgbwLnNNqpVHOpm1s38TdYm+UPEzCYrB7yZWaYc8GZmmXLAm5llqqsCvl1fejIzm4y6KuDbyWFvZpONA97MLFMOeDOzTDngzcwy5YA3M8tUlgE/UQc8faDVzCaTLAN+vDnYzWwycsCbmWXKAW9mlikHvJlZphzwZmaZcsC3Ue+qm30A1swmjAO+xRzoZjZZjPqTfSORtA14BtgP7IuIPkmzgC8CvcA24F0Rsbe5ZnYWh7yZTQat2IP/5YhYFBF96fEq4JaIWADckh6bmdk4a8cQzTJgTZpeA5zZhm2Ymdkomg34AL4p6S5JK1PZ7IjYlaYfBWbXWlDSSkn9kvoHBwebbIaZmQ3X1Bg8cGJE7JT0M8B6SQ+UZ0ZESIpaC0bEamA1QF9fX806ZmY2dk3twUfEznS/B/gqcDywW9IcgHS/p9lGdrryQdehaZ9CaWbtNuaAl3SIpMOGpoG3AvcBa4EVqdoK4KZmG2lmZo1rZg9+NnC7pHuAO4GbI+L/A5cAp0raCrwlPbaSWnv0ZmatNuYx+Ih4GHh9jfLHgVOaaZSZmTXP32Q1M8uUA36ceCjGzMabA34cOeTNbDxlF/CdGKI+ddLM2iG7gDczs4IDfpLwnruZtZoD3swsUw74SWj43ny9L0Z5r9/MRuKA71AOdzMbjQN+kiqfWWNmNhbNXi7YxomHZsysUd6Dn8TGGuS5fgDk2i+zdnHAd7iRvhzlQDTrblkFvAPtpfycmHWvrALeXlQr2B32Zt3FAZ+Jkc66Ga1s+LKT8QyeydQWs07hgM9IK0Kw6jocuGaTnwO+S1Tdi68yr8q6zWzitS3gJS2R9KCkAUmr2rUda8zws24aDeeR6te77PFYLoM8GYeJzDpNWwJe0hTgr4DTgYXAWZIWtmNbQxwE7VErtEe6fn2VD4+R1uMvdJm1Trv24I8HBiLi4Yj4T+B6YFk7NuQfyZgcqrwG9T4kRltH1f8G6n1QVG1rO44/jPQB2C7NbMPvpbwoIlq/UukdwJKIeH96fDZwQkScV6qzEliZHr4GeHCMmzsCeKyJ5naqbux3N/YZ3O9u0mifj4qInnozJ+xaNBGxGljd7Hok9UdEXwua1FG6sd/d2Gdwvye6HeOp1X1u1xDNTmB+6fG8VGZmZuOkXQH/fWCBpKMlvRxYDqxt07bMzKyGtgzRRMQ+SecB3wCmAFdFxOZ2bIsWDPN0qG7sdzf2GdzvbtLSPrflIKuZmU08f5PVzCxTDngzs0x1bMDnfikESdsk3Stpo6T+VDZL0npJW9P9zFQuSZel52KTpOMmtvXVSbpK0h5J95XKGu6npBWp/lZJKyaiL42o0++LJO1Mr/lGSUtL8y5I/X5Q0mml8o55H0iaL+lWSfdL2izp/FSe9es9Qr/b/3pHRMfdKA7cPgS8Eng5cA+wcKLb1eI+bgOOGFb2J8CqNL0K+HSaXgp8HRCwGNgw0e1voJ9vAo4D7htrP4FZwMPpfmaanjnRfRtDvy8Cfq9G3YXpb3w6cHT625/Sae8DYA5wXJo+DPi31LesX+8R+t3217tT9+DH7VIIk8wyYE2aXgOcWSq/Jgp3ADMkzZmA9jUsIr4LPDGsuNF+ngasj4gnImIvsB5Y0vbGN6FOv+tZBlwfEc9FxCPAAMV7oKPeBxGxKyLuTtPPAFuAuWT+eo/Q73pa9np3asDPBbaXHu9g5CesEwXwTUl3pcs6AMyOiF1p+lFgdprO7flotJ859f+8NBxx1dBQBRn2W1IvcCywgS56vYf1G9r8endqwHeDEyPiOIorcp4r6U3lmVH8L5f9Oa7d0s/kcuBVwCJgF/CZCW1Nm0g6FPgy8OGIeLo8L+fXu0a/2/56d2rAZ38phIjYme73AF+l+Pds99DQS7rfk6rn9nw02s8s+h8RuyNif0S8AFxB8ZpDRv2WNI0i5L4QEV9Jxdm/3rX6PR6vd6cGfNaXQpB0iKTDhqaBtwL3UfRx6IyBFcBNaXot8N501sFi4KnSv7ydqNF+fgN4q6SZ6d/ct6ayjjLsuMnbKV5zKPq9XNJ0SUcDC4A76bD3gSQBVwJbIuKzpVlZv971+j0ur/dEH2Fu4sj0Uoqj0Q8BF050e1rct1dSHCG/B9g81D/gcOAWYCvwLWBWKhfFD6w8BNwL9E10Hxro63UU/54+TzGmeM5Y+gn8JsXBqAHgfRPdrzH2+9rUr03pjTunVP/C1O8HgdNL5R3zPgBOpBh+2QRsTLelub/eI/S77a+3L1VgZpapTh2iMTOzUTjgzcwy5YA3M8uUA97MLFMOeDOzTDngzcwy5YA3M8vUfwHOuY5I6aZ5OwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = lengths_words\n",
    "data = np.array(data)\n",
    "\n",
    "d = np.diff(np.unique(data)).min()\n",
    "left_of_first_bin = data.min() - float(d)/2\n",
    "right_of_last_bin = data.max() + float(d)/2\n",
    "plt.hist(data, np.arange(left_of_first_bin, right_of_last_bin + d, d))\n",
    "plt.title('Lengths of reviews in words')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56b29214",
   "metadata": {},
   "source": [
    "We see that most reviews are under 512 words. This is good, since the tokenizer that we will use has a maximum length of 512 words, meaning that any words after this limit will be ignored by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd7759cd",
   "metadata": {},
   "source": [
    "# Preparing the data for batch training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3415588e",
   "metadata": {},
   "source": [
    "The text data are still in ```.txt``` files in four directories:\n",
    "```\n",
    "train/pos\n",
    "train/neg\n",
    "test/pos\n",
    "test/neg\n",
    "```\n",
    "These can be converted into a Dataset, that the pretrained NLP model can efficiently process in batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "40a68093",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "71c4af7a8e9e4c1f921864d6fea9835d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Resolving data files:   0%|          | 0/25000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dd3c9c7f80f746518e3dd30e9b202b77",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Resolving data files:   0%|          | 0/25000 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default-208475e3491f36cc\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading and preparing dataset text/default to /home/mark/.cache/huggingface/datasets/text/default-208475e3491f36cc/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5...\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "caf9bf08075c4ffd887029b29a4d7abb",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "42e4f0317d84420ca9d092384e80dcce",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset text downloaded and prepared to /home/mark/.cache/huggingface/datasets/text/default-208475e3491f36cc/0.0.0/e16f44aa1b321ece1f87b07977cc5d70be93d69b20486d6dacd62e12cf25c9a5. Subsequent calls will reuse this data.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ddf31b5d436847daa35ce837390955e1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/2 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from datasets import load_dataset\n",
    "\n",
    "data = load_dataset(\"text\", data_files={'train': ['data/aclImdb/train/pos/*.txt', 'data/aclImdb/train/neg/*.txt'], 'test': ['data/aclImdb/test/pos/*.txt', 'data/aclImdb/test/neg/*.txt']})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3664516e",
   "metadata": {},
   "source": [
    "NLP models can be very expensive to train, often involving a GPU running for long periods of time. So when setting up the training, and even when tuning hyperparameters, I want to have the option to do this on a subset of the data to speed up the process.\n",
    "\n",
    "I will define a function to return a subset of the dataset, with a given number of rows, but ensuring that this subset includes rows from both the positive and negative parts of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "50b25693",
   "metadata": {},
   "outputs": [],
   "source": [
    "def data_subset(data, max_rows):\n",
    "    return data.filter(lambda e, i: (i in range(int(max_rows / 2)) or i in range(int(len(data['train']) / 2), int(len(data['train']) / 2) + int(max_rows / 2))), with_indices=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b15b6bd5",
   "metadata": {},
   "source": [
    "### Save the data to Amazon S3\n",
    "I next saved the data to Amazon S3 storage, where it could be accessed from elsewhere as I trained the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c56840b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p ~/.aws\n",
    "!cp ./aws_config ~/.aws/config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcbd8c85",
   "metadata": {},
   "outputs": [],
   "source": [
    "from botocore.session import Session\n",
    "\n",
    "s3_session = Session()\n",
    "s3 = datasets.filesystems.S3FileSystem(session=s3_session)\n",
    "data.save_to_disk('s3://mlzoomcamp-capstone', fs=s3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad75a352",
   "metadata": {},
   "source": [
    "# Base Model and Tokenizer\n",
    "Since training an NLP model from scratch requires a huge amount of time and computing resources, I will use a base transformer model to process the text data. I will then add my own custom layers on top of the base model, and train these custom layers on the target data - the ratings.\n",
    "\n",
    "I decided to use DistilBert as the base transformer. According to its [Hugging Face model card](https://huggingface.co/docs/transformers/model_doc/distilbert), DistilBERT is \"a small, fast, cheap and light Transformer model trained by distilling BERT base\". Specifically I will use the \"distilbert-base-uncased\" model, which is a base model (designed to be finetuned for various tasks) that is uncased (meaning that it does not distinguish between upper and lower case)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "1aa30345",
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint = 'distilbert-base-uncased'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5cb933b",
   "metadata": {},
   "source": [
    "In order to be inputted into the Transformer model, the data needs to be processed by a tokenizer. The tokenizer takes the words and basically assigns an integer to each word. This tokenizer is defined by the Transformer model, so it is important that we use the tokenizer for the particular model that we will use."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "5642e9d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(checkpoint)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d6348ea",
   "metadata": {},
   "source": [
    "I want to truncate the data (discarding words after the ```max_length``` has been reached) and add padding (zeros) up until the ```max_length```. This ```max_length``` is the maximum number of words from the text that will be inputted into the model. This is a hyperparameter that I will tune, up to the maximum value of 512 which is the maximum length that ```distilbert-base-uncased``` can read."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "cad2c4e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize_function(data):\n",
    "    return tokenizer(data[\"text\"], truncation=True, padding=True, max_length=params['max_length'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa8e9a3b",
   "metadata": {},
   "source": [
    "The ```data_collator``` function will take this tokenized data and return tensorflow tensors, which are the format that the base model expects as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "289fb6ef",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorWithPadding(tokenizer=tokenizer, return_tensors=\"tf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e6f15e5",
   "metadata": {},
   "source": [
    "I have defined a single ```prepare_data()``` function below, that takes the hyperparameters as input and returns the tensorflow tensors that can be inputted into the model.\n",
    "\n",
    "I am using the 25,000 ```train``` reviews as training data and the 25,000 ```test``` reviews as validation data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "id": "c6baa2c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "def prepare_data(params):\n",
    "    max_rows = params['max_rows']\n",
    "    max_length = params['max_length']\n",
    "    smaller_data = data.filter(lambda e, i: (i in range(int(max_rows / 2)) or i in range(12500, 12500 + int(max_rows / 2))), with_indices=True)\n",
    "    tokenized_data = smaller_data.map(tokenize_function, batched=True)\n",
    "    \n",
    "    tf_train_dataset = tokenized_data[\"train\"].to_tf_dataset(\n",
    "    columns=[\"attention_mask\", \"input_ids\"],\n",
    "    label_cols=[\"rating\"],\n",
    "    shuffle=True,\n",
    "    collate_fn=data_collator,\n",
    "    batch_size=params['batch_size'],\n",
    "        )\n",
    "    tf_val_dataset = tokenized_data[\"test\"].to_tf_dataset(\n",
    "        columns=[\"attention_mask\", \"input_ids\"],\n",
    "        label_cols=[\"rating\"],\n",
    "        shuffle=True,\n",
    "        collate_fn=data_collator,\n",
    "        batch_size=params['batch_size'],\n",
    "        )\n",
    "    return tf_train_dataset, tf_val_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387a74e7",
   "metadata": {},
   "source": [
    "The function ```config_model``` again takes the various hyperparameters from ```params``` and configures the base model accordingly. It also sets the base layers as not trainable, since I don't want to train the base model - I only want to train the custom layers that I am adding.\n",
    "\n",
    "The following two functions were mostly copied from [this post](https://towardsdatascience.com/hugging-face-transformers-fine-tuning-distilbert-for-binary-classification-tasks-490f1d192379) from Towards Data Science."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "5e50ba19",
   "metadata": {},
   "outputs": [],
   "source": [
    "def config_model(params):\n",
    " \n",
    "    # Configure DistilBERT's initialization\n",
    "    config = DistilBertConfig(dropout=params['layer_dropout'] , \n",
    "                              attention_dropout=params['att_dropout'], \n",
    "                              output_hidden_states=True)\n",
    "\n",
    "    # The bare, pre-trained DistilBERT transformer model outputting raw hidden-states \n",
    "    # and without any specific head on top.\n",
    "    distilBERT = TFDistilBertModel.from_pretrained(checkpoint, config=config)\n",
    "\n",
    "    # Make DistilBERT layers untrainable\n",
    "    for layer in distilBERT.layers:\n",
    "        layer.trainable = False\n",
    "    return distilBERT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86355ca2",
   "metadata": {},
   "source": [
    "The function ```build_model()``` takes this base transformer model as an input, along with the hyperparameters ```params``` and adds the following customs layers:\n",
    "\n",
    "* A GlobalAvgPool1D layer to reduce the dimension to a single vector\n",
    "* A 128-unit dense layer, with RELU activation\n",
    "* A 128-unit dense layer, with RELU activation\n",
    "* A 64-unit dense layer, with RELU activation\n",
    "* A 10-unit dense layer, with RELU activation\n",
    "* A 1-unit dense layer, with no activation, to output the single rating prediction\n",
    "\n",
    "I did not have time to systematically vary this architecture, so I made a few trial and error assessments, and went with this. If I had more time, tuning this architecture would be the next thing I would focus on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37631b4d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model(transformer, params):\n",
    "    \"\"\"\n",
    "    Template for building a model off of the BERT or DistilBERT architecture\n",
    "    for a binary classification task.\n",
    "    \n",
    "    Input:\n",
    "      - transformer:  a base Hugging Face transformer model object (BERT or DistilBERT)\n",
    "                      with no added classification head attached.\n",
    "      - max_length:   integer controlling the maximum number of encoded tokens \n",
    "                      in a given sequence.\n",
    "    \n",
    "    Output:\n",
    "      - model:        a compiled tf.keras.Model with added classification layers \n",
    "                      on top of the base pre-trained model architecture.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Define weight initializer with a random seed to ensure reproducibility\n",
    "    weight_initializer = tf.keras.initializers.GlorotNormal(seed=RANDOM_STATE) \n",
    "    \n",
    "    # Define input layers\n",
    "    input_ids_layer = tf.keras.layers.Input(shape=(params['max_length'],), \n",
    "                                            name='input_ids', \n",
    "                                            dtype='int32')\n",
    "    input_attention_layer = tf.keras.layers.Input(shape=(params['max_length'],), \n",
    "                                                  name='attention_mask', \n",
    "                                                  dtype='int32')\n",
    "    \n",
    "    # DistilBERT outputs a tuple where the first element at index 0\n",
    "    # represents the hidden-state at the output of the model's last layer.\n",
    "    # It is a tf.Tensor of shape (batch_size, sequence_length, hidden_size=768).\n",
    "    last_hidden_state = transformer([input_ids_layer, input_attention_layer])[0]\n",
    "    \n",
    "    ##                                                 ##\n",
    "    ## Define additional dropout and dense layers here ##\n",
    "    ##                                            \n",
    "    \n",
    "    # vector = tf.reshape(last_hidden_state, [-1])\n",
    "\n",
    "    vector = tf.keras.layers.GlobalAvgPool1D()(last_hidden_state)\n",
    "\n",
    "    output1 = tf.keras.layers.Dense(128, \n",
    "                                   activation='relu',\n",
    "                                   kernel_initializer=weight_initializer,  \n",
    "                                   kernel_constraint=None,\n",
    "                                   bias_initializer='zeros'\n",
    "                                   )(vector)\n",
    "    \n",
    "    output2 = tf.keras.layers.Dense(128, \n",
    "                                   activation='relu',\n",
    "                                   kernel_initializer=weight_initializer,  \n",
    "                                   kernel_constraint=None,\n",
    "                                   bias_initializer='zeros'\n",
    "                                   )(output1)\n",
    "    \n",
    "    output3 = tf.keras.layers.Dense(64, \n",
    "                                   activation='relu',\n",
    "                                   kernel_initializer=weight_initializer,  \n",
    "                                   kernel_constraint=None,\n",
    "                                   bias_initializer='zeros'\n",
    "                                   )(output2)\n",
    "\n",
    "    output4 = tf.keras.layers.Dense(10, \n",
    "                                   activation='relu',\n",
    "                                   kernel_initializer=weight_initializer,  \n",
    "                                   kernel_constraint=None,\n",
    "                                   bias_initializer='zeros'\n",
    "                                   )(output3)\n",
    "    \n",
    "    # Define a single node that makes up the output layer (for a numerical output)\n",
    "    output5 = tf.keras.layers.Dense(1, \n",
    "                                   activation=None,\n",
    "                                   kernel_initializer=weight_initializer,  \n",
    "                                   kernel_constraint=None,\n",
    "                                   bias_initializer='zeros'\n",
    "                                   )(output4)\n",
    "    \n",
    "    # Define the model\n",
    "    model = tf.keras.Model([input_ids_layer, input_attention_layer], output5)\n",
    "    \n",
    "    # Compile the model\n",
    "    model.compile(tf.keras.optimizers.Adam(learning_rate=params['lr']), \n",
    "                  loss=MeanSquaredError(),\n",
    "                  metrics=['mse'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "e5315886",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some layers from the model checkpoint at distilbert-base-uncased were not used when initializing TFDistilBertModel: ['vocab_projector', 'vocab_layer_norm', 'activation_13', 'vocab_transform']\n",
      "- This IS expected if you are initializing TFDistilBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFDistilBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFDistilBertModel were initialized from the model checkpoint at distilbert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertModel for predictions without further training.\n"
     ]
    }
   ],
   "source": [
    "distilBERT = config_model(params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "id": "fbc893af",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = build_model(distilBERT, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "id": "35eb2b62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " input_ids (InputLayer)         [(None, 512)]        0           []                               \n",
      "                                                                                                  \n",
      " attention_mask (InputLayer)    [(None, 512)]        0           []                               \n",
      "                                                                                                  \n",
      " tf_distil_bert_model_1 (TFDist  TFBaseModelOutput(l  66362880   ['input_ids[0][0]',              \n",
      " ilBertModel)                   ast_hidden_state=(N               'attention_mask[0][0]']         \n",
      "                                one, 512, 768),                                                   \n",
      "                                 hidden_states=((No                                               \n",
      "                                ne, 512, 768),                                                    \n",
      "                                 (None, 512, 768),                                                \n",
      "                                 (None, 512, 768),                                                \n",
      "                                 (None, 512, 768),                                                \n",
      "                                 (None, 512, 768),                                                \n",
      "                                 (None, 512, 768),                                                \n",
      "                                 (None, 512, 768)),                                               \n",
      "                                 attentions=None)                                                 \n",
      "                                                                                                  \n",
      " global_average_pooling1d_1 (Gl  (None, 768)         0           ['tf_distil_bert_model_1[0][7]'] \n",
      " obalAveragePooling1D)                                                                            \n",
      "                                                                                                  \n",
      " dense_5 (Dense)                (None, 128)          98432       ['global_average_pooling1d_1[0][0\n",
      "                                                                 ]']                              \n",
      "                                                                                                  \n",
      " dense_6 (Dense)                (None, 128)          16512       ['dense_5[0][0]']                \n",
      "                                                                                                  \n",
      " dense_7 (Dense)                (None, 64)           8256        ['dense_6[0][0]']                \n",
      "                                                                                                  \n",
      " dense_8 (Dense)                (None, 10)           650         ['dense_7[0][0]']                \n",
      "                                                                                                  \n",
      " dense_9 (Dense)                (None, 1)            11          ['dense_8[0][0]']                \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 66,486,741\n",
      "Trainable params: 123,861\n",
      "Non-trainable params: 66,362,880\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5c69075",
   "metadata": {},
   "source": [
    "# Model Training\n",
    "As I do not have a GPU on my local machine, at this point I moved over to Kaggle to train the model:\n",
    "\n",
    "[MLZoomcamp Capstone NLP IMDB Rating Predictor\n",
    "](https://www.kaggle.com/markwoodward/mlzoomcamp-capstone-nlp-imdb-rating-predictor?scriptVersionId=82297560)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MLZoomCamp",
   "language": "python",
   "name": "mlzoomcamp"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}