C3W4_Predicting the Next Word.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bFWbEb6uGbN-"
      },
      "source": [
        "# Week 4: Predicting the next word\n",
        "\n",
        "Welcome to this assignment! During this week you saw how to create a model that will predict the next word in a text sequence, now you will implement such model and train it using a corpus of Shakespeare's sonnets, while also creating some helper functions to pre-process the data.\n",
        "\n",
        "\n",
        "Let's get started!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "AkcyE13FLAPZ",
        "outputId": "2f6471b4-362e-4fb2-f2b7-60203abcb7ee",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Reading package lists... Done\n",
            "Building dependency tree       \n",
            "Reading state information... Done\n",
            "The following package was automatically installed and is no longer required:\n",
            "  libnvidia-common-460\n",
            "Use 'apt autoremove' to remove it.\n",
            "The following packages will be REMOVED:\n",
            "  libcudnn8-dev\n",
            "The following held packages will be changed:\n",
            "  libcudnn8\n",
            "The following packages will be upgraded:\n",
            "  libcudnn8\n",
            "1 upgraded, 0 newly installed, 1 to remove and 18 not upgraded.\n",
            "Need to get 420 MB of archives.\n",
            "After this operation, 3,369 MB disk space will be freed.\n",
            "Get:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  libcudnn8 8.4.1.50-1+cuda11.6 [420 MB]\n",
            "Fetched 420 MB in 7s (64.0 MB/s)\n",
            "(Reading database ... 155569 files and directories currently installed.)\n",
            "Removing libcudnn8-dev (8.0.5.39-1+cuda11.1) ...\n",
            "(Reading database ... 155547 files and directories currently installed.)\n",
            "Preparing to unpack .../libcudnn8_8.4.1.50-1+cuda11.6_amd64.deb ...\n",
            "Unpacking libcudnn8 (8.4.1.50-1+cuda11.6) over (8.0.5.39-1+cuda11.1) ...\n",
            "Setting up libcudnn8 (8.4.1.50-1+cuda11.6) ...\n"
          ]
        }
      ],
      "source": [
        "# Install this package to use Colab's GPU for training\n",
        "!apt install --allow-change-held-packages libcudnn8=8.4.1.50-1+cuda11.6"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "BOwsuGQQY9OL",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "import numpy as np \n",
        "import matplotlib.pyplot as plt\n",
        "from tensorflow.keras.models import Sequential\n",
        "from tensorflow.keras.utils import to_categorical \n",
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from tensorflow.keras.layers import Embedding, LSTM, Dense, Bidirectional"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BTxqlHqKHzhr"
      },
      "source": [
        "For this assignment you will be using the [Shakespeare Sonnets Dataset](https://www.opensourceshakespeare.org/views/sonnets/sonnet_view.php?range=viewrange&sonnetrange1=1&sonnetrange2=154), which contains more than 2000 lines of text extracted from Shakespeare's sonnets."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "WZ4qOUzujMP6",
        "tags": [
          "graded"
        ],
        "outputId": "4a2d811e-4e4c-48ad-b742-ecf6294bc682",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/usr/local/lib/python3.7/dist-packages/gdown/cli.py:131: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.\n",
            "  category=FutureWarning,\n",
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=108jAePKK4R3BVYBbYJZ32JWUwxeMg20K\n",
            "To: /content/sonnets.txt\n",
            "100% 93.6k/93.6k [00:00<00:00, 79.6MB/s]\n"
          ]
        }
      ],
      "source": [
        "# sonnets.txt\n",
        "!gdown --id 108jAePKK4R3BVYBbYJZ32JWUwxeMg20K"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "Pfd-nYKij5yY",
        "tags": [
          "graded"
        ],
        "outputId": "16033609-d967-4390-b589-8fc495bd6ffa",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 2159 lines of sonnets\n",
            "\n",
            "The first 5 lines look like this:\n",
            "\n",
            "from fairest creatures we desire increase,\n",
            "that thereby beauty's rose might never die,\n",
            "but as the riper should by time decease,\n",
            "his tender heir might bear his memory:\n",
            "but thou, contracted to thine own bright eyes,\n"
          ]
        }
      ],
      "source": [
        "# Define path for file with sonnets\n",
        "SONNETS_FILE = './sonnets.txt'\n",
        "\n",
        "# Read the data\n",
        "with open('./sonnets.txt') as f:\n",
        "    data = f.read()\n",
        "\n",
        "# Convert to lower case and save as a list\n",
        "corpus = data.lower().split(\"\\n\")\n",
        "\n",
        "print(f\"There are {len(corpus)} lines of sonnets\\n\")\n",
        "print(f\"The first 5 lines look like this:\\n\")\n",
        "for i in range(5):\n",
        "  print(corpus[i])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "imB15zrSNhA1"
      },
      "source": [
        "## Tokenizing the text\n",
        "\n",
        "Now fit the Tokenizer to the corpus and save the total number of words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "AAhM_qAZk0o5",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "tokenizer = Tokenizer()\n",
        "tokenizer.fit_on_texts(corpus)\n",
        "total_words = len(tokenizer.word_index) + 1"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77-0sA46OETa"
      },
      "source": [
        "When converting the text into sequences you can use the `texts_to_sequences` method as you have done throughout this course.\n",
        "\n",
        "In the next graded function you will need to process this corpus one line at a time. Given this, it is important to keep in mind that the way you are feeding the data unto this method affects the result. Check the following example to make this clearer.\n",
        "\n",
        "The first example of the corpus is a string and looks like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "tqhPxdeXlfjh",
        "tags": [
          "graded"
        ],
        "outputId": "f6bef1ad-313a-4994-cddc-1559caeeb2ce",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 35
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'from fairest creatures we desire increase,'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "corpus[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XFMP4z11O3os"
      },
      "source": [
        "If you pass this text directly into the `texts_to_sequences` method you will get an unexpected result:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "EMSEhmbzNZCE",
        "tags": [
          "graded"
        ],
        "outputId": "f50c8e01-34ad-402c-d1a0-49f81dac2ac8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[],\n",
              " [],\n",
              " [58],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [17],\n",
              " [6],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [17],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [6],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [6],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [],\n",
              " [17],\n",
              " [],\n",
              " [],\n",
              " []]"
            ]
          },
          "metadata": {},
          "execution_count": 7
        }
      ],
      "source": [
        "tokenizer.texts_to_sequences(corpus[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pPZmZtpEPEeI"
      },
      "source": [
        "This happened because `texts_to_sequences` expects a list and you are providing a string. However a string is still and `iterable` in Python so you will get the word index of every character in the string.\n",
        "\n",
        "Instead you need to place the example whithin a list before passing it to the method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "Qmgo-vXhk4nd",
        "tags": [
          "graded"
        ],
        "outputId": "dee53fe1-c403-458f-a06d-a6aed343f587",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[34, 417, 877, 166, 213, 517]]"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "tokenizer.texts_to_sequences([corpus[0]])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0DU7wK-eQ5dc"
      },
      "source": [
        "Notice that you received the sequence wrapped inside a list so in order to get only the desired sequence you need to explicitly get the first item in the list like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "kpTy8WmIQ57P",
        "tags": [
          "graded"
        ],
        "outputId": "133afe53-f509-4c24-ad80-175191ae6992",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[34, 417, 877, 166, 213, 517]"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "tokenizer.texts_to_sequences([corpus[0]])[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-oqy9KjXRJ9A"
      },
      "source": [
        "## Generating n_grams\n",
        "\n",
        "Now complete the `n_gram_seqs` function below. This function receives the fitted tokenizer and the corpus (which is a list of strings) and should return a list containing the `n_gram` sequences for each line in the corpus:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "iy4baJMDl6kj",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# GRADED FUNCTION: n_gram_seqs\n",
        "def n_gram_seqs(corpus, tokenizer):\n",
        "    \"\"\"\n",
        "    Generates a list of n-gram sequences\n",
        "    \n",
        "    Args:\n",
        "        corpus (list of string): lines of texts to generate n-grams for\n",
        "        tokenizer (object): an instance of the Tokenizer class containing the word-index dictionary\n",
        "    \n",
        "    Returns:\n",
        "        input_sequences (list of int): the n-gram sequences for each line in the corpus\n",
        "    \"\"\"\n",
        "    input_sequences = []\n",
        "    \n",
        "    ### START CODE HERE\n",
        "    for line in corpus:\n",
        "      token_list = tokenizer.texts_to_sequences([line])[0]\n",
        "\n",
        "      for i in range(1, len(token_list)):\n",
        "        n_grams_seq = token_list[:i+1]\n",
        "        input_sequences.append(n_grams_seq)\n",
        "    ### END CODE HERE\n",
        "    \n",
        "    return input_sequences"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "DlKqW2pfM7G3",
        "tags": [
          "graded"
        ],
        "outputId": "c7a83bdf-3fad-429c-d89e-469e6a8ff843",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "n_gram sequences for first example look like this:\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[34, 417],\n",
              " [34, 417, 877],\n",
              " [34, 417, 877, 166],\n",
              " [34, 417, 877, 166, 213],\n",
              " [34, 417, 877, 166, 213, 517]]"
            ]
          },
          "metadata": {},
          "execution_count": 11
        }
      ],
      "source": [
        "# Test your function with one example\n",
        "first_example_sequence = n_gram_seqs([corpus[0]], tokenizer)\n",
        "\n",
        "print(\"n_gram sequences for first example look like this:\\n\")\n",
        "first_example_sequence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0HL8Ug6UU0Jt"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "n_gram sequences for first example look like this:\n",
        "\n",
        "[[34, 417],\n",
        " [34, 417, 877],\n",
        " [34, 417, 877, 166],\n",
        " [34, 417, 877, 166, 213],\n",
        " [34, 417, 877, 166, 213, 517]]\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "id": "wtPpCcBjNc4c",
        "tags": [
          "graded"
        ],
        "outputId": "133a8d0e-4ac4-4427-94ae-90fbba50a337",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "n_gram sequences for next 3 examples look like this:\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[[8, 878],\n",
              " [8, 878, 134],\n",
              " [8, 878, 134, 351],\n",
              " [8, 878, 134, 351, 102],\n",
              " [8, 878, 134, 351, 102, 156],\n",
              " [8, 878, 134, 351, 102, 156, 199],\n",
              " [16, 22],\n",
              " [16, 22, 2],\n",
              " [16, 22, 2, 879],\n",
              " [16, 22, 2, 879, 61],\n",
              " [16, 22, 2, 879, 61, 30],\n",
              " [16, 22, 2, 879, 61, 30, 48],\n",
              " [16, 22, 2, 879, 61, 30, 48, 634],\n",
              " [25, 311],\n",
              " [25, 311, 635],\n",
              " [25, 311, 635, 102],\n",
              " [25, 311, 635, 102, 200],\n",
              " [25, 311, 635, 102, 200, 25],\n",
              " [25, 311, 635, 102, 200, 25, 278]]"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "# Test your function with a bigger corpus\n",
        "next_3_examples_sequence = n_gram_seqs(corpus[1:4], tokenizer)\n",
        "\n",
        "print(\"n_gram sequences for next 3 examples look like this:\\n\")\n",
        "next_3_examples_sequence"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EIzecMczU9UB"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "n_gram sequences for next 3 examples look like this:\n",
        "\n",
        "[[8, 878],\n",
        " [8, 878, 134],\n",
        " [8, 878, 134, 351],\n",
        " [8, 878, 134, 351, 102],\n",
        " [8, 878, 134, 351, 102, 156],\n",
        " [8, 878, 134, 351, 102, 156, 199],\n",
        " [16, 22],\n",
        " [16, 22, 2],\n",
        " [16, 22, 2, 879],\n",
        " [16, 22, 2, 879, 61],\n",
        " [16, 22, 2, 879, 61, 30],\n",
        " [16, 22, 2, 879, 61, 30, 48],\n",
        " [16, 22, 2, 879, 61, 30, 48, 634],\n",
        " [25, 311],\n",
        " [25, 311, 635],\n",
        " [25, 311, 635, 102],\n",
        " [25, 311, 635, 102, 200],\n",
        " [25, 311, 635, 102, 200, 25],\n",
        " [25, 311, 635, 102, 200, 25, 278]]\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dx3V_RjFWQSu"
      },
      "source": [
        "Apply the `n_gram_seqs` transformation to the whole corpus and save the maximum sequence length to use it later:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "id": "laMwiRUpmuSd",
        "tags": [
          "graded"
        ],
        "outputId": "803f5dd5-e1ec-4cf0-b47d-a1e763e675db",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "n_grams of input_sequences have length: 15462\n",
            "maximum length of sequences is: 11\n"
          ]
        }
      ],
      "source": [
        "# Apply the n_gram_seqs transformation to the whole corpus\n",
        "input_sequences = n_gram_seqs(corpus, tokenizer)\n",
        "\n",
        "# Save max length \n",
        "max_sequence_len = max([len(x) for x in input_sequences])\n",
        "\n",
        "print(f\"n_grams of input_sequences have length: {len(input_sequences)}\")\n",
        "print(f\"maximum length of sequences is: {max_sequence_len}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2OciMdmEdE9L"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "n_grams of input_sequences have length: 15462\n",
        "maximum length of sequences is: 11\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zHY7HroqWq12"
      },
      "source": [
        "## Add padding to the sequences\n",
        "\n",
        "Now code the `pad_seqs` function which will pad any given sequences to the desired maximum length. Notice that this function receives a list of sequences and should return a numpy array with the padded sequences: "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "cellView": "code",
        "id": "WW1-qAZaWOhC",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# GRADED FUNCTION: pad_seqs\n",
        "def pad_seqs(input_sequences, maxlen):\n",
        "    \"\"\"\n",
        "    Pads tokenized sequences to the same length\n",
        "    \n",
        "    Args:\n",
        "        input_sequences (list of int): tokenized sequences to pad\n",
        "        maxlen (int): maximum length of the token sequences\n",
        "    \n",
        "    Returns:\n",
        "        padded_sequences (array of int): tokenized sequences padded to the same length\n",
        "    \"\"\"\n",
        "    ### START CODE HERE\n",
        "    padded_sequences = pad_sequences(input_sequences, maxlen= maxlen, padding= 'pre')\n",
        "    \n",
        "    return padded_sequences\n",
        "    ### END CODE HERE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "id": "IqVQ0pb3YHLr",
        "tags": [
          "graded"
        ],
        "outputId": "de031791-2011-4587-c8b9-47ae87b76120",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0,  34, 417],\n",
              "       [  0,   0,  34, 417, 877],\n",
              "       [  0,  34, 417, 877, 166],\n",
              "       [ 34, 417, 877, 166, 213],\n",
              "       [417, 877, 166, 213, 517]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 15
        }
      ],
      "source": [
        "# Test your function with the n_grams_seq of the first example\n",
        "first_padded_seq = pad_seqs(first_example_sequence, len(first_example_sequence))\n",
        "first_padded_seq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Re_avDznXRnU"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "array([[  0,   0,   0,  34, 417],\n",
        "       [  0,   0,  34, 417, 877],\n",
        "       [  0,  34, 417, 877, 166],\n",
        "       [ 34, 417, 877, 166, 213],\n",
        "       [417, 877, 166, 213, 517]], dtype=int32)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "id": "j56_UCOBYzZt",
        "tags": [
          "graded"
        ],
        "outputId": "c2bdfd61-5f0e-495c-f1e0-725a808daf83",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0,   0,   0,   0,   8, 878],\n",
              "       [  0,   0,   0,   0,   0,   8, 878, 134],\n",
              "       [  0,   0,   0,   0,   8, 878, 134, 351],\n",
              "       [  0,   0,   0,   8, 878, 134, 351, 102],\n",
              "       [  0,   0,   8, 878, 134, 351, 102, 156],\n",
              "       [  0,   8, 878, 134, 351, 102, 156, 199],\n",
              "       [  0,   0,   0,   0,   0,   0,  16,  22],\n",
              "       [  0,   0,   0,   0,   0,  16,  22,   2],\n",
              "       [  0,   0,   0,   0,  16,  22,   2, 879],\n",
              "       [  0,   0,   0,  16,  22,   2, 879,  61],\n",
              "       [  0,   0,  16,  22,   2, 879,  61,  30],\n",
              "       [  0,  16,  22,   2, 879,  61,  30,  48],\n",
              "       [ 16,  22,   2, 879,  61,  30,  48, 634],\n",
              "       [  0,   0,   0,   0,   0,   0,  25, 311],\n",
              "       [  0,   0,   0,   0,   0,  25, 311, 635],\n",
              "       [  0,   0,   0,   0,  25, 311, 635, 102],\n",
              "       [  0,   0,   0,  25, 311, 635, 102, 200],\n",
              "       [  0,   0,  25, 311, 635, 102, 200,  25],\n",
              "       [  0,  25, 311, 635, 102, 200,  25, 278]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 16
        }
      ],
      "source": [
        "# Test your function with the n_grams_seq of the next 3 examples\n",
        "next_3_padded_seq = pad_seqs(next_3_examples_sequence, max([len(s) for s in next_3_examples_sequence]))\n",
        "next_3_padded_seq"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3rmcDluOXcIU"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "array([[  0,   0,   0,   0,   0,   0,   8, 878],\n",
        "       [  0,   0,   0,   0,   0,   8, 878, 134],\n",
        "       [  0,   0,   0,   0,   8, 878, 134, 351],\n",
        "       [  0,   0,   0,   8, 878, 134, 351, 102],\n",
        "       [  0,   0,   8, 878, 134, 351, 102, 156],\n",
        "       [  0,   8, 878, 134, 351, 102, 156, 199],\n",
        "       [  0,   0,   0,   0,   0,   0,  16,  22],\n",
        "       [  0,   0,   0,   0,   0,  16,  22,   2],\n",
        "       [  0,   0,   0,   0,  16,  22,   2, 879],\n",
        "       [  0,   0,   0,  16,  22,   2, 879,  61],\n",
        "       [  0,   0,  16,  22,   2, 879,  61,  30],\n",
        "       [  0,  16,  22,   2, 879,  61,  30,  48],\n",
        "       [ 16,  22,   2, 879,  61,  30,  48, 634],\n",
        "       [  0,   0,   0,   0,   0,   0,  25, 311],\n",
        "       [  0,   0,   0,   0,   0,  25, 311, 635],\n",
        "       [  0,   0,   0,   0,  25, 311, 635, 102],\n",
        "       [  0,   0,   0,  25, 311, 635, 102, 200],\n",
        "       [  0,   0,  25, 311, 635, 102, 200,  25],\n",
        "       [  0,  25, 311, 635, 102, 200,  25, 278]], dtype=int32)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "id": "rgK-Q_micEYA",
        "tags": [
          "graded"
        ],
        "outputId": "35e04fd4-fc84-4e76-8030-8920408d5b70",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "padded corpus has shape: (15462, 11)\n"
          ]
        }
      ],
      "source": [
        "# Pad the whole corpus\n",
        "input_sequences = pad_seqs(input_sequences, max_sequence_len)\n",
        "\n",
        "print(f\"padded corpus has shape: {input_sequences.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "59RD1YYNc7CW"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "padded corpus has shape: (15462, 11)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZbOidyPrXxf7"
      },
      "source": [
        "## Split the data into features and labels\n",
        "\n",
        "Before feeding the data into the neural network you should split it into features and labels. In this case the features will be the padded n_gram sequences with the last word removed from them and the labels will be the removed word.\n",
        "\n",
        "Complete the `features_and_labels` function below. This function expects the padded n_gram sequences as input and should return a tuple containing the features and the one hot encoded labels.\n",
        "\n",
        "Notice that the function also receives the total of words in the corpus, this parameter will be very important when one hot enconding the labels since every word in the corpus will be a label at least once. If you need a refresh of how the `to_categorical` function works take a look at the [docs](https://www.tensorflow.org/api_docs/python/tf/keras/utils/to_categorical)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 20,
      "metadata": {
        "cellView": "code",
        "id": "9WGGbYdnZdmJ",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# GRADED FUNCTION: features_and_labels\n",
        "def features_and_labels(input_sequences, total_words):\n",
        "    \"\"\"\n",
        "    Generates features and labels from n-grams\n",
        "    \n",
        "    Args:\n",
        "        input_sequences (list of int): sequences to split features and labels from\n",
        "        total_words (int): vocabulary size\n",
        "    \n",
        "    Returns:\n",
        "        features, one_hot_labels (array of int, array of int): arrays of features and one-hot encoded labels\n",
        "    \"\"\"\n",
        "    ### START CODE HERE\n",
        "    features = input_sequences[:,:-1]\n",
        "    labels = input_sequences[:,-1]\n",
        "    one_hot_labels = to_categorical(labels, num_classes= total_words)\n",
        "    ### END CODE HERE\n",
        "\n",
        "    return features, one_hot_labels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 21,
      "metadata": {
        "id": "23DolaBRaIAZ",
        "tags": [
          "graded"
        ],
        "outputId": "a40acb4b-07bb-4f57-ca19-7ca89c43c298",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "labels have shape: (5, 3211)\n",
            "\n",
            "features look like this:\n",
            "\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[  0,   0,   0,  34],\n",
              "       [  0,   0,  34, 417],\n",
              "       [  0,  34, 417, 877],\n",
              "       [ 34, 417, 877, 166],\n",
              "       [417, 877, 166, 213]], dtype=int32)"
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ],
      "source": [
        "# Test your function with the padded n_grams_seq of the first example\n",
        "first_features, first_labels = features_and_labels(first_padded_seq, total_words)\n",
        "\n",
        "print(f\"labels have shape: {first_labels.shape}\")\n",
        "print(\"\\nfeatures look like this:\\n\")\n",
        "first_features"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7t4yAx2UaQ43"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "labels have shape: (5, 3211)\n",
        "\n",
        "features look like this:\n",
        "\n",
        "array([[  0,   0,   0,  34],\n",
        "       [  0,   0,  34, 417],\n",
        "       [  0,  34, 417, 877],\n",
        "       [ 34, 417, 877, 166],\n",
        "       [417, 877, 166, 213]], dtype=int32)\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "GRTuLEt3bRKa",
        "tags": [
          "graded"
        ],
        "outputId": "fd894fbb-ebe9-4e19-cf7b-9fff22f1eaa2",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "features have shape: (15462, 10)\n",
            "labels have shape: (15462, 3211)\n"
          ]
        }
      ],
      "source": [
        "# Split the whole corpus\n",
        "features, labels = features_and_labels(input_sequences, total_words)\n",
        "\n",
        "print(f\"features have shape: {features.shape}\")\n",
        "print(f\"labels have shape: {labels.shape}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xXSMK_HpdLns"
      },
      "source": [
        "**Expected Output:**\n",
        "\n",
        "```\n",
        "features have shape: (15462, 10)\n",
        "labels have shape: (15462, 3211)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ltxaOCE_aU6J"
      },
      "source": [
        "## Create the model\n",
        "\n",
        "Now you should define a model architecture capable of achieving an accuracy of at least 80%.\n",
        "\n",
        "Some hints to help you in this task:\n",
        "\n",
        "- An appropriate `output_dim` for the first layer (Embedding) is 100, this is already provided for you.\n",
        "- A Bidirectional LSTM is helpful for this particular problem.\n",
        "- The last layer should have the same number of units as the total number of words in the corpus and a softmax activation function.\n",
        "- This problem can be solved with only two layers (excluding the Embedding) so try out small architectures first."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 34,
      "metadata": {
        "cellView": "code",
        "id": "XrE6kpJFfvRY",
        "tags": [
          "graded"
        ]
      },
      "outputs": [],
      "source": [
        "# GRADED FUNCTION: create_model\n",
        "def create_model(total_words, max_sequence_len):\n",
        "    \"\"\"\n",
        "    Creates a text generator model\n",
        "    \n",
        "    Args:\n",
        "        total_words (int): size of the vocabulary for the Embedding layer input\n",
        "        max_sequence_len (int): length of the input sequences\n",
        "    \n",
        "    Returns:\n",
        "        model (tf.keras Model): the text generator model\n",
        "    \"\"\"\n",
        "    model = Sequential()\n",
        "    ### START CODE HERE\n",
        "    model.add(Embedding(total_words, 100, input_length= max_sequence_len-1))\n",
        "    model.add(Bidirectional(LSTM(100)))\n",
        "    model.add(Dense(total_words, activation='softmax'))\n",
        "\n",
        "\n",
        "    # Compile the model\n",
        "    model.compile(loss= 'categorical_crossentropy',\n",
        "                  optimizer= 'adam',\n",
        "                  metrics=['accuracy'])\n",
        "    \n",
        "    ### END CODE HERE\n",
        "\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "0IpX_Gu_gISk",
        "tags": [],
        "outputId": "1340c731-8704-4406-a4d2-032ad64573b9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/50\n",
            "484/484 [==============================] - 6s 7ms/step - loss: 6.8872 - accuracy: 0.0244\n",
            "Epoch 2/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 6.4448 - accuracy: 0.0319\n",
            "Epoch 3/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 6.2311 - accuracy: 0.0395\n",
            "Epoch 4/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 5.9904 - accuracy: 0.0477\n",
            "Epoch 5/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 5.7407 - accuracy: 0.0568\n",
            "Epoch 6/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 5.4672 - accuracy: 0.0693\n",
            "Epoch 7/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 5.1717 - accuracy: 0.0816\n",
            "Epoch 8/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 4.8545 - accuracy: 0.1014\n",
            "Epoch 9/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 4.5378 - accuracy: 0.1256\n",
            "Epoch 10/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 4.2235 - accuracy: 0.1638\n",
            "Epoch 11/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 3.9157 - accuracy: 0.2108\n",
            "Epoch 12/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 3.6307 - accuracy: 0.2552\n",
            "Epoch 13/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 3.3654 - accuracy: 0.3060\n",
            "Epoch 14/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 3.1167 - accuracy: 0.3530\n",
            "Epoch 15/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 2.8865 - accuracy: 0.3981\n",
            "Epoch 16/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 2.6789 - accuracy: 0.4367\n",
            "Epoch 17/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 2.4889 - accuracy: 0.4784\n",
            "Epoch 18/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 2.3161 - accuracy: 0.5119\n",
            "Epoch 19/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 2.1618 - accuracy: 0.5503\n",
            "Epoch 20/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 2.0122 - accuracy: 0.5781\n",
            "Epoch 21/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.8833 - accuracy: 0.6082\n",
            "Epoch 22/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.7559 - accuracy: 0.6342\n",
            "Epoch 23/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.6428 - accuracy: 0.6595\n",
            "Epoch 24/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.5388 - accuracy: 0.6833\n",
            "Epoch 25/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.4433 - accuracy: 0.7000\n",
            "Epoch 26/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.3597 - accuracy: 0.7200\n",
            "Epoch 27/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.2770 - accuracy: 0.7377\n",
            "Epoch 28/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.2056 - accuracy: 0.7500\n",
            "Epoch 29/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.1427 - accuracy: 0.7612\n",
            "Epoch 30/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.0804 - accuracy: 0.7760\n",
            "Epoch 31/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 1.0258 - accuracy: 0.7847\n",
            "Epoch 32/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.9829 - accuracy: 0.7937\n",
            "Epoch 33/50\n",
            "484/484 [==============================] - 4s 8ms/step - loss: 0.9255 - accuracy: 0.8072\n",
            "Epoch 34/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.8872 - accuracy: 0.8091\n",
            "Epoch 35/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.8592 - accuracy: 0.8161\n",
            "Epoch 36/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.8147 - accuracy: 0.8240\n",
            "Epoch 37/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.7918 - accuracy: 0.8280\n",
            "Epoch 38/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.7640 - accuracy: 0.8306\n",
            "Epoch 39/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.7443 - accuracy: 0.8360\n",
            "Epoch 40/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.7307 - accuracy: 0.8340\n",
            "Epoch 41/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6990 - accuracy: 0.8403\n",
            "Epoch 42/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6869 - accuracy: 0.8414\n",
            "Epoch 43/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6688 - accuracy: 0.8414\n",
            "Epoch 44/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6621 - accuracy: 0.8442\n",
            "Epoch 45/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6472 - accuracy: 0.8453\n",
            "Epoch 46/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6385 - accuracy: 0.8457\n",
            "Epoch 47/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6276 - accuracy: 0.8441\n",
            "Epoch 48/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6161 - accuracy: 0.8473\n",
            "Epoch 49/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6139 - accuracy: 0.8455\n",
            "Epoch 50/50\n",
            "484/484 [==============================] - 3s 7ms/step - loss: 0.6071 - accuracy: 0.8467\n"
          ]
        }
      ],
      "source": [
        "# Get the untrained model\n",
        "model = create_model(total_words, max_sequence_len)\n",
        "\n",
        "# Train the model\n",
        "history = model.fit(features, labels, epochs=50, verbose=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gy72RPgly55q"
      },
      "source": [
        "**To pass this assignment, your model should achieve a training accuracy of at least 80%**. If your model didn't achieve this threshold, try training again with a different model architecture, consider increasing the number of unit in your `LSTM` layer."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "1fXTEO3GJ282",
        "tags": [],
        "outputId": "c619f862-c347-4656-9db5-ec0a17f58120",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 545
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAfVUlEQVR4nO3deZgU1dn+8e/jyA6yCCqyCPojMS5RkhHRaBRFBReMignuGhXjgiFGhbiwGPV9DbhHfWVzV8CgMCqbCwQXQIa4ICgICAHEgIiiqCzD8/vjNKEdZ5hmprtruvr+XFddM1VdM/0UzNwcTp1zytwdERHJfTtFXYCIiKSHAl1EJCYU6CIiMaFAFxGJCQW6iEhMKNBFRGJCgS7ViplNMLML0n2uSD4wjUOXqjKzb5J26wIbgJLE/mXu/lT2qxLJPwp0SSszWwJc4u6vlPHazu6+OftV5Rb9OUllqctFMsbMjjaz5WbWx8w+Ax4xs8Zm9qKZrTaztYnPWyZ9zVQzuyTx+YVm9oaZDU6c+4mZda3kuW3NbJqZfW1mr5jZA2b2ZDl1V1RjEzN7xMw+Tbw+Num1U83sXTNbZ2aLzKxL4vgSM+ucdN6Are9vZm3MzM3sYjP7N/Ba4vizZvaZmX2VqH3/pK+vY2Z3mtnSxOtvJI69ZGa9Sl3P+2Z22o7+/UnuUaBLpu0BNAH2AnoSfuYeSey3Br4D/r6drz8UmA80Bf4GDDczq8S5TwNvA7sCA4DztvOeFdX4BKFraX9gN+BuADPrADwOXAc0An4NLNnO+5R2FPAz4ITE/gSgXeI9/gUkd10NBn4JHE74870e2AI8Bpy79SQzOwhoAby0A3VIrnJ3bdrSthECrHPi86OBjUDt7Zx/MLA2aX8qocsG4EJgYdJrdQEH9tiRcwmhvBmom/T6k8CTKV7Tf2sEmhOCs3EZ5z0M3F3Rn0tif8DW9wfaJGrdezs1NEqc05DwD853wEFlnFcbWAu0S+wPBh6M+udCW3Y2tdAl01a7+/dbd8ysrpk9nOgqWAdMAxqZWUE5X//Z1k/c/dvEp/V38Nw9gS+SjgEsK6/gCmpslfhea8v40lbAovK+bwr+W5OZFZjZ/ya6bdaxraXfNLHVLuu9En/Wo4BzzWwn4CzC/ygkDyjQJdNK33X/M/BT4FB334XQLQFQXjdKOqwEmphZ3aRjrbZz/vZqXJb4Xo3K+LplwD7lfM/1hP81bLVHGeck/1mdDZwKdCa0ytsk1fA58P123usx4BzgWOBbd59eznkSMwp0ybYGhO6CL82sCdA/02/o7kuBYmCAmdU0s8OAUypTo7uvJPRtP5i4eVrDzLYG/nDgIjM71sx2MrMWZrZv4rV3gR6J8wuB7hWU3YAw/HMN4R+C25Nq2AKMAO4ysz0TrfnDzKxW4vXphG6hO1HrPK8o0CXb7gHqEFqZM4CJWXrfc4DDCAF5K6FbYkM551ZU43nAJuAjYBXQG8Dd3wYuItwk/Qr4J+HGKsDNhBb1WmAg4Sbt9jwOLAVWAPMSdSS7FpgDzAK+AO7gh7/PjwMHEu4VSJ7QOHTJS2Y2CvjI3TP+P4QomNn5QE93PyLqWiR71EKXvGBmh5jZPomukC6E/umxFX1dLkrcK7gCGBJ1LZJdCnTJF3sQhjl+A9wHXO7u70RaUQaY2QnAauA/VNytIzGjLhcRkZhQC11EJCZ2juqNmzZt6m3atInq7UVEctLs2bM/d/dmZb0WWaC3adOG4uLiqN5eRCQnmdnS8l5Tl4uISEwo0EVEYkKBLiISEwp0EZGYUKCLiMSEAl1EJCYU6CIiMRHZOHQRkahs2hS2zZu3bSUl4WOdOtCwIdSoUfbXfv01LF0KS5aEbe1aaNwYmjTZtjVuDLvsAt99B998E77mm2+2bR06wE9/mv7rUqCLSLW3aROsXx8Ctyzr18Py5T/eVq4MAbp+PXz7bfi4ve+TrF49aNRo2/b99yHA16yp+vU8+KACXURiau1aePNNmDYN3noLVq/eFr7r14dA3xH16kGrVtC8Oey1F9StG45t3erWhZo1Yeedt20FBWH77jv48ssfbmvXQv36cMgh0KbND7cmTcI5X3zxw23duvA+9etv2xo0CB933z39f4agQBeRNPvuO3j3XZg1K2yffBK6MJo2hV133bbVqQPFxSHE58wB99DNUVgI7dv/MIC3buV1g9SuDS1bhq1Vq9DdYZl8Sm0pzZqFLWoKdBHZYevXw6efhi6NTz8N2/z5IcDnzNnWpbHHHqFr4bPP4IMPQnfF+vXbvk+9enD44dC9Oxx5JBx6aAh6qRwFuoiUad26ENLz58NHH4Vt/nz497/Da6U1ahRa19ddF7omDjkEWrT4cUt5w4YQ7F9/DXvvXX6rW3acAl1EgBC0U6ZAURGMHx9GcmxVUAD77BNa2506wZ57hq15822fN2qUWjdHrVrhfEk/BbpIHluzJoT3uHEwaVIYEVK3Lhx/PPzhDyHA9903hHnNmlFXKxVRoIvkkU2bYOZMmDwZXn4Z3n4btmwJLe1zzoFu3eCYY8JNRsk9KQV64inp9wIFwDB3/99Sr7cGHgMaJc7p6+7j01yriKTAPfRPr1kThs+tWQMffxwC/LXXwms77RQmt9x0E5xyCvziF+GY5LYKA93MCoAHgOOA5cAsMyty93lJp90EjHb3h8xsP2A80CYD9YpIGebOhf794fXXQ4iXNXGmbVs4++zQndKpU5jNKPGSSgu9A7DQ3RcDmNlI4FQgOdAd2CXxeUPg03QWKSJlW7QIBgyAp54Kk1a6dw+TVnbdNUx42Trmu0WLMAlG4i2VQG8BLEvaXw4cWuqcAcBkM+sF1AM6p6U6ESnTihVw660wbFiY5XjddXD99SG8JX+l66boWcCj7n6nmR0GPGFmB7j7luSTzKwn0BOgdevWaXprkfzgHmZWPvEEDB0aulV69oQbb9QwQAlSCfQVQKuk/ZaJY8kuBroAuPt0M6sNNAVWJZ/k7kOAIQCFhYVeyZpF8sq8efDMMzByJCxcGCbinHVW6DPfe++oq5PqJJVAnwW0M7O2hCDvAZxd6px/A8cCj5rZz4DawOp0FiqSTxYvhmefDUH+3nthBEqnTtC3L5x+um5oStkqDHR332xmVwGTCEMSR7j7XDO7BSh29yLgz8BQM/sT4Qbphe6uFrjIDli6NIT46NFhTRSAjh3h3nvht78N66KIbI9FlbuFhYVeXFwcyXuLVBfr1sGIETBqFMyYEY4VFoYAP/NMjUyRHzOz2e5eWNZrmikqEoGSEnjkkXBDc9WqsFzs//xPCPF99om6OslVCnSRLJs6FXr3Dn3jhx8OL7wQZm2KVJUm+4pkyaJF4YZmp07hCTgjR8IbbyjMJX3UQhfJsA8/hPvuC33lNWqECUHXXKMHOUj6KdBFMmDLlrAc7b33ho+1asH558PAgZoEJJmjQBdJo/Xr4fHHQ5DPnx+Wpf3rX+Gyy6rHMycl3hToImmwZg3cf3/YvvgiDD188skwakUPhpBsUaCLVMGKFXDXXfDww6F13q1bWCTr8MOz+9R5EVCgi1TKwoVwxx3w2GOhv/yss6BPHzjggKgrk3ymQBfZASUlMGgQ9OsX1le59FK49trw8AiRqCnQRVL08cdwwQUwfTqccUboL2/ePOqqRLbRxCKRCrjDgw/CwQeHMeVPPhkW0VKYS3WjFrrIdixfDhdfDJMnh2dxDh8OLVtGXZVI2dRCFylHUREceGCYnv/ggzBxosJcqjcFukgpmzfDDTfAqaeGlQ/few8uv1zDEKX6U5eLSJJVq8IQxNdeCyNY7rsPateOuiqR1CjQRRKmTw8zO9esCQtpXXRR1BWJ7Bh1uUjecw9DEH/96zBN/623FOaSmxTokteWLYNTToGrr4YTToDZs8PTg0RykQJd8tKWLWHkyv77w5QpYT2WoiJo3DjqykQqT33oknfmz4dLLgnDETt3hiFDNHVf4kEtdMkbmzbB7bfDQQfBBx+EG5+TJyvMJT7UQpe8sG5dGFc+dSp07x5ugu6xR9RViaSXAl1ib9Uq6NIF5swJy92ef37UFYlkhgJdYm3JkrAGy/Ll4aZn165RVySSOQp0ia25c0OYf/stvPJKeIqQSJzppqjE0vTpcOSRYdLQtGkKc8kPCnSJnUmTwnDEJk3gzTfDioki+UCBLrFSVBRmfv7kJyHMNSRR8okCXWJjzJjwaLj27cPsz913j7oikexSoEssjBoFv/sddOgQJgs1ahR1RSLZp0CXnPfkk3D22eHG58SJ0LBh1BWJREOBLjntkUfCRKGjjoIJE6BBg6grEomOAl1y1tCh8Pvfw3HHwYsvQr16UVckEi0FuuSkIUOgZ0848UQYNw7q1o26IpHoKdAl5wwdCpddBiedBM89p2d+imylQJecMmzYtpb5mDFQq1bUFYlUHwp0yRnDh8Oll4YFthTmIj+mQJecMGJECPMuXdTNIlKelALdzLqY2XwzW2hmfcs557dmNs/M5prZ0+ktU/LZo4+GR8Yddxw8/7zCXKQ8FS6fa2YFwAPAccByYJaZFbn7vKRz2gF/AX7l7mvNbLdMFSz5ZcyYMDSxc2cYO1ZhLrI9qbTQOwAL3X2xu28ERgKnljrnUuABd18L4O6r0lum5KOZM+Hcc6FjxzA0sU6dqCsSqd5SCfQWwLKk/eWJY8l+AvzEzN40sxlm1iVdBUp+WrIEunWDPfdUmIukKl1PLNoZaAccDbQEppnZge7+ZfJJZtYT6AnQunXrNL21xM2XX4Yx5hs3wksvQbNmUVckkhtSaaGvAFol7bdMHEu2HChy903u/gmwgBDwP+DuQ9y90N0Lm+m3VMqwaROceSYsWBBGs+y7b9QVieSOVAJ9FtDOzNqaWU2gB1BU6pyxhNY5ZtaU0AWzOI11Sh5whyuuCM//HDoUOnWKuiKR3FJhoLv7ZuAqYBLwITDa3eea2S1m1i1x2iRgjZnNA6YA17n7mkwVLfE0aFCYCXrjjXDhhVFXI5J7zN0jeePCwkIvLi6O5L2l+hk1Cnr0CNtTT8FOmvImUiYzm+3uhWW9pl8bidzEiXDeeXDkkWF9c4W5SOXoV0ci9eabcPrpsP/+8MILmjgkUhUKdInMe++F4YmtWsGkSXp0nEhVKdAlEh9/DCecEB4ZN3ky7KbFIkSqLF0Ti0RStmJFWGirpASmToW99oq6IpF4UKBLVn3+eQjzL76AKVM0cUgknRTokjVbtoRhiYsXhz7zX/4y6opE4kWBLlnz0EPw6qvhAc9HHRV1NSLxo5uikhWLFsH114cboZdcEnU1IvGkQJeM27IFLroIatQIU/vNoq5IJJ7U5SIZd9998Prr4VFyLVtGXY1IfKmFLhk1fz785S9w8slw/vlRVyMSbwp0yZiSkrBqYp064UaoulpEMktdLpIxd94JM2bA009D8+ZRVyMSf2qhS0bMmwc33xwW3urRI+pqRPKDAl3S7vvvw3K4u+wSxp6rq0UkO9TlImnXuzf8618wbpwW3RLJJrXQJa0efxwefhj69IFu3So+X0TSR4EuafP++/CHP8DRR8Ott0ZdjUj+UaBLWnz1FZxxBjRqBCNHws7qzBPJOv3aSZW5h/Hmn3wS1jffffeoKxLJTwp0qbI774SxY+Guu+CII6KuRiR/qctFquSf/4S+faF79zC6RUSio0CXSluzJkwa2mcfGD5c481FoqYuF6m0Pn1g9WqYODFMIhKRaKmFLpXyxhuhVX7NNXDQQVFXIyKgQJdK2LgxjDdv3Rr694+6GhHZSl0ussPuvhvmzoWiIqhXL+pqRGQrtdBlh3zyCQwcCKedBqecEnU1IpJMgS4pc4erroKCArj33qirEZHS1OUiKXvuORg/PkwgatUq6mpEpDS10CUl69bB1VfDwQdDr15RVyMiZVELXVJy882wciU8/7wW3hKprtRClwoVF8Pf/w6XXw4dOkRdjYiUR4Eu27VhA1xwAeyxB9x2W9TViMj26D/Psl0DBoQHPo8fH9Y6F5HqSy10KdfMmfC3v8HFF0PXrlFXIyIVUaBLmb7/Pjy0okWLsN65iFR/6nKRMvXrBx99BJMnQ8OGUVcjIqlQC11+5K23YPBguOwyOO64qKsRkVSlFOhm1sXM5pvZQjPru53zzjAzN7PC9JUo2fTdd3DRRWElxUGDoq5GRHZEhV0uZlYAPAAcBywHZplZkbvPK3VeA+CPwMxMFCrZcdNNsGABvPIKNGgQdTUisiNSaaF3ABa6+2J33wiMBE4t47y/AncA36exPsmiN98MS+NecQUce2zU1YjIjkol0FsAy5L2lyeO/ZeZ/QJo5e4vbe8bmVlPMys2s+LVq1fvcLGSOSUlcOWVYdGtO+6IuhoRqYwq3xQ1s52Au4A/V3Suuw9x90J3L2zWrFlV31rSaMQIeO+90G9ev37U1YhIZaQS6CuA5MVSWyaObdUAOACYamZLgI5AkW6M5o6vvoIbb4QjjoAzz4y6GhGprFTGoc8C2plZW0KQ9wDO3vqiu38FNN26b2ZTgWvdvTi9pUqm3HYbfP45TJgAZlFXIyKVVWEL3d03A1cBk4APgdHuPtfMbjGzbpkuUDJr4UK4554wK/SXv4y6GhGpipRmirr7eGB8qWP9yjn36KqXJdly3XVQq5ZWUhSJA80UzWOvvQZjx8INN0Dz5lFXIyJVpUDPUyUl0Ls3tG0Lf/pT1NWISDpoca48NWwYzJkD//gH1K4ddTUikg5qoeehL78MU/x//Ws4/fSoqxGRdFGg56Fbb4U1a8LoFg1TFIkPBXqe+eQTuP/+sKJi+/ZRVyMi6aRAzzM33AAFBXDLLVFXIiLppkDPI7NmwciRcO214dFyIhIvCvQ84R6CfLfdwmQiEYkfDVvMEy+8ANOmwUMP6cEVInGlFnoe2LwZ+vSBffeFSy6JuhoRyRS10PPAsGHw0UcwbhzsrL9xkdhSCz3mvv4a+vcPk4hOOSXqakQkk9Rei7lBg2DVKnjxRU0iEok7tdBj7NNP4c47oUcPOOSQqKsRkUxToMdYv37hhujtt0ddiYhkgwI9phYsgEcegSuvDEvkikj8KdBjauBAqFMH+vaNuhIRyRYFegzNmwfPPAO9eoWZoSKSHxToMTRwINSrF6b6i0j+UKDHzJw5MHp0eLzcrrtGXY2IZJMCPWYGDIBddoFrrom6EhHJNgV6jLzzDjz3XAjzxo2jrkZEsk2BHiP9+4cg79076kpEJAoK9JiYNSsskXvttdCwYdTViEgUFOgx0b9/uAnaq1fUlYhIVBToMTB9OkyYANdfr4dXiOQzBXoM9OsXJhBdeWXUlYhIlLR8bo57+WV45ZWwqmK9elFXIyJRUgs9h23aBH/8I+yzj1rnIqIWek578EH48MPwaLlataKuRkSiphZ6jlq9OoxsOf54PVpORAIFeo666SZYvx7uuUePlhORQIGeg955B4YOhauugp/9LOpqRKS6UKDnGHe4+mpo2jR0uYiIbKWbojlm1Ch44w0YMgQaNYq6GhGpTtRCzyHr18N110H79vD730ddjYhUN2qh55A77oDly+Hpp6GgIOpqRKS6UQs9RyxZAoMGQY8ecOSRUVcjItVRSoFuZl3MbL6ZLTSzHz1H3syuMbN5Zva+mb1qZnulv9T8dsMNYXji3/4WdSUiUl1VGOhmVgA8AHQF9gPOMrP9Sp32DlDo7j8H/gEodtKouBieeQb+9Cdo1SrqakSkukqlhd4BWOjui919IzASODX5BHef4u7fJnZnAC3TW2b+cg8PrWjWDPr0iboaEanOUgn0FsCypP3liWPluRiYUNYLZtbTzIrNrHj16tWpV5nHXnoJ/vnPMOZ8l12irkZEqrO03hQ1s3OBQmBQWa+7+xB3L3T3wmbNmqXzrWNp8+bw0Ip27aBnz6irEZHqLpVhiyuA5J7bloljP2BmnYEbgaPcfUN6ystvI0aE1RTHjIEaNaKuRkSqu1Ra6LOAdmbW1sxqAj2AouQTzKw98DDQzd1Xpb/M/PPNN+FJRL/6FZx2WtTViEguqLCF7u6bzewqYBJQAIxw97lmdgtQ7O5FhC6W+sCzFpb++7e7d8tg3bE3eDD85z/w/PNaTVFEUmPuHskbFxYWenFxcSTvXd2tXBn6zbt2hWefjboaEalOzGy2uxeW9ZpmilZDAwbAhg1w++1RVyIiuUSBXs3MmwfDhsHll4dWuohIqhTo1ciWLXDFFVC/Ptx8c9TViEiu0WqL1cj994dJRMOHh5mhIiI7Qi30amL+fOjbF046CS66KOpqRCQXKdCrgZISuPBCqFMnPCtUwxRFpDLU5VINDB4MM2aEB1c0bx51NSKSq9RCj9gHH4QZoWecER5eISJSWQr0CG3aBOefDw0bwkMPqatFRKpGXS4Ruu02eOcdeO45jWoRkapTCz0is2eHQD/3XC2+JSLpoUCPwKJF0L077LYb3Hdf1NWISFyoyyXL5syB448P/ecTJ0LjxlFXJCJxoRZ6Fs2YAUcdBTvtBNOmQWGZ66WJiFSOAj1LXn0VOneGJk3gjTdgv/2irkhE4kaBngXPPw8nngh77w2vvw5t20ZdkYjEkQI9wx57DM48E37xC5g6VTNBRSRzFOgZsnEj9O4d1mg5+mh4+eXQ3SIikikK9AxYtizc/Lz3Xrj6ahg/PqxxLiKSSRq2mGaTJsE554RHyI0eHbpbRESyQS30NCkpCc8C7do19JMXFyvMRSS7FOhpMHcudOkCAwfCeefBzJnw059GXZWI5BsFehXMnAm/+Q0ccABMnw5DhsCjj0LdulFXJiL5SIG+g9zDJKFjj4WOHcOMz/79YelSuPRSLYErItHRTdEUucMLL4QVEt9+O/STDx4MPXtCgwZRVyciokCvUEkJjBkTgvz998Msz//7P7jgAqhdO+rqRES2UZdLOTZvhieeCP3jv/tdGIb4+OOwYAFcdpnCXESqH7XQS1m5EkaOhL//HRYvhp//PIwnP/10KCiIujoRkfIp0IG1a8Nj4J5+GqZMCf3lhx4Kd98NJ58clrsVEanu8jbQ160LU/JHjgwfN22Cdu2gXz846yyNIxeR3JNXgb5qFYwbF5azffXVsIDWnntCr15w9tlhRUQNOxSRXBX7QF+wAF58EcaODQ+WcA8jVXr1Cg9n7thRfeMiEg+xC/SNG8NDJF56KQT5xx+H4wceGLpTTjst3OhUS1xE4iYWgf7ZZzBhQgjxyZPh66+hZk3o1CksX3vSSXpKkIjEX04GekkJzJoVbmaOHw+zZ4fje+4JPXqEAD/2WK1BLiL5JecCffhw6NsXPv88DCc87DC4/fbwzE51pYhIPsu5QG/RIqw5fuKJcPzxeqybiMhWORfoXbqETUREfkhzIEVEYiKlQDezLmY238wWmlnfMl6vZWajEq/PNLM26S5URES2r8JAN7MC4AGgK7AfcJaZ7VfqtIuBte7+/4C7gTvSXaiIiGxfKi30DsBCd1/s7huBkcCppc45FXgs8fk/gGPNNN5ERCSbUgn0FsCypP3liWNlnuPum4GvgF1LfyMz62lmxWZWvHr16spVLCIiZcrqTVF3H+Luhe5e2KxZs2y+tYhI7KUS6CuAVkn7LRPHyjzHzHYGGgJr0lGgiIikJpVAnwW0M7O2ZlYT6AEUlTqnCLgg8Xl34DV39/SVKSIiFbFUctfMTgTuAQqAEe5+m5ndAhS7e5GZ1QaeANoDXwA93H1xBd9zNbC0knU3BT6v5Nfmsny9bsjfa9d155dUrnsvdy+zzzqlQK9uzKzY3QujriPb8vW6IX+vXdedX6p63ZopKiISEwp0EZGYyNVAHxJ1ARHJ1+uG/L12XXd+qdJ152QfuoiI/FiuttBFRKQUBbqISEzkXKBXtJRvXJjZCDNbZWYfJB1rYmYvm9nHiY+No6wxE8yslZlNMbN5ZjbXzP6YOB7razez2mb2tpm9l7jugYnjbRNLUi9MLFFdM+paM8HMCszsHTN7MbEf++s2syVmNsfM3jWz4sSxKv2c51Sgp7iUb1w8CpR+NlNf4FV3bwe8mtiPm83An919P6AjcGXi7zju174BOMbdDwIOBrqYWUfCUtR3J5amXktYqjqO/gh8mLSfL9fdyd0PThp7XqWf85wKdFJbyjcW3H0aYdZtsuRlih8DfpPVorLA3Ve6+78Sn39N+CVvQcyv3YNvErs1EpsDxxCWpIYYXjeAmbUETgKGJfaNPLjuclTp5zzXAj2VpXzjbHd3X5n4/DNg9yiLybTEk6/aAzPJg2tPdDu8C6wCXgYWAV8mlqSG+P683wNcD2xJ7O9Kfly3A5PNbLaZ9Uwcq9LPec49JFoCd3czi+2YUzOrD4wBerv7uuTnpcT12t29BDjYzBoBzwP7RlxSxpnZycAqd59tZkdHXU+WHeHuK8xsN+BlM/so+cXK/JznWgs9laV84+w/ZtYcIPFxVcT1ZISZ1SCE+VPu/lzicF5cO4C7fwlMAQ4DGiWWpIZ4/rz/CuhmZksIXajHAPcS/+vG3VckPq4i/APegSr+nOdaoKeylG+cJS9TfAEwLsJaMiLRfzoc+NDd70p6KdbXbmbNEi1zzKwOcBzh/sEUwpLUEMPrdve/uHtLd29D+H1+zd3PIebXbWb1zKzB1s+B44EPqOLPec7NFC1rKd+IS8oIM3sGOJqwnOZ/gP7AWGA00Jqw9PBv3b30jdOcZmZHAK8Dc9jWp3oDoR89ttduZj8n3AQrIDS0Rrv7LWa2N6Hl2gR4BzjX3TdEV2nmJLpcrnX3k+N+3Ynrez6xuzPwdGJZ8l2pws95zgW6iIiULde6XEREpBwKdBGRmFCgi4jEhAJdRCQmFOgiIjGhQBcRiQkFuohITPx/RUXNMTfd+zYAAAAASUVORK5CYII=\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAEICAYAAAB25L6yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXyU5bn/8c9FCIsEASGAEBA49YAIGCRstlXQqigerAUXilatCvaoqNXidqxocf251a3uS6ui9iBaxWrVquCKQUBF8NQiFJA1SgRl5/r9cU+EYiCTZCbPMzPf9+v1vGafuR4Zvtzecy/m7oiISHzVi7oAERHZNQW1iEjMKahFRGJOQS0iEnMKahGRmFNQi4jEnIJaYs/M/mpmJ6f6udWsYZCZLU71+4oko37UBUh2MrO1293cDdgAbEncHuPujyX7Xu5+RDqeK5IpFNSSFu5eUHHdzBYAp7v7Kzs+z8zqu/vmuqxNJNOo60PqVEUXgpldZGbLgIfMrIWZPW9mK83sq8T1ou1e87qZnZ64foqZvWlmNyae+7mZHVHD53Y2s6lmtsbMXjGzO83s0STPY5/EZ602szlmNmy7x440s08S77vEzC5M3N8qcW6rzexLM5tmZvo7KFXSl0Si0BbYA9gLGE34Hj6UuN0RWAfcsYvX9wc+BVoBNwAPmJnV4LmPA9OBlsB44KRkijezfOA54G9Aa+Ac4DEz65p4ygOE7p2mQA/g74n7LwAWA4VAG+BSQGs4SJUU1BKFrcAV7r7B3de5e5m7T3L3b919DXA1cNAuXr/Q3e9z9y3AI8CehOBL+rlm1hHoC/zW3Te6+5vAX5KsfwBQAFyXeO3fgeeBkYnHNwHdzWx3d//K3T/Y7v49gb3cfZO7T3MttiNJUFBLFFa6+/qKG2a2m5ndY2YLzexrYCrQ3MzydvL6ZRVX3P3bxNWCaj63HfDldvcBLEqy/nbAInffut19C4H2ievDgSOBhWb2hpkNTNz//4DPgL+Z2XwzuzjJz5Mcp6CWKOzYirwA6Ar0d/fdgQMT9++sOyMVlgJ7mNlu293XIcnXfgF02KF/uSOwBMDd33f3owndIs8ATyXuX+PuF7h7F2AY8GszO6SW5yE5QEEtcdCU0C+92sz2AK5I9we6+0KgFBhvZg0Srd7/SvLl7wHfAuPMLN/MBiVe+0TivUaZWTN33wR8TejqwcyOMrMfJPrIywnDFbdW/hEi2yioJQ5uBRoDq4B3gRfr6HNHAQOBMmAC8CRhvPcuuftGQjAfQaj5LuAX7j4v8ZSTgAWJbpwzE58DsDfwCrAWeAe4y91fS9nZSNYy/ZYhEpjZk8A8d097i16kOtSilpxlZn3N7D/MrJ6ZDQGOJvQpi8SKZiZKLmsLPE0YR70Y+JW7z4y2JJHvU9eHiEjMqetDRCTm0tL10apVK+/UqVM63lpEJCvNmDFjlbsXVvZYWoK6U6dOlJaWpuOtRUSykpkt3Nlj6voQEYk5BbWISMxVGdRm1tXMZm13fG1m59VFcSIikkQftbt/ChQDJFYzWwJMTnNdIpJCmzZtYvHixaxfv77qJ0taNWrUiKKiIvLz85N+TXV/TDwE+GdiQRsRyRCLFy+madOmdOrUiZ3vsSDp5u6UlZWxePFiOnfunPTrqttHfQIwsbIHzGy0mZWaWenKlSur+bYikk7r16+nZcuWCumImRktW7as9v/ZJB3UZtaAsIbunyt73N3vdfcSdy8pLKx0KKCIREghHQ81+XOoTov6COADd19e7U9Jwrp1cNNN8MYb6Xh3EZHMVZ2gHslOuj1SIS8vBPXVV6frE0QkKmVlZRQXF1NcXEzbtm1p3779d7c3bty4y9eWlpYyduzYKj/jgAMOSEmtr7/+OkcddVRK3itVkvox0cyaAIcCY9JVSIMGcM45cOml8NFH0LNnuj5JROpay5YtmTVrFgDjx4+noKCACy+88LvHN2/eTP36lcdRSUkJJSUlVX7G22+/nZpiYyipFrW7f+PuLd29PJ3FjBkDu+0Gt9ySzk8RkTg45ZRTOPPMM+nfvz/jxo1j+vTpDBw4kN69e3PAAQfw6aefAv/ewh0/fjy//OUvGTRoEF26dOG222777v0KCgq+e/6gQYMYMWIE3bp1Y9SoUVSsEvrCCy/QrVs3+vTpw9ixY6vVcp44cSI9e/akR48eXHTRRQBs2bKFU045hR49etCzZ09uSYTXbbfdRvfu3enVqxcnnHBCrf9bxWo96j32gFNPhfvug2uugbZto65IJPucdx4kGrcpU1wMt95a/dctXryYt99+m7y8PL7++mumTZtG/fr1eeWVV7j00kuZNGnS914zb948XnvtNdasWUPXrl351a9+9b0xyTNnzmTOnDm0a9eOH/7wh7z11luUlJQwZswYpk6dSufOnRk5cmTSdX7xxRdcdNFFzJgxgxYtWnDYYYfxzDPP0KFDB5YsWcLHH38MwOrVqwG47rrr+Pzzz2nYsOF399VG7KaQn3subNoEd94ZdSUikm7HHnsseXl5AJSXl3PsscfSo0cPzj//fObMmVPpa4YOHUrDhg1p1aoVrVu3Zvny749v6NevH0VFRdSrV4/i4mIWLFjAvHnz6NKly3fjl6sT1O+//z6DBg2isLCQ+vXrM2rUKKZOnUqXLl2YP38+55xzDi+++CK77747AL169WLUqFE8+uijO+3SqY5YtagB9t4bhg2DP/wBLrkkdIWISOrUpOWbLk2aNPnu+uWXX87gwYOZPHkyCxYsYNCgQZW+pmHDht9dz8vLY/PmzTV6Tiq0aNGC2bNn89JLL3H33Xfz1FNP8eCDDzJlyhSmTp3Kc889x9VXX81HH31Uq8COXYsa4Ne/hrIy+NOfoq5EROpKeXk57du3B+Dhhx9O+ft37dqV+fPns2DBAgCefPLJpF/br18/3njjDVatWsWWLVuYOHEiBx10EKtWrWLr1q0MHz6cCRMm8MEHH7B161YWLVrE4MGDuf766ykvL2ft2rW1qj12LWqAH/8YSkrg5pvhjDOgXiz/ORGRVBo3bhwnn3wyEyZMYOjQoSl//8aNG3PXXXcxZMgQmjRpQt++fXf63FdffZWioqLvbv/5z3/muuuuY/Dgwbg7Q4cO5eijj2b27NmceuqpbN26FYBrr72WLVu2cOKJJ1JeXo67M3bsWJo3b16r2tOyZ2JJSYnXduOAiRPh5z+H556DmA1pFMk4c+fOZZ999om6jMitXbuWgoIC3J2zzjqLvffem/PPP7/O66jsz8PMZrh7peMQY9tWHTECiopCq1pEJBXuu+8+iouL2XfffSkvL2fMmLRNDUmpWHZ9AOTnw9ixMG4czJwJvXtHXZGIZLrzzz8/khZ0bcW2RQ2hf7qgQK1qkVRIRzenVF9N/hxiHdTNm8Npp8ETT8CSJVFXI5K5GjVqRFlZmcI6YhXrUTdq1Khar4tt10eFsWPh9tvDtPIbb4y6GpHMVFRUxOLFi9Fa8dGr2OGlOmIf1F26wIknhu6PkhJIwbR5kZyTn59frR1FJF5iH9QAd98Nn38OJ50UukOGDIm6IhGRuhPrPuoKjRuH8dT77gvDh0MWr2YoIvI9GRHUAM2awUsvQbt2MHRoWLNaRCQXZExQA7RpAy+/HBZqOvzw0B0iIpLtMiqoATp1Ci3r9evh0EOhkhUORUSySsYFNUCPHvDCC7B0afhhsZYLU4mIxFpGBjXAgAEwaRJ8+CH84heQWLxKRCTrZGxQQ2hN33gjTJ4MV10VdTUiIumREeOod+W880Kr+sorw87lw4dHXZGISGpldIsawCxMiBk4MHSBzJ4ddUUiIqmVVFCbWXMz+18zm2dmc81sYLoLq46GDeHpp8Mu5sOGwYoVUVckIpI6ybaofw+86O7dgP2AuekrqWbatoVnngkhPWIEbNwYdUUiIqlRZVCbWTPgQOABAHff6O6r011YTfTpAw89BNOmwdlng1Z0FJFskEyLujOwEnjIzGaa2f1m1mTHJ5nZaDMrNbPSKJdSPOEEuPRSuO8+uOeeyMoQEUmZZIK6PrA/8Ad37w18A1y845Pc/V53L3H3ksLCwhSXWT1XXQVHHBHWsn7nnUhLERGptWSCejGw2N3fS9z+X0Jwx1ZeHjz2GHToEPqrly2LuiIRkZqrMqjdfRmwyMy6Ju46BPgkrVWlQIsWYSLMV1/BccfBpk1RVyQiUjPJjvo4B3jMzD4EioFr0ldS6vTqBfffH35cvPDCqKsREamZpGYmuvssoCTNtaTFz38O778Pt94KffuGbb1ERDJJxs9MTMYNN8BBB8Ho0TBrVtTViIhUT04EdX4+PPlkmLl4zDFQVhZ1RSIiycuJoIawO8ykSbBkCZx2mibDiEjmyJmgBujfH669Fp59VpNhRCRz5FRQA5x/Phx2WLicMyfqakREqpZzQV2vHjzyCOy+O4wcGfZeFBGJs5wLaggr7T30EHz0EfzmN1FXIyKyazkZ1ABHHgnnngt33AHPPx91NSIiO5ezQQ1w/fWw335w6qlhR3MRkTjK6aBu2BAmToRvvtFO5iISXzkd1AD77AO//z288grcfHPU1YiIfF/OBzXA6afDT38K//M/MDd2m4yJSK5TULNtJ/OCAjjlFNi8OeqKRES2UVAntGkTRoBMnw433RR1NSIi2yiot3P88fCzn8FvfwufxH5rBBHJFQrq7ZjBXXdB06bqAhGR+FBQ76BNG7jzzrDZgLpARCQOFNSVOO44GD5cXSAiEg8K6kpUdIHsvru6QEQkegrqnWjdelsXyI03Rl2NiOQyBfUuHHccjBgB48fDZ59FXY2I5CoFdRVuuy2sCfKrX2n7LhGJRlJBbWYLzOwjM5tlZqXpLipO9twTrrkmrAUycWLU1YhILqpOi3qwuxe7e0naqompM8+Efv3C9l1ffRV1NSKSa9T1kYS8vLAZblkZXHJJ1NWISK5JNqgd+JuZzTCz0eksKK6Ki8OOMPfcA2+/HXU1IpJLzJP4hczM2rv7EjNrDbwMnOPuU3d4zmhgNEDHjh37LFy4MB31RmrtWujeHZo1gw8+gPz8qCsSkWxhZjN21rWcVIva3ZckLlcAk4F+lTznXncvcfeSwsLC2tQbWwUFcPvt8PHHcMstUVcjIrmiyqA2syZm1rTiOnAY8HG6C4uro48Ox/jxsGBB1NWISC5IpkXdBnjTzGYD04Ep7v5iesuKt9tvh3r14KyzNLZaRNKvyqB29/nuvl/i2Nfdr66LwuKsQwf43e/ghRdg0qSoqxGRbKfheTV0zjlhJMjYsVBeHnU1IpLNFNQ1VL8+3HsvLFsWNsUVEUkXBXUt9O0b+qkrVtkTEUkHBXUtTZgQ1gMZPVrrVotIeiioa6lZM/j972HWrDAaREQk1RTUKTB8OAwdCpdfDv/6V9TViEi2UVCngBnccQds3RpGgYiIpJKCOkU6dYIrr4Rnn4Vnnom6GhHJJgrqFDrvPOjZM4yxXrMm6mpEJFsoqFMoPz+MrV6yJPRXi4ikgoI6xQYMCDvC3H47lObUpmUiki4K6jS49lpo00Zjq0UkNRTUadCsWdi9fObMcCkiUhsK6jQZPhyOOir0VWfhZjciUocU1GlSMbYa4OyztW61iNScgjqN9torrFv9/PNat1pEak5BnWZjx8L++2vdahGpOQV1mlWsW718OVx6adTViEgmUlDXgT59wmzFP/wB3nkn6mpEJNMoqOvI734HRUVwxhmwcWPU1YhIJlFQ15GmTUOLes4cuP76qKsRkUyioK5DQ4fCyJFhV5i5c6OuRkQyhYK6jt16KxQUwOmnh/WrRUSqknRQm1memc00s+fTWVC2a90abrkF3n4b7r476mpEJBNUp0V9LqD/YU+Bk06Cww6Diy+GRYuirkZE4i6poDazImAocH96y8kNZqE1vWUL/Pd/a3q5iOxasi3qW4FxwE57Vc1stJmVmlnpypUrU1JcNuvcOfyo+Pzz8NRTUVcjInFWZVCb2VHACnefsavnufu97l7i7iWFhYUpKzCbjR0LffuGyTBlZVFXIyJxlUyL+ofAMDNbADwBHGxmj6a1qhyRlwf33w9ffRX2WxQRqUyVQe3ul7h7kbt3Ak4A/u7uJ6a9shzRqxdcdhk8+qhW2BORymkcdQxcdhmUlMCYMbB0adTViEjcVCuo3f11dz8qXcXkqvx8+NOf4JtvwkQYjQIRke2pRR0T3brBDTfACy+EZVFFRCooqGPkrLPgJz+BX/8aPvss6mpEJC4U1DFSrx489BA0aAC/+AVs3hx1RSISBwrqmCkqgrvuChsM3HBD1NWISBwoqGNo5Eg4/ni44gr44IOoqxGRqCmoY+quu8JKeyeeGEaDiEjuUlDH1B57wB//CPPmhanmIpK7FNQxdsghYTLMgw+GmYsikpsU1DF3xRXw4x/DmWfCp59GXY2IREFBHXP168PEidCoERx3HKxbF3VFIlLXFNQZoH37MMX8ww/DZBgRyS0K6gxxxBEwblzYGUYbDYjkFgV1BpkwAQYMCAs3/fOfUVcjInVFQZ1B8vPhiSdCv/Xxx8P69VFXJCJ1QUGdYfbaCx5+GGbMgLPP1pKoIrlAQZ2Bhg0L46sfeEBLoorkAgV1hrryShgyJGyM+847UVcjIumkoM5QeXnw+OPQoQOMGAHLlkVdkYiki4I6g7VoAc88A6tXw7HHwsaNUVckIumgoM5wPXuGvuo339RkGJFsVT/qAqT2TjgBSkvhppugb184+eSoKxKRVFKLOktcdx0cfDCMGQPTp0ddjYikUpVBbWaNzGy6mc02szlmdmVdFCbVU78+PPkktGsXhu8tXBh1RSKSKsm0qDcAB7v7fkAxMMTMBqS3LKmJVq1gypQwY3HoUCgvj7oiEUmFKoPag7WJm/mJQ/PhYmqffeDpp8Pa1cceC5s2RV2RiNRWUn3UZpZnZrOAFcDL7v5eesuS2jj4YLjnHnj5ZU0zF8kGSQW1u29x92KgCOhnZj12fI6ZjTazUjMrXblyZarrlGr65S/hkkvCFPObboq6GhGpjWqN+nD31cBrwJBKHrvX3UvcvaSwsDBV9UktTJgQdoUZNy50h4hIZkpm1EehmTVPXG8MHArMS3dhUnv16oWV9vr3hxNPhPfUYSWSkZJpUe8JvGZmHwLvE/qon09vWZIqjRvDs8/CnnuGXWI+/DDqikSkuqqcmejuHwK966AWSZPWreGVV8Ju5oceCtOmwX/+Z9RViUiyNDMxR3TuHMLaHX7yE02IEckkCuoc0q1bGLK3Zg0ccggsXRp1RSKSDAV1jtlvP/jrX8P61YceCqtWRV2RiFRFQZ2DBgyA554LO5kPGaKp5iJxp6DOUYMHw6RJYRTIkCFh8wERiScFdQ478siw4t6MGaHPuqws6opEpDIK6hx3zDFhO685c2DQIFi+POqKRGRHCmrhyCPD8qjz58NBB8GSJVFXJCLbU1ALELo+XnoJvvgCDjwQFiyIuiIRqaCglu/86EdhUsyXX4aw/sc/oq5IREBBLTvo1w9eew3WrQtTzmfMiLoiEVFQy/cUF8PUqdCoUWhZT5kSdUUiuU1BLZXaZx94990w7XzYsLBjjIhEQ0EtO9W2LbzxRpgQc+aZYceYrVujrkok9yioZZcKCsJ61qNHw3XXhQ0INmyIuiqR3FLletQi9evD3XeHpVIvuSQM4fvzn0E7ronUDbWoJSlmcPHF8Nhjoe+6Tx94//2oqxLJDQpqqZaf/xzeeivsx/ijH8F994XNCEQkfRTUUm19+oTx1YMHh77r008P465FJD0U1FIjLVuG8dWXXw4PPhha15p2LpIeCmqpsbw8uOqqbZsQ9OkDL7wQdVUi2UdBLbV21FFQWgodOsDQoXDppbB5c9RViWQPBbWkxA9+AO+8A2ecAddeG3Y61+a5IqlRZVCbWQcze83MPjGzOWZ2bl0UJpmncWO491744x/D0L3iYvj736OuSiTzJdOi3gxc4O7dgQHAWWbWPb1lSSY76aQQ1C1bhpb1736nqecitVFlULv7Unf/IHF9DTAXaJ/uwiSzde8O06fDqFHw29+GoXxa31qkZqrVR21mnYDewHuVPDbazErNrHTlypWpqU4yWkFB6AZ56CGYPRt69YIbb4QtW6KuTCSzJB3UZlYATALOc/evd3zc3e919xJ3LynUIhCSYAannAKffAKHHw6/+Q0MHAgffxx1ZSKZI6mgNrN8Qkg/5u5Pp7ckyUbt2sHkyfDEE/D557D//nDllbBxY9SVicRfMqM+DHgAmOvuN6e/JMlWZnD88aF1PWIEjB8PJSVa3EmkKsm0qH8InAQcbGazEseRaa5LslhhITz+eFjnuqwMBgyACy6Ab76JujKReEpm1Meb7m7u3svdixOHJgpLrQ0bFlrXZ5wBN98MPXvCq69GXZVI/GhmokSqWbOwKcHrr4e1Q37yEzjtNPjqq6grE4kPBbXEwkEHwYcfwkUXwSOPhM11H3xQQ/lEQEEtMdK4cdiXcfp06NIltKz79oVp06KuTCRaCmqJnf33D7vIPP44rFoFBx4Ixx2n9a4ldymoJZbMYORImDcvjLeeMgW6dYPLLoPy8qirE6lbCmqJtd12C2uFfPopHHssXHNN6Ba56SZt/yW5Q0EtGaGoCP70p7BBQd++cOGFsPfeYVnVTZuirk4kvRTUklH69IEXXwzD+Tp2hDFjYN99w9R0LaUq2UpBLRnpoIPCD45/+Qs0ahT6s3v2DD9AahswyTYKaslYZvBf/wWzZsHEiVCvXlj/umIMthZ8kmyhoJaMV68enHBCWPN68uQw2/G008I+jnfeqR8dJfMpqCVr1KsHP/1pWI3vr38Nu6KffXa4vOQSWLgw6gpFakZBLVnHDIYMgTffDD86Hngg3HBDGNZ3zDFhw133qKsUSZ6CWrKWWfjR8emnw2YF48aF6eiHHBJGitx1F6xZE3WVIlVTUEtO6NgRrr0WFi+Ghx8OE2nOOgvat4dzzoG5c6OuUGTnFNSSUxo1gpNPDv3Y774LRx8dJs107x6WWJ08WcP7JH4U1JKTzKB//zDbcdGiMDX9//4PfvYz6NQJLr5YG/BKfCioJee1bh1GhcyfH1rU++0HN94YJtD07h12n1m6NOoqJZcpqEUS6tcPw/umTIEvvoDbboP8/LCfY1ERHHYY3HcfrFgRdaWSaxTUIpVo3Tr8yDh9elhq9bLLQot79GjYc88w5O/WWzU2W+qGeRoGlJaUlHhpaWnK31ckSu7w0UdhuN/TT4frEDY6OOaYcHTvHvq/RarLzGa4e0mljymoRWrms89Cn/bTT4cRJBCWXq0I7X79wmxJkWTUKqjN7EHgKGCFu/dI5gMV1JJrvvgCnn0WnnkmzHzcvDl0kQwdCocfHibZtGgRdZUSZ7UN6gOBtcAfFdQiVVu9OvwgOXkyvPwyfP11aFn37x9C+/DDw+YHeXlRVypxUuuuDzPrBDyvoBapnk2bwg+SL70UjvffD33dTZuGsB4wIBz9+4cfMCV3KahFYqKsDF55BaZODf3as2fDli3hsc6dYeDAsKDUkCFQWBhtrVK36iSozWw0MBqgY8eOfRZq3JJIlb79Fj74IIT2u++GRaNWrNg2c/LII0M/d+/eGk2S7dSiFskQW7eG4J4yBV54YVtXSdu2Ibj33z+E9v77Q7t2Cu9ssqugrl/XxYjIztWrByUl4bjiitC6fvFF+NvfYMaMsEdkRduqsDCEdvfuYVjgD34Qjo4dwyxLyR7JjPqYCAwCWgHLgSvc/YFdvUYtapH0WLs29GvPnBla3jNnhsWkvv1223Py80N/d48ecMAB4dh/f2jYMLq6pWqa8CKSxdzDolGffRaOf/wjHDNnhmnvAA0aQJ8+IbT79AkrBHboEMZ6a5hgPKjrQySLmYX+6nbtwhok21u2DN55B95+O1zecQds2LDt8fr1w+YJHTuG4z/+49+7UVq2VD94HKhFLZJDNmwIre1Fi+Bf/9p2LFoECxaE69tHQrNmIbwrWt/t2v37ZWEh7LEHNG6sQK8ttahFBAj91D16hKMyGzaEwK7oRqk45s+Ht96CVasqf12DBiGwW7QIl+3awT77bDu6dg2760jNKKhF5DsNG4ZQ7dq18sc3bAjdKUuXhvVNVq2Cr74Kx5dfbrucORMmTQrDDSGMZuncObTO99gjtNSbNw9Hs2Yh4Fu3DsMQ27QJz1ELfRsFtYgkrWFD2GuvcFRl/fowImXu3HB88knYDf7zz8N6KKtXhyn2lcnPD8Hdpk2Ybt+4cWiRN2687XqzZqEPvVWrcFlxvXlzaNIku7pjFNQikhaNGkGvXuGojHsI89WrQ0t8xYrQWl++fNvl8uXwzTehlb5uXXj+unXhKC/f9UbEZiGwmzSBgoJw7L575UezZpUfDRqEUTF5eeH/CiquN2gQ/tGqq38IFNQiEgmzbS3kPfcME3eqwz2sTFhWFo5Vq8Ll6tUh3L/5Jow7r7i+Zk04li0LLf3y8vD69etrfg7bt/IrzmPatJq/384oqEUkI5lta/l26VLz99m4MQR2efn3j02bwqJZOx4bN36/hb9uHey2W+rOb3sKahHJaQ0ahL7tVq2irmTntFGQiEjMKahFRGJOQS0iEnMKahGRmFNQi4jEnIJaRCTmFNQiIjGnoBYRibm0rEdtZiuBmm5D3grYyWKKWU3nnVt03rklmfPey90LK3sgLUFdG2ZWurPFs7OZzju36LxzS23PW10fIiIxp6AWEYm5OAb1vVEXEBGdd27ReeeWWp137PqoRUTk38WxRS0iIttRUIuIxFxsgtrMhpjZp2b2mZldHHU96WRmD5rZCjP7eLv79jCzl83sH4nLFlHWmGpm1sHMXjOzT8xsjpmdm7g/q88bwMwamdl0M5udOPcrE/d3NrP3Et/5J82sQdS1ppqZ5ZnZTDN7PnE7688ZwMwWmNlHZjbLzEoT99X4ux6LoDazPOBO4AigOzDSzKq5g1pGeRgYssN9FwOvuvvewKuJ29lkM3CBu3cHBgBnJf6Ms/28ATYAB7v7fkAxMMTMBgDXA7e4+w+Ar4DTIqwxXc4F5m53OxfOucJgdy/ebvx0jb/rsQhqoO1D+GUAAAJcSURBVB/wmbvPd/eNwBPA0RHXlDbuPhX4coe7jwYeSVx/BPhpnRaVZu6+1N0/SFxfQ/jL254sP28AD9YmbuYnDgcOBv43cX/WnbuZFQFDgfsTt40sP+cq1Pi7Hpegbg8s2u724sR9uaSNuy9NXF8GtImymHQys05Ab+A9cuS8E10As4AVwMvAP4HV7r458ZRs/M7fCowDtiZutyT7z7mCA38zsxlmNjpxX42/69rcNobc3c0sK8dNmlkBMAk4z92/Do2sIJvP2923AMVm1hyYDHSLuKS0MrOjgBXuPsPMBkVdTwR+5O5LzKw18LKZzdv+wep+1+PSol4CdNjudlHivlyy3Mz2BEhcroi4npQzs3xCSD/m7k8n7s76896eu68GXgMGAs3NrKKxlG3f+R8Cw8xsAaEr82Dg92T3OX/H3ZckLlcQ/mHuRy2+63EJ6veBvRO/CDcATgD+EnFNde0vwMmJ6ycDz0ZYS8ol+icfAOa6+83bPZTV5w1gZoWJljRm1hg4lNBH/xowIvG0rDp3d7/E3YvcvRPh7/Pf3X0UWXzOFcysiZk1rbgOHAZ8TC2+67GZmWhmRxL6tPKAB9396ohLShszmwgMIix9uBy4AngGeAroSFgi9jh33/EHx4xlZj8CpgEfsa3P8lJCP3XWnjeAmfUi/HiUR2gcPeXuV5lZF0Jrcw9gJnCiu2+IrtL0SHR9XOjuR+XCOSfOcXLiZn3gcXe/2sxaUsPvemyCWkREKheXrg8REdkJBbWISMwpqEVEYk5BLSIScwpqEZGYU1CLiMScglpEJOb+PxTdHuzhdkunAAAAAElFTkSuQmCC\n"
          },
          "metadata": {
            "needs_background": "light"
          }
        }
      ],
      "source": [
        "# Take a look at the training curves of your model\n",
        "\n",
        "acc = history.history['accuracy']\n",
        "loss = history.history['loss']\n",
        "\n",
        "epochs = range(len(acc))\n",
        "\n",
        "plt.plot(epochs, acc, 'b', label='Training accuracy')\n",
        "plt.title('Training accuracy')\n",
        "\n",
        "plt.figure()\n",
        "\n",
        "plt.plot(epochs, loss, 'b', label='Training Loss')\n",
        "plt.title('Training loss')\n",
        "plt.legend()\n",
        "\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OjvED5A3qrn2"
      },
      "source": [
        "Before closing the assignment, be sure to also download the `history.pkl` file which contains the information of the training history of your model and will be used to compute your grade. You can download this file by running the cell below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 37,
      "metadata": {
        "id": "9QRG73l6qE-c",
        "tags": [],
        "outputId": "e74d8c30-3ac3-49a6-e2e9-c4686e7481b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        }
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ],
            "application/javascript": [
              "download(\"download_92565508-051b-4a2d-ab96-efdaa9625978\", \"history.pkl\", 944)"
            ]
          },
          "metadata": {}
        }
      ],
      "source": [
        "def download_history():\n",
        "  import pickle\n",
        "  from google.colab import files\n",
        "\n",
        "  with open('history.pkl', 'wb') as f:\n",
        "    pickle.dump(history.history, f)\n",
        "\n",
        "  files.download('history.pkl')\n",
        "\n",
        "download_history()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wdsMszk9zBs_"
      },
      "source": [
        "## See your model in action\n",
        "\n",
        "After all your work it is finally time to see your model generating text. \n",
        "\n",
        "Run the cell below to generate the next 100 words of a seed text.\n",
        "\n",
        "After submitting your assignment you are encouraged to try out training for different amounts of epochs and seeing how this affects the coherency of the generated text. Also try changing the seed text to see what you get!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6Vc6PHgxa6Hm",
        "tags": []
      },
      "outputs": [],
      "source": [
        "seed_text = \"Help me Obi Wan Kenobi, you're my only hope\"\n",
        "next_words = 100\n",
        "  \n",
        "for _ in range(next_words):\n",
        "\t# Convert the text into sequences\n",
        "\ttoken_list = tokenizer.texts_to_sequences([seed_text])[0]\n",
        "\t# Pad the sequences\n",
        "\ttoken_list = pad_sequences([token_list], maxlen=max_sequence_len-1, padding='pre')\n",
        "\t# Get the probabilities of predicting a word\n",
        "\tpredicted = model.predict(token_list, verbose=0)\n",
        "\t# Choose the next word based on the maximum probability\n",
        "\tpredicted = np.argmax(predicted, axis=-1).item()\n",
        "\t# Get the actual word from the word index\n",
        "\toutput_word = tokenizer.index_word[predicted]\n",
        "\t# Append to the current text\n",
        "\tseed_text += \" \" + output_word\n",
        "\n",
        "print(seed_text)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oQVDbdcYqSux"
      },
      "source": [
        "You will also need to submit this notebook for grading. To download it, click on the File tab in the upper left corner of the screen then click on Download -> Download .ipynb. You can name it anything you want as long as it is a valid .ipynb (jupyter notebook) file.\n",
        "\n",
        "**Congratulations on finishing this week's assignment!**\n",
        "\n",
        "You have successfully implemented a neural network capable of predicting the next word in a sequence of text!\n",
        "\n",
        "**We hope to see you in the next course of the specialization! Keep it up!**"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "dlai_version": "1.2.0",
    "jupytext": {
      "main_language": "python"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}