demo08-word-embedding.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/rtealwitter/dl-demos/blob/main/demo08-word-embedding.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YpYIBU4rQG4t"
      },
      "source": [
        "# Harry Potter and Word Embedding\n",
        "\n",
        "We view words as having meaning but our representation of them is arbitrary and meaningless to computers.\n",
        "When working with natural language, the first step is to convert words into vectors.\n",
        "\n",
        "Our goal is to represent words in a vector space where similar words are close to each other. The problem is challenging because the meaning of words is not transitive. For example, the word \"wizard\" is similar to both \"witch\" and \"man\" but \"witch\" is not similar to \"man\".\n",
        "\n",
        "### word2vec\n",
        "\n",
        "A popular approach for getting meaninful word embeddings is called word2vec and first proposed in [this](https://arxiv.org/pdf/1301.3781.pdf) paper. The key concept is to find words that are similar to each other. We do this by scanning text and identifying words that appear next to each other as similar. Consider two similar words $x$ and $y$.\n",
        "\n",
        "![](https://lilianweng.github.io/posts/2017-10-15-word-embedding/word2vec-skip-gram.png)\n",
        "\n",
        "The idea is to represent $x$ as a one-hot encoding (similar to how we represent classes). Call the dimension of the space $V$ (this is the number of unique words). Then we apply a fully connected layer to represent the one-hot encoded word in a small hidden dimension $N$. The hidden dimension representation is the vector representing the word.\n",
        "\n",
        "In order to make the representation meaningful, we will apply a second fully connected layer to convert the hidden dimension back to the one-hot encoded dimension $V$. Now, we train both layers using cross entropy loss so that on input $x$, we get output close to $y$.\n",
        "\n",
        "Now there are two issues with this approach:\n",
        "\n",
        "* We train two layers but we only use the first word one to encode a word.\n",
        "\n",
        "* We really want the embedding of the two similar words $x$ and $y$ to be close. However, the loss function we use is more complicated.\n",
        "\n",
        "### Contrastive Learning\n",
        "\n",
        "A more direct approach to get meaningful vector representation is to use *contrastive learning*. The key concept is that we have positive pairs $(x,y)$ whose vector representations we want to be close and negative pairs $(x,z)$ whose vector representations we want to be far apart.\n",
        "\n",
        "Let $f$ be the network we apply to the words to embed them. If we use the inner product as our notion of similarity, then the loss function is\n",
        "\n",
        "$$ L(f) =\n",
        "\\sum_{\\text{positive pairs $(x,y)$}} (\\langle f(x),f(y) \\rangle-1)^2\n",
        "+ \\sum_{\\text{negative pairs $(x,z)$}} (\\langle f(x),f(z) \\rangle)^2\n",
        "$$\n",
        "\n",
        "This forces the inner product between positive pairs to be large and the inner product between negative pairs to be small.\n",
        "\n",
        "Note: The loss function we just described is one of many loss functions used for contrastive learning.\n",
        "\n",
        "Because contrastive learning tends to work better and is more direct, we will show how to use it to embed words today. The text we'll use for finding positive and negative pairs of words is Harry Potter!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HYiA684zOt4F"
      },
      "source": [
        "## Libraries\n",
        "\n",
        "As always, we rely on a lot of libraries."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "fq8pZAK27DDz"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import re\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import numpy as np\n",
        "import matplotlib\n",
        "from matplotlib.patches import Rectangle\n",
        "import matplotlib.pyplot as plt\n",
        "import urllib.request\n",
        "import numpy\n",
        "from torch.utils.data import DataLoader\n",
        "from torch.utils.data import Dataset\n",
        "import torch.nn.functional as F"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nLHxU1tqOyvA"
      },
      "source": [
        "## Loading Text\n",
        "\n",
        "We load the text of Harry Potter from a website and preprocess it. In particular, we only want the chapters of the book and we make all words lowercase. Finally, we remove annoying characters like new lines."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "wz2yu_chQG2z"
      },
      "outputs": [],
      "source": [
        "url1 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20the%20Sorcerer's%20Stone.txt\"\n",
        "url2 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20the%20Chamber%20of%20Secrets.txt\"\n",
        "url3 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20the%20Prisoner%20of%20Azkaban%20.txt\"\n",
        "url4 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20the%20Goblet%20of%20Fire.txt\"\n",
        "url5 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20the%20Order%20of%20the%20Phoenix.txt\"\n",
        "url6 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20The%20Half-Blood%20Prince.txt\"\n",
        "url7 = \"https://raw.githubusercontent.com/bobdeng/owlreader/7c3e507853b07273421d83e220c2ef219b8b6122/ERead/assets/books/Harry%20Potter%20and%20the%20Deathly%20Hallows%20.txt\"\n",
        "\n",
        "def read_file(url):\n",
        "  remove = ['\\u3000', '\\n', '¬', ',', '.']\n",
        "  with urllib.request.urlopen(url) as webpage:\n",
        "    text = webpage.read().decode(\"utf8\").lower()\n",
        "    for to_remove in remove:\n",
        "      text = text.replace(to_remove, '')\n",
        "  return text\n",
        "\n",
        "text1 = read_file(url1)\n",
        "text2 = read_file(url2)\n",
        "text3 = read_file(url3)\n",
        "text4 = read_file(url4)\n",
        "text5 = read_file(url5)\n",
        "text6 = read_file(url6)\n",
        "text7 = read_file(url7)\n",
        "\n",
        "text = text1 + text2 + text3 + text4 + text5 + text6 + text7"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nTehzSQtPGaZ"
      },
      "source": [
        "## Tokenizing\n",
        "\n",
        "Now we tokenize the text. That is, we turn the string into a list of words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "ITb7-sfhahjs"
      },
      "outputs": [],
      "source": [
        "def get_frequent(tokens, threshold=50):\n",
        "  frequency = {token : 0 for token in set(tokens)}\n",
        "  for token in tokens:\n",
        "    frequency[token] += 1\n",
        "\n",
        "  frequent_tokens = []\n",
        "  for token in tokens:\n",
        "    if frequency[token] >= threshold:\n",
        "      frequent_tokens += [token]\n",
        "\n",
        "  return frequent_tokens, len(set(tokens))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eJEVyoWf8J73",
        "outputId": "6160be7a-0e9e-48e1-d1a7-32d3aaaeddd5"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "['harry', 'potter', 'and', 'the', 'boy', 'who', 'and', 'mrs', 'dursley', 'of', 'number', 'four', 'privet', 'drive', 'were', 'proud', 'to', 'say', 'that', 'they', 'were', 'perfectly', 'normal', 'thank', 'you', 'very', 'much', 'they', 'were', 'the', 'last', 'people', 'youd', 'expect', 'to', 'be', 'in', 'anything', 'strange', 'or', 'because', 'they', 'just', 'didnt', 'hold', 'with', 'such', 'dursley', 'was', 'the', 'of', 'a', 'called', 'which', 'made', 'he', 'was', 'a', 'big', 'man', 'with', 'hardly', 'any', 'neck', 'although', 'he', 'did', 'have', 'a', 'very', 'large', 'mrs', 'dursley', 'was', 'thin', 'and', 'and', 'had', 'nearly', 'twice', 'the', 'usual', 'amount', 'of', 'neck', 'which', 'came', 'in', 'very', 'useful', 'as', 'she', 'spent', 'so', 'much', 'of', 'her', 'time', 'over', 'garden']\n"
          ]
        }
      ],
      "source": [
        "letters = 'abcdefghijklmnopqrstuvwxyz'\n",
        "def process_word(word):\n",
        "  return ''.join([letter for letter in word if letter in letters])\n",
        "tokens = [process_word(word) for word in text.split(' ')]\n",
        "\n",
        "tokens, num_tokens = get_frequent(tokens)\n",
        "print(tokens[:100])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HaRAaxLtNlk9",
        "outputId": "5102c043-8e20-480e-9509-610040884d2e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "There are 971182 total tokens and 38813 unique tokens.\n"
          ]
        }
      ],
      "source": [
        "print(f'There are {len(tokens)} total tokens and {num_tokens} unique tokens.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wY_eeC-gPcK8"
      },
      "source": [
        "Now we map from each word (token) to a unique index and back. We will use this for the one hot encoding of the word."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "eyw7PkfQ-iAb"
      },
      "outputs": [],
      "source": [
        "def mapping(tokens):\n",
        "  word_to_id = {}\n",
        "  id_to_word = {}\n",
        "\n",
        "  for i, token in enumerate(set(tokens)):\n",
        "    word_to_id[token] = i\n",
        "    id_to_word[i] = token\n",
        "\n",
        "  return word_to_id, id_to_word\n",
        "\n",
        "WORD_TO_ID, ID_TO_WORD = mapping(tokens)\n",
        "num_tokens = len(set(tokens))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SNu4q1OsPkG6"
      },
      "source": [
        "The next step is to get the dataset we will use. We do this by iterating through the tokens with a sliding window. For each word $w$, we look at its neighbors $u$ and add a data point $(w,u)$ where we need to predict $u$ on input $w$. For now, we store the data as indices to avoid carrying around a bunch of zeros in the one hot encoding."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "g0jJf78x_bRh"
      },
      "outputs": [],
      "source": [
        "def convert(x):\n",
        "  return torch.tensor(np.asarray(x)).long()\n",
        "\n",
        "def process(tokens, window=2, threshold=100):\n",
        "  frequency = {WORD_TO_ID[token]: 0 for token in set(tokens)}\n",
        "  X, y = [], []\n",
        "  for i in range(len(tokens)-1):\n",
        "    index_i = WORD_TO_ID[tokens[i]]\n",
        "    # Ensure common words like \"the\" don't appear too frequently\n",
        "    if frequency[index_i] < threshold:\n",
        "      for j in list(range(max(0,i-window), min(len(tokens), i+window+1))):\n",
        "        index_j = WORD_TO_ID[tokens[j]]\n",
        "        #if frequency[index_j] < threshold:\n",
        "        X += [index_i]\n",
        "        y += [index_j]\n",
        "        frequency[index_i] += 1\n",
        "        frequency[index_j] += 1\n",
        "  return X, y"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S8s2bBqjXMO3"
      },
      "source": [
        "# Initialization\n",
        "\n",
        "Now we define the custom class we need so we can wrap the nice DataLoader function around our dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "id": "tTdzbQ9nWck0"
      },
      "outputs": [],
      "source": [
        "class DataSet(Dataset):\n",
        "  def __init__(self, tokens):\n",
        "    X,y = process(tokens)\n",
        "    self.X = X\n",
        "    self.y = y\n",
        "\n",
        "  def __len__(self):\n",
        "    return len(self.X)\n",
        "\n",
        "  def __getitem__(self, idx):\n",
        "    return self.X[idx], self.y[idx]\n",
        "\n",
        "dataloader = DataLoader(DataSet(tokens), batch_size = 128, shuffle=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "id": "fI-iOCP0UivK"
      },
      "outputs": [],
      "source": [
        "class Model(nn.Module):\n",
        "  def __init__(self, num_tokens, embed_dim):\n",
        "    super(Model, self).__init__()\n",
        "    self.embedding = nn.Embedding(num_tokens, embed_dim)\n",
        "\n",
        "  def forward(self, x):\n",
        "    return self.embedding(x)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "4g_v-qYm88dT"
      },
      "outputs": [],
      "source": [
        "\n",
        "def contrastive_loss(X_embedded, y_embedded):\n",
        "  # X_embedded (batch_size, embed_dim)\n",
        "  # y_embedded (batch_size, embed_dim)\n",
        "  batch_size = len(X_embedded)\n",
        "  gram = X_embedded @ y_embedded.T # (batch_size, batch_size)\n",
        "  difference = gram - torch.eye(batch_size).to(DEVICE)\n",
        "  return torch.sqrt(torch.square(difference).sum()/batch_size**2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_W0DM6ZXQuXR"
      },
      "source": [
        "Finally, we're ready to initialize the architecture, loss, and optimizer. Notice the simplicity of the model!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "E5bwwa809pd5"
      },
      "outputs": [],
      "source": [
        "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "embed_dim = 64\n",
        "\n",
        "model = Model(num_tokens, embed_dim).to(DEVICE)\n",
        "\n",
        "losses = []\n",
        "\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6-kIbSDIXGqF"
      },
      "source": [
        "## Training\n",
        "\n",
        "Now let's train."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Zlic9nmfD2tC",
        "outputId": "c49272c7-e5fb-445d-b209-865174d0d224"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch: 1 \t Loss: 5.027184009552002\n",
            "Epoch: 2 \t Loss: 3.192918300628662\n",
            "Epoch: 3 \t Loss: 1.7395877838134766\n",
            "Epoch: 4 \t Loss: 1.1821469068527222\n",
            "Epoch: 5 \t Loss: 0.714796245098114\n",
            "Epoch: 6 \t Loss: 0.395793616771698\n",
            "Epoch: 7 \t Loss: 0.29397016763687134\n",
            "Epoch: 8 \t Loss: 0.18231192231178284\n",
            "Epoch: 9 \t Loss: 0.13821038603782654\n",
            "Epoch: 10 \t Loss: 0.13866692781448364\n",
            "Epoch: 11 \t Loss: 0.10848765075206757\n",
            "Epoch: 12 \t Loss: 0.10664348304271698\n",
            "Epoch: 13 \t Loss: 0.10389314591884613\n",
            "Epoch: 14 \t Loss: 0.10367842018604279\n",
            "Epoch: 15 \t Loss: 0.10630057752132416\n",
            "Epoch: 16 \t Loss: 0.10346873849630356\n",
            "Epoch: 17 \t Loss: 0.10473118722438812\n",
            "Epoch: 18 \t Loss: 0.10415786504745483\n",
            "Epoch: 19 \t Loss: 0.10364694893360138\n",
            "Epoch: 20 \t Loss: 0.1035253033041954\n"
          ]
        }
      ],
      "source": [
        "num_epochs = 20\n",
        "\n",
        "for epoch in range(1, num_epochs+1):\n",
        "  total_loss = 0\n",
        "  for (X,y) in dataloader:\n",
        "    X = X.to(DEVICE)\n",
        "    y = y.to(DEVICE)\n",
        "    optimizer.zero_grad()\n",
        "    X_embedded = model(X)\n",
        "    y_embedded = model(y)\n",
        "    loss = contrastive_loss(X_embedded, y_embedded)\n",
        "    loss.backward()\n",
        "    optimizer.step()\n",
        "    total_loss += loss.item()\n",
        "  losses += [total_loss/len(dataloader)]\n",
        "  print(f'Epoch: {epoch} \\t Loss: {loss}')\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8x_ziNQHRAVt"
      },
      "source": [
        "## Exploring the Embeddings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "70TH9Lcej1SF"
      },
      "source": [
        "What is the next word predicted?"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "zB3kOnsbkJHA",
        "outputId": "ea1361ae-4831-490a-aab2-783f595d5a4c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "witch\n",
            "tall\n",
            "fully\n",
            "dressed\n",
            "phineas\n",
            "nigellus\n",
            "wizard\n",
            "smiling\n",
            "grindelwald\n",
            "circle\n"
          ]
        }
      ],
      "source": [
        "target = torch.tensor([WORD_TO_ID['wizard']]).to(DEVICE)\n",
        "candidates = torch.tensor(list(ID_TO_WORD.keys())).to(DEVICE)\n",
        "candidates_embedded = model(candidates) # (num_tokens, embed_dim)\n",
        "target_embedded = model(target) # (embed_dim)\n",
        "\n",
        "similarities = candidates_embedded @ target_embedded.T\n",
        "_, indices = torch.topk(similarities.squeeze(), 10)\n",
        "\n",
        "for index in indices:\n",
        "  print(ID_TO_WORD[index.item()])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oogwe4ErUFNd"
      },
      "source": [
        "To investigate the model, let's visualize the embeddings of the words."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 406
        },
        "id": "eT_s4422gMUw",
        "outputId": "72612335-8f9f-44ca-df26-b2f07547f8bf"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 7 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAnEAAAGFCAYAAACSZqWqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABTGElEQVR4nO3de7xVc+L/8dfa13PrnNOpTld1uggVohsqhXR1iaEM4YSUxGQGDUZOLqXcGhEZqUyNUQ1+Y4imUSQxmYRISoopdbqe6pyzz76s9fujaX9te31ORWTNvJ+PR4+HvT7r81mf9VmX/bbW2utYjuM4iIiIiIin+I50B0RERETk0CnEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHhQ4GBnrJh+NyTirmV2PE5k4xZj3VhWTdb1+x2OP5hWFonCu5+GsB3LtW5ObDuXrL2DgOO+7GhWTdb2G+PaNkDQidAm/i98uLzTOFZFYO3HWIb3HTvxOFUbvzGsFWCBLxzCstz7Hs+sweYzh4A/fZj9iSoa7FyJ5dYvIOELUlpwLFjuObsq7udf3zTAMeTwLHsP/Sqew08ivdBxwLYNKwVldg5T9gwkYdg9cq09DA0/S8ByaRuIBrP59OgBOL70+kG7iqMrl7tvDyBqhVmd0Q7HsN72ls1UXHoxRKOu5YG6tWk653F84VBamS8RpVbpZ8Yxd6oixFcs2zc+LuKZuXzdY4RxXwtUlnHUoifx2enjksiswbYzrnDdFwASjo/SqpqA+77kOPznGHEv91k2tcK7cdsVA9Fyjvr0ZXyO+zaPBbNZd9z5rtsLoDxi8co/s7Ft92XnBiq5vsUigr709h3LIpaRh2vH/rNiPjtmWCvAtvFXlBnLHQCf37V9Bws7nGlcdmmZxTWPZxKLu5cXZlcx7fyVhAOG/SWRwC4rc+9YIIjV9Ggsn+H43V7Git89hRN3P4ZCNTI5YcgZ+AJ+1/JYeYQv/t9SHJfj2K5ZSKRkGgTD7m1Hd3P8J9PxOe7LdvwBaNDYeO7ZWZXB/R91Je6k9y3fKuPWzKcIGs4NdjiLilP6gt99vfZUBfjj8hYkHPdlBwLQvEkQny99m2Xae+ldPtv9nAfEApmsa3I2js992RY2OdZe464a21TKF72H4ERj6f2qX4fmr/3B9bwD+47dSrIwHb97Ky2eXxgmYTjGgkGLVsfkuK93dCe9V4/Db/iOBMgd+bCxTLzt4K/EGQLcvjJzIABIhLONX3yxuGUMcAAZ8b3GALev7Rxj2wAB4sbAQDxuDHAAzgHWCyxjgAOwQ1nGL22fEzeGCQDbFzCeRAFitt8Y4ADCTsR4MjuQSifDGOAAMqk0BjiAuD/DGAj8Tsy8PYA4AWOAA3B27TIGOAB/fq7xROqzqx9z4jFjgANIhLKq3dd80UrXAAdghzKM+wJUH9D+j7nch2388vHFq4wBDiARCBu3F0BVzDIGOICsQNQ1wAH79tFqjpF9Uau6YvuAo2Js37KqXfbuCssY4AByw3FjgNvXt2rK/H5jgAOI7600BjiAQFbIGOAA4pGoa4ADICfXGOAAAvFKY4AD/hOKzX0vj4dcAxxAtlVpDHAATjBsDHAAlbGAMcABBPyWa5CBA5/zEv6QMcABWDjV7qrxnbtdAxxAoKb5vAP793Jz45EoxgAHEAhUs97x8moDnPx30+1UEREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREROWjFxcUUFRUd1ja7d+9OmzZtDmub1bEsi5KSkp9seT8WhTgRERH50W3atImSkhJWrFhxpLvyX+Og/3aqiIiIyPe1adMmxowZQ1FREW3btj2ifamsrCQQ8H4E0pU4ERER+a9n2zaRSASAjIwMhTgRERH577Jnzx5GjhxJUVER4XCYwsJCzj77bJYvX542r+M4FBUVcf7556eVRSIR8vLyGDp0KIsWLaJDhw4ADB48GMuysCyL6dOnp9T59NNPOeOMM8jKyqJhw4ZMmDAhrd2qqiruuusuWrRoQTgc5qijjuLWW2+lqqoqZT7LshgxYgSzZs2idevWhMNhXnvttWTZt5+JKykpwbIs1q5dS3FxMfn5+eTl5TF48GAqKioOdQh/Mt6PoSIiInLYDBs2jLlz5zJixAhatWrF9u3befvtt1m1ahUnn3xyyryWZTFo0CAmTJjAjh07KCgoSJa9/PLL7N69m0GDBtGyZUvuvvtuRo8ezbXXXkvXrl0BOO2005Lz79y5k969e3PhhRcyYMAA5s6dy6hRozj++OPp06cPsO9q2nnnncfbb7/Ntddey3HHHcfHH3/MI488wueff85LL72U0r833niD2bNnM2LECGrXrn3AH2QMGDCApk2bMm7cOJYvX87TTz9NYWEh48eP/wEj+uNRiBMREZGkV155hSFDhvDQQw8lp916663G+a+44gruu+8+Zs+ezbBhw5LTZ86cSVFREV26dMGyLPr06cPo0aM59dRTGTRoUFo7mzZt4tlnn+Xyyy8H4Oqrr6ZJkyZMnTo1GeL+9Kc/sWDBAt588026dOmSrNumTRuGDRvGO++8kxIMV69ezccff0yrVq0Oat1POukkpk6dmvy8fft2pk6d+rMNcbqdKiIiIkn5+fm89957bNq06aDmb9myJZ06dWLWrFnJaTt27GDevHlcdtllWJZ1UO3k5OSkhLtQKETHjh1Zt25dctqcOXM47rjjOPbYY9m2bVvy35lnngnAwoULU9rs1q3bQQc4ICWEAnTt2pXt27eze/fug27jp6QQJyIiIkkTJkxg5cqVHHXUUXTs2JGSkpKUIOXmiiuuYMmSJWzYsAHYF7ZisVjyqtrBaNSoUVrgq1mzJjt37kx+XrNmDZ988gl16tRJ+deyZUsASktLU+o3bdr0oJcP0Lhx47TlAyl9+DlRiBMREZGkAQMGsG7dOiZNmkSDBg144IEHaN26NfPmzTPWueSSSwgGg8mrcTNnzqR9+/Ycc8wxB71cv9/vOt1xnOR/27bN8ccfz9///nfXf8OHD0+pm5mZedDLP9g+/JzomTgRERFJUb9+fYYPH87w4cMpLS3l5JNP5r777ks+m/ZdBQUF9OvXj1mzZnHZZZexZMkSJk6cmDLPwd5WrU7z5s358MMPOeussw5Le16nK3EiIiICQCKRoKysLGVaYWEhDRo0SHuFx3ddfvnlfPrpp9xyyy34/X4uueSSlPLs7GwAdu3a9b37N2DAADZu3Mgf/vCHtLLKykrKy8u/d9tepCtxIiIiAux7R1yjRo246KKLOPHEE8nJyWHBggUsW7Ys5deqbvr160etWrWYM2cOffr0obCwMKW8efPm5Ofn8+STT1KjRg2ys7Pp1KnTIT23dvnllyd/Bbtw4UI6d+5MIpHgs88+Y/bs2bz++uu0b9/+e627FynEiYiICABZWVkMHz6c+fPn88ILL2DbNi1atGDy5Mlcd9111dYNhUIMHDiQyZMnu/6gIRgMMmPGDG677TaGDRtGPB5n2rRphxTifD4fL730Eo888gjPPvssL774IllZWTRr1oxf/epXyR84/K9QiBMRERFgXxCbMGGC619K2O+7f2Xhu/Vr1Kjh+hccAM477zzOO++8tOmLFi066GUFg0FuvfXWat9dB9X/GOG7ZSUlJSl/wWG/4uJiiouLq13OkaRn4kREROQHi0QizJw5k1/84hdkZWUd6e78T9CVOBEREfneSktLWbBgAXPnzmX79u386le/OtJd+p+hECciIiLf26effspll11GYWEhjz76KG3btj3SXfqfoRAnIiIi31v37t1/ti/D/W+nZ+JEREREPEghTkRERMSDFOJEREREPEghTkRERMSDFOJEREREPEghTkRERMSDFOJEREREPEghTkRERMSDFOJEREREPEghTkRERMSDFOJEREREPEghTkRERMSDFOJEREREPEghTkRERMSDFOJEREREPEghTkRERMSDLMdxnCPdCRERERE5NLoSJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHhQ42Bn3LJsHju1eaCcIlG3FMtRN+EOUNTwefOmZcXckwLPLmpCw3fNkLmWM8D9F0Eq4lsfCOXzV/jIcn/uqRGI+lm0owHHSe+ffuolG1/XGF4u61g3mBGl9RRt8Afe+WZlZZPfph+X3uy+bDN4PnY5jpZfbNuyp8oNh1EK+OG1q/xufYVAtO0FulXnMrapKwv9aiOWyzRLhbLafdiH4DWMWD7Dsm8bYhowf2lPKSc9dhS8Rcy3fHarFjDYTSfhCaWXBoEWb1rn4DCuW5ezl/MRc/Lhvb9sfZE+D48By71t5LMj/+7I1tpNensNeLgvPJWDYlyp92byZ9wtsy31cHAcShkMAIDO+mzM3T3fte3RPBWtmvoZjaqCgNqH7JmOF0scMIBCroMlXC/EZjkHHsnDCWWClj6vtC7C3dlPjmEWdAGtiLXAM2zsc30u7TX/BZ9gm8UAGpc07gy99P09s28aOW34DMfd9xY7alK3dC4ZXjgfq1abZi0/gC7uPC7t3YU19ECsRTyvyZWZSq08P4/G5O5HFU6XnkMC9vAa7uTYw3bi/OPE4sS2lrmXlwXxePeYObF/QtTyLcs63XsJvuW/PRHkF2195fd+JwkV0b4w1L3yBk0gfuHDtGpz6xFX4Q4bje+tull73DHbMfb2C9WrT6pXfG8e8PBbklXUtXY+xzGCMXi3W4/e5b1BfLELt9ctcz0sAZYlsJm+/gIThq8lxHBJx97ZzwjEu77CBgN+9fHuZw+1P2BhWm1o1bCYW78YwbFBViX/F2+7n1Koo21d8vu8k4aI8ozb/77TJ2H73MQ0G4OgmlvG8WFmZYOFbO113h5oZVdxx+scEDesNUKNjP2OZeNvBX4kzBTgA2zaGCQDHH3ANcACRmN8Y4ACyqDQGOIBEIMMY4ABiCZ9rgAPw795pDHAAgcyAMcABWOGw8QsCIG6FXAMc7P/OMo9awJcwBjgAH9WPObGo8URphzKMAQ4gZvuNAQ4gGCkzBjiASCDXNcABBAI+44kKIIOIMcAB+7a1IYwARBIB1y8XgAwrYvxCBohaGcYAdzBCdqWx74nKKnOAA6ycXGOAA/AnosYA958WXAMcgOPzVztmCSdgDHAAATtiDHAAdiDoGuAA7D17jAEOwE44xgAH4M/PNQc4gMpy1wAH4AuHqj0+K+ywMcDBvnNPdfuLYwhYAFWBHGOAg//s54YAB2BXVRkDHEAiEncNcACh3ExjgAOI7q40BjiAQM0a1Y55VcJvPMbC/oQxwAH4EjHjeQn2b5PvdwxmBhPGAAewtwJjgAPIzbTNAQ4gbu67HU8YAxxAJJhrDHAAfj/VnhejUce4O2SH4tUGOPnvptupIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIickRs2bKFiy66iFq1amFZFhMnTgRgzZo19OzZk7y8PCzL4qWXXmL69OlYlsX69esP2/J/jDZ/St//9fQiIiIiP8BNN93E66+/zl133UW9evVo3749AFdeeSVffvkl9913H/n5+bRv354FCxYc4d7+/CjEiYiIyBHxxhtvcP7553PzzTcnp1VWVrJ06VLuuOMORowYkZx++eWXc8kllxAOh49EV3+WdDtVREREfjDbtolEIodUp7S0lPz8/JRpW7duBUib7vf7ycjIwDL8neifg4qKip90eQpxIiIikmLRokW0b9+ejIwMmjdvzpQpUygpKUkJUJZlMWLECGbNmkXr1q0Jh8PMmzePoqIizj///LQ2I5EIeXl5DB06NPksmuM4PP7441iWhWVZlJSU0KRJEwBuueUWLMuiqKgIcH9+raioiHPOOYe3336bjh07kpGRQbNmzXj22WfTlv/JJ59w5plnkpmZSaNGjbj33nuxbdt1/SdPnpxcpwYNGnD99deza9eulHm6d+9OmzZt+Ne//sXpp59OVlYWt99++yGO9A+j26kiIiKS9MEHH9C7d2/q16/PmDFjSCQS3H333dSpUydt3jfeeIPZs2czYsQIateuTdOmTRk0aBATJkxgx44dFBQUJOd9+eWX2b17N4MGDaJhw4b88Y9/5PLLL+fss8/miiuuAOCEE04gPz+fm266iV/+8pf07duXnJycavu7du1aLrroIq6++mquvPJKnnnmGYqLi2nXrh2tW7cGYPPmzZxxxhnE43F++9vfkp2dzVNPPUVmZmZaeyUlJYwZM4YePXpw3XXXsXr1ap544gmWLVvGkiVLCAaDyXm3b99Onz59uOSSSxg0aBB169b9XmP+fSnEiYiISNJdd92F3+9nyZIlNGjQAIABAwZw3HHHpc27evVqPv74Y1q1apWclpWVxX333cfs2bMZNmxYcvrMmTMpKiqiS5cuWJZFs2bNuPzyy2nZsiWDBg1Kzpebm8tNN93EySefnDLdZPXq1bz11lt07do12dejjjqKadOm8eCDDwIwfvx4tm7dynvvvUfHjh2BfT+eOProo1Pa2rp1K+PGjaNnz57MmzcPn2/fDctjjz2WESNGMHPmTAYPHpycf/PmzTz55JMMHTr0gP38Meh2qoiIiACQSCRYsGAB/fv3TwY4gBYtWtCnT5+0+bt165YS4ABatmxJp06dmDVrVnLajh07mDdvHpdddtlhf6atVatWyQAHUKdOHY455hjWrVuXnPbqq69yyimnJAPc/vkuu+yylLYWLFhANBpl5MiRyQAHMGTIEHJzc3nllVdS5g+Hwymh7qemECciIiLAvh8aVFZW0qJFi7Qyt2lNmzZ1beeKK65gyZIlbNiwAYA5c+YQi8W4/PLLD2+HgcaNG6dNq1mzJjt37kx+3rBhQ9pVN4Bjjjkm5fP+/n53eigUolmzZsny/Ro2bEgoFPreff+hFOJERETke3F7pgzgkksuIRgMJq/GzZw5k/bt26eFo8PB7/e7Tncc57Av67tM6/9TUYgTERERAAoLC8nIyGDt2rVpZW7TTAoKCujXrx+zZs1iw4YNLFmy5Ee5CnewmjRpwpo1a9Kmr169Om0+t+nRaJQvv/wyWf5zoRAnIiIiwL6rWj169OCll15i06ZNyelr165l3rx5h9TW5Zdfzqeffsott9yC3+/nkksuOdzdPWh9+/bl3Xff5Z///Gdy2tatW1Oe2wPo0aMHoVCIRx99NOVK3tSpUykrK6Nfv34/WZ8PhkKciIiIJJWUlBCPx+ncuTMTJkxg3LhxdOvWjTZt2hxSO/369aNWrVrMmTOHs88+m8LCwh+pxwd26623UqtWLXr37s2YMWN48MEH6dy5c9qVtTp16nDbbbfx2muv0bt3bx5//HFuvPFGbrjhBjp06HBQv5b9KSnEiYiISFK7du2YN28eNWvW5M4772Tq1KncfffdnHXWWWRkZBx0O6FQiIEDBwIc0VupAPXr12fhwoWccMIJ3H///UycOJErrriCX/3qV2nzlpSU8Nhjj/HVV19x0003MXv2bK699lrmz5+f8o64nwO9J05ERERSnHnmmSxfvjxlWv/+/WnUqFHy88H8cCAUClGjRg3Xv+BgaqOoqMh1enFxMcXFxSnTvv3XG75t0aJFadOOP/541+lXXXVV2rTrr7+e66+/3rXt6pbxU9OVOBEREUlRWVmZ8nnNmjW8+uqrdO/e/aDbiEQizJw5k1/84hdkZWUd5h4K6EqciIiIfEezZs0oLi5OvhvtiSeeIBQKceuttx6wbmlpKQsWLGDu3Lls377d9ZalHB4KcSIiIpKid+/ePPfcc2zevJlwOMypp57K2LFjXV+Y+12ffvopl112GYWFhTz66KO0bdv2x+/w/yiFOBEREUkxbdq07123e/fuP8mLdkXPxImIiIh4kkKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kOU4jnOkOyEiIiIih0ZX4kREREQ8SCFORERExIMU4kREREQ8SCFORERExIMU4kREREQ8SCFORERExIMU4kREREQ8SCFORERExIMCBzvjxt8MgnjMtSy6J8qqP63CSbi/NzhUrxbH//VBfOFQWtmOvT7ue7EW8YTlWjdrdynnTy/Gn3BftlO3PtHn5kE47FoepoqO4X/ht9L7VvXNdj648A6caNy9bq1sTh3/C/wh92Gyg2EqWrYHn9+1PEaAr/wtcKz0rByJWry3OgvbcV/vGs5urow9RYCEcdlVx7QHn3sOt/fspnzmM5BIr+/LCFO7+2lYfvd+77VyeCk8iITlvt7+0k3Uu7YXVizqWl6RW5f51/8JO5i+TXKzbIadV07AfdFYdoIakW24jwo45Xtw/voc2O7j4mTXwO4/GALpfd9bFeT5lS1IOO5jVsPaw7XhmQQs97YTwQy2HtvduL2jm0pZ22cYTjR9Xw3nZ9B+1Bn4g4YV9/sJ1KuHZbmvuR0IUdG0rXF7w7792622bfnYE64NhrZjtp91exrgGEY9RJTjQ5/gczmGABL4KQ02Apf93N5WSuSWoRBzP34tv0VOYbZ5vRM2kbKIaxlAICtMs3M64nPZlyM7Klh2z3zsuO1atyKvLgtHznbdTwHynF38JvEwQdzPD1gWgVz3vleG8njn+N9g+4LudTdvIvjLvlhR92PIH/ZRt1MtLL/7uMQjCba8tx1cVi1cN58OfxqFP+y+7IQvwM78pq7bC8AX2UvB0pewDMdYuT+Plxre5LpuWb5KzstfjN9yH3McB+y48fgutzN5peIMbNyPE8cBl8MLgGz2cpE113j8VljZLKxxIbbhvBa2orTLXW3czy07Tvaeb9z7btv4qiqM6xWxMvlXXk8cy329bBvKIgHcj2DYUxbh2SffJ+HyHVuQ72fcHQ0JBc3XZI5u3sRYJt528FfiDAEOIB6JGwMcQCC/hmuAAyiv8hkDHEBGZZkxwAE4eTWNAQ4gaMVcAxxAbNdeY4ADCOVkGAMcgOMPGr/QARIEXAMcQCxuGQMcQIZTYQxwwL6QYvxCB6ey0jXAAfhCIWOAA6gi0xjgAHy7dxoDHEBVVp7xizEz7BgDHICFbTwR7ms8YgxwAIQzXQMcQCTuNwY4gCwqjV8AsC9IVbu9d+52DXAAweyQOcAB+HzGIAPg+Kvf3mA6/YODzxjgABKOzxjgAAJW3PjFBmBbfmMgYM9uY4ADsHxWtevNAf6ejD8cdA1wALHyKmOAA4hm5Rv3U4Bsys0BDqCavscC2eYAB1hlu4wBDsAX9BkDHIAds10DHEAwL9sY4IB9QcK0vQBftMoY4AAifvO6ha2oOcD9R3XHd5UTMgY42BfiTDKIVHv8Rq0MY4ADCPiq388tp7pzU3VHEMR9YWOAg/2b0txCZWXcNcAB1MjxVxvg5L+btryIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiICAAlJSVYlsW2bdsOW5vLli3jtNNOIzt738uxV6xYAcBrr71G27ZtycjIwLIsdu3aRXFxMUVFRYdt2YfDAw88QLNmzfD7/bRt2xaAeDzOrbfeylFHHYXP56N///5HpG8H/RcbRERERA5FLBbj4osvJiMjg0ceeYSsrCyaNGnC9u3bGTBgAK1bt+bxxx8nHA6TnZ19pLubZv78+dx6660MGjSIkpISateuDcAzzzzDAw88wMiRIzn55JNp3LjxEemfQpyIiIj8KL744gs2bNjAH/7wB6655prk9Ndee409e/Zwzz330KNHjyPYw+q98cYb+Hw+pk6dSigUSpnesGFDHnnkkSPYO91OFRERkR9JaWkpAPn5+Qc1/eemtLSUzMzMlAC3f/rPoe8KcSIiIpJi//Np+fn55OXlMXjwYCoqKgBYv349lmUxffr0tHqWZVFSUgJAcXEx3bp1A+Diiy/Gsiy6d+9O9+7dufLKKwHo0KEDlmVRXFxs7Itt20ycOJHWrVuTkZFB3bp1GTp0KDt37kyZr6ioiHPOOYf58+cnn7Vr1aoVL7zwQsp8+5/7+67p06djWRbr169Prsu0adMoLy/HsqzkOluWxcKFC/nkk0+S0xctWnQQo3r46XaqiIiIpBgwYABNmzZl3LhxLF++nKeffprCwkLGjx9/0G0MHTqUhg0bMnbsWG688UY6dOhA3bp1ATjmmGN46qmnuPvuu2natCnNmzevtp3p06czePBgbrzxRr788ksee+wxPvjgA5YsWUIwGEzOu2bNGgYOHMiwYcO48sormTZtGhdffDGvvfYaZ5999iGNwR//+Eeeeuop/vnPf/L0008DcNJJJ/HHP/6R++67j7179zJu3DgAjjvuuENq+3BRiBMREZEUJ510ElOnTk1+3r59O1OnTj2kEHfqqadSVVXF2LFj6dq1KxdddFGybOPGjTz11FP06dOH9u3bG9t4++23efrpp5k1axaXXnppcvoZZ5xB7969mTNnTsr0zz//nL/85S9ceOGFAFx99dUce+yxjBo16pBD3KBBg1iwYAHLly9n0KBByemtW7fm6aefxu/3p0w/EnQ7VURERFIMGzYs5XPXrl3Zvn07u3fv/kn7MWfOHPLy8jj77LPZtm1b8l+7du3Iyclh4cKFKfM3aNCACy64IPk5NzeXK664gg8++IDNmzf/pH3/KehKnIiIiKT47iszatasCZD2HNqPbc2aNZSVlVFYWOhavv8HEvu1aNEi7Xm3li1bAvue5atXr96P09EjRCFOREREUvj9ftfpjuO4/igAIJFIHPZ+2LZNYWEhs2bNci2vU6fOIbf5U/b/x6YQJyIiIgdt/1W5Xbt2pUzfsGHDYV9W8+bNWbBgAZ07dyYzM/OA869duzYtaH7++ecAyb8E8e3+f/s1IT9G/39seiZOREREDlpubi61a9fmrbfeSpk+efLkw76sAQMGkEgkuOeee9LK4vF4WpDctGkTL774YvLz7t27efbZZ2nbtm3yVur+X8J+u//l5eXMmDHjsPf/x6YrcSIiInJIrrnmGu6//36uueYa2rdvz1tvvZW84nU4devWjaFDhzJu3DhWrFhBz549CQaDrFmzhjlz5vD73/8+5VevLVu25Oqrr2bZsmXUrVuXZ555hi1btjBt2rTkPD179qRx48ZcffXV3HLLLfj9fp555hnq1KnDV199ddjX4cekECciIiKHZPTo0WzdupW5c+cye/Zs+vTpw7x584w/QPghnnzySdq1a8eUKVO4/fbbCQQCFBUVMWjQIDp37pwy79FHH82kSZO45ZZbWL16NU2bNuX555+nV69eyXmCwSAvvvgiw4cP584776RevXqMHDmSmjVrMnjw4MPe/x+TQpyIiIgA+/6awf6/uPBtxcXFKX9VITMzk6effjr5Etz9HMdJ+dy9e/e0aW7t7ef2VyAAhgwZwpAhQw7Yf9h3pa1nz57VznPyySfz7rvvuvbru/1x69OR+gsN36Vn4kREREQ8SCFORERExIMU4kREREQ8SM/EiYiIiOetX7/+SHfhJ6crcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIeZDmO4xzpToiIiIjIodGVOBEREREPUogTERER8SCFOBEREREPUogTERER8SCFOBEREREPUogTERER8SCFOBEREREPUogTERER8aDAwc64pMMpONGoa5kds9mzthIMrw0O1avF8X99EF84lFa2Y6+Pe/5SQDxhudbN3l3K+c8OJpCIuZYnCuuza9rrEA67L9sf5/h6W/C5NJ/YWsqO64ZCzL3tYGaQFr3b4PO7Z10rI4Pwaadj+f2u5XsTmcwu60mC9HK/3+GoQtu1XwAZib2ctuNF/CTcZ7ATULbLvQxIVFWx818rweVdzr7MTPJ79TD328nixeh5rv0G8JduosGwXlgx9/2hLKOAKV3vIeEPppXl1AhzyZD2BALubWdGd9Hj84fwO3HX8uieSlY9vwQnYbuW+2rXpuCxJ7BC6fta3PaxOVIAuA96yIlwQvSf+HBvO2qF+TTcHsdy3x/sb76h/PwLwOU4CdUI0WbwCfgChn0pFKJGu7ZYPvdyJx4nsfkb1zKAREY2uzpfCP70Q9rGIkIWWO7rvX1njN/d9xXxuPsBXDM/wOjbWhAMuvctEN1Ls09ewOek76uO4+BUVRn7jeXDysvHMvQtumsvKyf9BSfufhz4g34KW9XFcjmQYuUx1r70JY7tvl7V7acAhVlVTOv/MWG/e307GqNizReux5iVkUHGaV2Nx1hkyy7eu3IidtR9Pw/lhml7XXvj/lJVFmHFE+/jJNKXHa6Ty6lPXYs/5H56dywLOzvPuD8QrYLVH7uuF8Befx5/qTeShJU+btnxnVzw9QMEDMdvPBKl9F+rjW1vt3O5IzKMmOGrqSDfz313HEXIZV/02TFql3+NZfgiqkiEeW1XR2zDeS3sj9Ou4WYMhyD+RJR6Oz5xbd+pisDH7xvXqzKYz6I2t2D73Pe1PWURZk5ZRsJlex5IYU6caZd/TThgrpvRc/AhtyvecNBX4kwBDsCJO8YABxDIr+Ea4ADKIz5jgAMIR8qMAQ7Aya1pDHAAAZ85KNm7dxsDHIA/HDAGOACCIeNJGiDihM1ByIexXwBBu8oc4AAMX0z7ObG48YRihavvd1U1/Qbw7d5pDHAAlaEc4xdjRlbQGOAAQolyY4ADiEdixgAH4MvNdQ1wADY+TAEOIODEjAEOIE7AGOAAnF27XAMcQCAzYPxCBrCCAWOAA8A29wvACWa4Brj/tG7+wgb27k0YAxxAdnbAGOAA/PGIa4Db17EDfCn5LGOAA4hXRIwBDsAX8LkGOIB4JGEMcFD9fgqQmxE3BjgAJ2E+xg50boiVVRgDHEAgq/r9JV4Rdw1wAKHcTGOAA8DyVbs/EK9mvYCIL8s1wAGEE+XGAAdgH6DtvWQaAxxATo7fNcAB+BzbGOAAqpygMcABBPy2McAB+Oy4uf0DrFc0kG0McACVlbHvFeAAcjMT1QY4+e+m26kiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIv8TiouLKSoqOtLdOGwU4kREREQ8SCFORERExIMU4kRERMSovLz8SHdBDBTiREREBICSkhIsy+LTTz/l0ksvpWbNmnTp0oWPPvqI4uJimjVrRkZGBvXq1eOqq65i+/btybpz587FsizefPPNtHanTJmCZVmsXLkyOe2zzz7joosuoqCggIyMDNq3b89f//rXlHrTp0/HsiyWLFnCr3/9a+rUqUN2djYXXHABW7duTVvOvHnz6NatGzVq1CA3N5cOHTrwpz/9qdp1fvDBBznttNOoVasWmZmZtGvXjrlz56bMc+GFF3LyySenTDv33HOxLCulz++99x6WZTFv3rxql3m4KMSJiIhIiosvvpiKigrGjh3LkCFD+Pvf/866desYPHgwkyZN4pJLLuHPf/4zffv2xXEcAPr160dOTg6zZ89Oa+/555+ndevWtGnTBoBPPvmEU045hVWrVvHb3/6Whx56iOzsbPr378+LL76YVv+GG27gww8/5K677uK6667j5ZdfZsSIESnzTJ8+nX79+rFjxw5uu+027r//ftq2bctrr71W7br+/ve/56STTuLuu+9m7NixBAIBLr74Yl555ZXkPF27duXDDz9k9+7dADiOw5IlS/D5fCxevDg53+LFi/H5fHTu3PkgR/qHCfwkSxERERHPOPHEE1OuYFVWVvKb3/wmZZ5TTjmFX/7yl7z99tt07dqVzMxMzj33XObOncujjz6K3+8HYPPmzbz55puUlJQk6/7qV7+icePGLFu2jHA4DMDw4cPp0qULo0aN4oILLkhZVq1atZg/fz6WZQFg2zaPPvooZWVl5OXlUVZWxo033kjHjh1ZtGgRGRkZybr7Q6bJ559/TmZmZvLziBEjOPnkk3n44Yfp168fsC/E2bbNkiVL6NOnDytXrmTnzp1cfPHFaSHuxBNPJDc394BjfDjoSpyIiIikGDZsWMrnb4ecSCTCtm3bOOWUUwBYvnx5smzgwIGUlpayaNGi5LS5c+di2zYDBw4EYMeOHbzxxhsMGDCAPXv2sG3bNrZt28b27dvp1asXa9asYePGjSnLv/baa5MBDvaFqkQiwYYNGwD4+9//zp49e/jtb3+bEuCAlHpuvr1uO3fupKysjK5du6as10knnUROTg5vvfUWsC+sNWrUiCuuuILly5dTUVGB4zjJQPtT0ZU4ERERSdG0adOUzzt27GDMmDH8+c9/prS0NKWsrKws+d+9e/cmLy+P559/nrPOOgvYdyu1bdu2tGzZEoC1a9fiOA533nknd955p+vyS0tLadiwYfJz48aNU8pr1qwJ7AtdAF988QVA8nbtofjb3/7Gvffey4oVK6iqqkpO/3b48/v9nHrqqcmrbosXL6Zr16506dKFRCLBu+++S926ddmxY4dCnIiIiBw53746BTBgwADeeecdbrnlFtq2bUtOTg62bdO7d29s207OFw6Hk8+1TZ48mS1btrBkyRLGjh2bnGf//DfffDO9evVyXX6LFi1SPu+/NftdB7pVeiCLFy/mvPPO4/TTT2fy5MnUr1+fYDDItGnT0n4Q0aVLF+677z4ikQiLFy/mjjvuID8/nzZt2rB48WLq1q0LoBAnIiIiPw87d+7kH//4B2PGjGH06NHJ6WvWrHGdf+DAgcyYMYN//OMfrFq1CsdxkrdSAZo1awZAMBikR48eh6WPzZs3B2DlypVpAbA6f/nLX8jIyOD1119PPpsHMG3atLR5u3btSjQa5bnnnmPjxo3JsHb66acnQ1zLli2TYe6noGfiRERExGj/VbDvXvWaOHGi6/w9evSgoKCA559/nueff56OHTum3J4tLCyke/fuTJkyhW+++SatvturQw6kZ8+e1KhRg3HjxhGJRFLKqrta5/f7sSyLRCKRnLZ+/XpeeumltHk7depEMBhk/PjxFBQU0Lp1a2BfuHv33Xd58803f9KrcKArcSIiIlKN3NxcTj/9dCZMmEAsFqNhw4bMnz+fL7/80nX+YDDIhRdeyJ///GfKy8t58MEH0+Z5/PHH6dKlC8cffzxDhgyhWbNmbNmyhaVLl/Lvf/+bDz/88JD7+Mgjj3DNNdfQoUOH5DvuPvzwQyoqKpgxY4ZrvX79+vHwww/Tu3dvLr30UkpLS3n88cdp0aIFH330Ucq8WVlZtGvXjnfffTf5jjjYdyWuvLyc8vLynzzE6UqciIiIVOtPf/oTvXr14vHHH+e2224jGAxW+0LbgQMHsnfvXmDf83Tf1apVK95//3369evH9OnTuf7663nyySfx+Xwpt2wPxdVXX81f//pXcnNzueeeexg1ahTLly+nT58+xjpnnnkmU6dOZfPmzYwcOZLnnnuO8ePHp73iZL/9Ia1Lly7JafXq1UvewtWVOBERETkiSkpKUt7ntl/Dhg154YUX0qabblX26NHjgD86aNasmfEK2X7FxcUUFxenTe/evbtr++eeey7nnnuusb3p06enTbvqqqu46qqr0qa7jcOECROYMGFC2nTT84E/Nl2JExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEgy3Ec50h3QkREREQOja7EiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBwUOdsbdk26FRNy1rDKUz+ITR2H7gu4L8SUoqrkLn+VSuGMrjBkB8ZhrXV/QT60W9bBcK0MksxZLz34Y2x9yLY/FHb78OobbK43DQYdT29j4DxBlLfdFE4rvpe3Gl/CRcC1PVEbYufhdsO20MrtGTcquuBMC7mNm7d5B7rQSLMOYA1gBP5ahc4mcmuwYNNq1/VjCx7odNXFwrxvwJWhaYNheADj4jLUhENlD4/dn4XPSx8UJBKF5K/C5D3ppmcVVT+YRS7i3npEVonO/dvgNG62GU8bV0ccJuGyTyI5y3rvzFex4+vYACNSvQ4vXnsIXdt+XfPEqam9eiYX7+7EryOIf/r7Ylj+tzLIccjMS5n2JKk70rcBnube9uyrE1I9PIOG4r3ekooplC1bg2On1s3LCnD+oA/5Aer8AQru30PaZS/Elou6dq1mL8JiHsYLu4+Kv2E3hwmlYtstxEApDu9OwfO7Ljlkh1tdoi2O5r1fCtiityAHD3mbhkBOKuo5reaXDq2/F3Q6//6vvt4zHUF4owm/a/pOgz72B2M4y1j04BSeevt7B7AxaXNwdn2HMKwO5vNN8KLbP/RQcdio4Nb4QP4bOJ+L4Sje6jkosmMVXx/TFMYy55djkUGY8fi07QU7lNmP5zooA9y1oTtxO32Z5WQl+06+UoPuiwbEJxSuMbe/bey3jSTdh+dkZru9aXrYXHpsNccO5o0a2xVW/yCAQcC+P2xZb9tbAtK+FqKJd+EPXY9SKR8n8+lOs6t6dHwwZ1yseyODfTU43brNIzM+yr+viOOn1/aWbqHdtL6yY4fgF+sVWm/slnnbwV+KqCRPRQLYxwAH4LcccCPbuNgY4AJ/fZwxwANFQDWOAA0gkHNcABxAK8L0DHEAgUWUMcABONOYa4ADsjGxjgAPwVe6tNsDt65u5c3amuf2EbVUTwcDvq2Z77V92NWX+eMQ1wAEQCBgDHEBZpc8Y4ACC4aAxwAFkOhWuAQ4gtrfKGOAAAjVzjQEOwGfHjQEOIErYNcDBvgOt2n2JuDHAAVTGA8YABxCLxl0DHEBGZtAY4AAClbvMAQ6wcmoYAxyAL1rpHuAAgiFjgANIWAFjgAOwHYvq9jbLcozjWhU1Hn7fqm9uOzsQMwY4gERFpWuAA/BnhIwBDiAWyDQGOIAQUXOAA7ATxlFJBMLGMABgYVd7/FpO9eV7owHXAAeQHbbNAY59ofsAp5ZqDxQHn7G8ImIOcACZGRgDHBx4Xwta5mPUSsSrD3CWOZgCJPyhardZzPa5BjgA3+6d1QY4+e+m26kiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIuJBCnEiIiIiHqQQJyIiIofd9OnTsSyL9evXH3DeoqIiiouLf/Q+/bdRiBMRERH5ASZPnsz06dN/8uUqxImIiIj8AApxIiIiIh5SUVFxRJevECciIiLMnTsXy7J4880308qmTJmCZVmsXLkSgDfeeIOuXbuSnZ1Nfn4+559/PqtWrTrgMhzH4d5776VRo0ZkZWVxxhln8Mknn7jOu2vXLkaOHMlRRx1FOBymRYsWjB8/HvtbfxR5/fr1WJbFgw8+yOOPP06zZs3IysqiZ8+efP311ziOwz333EOjRo3IzMzk/PPPZ8eOHWnLmjx5Mq1btyYcDtOgQQOuv/56du3alTJP9+7dadOmDf/61784/fTTycrK4vbbb6eoqIhPPvmEN998E8uysCyL7t27H3AsDgfzX2AWERGR/xn9+vUjJyeH2bNn061bt5Sy559/ntatW9OmTRsWLFhAnz59aNasGSUlJVRWVjJp0iQ6d+7M8uXLKSoqMi5j9OjR3HvvvfTt25e+ffuyfPlyevbsSTQaTZmvoqKCbt26sXHjRoYOHUrjxo155513uO222/jmm2+YOHFiyvyzZs0iGo1yww03sGPHDiZMmMCAAQM488wzWbRoEaNGjWLt2rVMmjSJm2++mWeeeSZZt6SkhDFjxtCjRw+uu+46Vq9ezRNPPMGyZctYsmQJwWAwOe/27dvp06cPl1xyCYMGDaJu3bp0796dG264gZycHO644w4A6tat+z23wqFRiBMREREyMzM599xzmTt3Lo8++ih+vx+AzZs38+abb1JSUgLALbfcQkFBAUuXLqWgoACA/v37c9JJJ3HXXXcxY8YM1/a3bt3KhAkT6NevHy+//DKWZQFwxx13MHbs2JR5H374Yb744gs++OADjj76aACGDh1KgwYNeOCBB/jNb37DUUcdlZx/48aNrFmzhry8PAASiQTjxo2jsrKS999/n0AgkOzDrFmzeOKJJwiHw2zdupVx48bRs2dP5s2bh8+37wblsccey4gRI5g5cyaDBw9OLmfz5s08+eSTDB06NKW/v/vd76hduzaDBg069IH/AXQ7VURERAAYOHAgpaWlLFq0KDlt7ty52LbNwIED+eabb1ixYgXFxcXJAAdwwgkncPbZZ/Pqq68a216wYEHyatn+AAcwcuTItHnnzJlD165dqVmzJtu2bUv+69GjB4lEgrfeeitl/osvvjgZ4AA6deoEwKBBg5IBbv/0aDTKxo0bU/o0cuTIZIADGDJkCLm5ubzyyispywmHwymh7kjTlTgREREBoHfv3uTl5fH8889z1llnAftupbZt25aWLVvy7rvvAnDMMcek1T3uuON4/fXXKS8vJzs7O618w4YNAMkra/vVqVOHmjVrpkxbs2YNH330EXXq1HHtZ2lpacrnxo0bp3zeH+i+fbXu29N37tyZ0qfvrk8oFKJZs2bJ8v0aNmxIKBRy7dORoBAnIiIiwL4rTf379+fFF19k8uTJbNmyhSVLlqTd7vyx2bbN2Wefza233upa3rJly5TP+2/9fpdpuuM436tfmZmZ36vej0UhTkRERJIGDhzIjBkz+Mc//sGqVatwHIeBAwcC0KRJEwBWr16dVu+zzz6jdu3arlfhvl13zZo1NGvWLDl969atyStj+zVv3py9e/fSo0ePw7JOJt9en2/3KRqN8uWXXx708r99e/inpGfiREREJKlHjx4UFBTw/PPP8/zzz9OxY0eaNm0KQP369Wnbti0zZsxIeQXHypUrmT9/Pn379q223WAwyKRJk1KuhH33l6YAAwYMYOnSpbz++utpZbt27SIej3//FfxOn0KhEI8++mhKn6ZOnUpZWRn9+vU7qHays7PTXknyU9CVOBEREUkKBoNceOGF/PnPf6a8vJwHH3wwpfyBBx6gT58+nHrqqVx99dXJV4zk5eUlf8Hqpk6dOtx8882MGzeOc845h759+/LBBx8wb948ateunTLvLbfcwl//+lfOOecciouLadeuHeXl5Xz88cfMnTuX9evXp9X5PurUqcNtt93GmDFj6N27N+eddx6rV69m8uTJdOjQ4aB/bdquXTueeOIJ7r33Xlq0aEFhYSFnnnnmD+7fgSjEiYiISIqBAwfy9NNPY1kWAwYMSCnr0aMHr732GnfddRejR48mGAzSrVs3xo8fn7xiZ3LvvfeSkZHBk08+ycKFC+nUqRPz589Pu+KVlZXFm2++ydixY5kzZw7PPvssubm5tGzZkjFjxqT8EvWHKikpoU6dOjz22GPcdNNNFBQUcO211zJ27NiUd8RVZ/To0WzYsIEJEyawZ88eunXrphAnIiIiP70ePXpU+/D/WWedlfz1qklxcTHFxcUp03w+H6NHj2b06NEp09evX59WPycnh7Fjx1b7o4qioiLXfnbv3t11ulufAK6//nquv/5643KAlNeufFfdunX529/+Vm39H4OeiRMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIMtxHOdId0JEREREDo2uxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcFDnbG3R/8AxzbvTARJ7hjE5ahbiIQZttR7cCXnhktJ0FmVZmxLo5NMLrX3LYVYFfuUWC559H/e5Nxegt7I37+8n59Eo576xkhmzPaVuE3RF3bgYpY2LVtgJAT4cToUny4jFs8RuCbDVhU867lUAgs97Yj/mzer3chjuW+CYPEaBlai89Kb393JMD0Zc1IOO4rlksZI3xTCFoJ1/J4OJstHS8Cv/uyrT27yPrTQ1iJeFqZk5NL4pLhEHCvm3D8bHUKMY2phU2Gr8o0LPicOLWr/u1a28GiKphtHNOoHWBNpAjH8P82QaK0Ca5yHVOAOH62BBq57otVMR8ffF2AY9jXMv1V9Kn3AX5D21Y8Stamz437S6Uvm8UFF2G77A8JG8rKfZjGNNOK0DPvXfyW+/FtW37KchoYx81XuZeCd+Zi2en7ix3OoqJjH/D73dvGojJYw9i2ZSfIjWw1Hv8xgqyzjsFxGXMLh7A/atxX9m2TfOM2Ce8t5dS/DMafiLkvu0Yd1l4zFScQSisLWVFOzHU//gB80Qj5a97FMpxT7WiUilWrwfAudjurBlX9r3I9Bn2xKgq+Xo5lqOtgQTjTOOa2L0hZYQvjObU8FuRvX7TENpw/QkFj02QE4nRu+o3xnOo4kMCPaV/1OQnyEttcS614lBqla4zHSJUvk49q98Kx3PdFHwlqh8vM5xY7TkGV+/dcWSTI4+8dQ8J2X7FwyKJj2zB+n3vjQV+clvkbMRQTTfj5fFcD13NTcM9Wjnt+BD7DfgpQ9PT/M5aJtx38lThTgAMs2zaHMMD2BVwDHIDlONXWPVC54/MZTzbfasV1aiTuMwY4gFAA48kG+M/J31w/4ETdAxyAnag+wFmW+UwIxH0ZxgAH4Lfixi+QypjfGOAAsqg0BjgAO5hhDHAAVqTcNcABkJFlDHAANuawkWy/mmKfkzDWdg4wpgn8xgAHEKhmTAFs/MZ9MWb7jGEBIOyLGQMc7Asz1e0vUV+Ga4CD/TngQMs2H9+25at23HyxiGuAA3CCIWOAgwNvEx/Vn1sSBFwD3H9ar3ZfidlWtdskFCkzBjiAeGaea4ADCPgS1e4rViJmDHAATjxuDHAAZGQaj0GfHTMGuH0Lr37MbZ95PwaoivuNAe4ATRP029WeU//TSjUl5v3hQMdI3Bc2BjgAn1X9/lLdvlgRCxgDHEAwgDHAAfithDHAwb7/uTWdmwKR3dUGOPnvptupIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiLiQQpxIiIiIh6kECciIiJJe/bsYeTIkRQVFREOhyksLOTss89m+fLlAHTv3p02bdrw6aefcsYZZ5CVlUXDhg2ZMGFCSjvRaJTRo0fTrl078vLyyM7OpmvXrixcuDBlvvXr12NZFg8++CCPPPIITZo0ITMzk27durFy5cq0/n322WdcdNFFFBQUkJGRQfv27fnrX//64w3Iz5hCnIiIiCQNGzaMJ554gl/84hdMnjyZm2++mczMTFatWpWcZ+fOnfTu3ZsTTzyRhx56iGOPPZZRo0Yxb9685Dy7d+/m6aefpnv37owfP56SkhK2bt1Kr169WLFiRdpyn332WR599FGuv/56brvtNlauXMmZZ57Jli1bkvN88sknnHLKKaxatYrf/va3PPTQQ2RnZ9O/f39efPHFH3Vcfo4CR7oDIiIi8vPxyiuvMGTIEB566KHktFtvvTVlnk2bNvHss89y+eWXA3D11VfTpEkTpk6dSp8+fQCoWbMm69evJxQKJesNGTKEY489lkmTJjF16tSUNteuXcuaNWto2LAhAL1796ZTp06MHz+ehx9+GIBf/epXNG7cmGXLlhEOhwEYPnw4Xbp0YdSoUVxwwQWHeTR+3nQlTkRERJLy8/N577332LRpk3GenJwcBg0alPwcCoXo2LEj69atS07z+/3JAGfbNjt27CAej9O+ffvkrdlv69+/fzLAAXTs2JFOnTrx6quvArBjxw7eeOMNBgwYwJ49e9i2bRvbtm1j+/bt9OrVizVr1rBx48YfvP5eohAnIiIiSRMmTGDlypUcddRRdOzYkZKSkpRwBtCoUSMsy0qZVrNmTXbu3JkybcaMGZxwwglkZGRQq1Yt6tSpwyuvvEJZWVnaco8++ui0aS1btmT9+vXAvit1juNw5513UqdOnZR/d911FwClpaU/ZNU9R7dTRUREJGnAgAF07dqVF198kfnz5/PAAw8wfvx4XnjhheStUr/f71rXcZzkf8+cOZPi4mL69+/PLbfcQmFhIX6/n3HjxvHFF18ccr9s2wbg5ptvplevXq7ztGjR4pDb9TKFOBEREUlRv359hg8fzvDhwyktLeXkk0/mvvvuS4a4gzF37lyaNWvGCy+8kHLVbv9Vs+9as2ZN2rTPP/+coqIiAJo1awZAMBikR48eh7A2/710O1VEREQASCQSabc6CwsLadCgAVVVVYfU1v6rdd++Ovfee++xdOlS1/lfeumllGfa/vnPf/Lee+8lg2NhYSHdu3dnypQpfPPNN2n1t27dekj9+2+gK3EiIiIC7HtHXKNGjbjooos48cQTycnJYcGCBSxbtizl16oH45xzzuGFF17gggsuoF+/fnz55Zc8+eSTtGrVir1796bN36JFC7p06cJ1111HVVUVEydOpFatWim/jH388cfp0qULxx9/PEOGDKFZs2Zs2bKFpUuX8u9//5sPP/zwB4+BlyjEiYiICABZWVkMHz6c+fPn88ILL2DbNi1atGDy5Mlcd911h9RWcXExmzdvZsqUKbz++uu0atWKmTNnMmfOHBYtWpQ2/xVXXIHP52PixImUlpbSsWNHHnvsMerXr5+cp1WrVrz//vuMGTOG6dOns337dgoLCznppJMYPXr0D119z1GIExEREWDfq0ImTJiQ9tcXvs0tgAFMnz495bNlWdx2223cdtttKdP79etnbPvXv/41v/71r6vtY7NmzZgxY0a18/yv0DNxIiIiIh6kECciIiLiQQpxIiIiIh6kZ+JERETkiCkqKkp5DYkcPF2JExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEghTgRERERD1KIExEREfEgy3Ec50h3QkREREQOja7EiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBynEiYiIiHiQQpyIiIiIBwUOdsYdK5eA6b3AlRX433kNy7Zdi2MZeaztdgOOP5hWFkxEaFm2FB/udausDFZkd8Ox/K7ljrO/W5Z7efK/0ssjVQ5LP7SxDauVHbbpf0oZAfdFE7N9rC+rjWNYdtCK0Sb3S3xW+gJ88Sj5pauxcF+4E43irPnEOOZOKAzHdwCfYVwsi1hGHljpfYs6QT512uAYMnzIruTk8oXGbRLzhfmioJNxm/j37qTBS/dj2fG0snhWPv/ufzu47AsACdtiV1UGpu0ZtGIcnbHedUwBcBz8Tty1dtwJ8DWNjesNDn4ctyEDwMImJ1BuLK+M+lm8ti62kz6D4zgkEobFAlnBKOe2WIPf575eCXzs8tV23Z4AsYSPL3YUGPdFMFYlg0q6+hbjt9y3d4WTwT+qTsfGfXtnOnvpXfUSflxW0HHAcW8XwInFiK9fZ9zPYxm5rOt8HY7f/VSVsC12VGbivr84BP3m7RmLw4bSkHHMMv1V9K7/IX7DvuZgEQ1kug5s3PGzMdEQ0/8n+0hQJ7zd2DfH4T/7qfsMPjtOfuQb19KE5WdnRgPjBrecBDViO8x7iuPgt6PG8t1VQaZ91IaEk75uWeEE53fYid9wzrQcmyxnj7HthONjO3UwrbeFTbavwrBqDha2se247WNzlbntoFPFMc5KfIZzMo5NIFbpWtu2/OzNqWscc4f/bE/TNrETZEXM28S2fFSG813rW5Fy8t77m/H7FyB76H3GMvG2g78SV90fdohVVbsDxUNZrgEOwO9EjWEBIG6FjGHh/5i/uKorj8YxBjiAcNA2BjiAhO2r9ksz4EsYw4Zlx40Bbl/j8erHPBA0Bjio/oQRJ1BNkIHggbaJL1jtNvFVlbsGOIBERo4xwMH+0G0eUz/mMd3P+AWBv9r1BnPQ2deuORAARBM+1wB3MML+hDHAQfXbE/Z9QX2fAAcQImoMcABRJ2QMcABhp8o9wB2MRKLa/TwRzDIGOOA/4236wq9+vRO2Ve2YhX1xY4CDfZHBtIAEfqo7vfosu9q+7VPN9qwmrBxoX7Ecc939LVRXXhkPuAY4gIygYwxw8J9jqJq27WqCa7J+tcfo9287QNwc4AAcc98dq/ox39e577c9ofp9zXeA71/576bbqSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIi4kEKcSIiIiIepBAnIiIiB1RSUoJ14BccHnaLFi3Csizmzp17wHmLi4spKio6rMsvKiqiuLj4sLZ5uCjEiYiIiHiQQpyIiIiIBynEiYiIiPwEIpEI9mH8M2kKcSIiIpLi7bffpkOHDmRkZNC8eXOmTJmSUr5+/Xosy2L69OlpdS3LoqSkJPl5/7N0n3/+OYMGDSIvL486depw55134jgOX3/9Neeffz65ubnUq1ePhx56yLVPiUSC22+/nXr16pGdnc15553H119/fcB1sW2biRMn0rp1azIyMqhbty5Dhw5l586dKfM5jsO9995Lo0aNyMrK4owzzuCTTz5xbXPdunVcfPHFFBQUkJWVxSmnnMIrr7ySMs/+Z/n+/Oc/87vf/Y6GDRuSlZXF7t27AXjvvffo3bs3eXl5ZGVl0a1bN5YsWXLA9fk281+WFhERkf85H3/8MT179qROnTqUlJQQj8e56667qFu37g9qd+DAgRx33HHcf//9vPLKK9x7770UFBQwZcoUzjzzTMaPH8+sWbO4+eab6dChA6effnpK/fvuuw/Lshg1ahSlpaVMnDiRHj16sGLFCjIzM43LHTp0KNOnT2fw4MHceOONfPnllzz22GN88MEHLFmyhGAwCMDo0aO599576du3L3379mX58uX07NmTaDSa0t6WLVs47bTTqKio4MYbb6RWrVrMmDGD8847j7lz53LBBRekzH/PPfcQCoW4+eabqaqqIhQK8cYbb9CnTx/atWvHXXfdhc/nY9q0aZx55pksXryYjh07HtSYKsSJiIhI0ujRo3Ech8WLF9O4cWMAfvGLX3D88cf/oHY7duyYvKJ37bXXUlRUxG9+8xvGjRvHqFGjAPjlL39JgwYNeOaZZ9JC3I4dO1i1ahU1atQA4OSTT2bAgAH84Q9/4MYbb3Rd5ttvv83TTz/NrFmzuPTSS5PTzzjjDHr37s2cOXO49NJL2bp1KxMmTKBfv368/PLLyV/h3nHHHYwdOzalzfvvv58tW7awePFiunTpAsCQIUM44YQT+PWvf83555+Pz/d/NzojkQjvv/9+Mmg6jsOwYcM444wzmDdvXnJZQ4cOpXXr1vzud79j/vz5BzWmup0qIiIiwL5blq+//jr9+/dPBjiA4447jl69ev2gtq+55prkf/v9ftq3b4/jOFx99dXJ6fn5+RxzzDGsW7curf4VV1yRDHAAF110EfXr1+fVV181LnPOnDnk5eVx9tlns23btuS/du3akZOTw8KFCwFYsGAB0WiUG264IeU1KiNHjkxr89VXX6Vjx47JAAeQk5PDtddey/r16/n0009T5r/yyitTrhSuWLGCNWvWcOmll7J9+/Zkn8rLyznrrLN46623Dvq5OV2JExEREQC2bt1KZWUlRx99dFrZMcccU21gOpBvh0KAvLw8MjIyqF27dtr07du3p9X/bp8sy6JFixasX7/euMw1a9ZQVlZGYWGha3lpaSkAGzZscF1GnTp1qFmzZsq0DRs20KlTp7S2jjvuuGR5mzZtktObNm2a1ifYF+5MysrK0pbrRiFOREREDonppb+JRMJYx+/3H9Q02HfL8XCwbZvCwkJmzZrlWl6nTp3DspzqfPd5vf1X2R544AHatm3rWicnJ+eg2laIExEREWBfqMnMzExeLfq21atXJ/97/1WiXbt2pcyz/4rWj+G7fXIch7Vr13LCCScY6zRv3pwFCxbQuXPnan/80KRJk+QymjVrlpy+devWtF+xNmnSJGUs9vvss89S2qquTwC5ubn06NGj2nkPRM/EiYiICLDvylivXr146aWX+Oqrr5LTV61axeuvv578nJubS+3atXnrrbdS6k+ePPlH69uzzz7Lnj17kp/nzp3LN998Q58+fYx1BgwYQCKR4J577kkri8fjyRDao0cPgsEgkyZNSrkKOHHixLR6ffv25Z///CdLly5NTisvL+epp56iqKiIVq1aVbse7dq1o3nz5jz44IPs3bs3rXzr1q3V1v82XYkTERGRpDFjxvDaa6/RtWtXhg8fTjweZ9KkSbRu3ZqPPvooOd8111zD/fffzzXXXEP79u156623+Pzzz3+0fhUUFNClSxcGDx7Mli1bmDhxIi1atGDIkCHGOt26dWPo0KGMGzeOFStW0LNnT4LBIGvWrGHOnDn8/ve/56KLLqJOnTrcfPPNjBs3jnPOOYe+ffvywQcfMG/evLRn9n7729/y3HPP0adPH2688UYKCgqYMWMGX375JX/5y19Sfpnqxufz8fTTT9OnTx9at27N4MGDadiwIRs3bmThwoXk5uby8ssvH9SYKMSJiIhI0gknnMDrr7/Or3/9a0aPHk2jRo0YM2YM33zzTUqIGz16NFu3bmXu3LnMnj2bPn36MG/ePOOPCH6o22+/nY8++ohx48axZ88ezjrrLCZPnkxWVla19Z588knatWvHlClTuP322wkEAhQVFTFo0CA6d+6cnO/ee+8lIyODJ598koULF9KpUyfmz59Pv379UtqrW7cu77zzDqNGjWLSpElEIhFOOOEEXn755bR5Tbp3787SpUu55557eOyxx9i7dy/16tWjU6dODB069KDHxHIO8unBHR+/bS7cvZPAe/8wFlfm1ufLLsNcyzLiuzl69zJj3XJfLh9ndzGWOw44jvsDlgD7Vs69fHe5w7KV5p/xFuTEObfjHmN5JB7gqz21jOWZ/gitarg/H+CPVpC/ba2xrlNZgfPFZ+byrBxo3c5Yblt+4pm5rmUVTiarHfPl3uxEGSdWLDaWVwZyWF+zg7E8uP3f1H/lEdeyqoJGfNPvN8a6cduirMr83EKGFaFF5lfGchyHgBN3X7YTZiNHmeviELDMh4OPBDnBCmN5WWWQpevcT16O41DN877UzKikT/MvjOVxApT5zftaZSzA+l3mXzIZnkEGIJcyTvMvNZbvsnN5M9rZWJ5vb6dH9G/uhY4DjvkYcyIR4l+tN5ZHatRj/alXG8tjCR+7Iu77i4VDMGDenpGoxdfbQsby/GA5PeqtNJbb+IgF3Zdd5YTYnGhgrBuwYtQO7zSWOw44uD/wDeC3o+RHtriWxa0guzLrG+v67Bg14ruM5Tg2ATtmLN5SnsmsT9zPHwU5cfq2M7ftcxJkOem3j/aLOQF2UttY7iNBtr/SUOrgw7yvRe0ApVXmtjOdco52VhnLsRME4xHXooQvSHmOObg4gGOZt6fPjpEV2WUsT1h+Ihn5rmX+PTvIff9117L9sofeV225eJeeiRMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIIU4EREREQ9SiBMRERHxIMtxHOdId0JEREREDo2uxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh4kEKciIiIiAcpxImIiIh40P8HU76eoEj0TOoAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "def embed(word):\n",
        "  return model(torch.tensor(WORD_TO_ID[word]).to(DEVICE)).detach().cpu().numpy()\n",
        "\n",
        "words = ['slytherin', 'gryffindor', 'hufflepuff', 'ravenclaw',\n",
        "         'voldemort', 'snape', 'dumbledore']\n",
        "embeddings = [embed(word) for word in words]\n",
        "num_embed = len(embeddings)\n",
        "cmap = matplotlib.cm.coolwarm\n",
        "fig, axes = plt.subplots(num_embed, 1)\n",
        "norm = matplotlib.colors.Normalize(vmin=embeddings[0].min(), vmax=embeddings[0].max())\n",
        "for idx in range(num_embed):\n",
        "  ax = axes[idx]\n",
        "  embedding = embeddings[idx]\n",
        "  for i in range(embed_dim):\n",
        "    ax.add_patch(Rectangle((i/embed_dim, 0),.05,1, color=cmap(norm(embedding[i]))))\n",
        "  ax.text(1.02, .5, words[idx], fontsize=12)\n",
        "  ax.axis('off')\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.10.0 64-bit ('3.10.0')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.10.0"
    },
    "vscode": {
      "interpreter": {
        "hash": "6aea8d84a86a1280a34780710c5e478b9c6b3c016841b907182e7484363820b1"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}