ML-Adversarial/adversarial_training_nlp.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XSk8LTPY1w54"
   },
   "source": [
    "# Adversarial training for language models\n",
    "\n",
    "<a href=\"https://colab.research.google.com/drive/1SrE5fYmWf0fjokgyTd-RSge8H_uMZ1I_\" target=\"_blank\">\n",
    "  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\">\n",
    "</a>\n",
    "\n",
    "Return to the [castle](https://github.com/Nkluge-correa/teeny-tiny_castle).\n",
    "\n",
    "Adversarial machine learning is a specialized area within machine learning dedicated to creating algorithms and techniques capable of resisting and effectively responding to adversarial attacks. This field aims to enhance the robustness of models by understanding potential vulnerabilities and developing strategies to mitigate risks posed by malicious inputs and adversarial intent.\n",
    "\n",
    "In this notebook, we will create an adversarial dataset to train and test the robustness of two different models. The technique used in this notebook is [`DeepWordBug`](https://arxiv.org/abs/1801.04354), an attack that performs simple character-level transformations to the highest-ranked tokens. As a baseline, we will use the Bi-LSTM trained in one of our other [notebooks](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Explainability/NLP/model_maker.ipynb). ALL models and datasets can be found on the Hugging Face Hub. 🤗"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "olOn0Aim8-eV",
    "outputId": "30391313-7f31-4a26-b577-32e73e0c63d3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m28.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m981.5/981.5 kB\u001b[0m \u001b[31m27.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Preparing metadata (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m445.7/445.7 kB\u001b[0m \u001b[31m19.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m61.1/61.1 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m471.6/471.6 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m776.5/776.5 kB\u001b[0m \u001b[31m47.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m769.7/769.7 kB\u001b[0m \u001b[31m51.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m143.3/143.3 kB\u001b[0m \u001b[31m13.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.1/139.1 kB\u001b[0m \u001b[31m12.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.8/44.8 kB\u001b[0m \u001b[31m3.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m202.6/202.6 kB\u001b[0m \u001b[31m17.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m44.9/44.9 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m12.6/12.6 MB\u001b[0m \u001b[31m96.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m82.6/82.6 kB\u001b[0m \u001b[31m7.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h  Building wheel for pinyin (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for word2number (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for docopt (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for langdetect (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for pptree (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for sqlitedict (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for wikipedia-api (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "  Building wheel for intervaltree (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m578.1/578.1 MB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m53.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m42.6/42.6 kB\u001b[0m \u001b[31m3.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m61.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.9/5.9 MB\u001b[0m \u001b[31m116.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m438.7/438.7 kB\u001b[0m \u001b[31m38.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m4.9/4.9 MB\u001b[0m \u001b[31m112.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m781.3/781.3 kB\u001b[0m \u001b[31m52.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n",
      "google-cloud-aiplatform 1.69.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-bigquery-connection 1.15.5 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-bigquery-storage 2.26.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-bigtable 2.26.0 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-functions 1.16.5 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-iam 2.15.2 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-language 2.13.4 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-pubsub 2.25.2 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-resource-manager 1.12.5 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "google-cloud-translate 3.15.5 requires protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "googleapis-common-protos 1.65.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0.dev0,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "grpc-google-iam-v1 0.13.1 requires protobuf!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.2, but you have protobuf 3.19.6 which is incompatible.\n",
      "pandas-gbq 0.23.2 requires google-auth-oauthlib>=0.7.0, but you have google-auth-oauthlib 0.4.6 which is incompatible.\n",
      "tensorflow-datasets 4.9.6 requires protobuf>=3.20, but you have protobuf 3.19.6 which is incompatible.\n",
      "tensorflow-metadata 1.16.0 requires protobuf<4.21,>=3.20.3; python_version < \"3.11\", but you have protobuf 3.19.6 which is incompatible.\n",
      "tf-keras 2.17.0 requires tensorflow<2.18,>=2.17, but you have tensorflow 2.10.1 which is incompatible.\u001b[0m\u001b[31m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install textattack -q\n",
    "!pip install tensorflow==2.10.1 keras==2.10.0 -q\n",
    "!pip install huggingface_hub -q"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 256,
     "referenced_widgets": [
      "99915648be134032ba3239be15ffd082",
      "714f68d4e3474f8db092fde8e303d1f2",
      "cca69fd51f8f4ef4974813c0d1751515",
      "3ffed0f2c9de45e792acfe1ca239ac79",
      "656d316d3517407eaed865011ce6cfdf",
      "fd8f12c286c7492db55194454ef65f96",
      "67e849bb46bc40daadfa833d3612c1fb",
      "7556ea1c50d645468502cbe37b46917c",
      "92be00d4ade547a3b191ee7e243c9b7b",
      "50abb8019e0648fb86c9ccbed62e29d2",
      "e6d5316cf65f4e218a357024672c88c1",
      "551e06f61bfc403ea1d0785b3d38c37b",
      "12f6000e28e14db88785688ae20d03c9",
      "b3917f6697684e8e9dfe7c6c03f8d471",
      "680d5c4c5c0a47f7b0a3a00196fe5834",
      "a3648f6d8b13440d8d4f837330fd9f4d",
      "9dbadda0deb14110970f68736f5071cd",
      "dbe9d443a7534b198c287f9e3643833b",
      "5b340ee63dc14cc6a6a5b56770155aed",
      "de70899085ba4f79831c0b48c3f8be3c",
      "6f00b32af7c846e8a5464fa9e3152fcf",
      "c9bdf61d9bdb41178cb9d8f91da8a05b"
     ]
    },
    "id": "CyEtvGK4vfhk",
    "outputId": "44bfe67e-b766-448e-9cf8-7fdf3ffae3f8"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:89: UserWarning: \n",
      "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
      "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
      "You will be able to reuse this secret in all of your notebooks.\n",
      "Please note that authentication is recommended but still optional to access public models or datasets.\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99915648be134032ba3239be15ffd082",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "BiLSTM-sentiment-classifier.h5:   0%|          | 0.00/10.1M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "551e06f61bfc403ea1d0785b3d38c37b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "(…)kenizer-BiLSTM-sentiment-classifier.json:   0%|          | 0.00/17.0M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'tokenizer-BiLSTM-sentiment-classifier.json'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "# Download the model\n",
    "hf_hub_download(repo_id=\"AiresPucrs/BiLSTM-sentiment-classifier\",\n",
    "                filename=\"BiLSTM-sentiment-classifier.h5\",\n",
    "                local_dir=\"./\",\n",
    "                repo_type=\"model\"\n",
    "                )\n",
    "\n",
    "# Download the tokenizer file\n",
    "hf_hub_download(repo_id=\"AiresPucrs/BiLSTM-sentiment-classifier\",\n",
    "                filename=\"tokenizer-BiLSTM-sentiment-classifier.json\",\n",
    "                local_dir=\"./\",\n",
    "                repo_type=\"model\"\n",
    "                )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bn5n5rVp1w56",
    "outputId": "91ff8774-157c-4d9b-da32-7a019bcd0221"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Review: \"this explanation is really bad\"\n",
      "(Negative 😊 95% | Positive 😔 5%)\n",
      "\n",
      "Review: \"i did not like this tutorial 2/10\"\n",
      "(Negative 😊 88% | Positive 😔 12%)\n",
      "\n",
      "Review: \"this tutorial is garbage i wont my money back\"\n",
      "(Negative 😊 89% | Positive 😔 11%)\n",
      "\n",
      "Review: \"is nice to see philosophers doing machine learning\"\n",
      "(Negative 😊 4% | Positive 😔 96%)\n",
      "\n",
      "Review: \"this is a great and wonderful example of nlp\"\n",
      "(Negative 😊 0% | Positive 😔 100%)\n",
      "\n",
      "Review: \"this tutorial is great one of the best tutorials ever made\"\n",
      "(Negative 😊 0% | Positive 😔 100%)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import torch\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import tensorflow as tf\n",
    "\n",
    "model = tf.keras.models.load_model('./BiLSTM-sentiment-classifier.h5')\n",
    "\n",
    "with open('./tokenizer-BiLSTM-sentiment-classifier.json') as fp:\n",
    "    data = json.load(fp)\n",
    "    tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(data)\n",
    "    fp.close()\n",
    "\n",
    "strings = [\n",
    "    'this explanation is really bad',\n",
    "    'i did not like this tutorial 2/10',\n",
    "    'this tutorial is garbage i wont my money back',\n",
    "    'is nice to see philosophers doing machine learning',\n",
    "    'this is a great and wonderful example of nlp',\n",
    "    'this tutorial is great one of the best tutorials ever made'\n",
    "]\n",
    "\n",
    "preds = model.predict(\n",
    "    tf.keras.preprocessing.sequence.pad_sequences(\n",
    "        tokenizer.texts_to_sequences(strings),\n",
    "        maxlen=250,\n",
    "        truncating='post'\n",
    "    ), verbose=0)\n",
    "\n",
    "for i, string in enumerate(strings):\n",
    "    print(f'Review: \"{string}\"\\n(Negative 😊 {preds[i][0] * 100:.2f}% | Positive 😔 {preds[i][1] * 100:.2f}%)\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zxCokbbD1w58"
   },
   "source": [
    "In this notebook, we will be exploring one of the functionalities of the [`textattack`](https://github.com/QData/TextAttack) library,\n",
    "a Python framework for adversarial attacks, data augmentation, and model training in NLP. First, let us wrap our model using the [`ModelWrapper`](https://textattack.readthedocs.io/en/latest/apidoc/textattack.models.wrappers.html#modelwrapper-class) class. Then, using the `call` method, we can create a function that gives us the prediction scores for our model output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "oW_s6RUl1w58",
    "outputId": "c97f35c6-0eec-4118-d5fb-6fef65f209ff"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.9498, 0.0502],\n",
       "        [0.8772, 0.1228],\n",
       "        [0.8881, 0.1119],\n",
       "        [0.0439, 0.9561],\n",
       "        [0.0026, 0.9974],\n",
       "        [0.0018, 0.9982]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from textattack.models.wrappers import ModelWrapper\n",
    "\n",
    "class ModelWrapper(ModelWrapper):\n",
    "    def __init__(self, model):\n",
    "        self.model = model\n",
    "\n",
    "    def __call__(self, text_input_list):\n",
    "        text_array = tokenizer.texts_to_sequences(text_input_list)\n",
    "        padded_text_array = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "                                                    text_array,\n",
    "                                                    maxlen=250,\n",
    "                                                    truncating='post'\n",
    "                                                )\n",
    "        preds = self.model.predict(padded_text_array, verbose=0)\n",
    "        logits = torch.tensor(preds)\n",
    "        logits = logits.squeeze(dim=-1)\n",
    "        return logits\n",
    "\n",
    "ModelWrapper(model)([\n",
    "    'this explanation is really bad',\n",
    "    'i did not like this tutorial 2/10',\n",
    "    'this tutorial is garbage i wont my money back',\n",
    "    'is nice to see philosophers doing machine learning',\n",
    "    'this is a great and wonderful example of nlp',\n",
    "    'this tutorial is great one of the best tutorials ever made'\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9mGP3Tkb1w59"
   },
   "source": [
    "That's exactly what we wanted. Now, we can just call an attack recipe from the attack recipes available on `textattack`. However, we need some sort of seed to create our adversarial. For this, we will use our [`sentiment analysis`](https://huggingface.co/datasets/AiresPucrs/sentiment-analysis) paired with the DeepWordBug recipe, a fast recipe for adversarial attacks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 113,
     "referenced_widgets": [
      "3a3e97ae56854cccbee68d0c5b063e7c",
      "cb35b4b8dafc40508a99a0f92d1eacfa",
      "2c7aac9423d743478cfa184e744114c4",
      "b6baba0c86f24721b71cf77e247dd2b6",
      "9c12bbc9380a430f80d93faccdc34cf4",
      "e4ed34e4db164a31959e77b28abd03cf",
      "7adbcd21eb6e42c1b9ea8ef48d2f7a96",
      "ab155fcec67d43afb117028d5b2b3fc9",
      "ee11f687caf64e32bde665244e897df8",
      "db6cf093283649409d19541b7e7d4fdc",
      "3330e529cf7645f0b07ba9a2adac6700",
      "5cd790662b764bc69e016da4123a3b11",
      "200847dc62fd40abbc0db0f9cce29d5a",
      "5d3f2952a9a940c6a93f4789af744bcb",
      "630d14a7a13d4e2f8a8ca839f9d7d50c",
      "5b5ac8675f544712b20d88c4740b2002",
      "76302a5601014b9cb1a144ff15e7a7d0",
      "82fbf497b6fa41cf8d66a6e6f169dc48",
      "dc6771e5c44646de80f0be1cee9a3bea",
      "da2e0232076d48dd9d0b7dc372a161c5",
      "af44d6cbf36f4a04aa900da10478269f",
      "61b30e389e6a4c888204766c3e3777fa",
      "2c01baa14f4d463cbcdbaca200baf5e2",
      "99d10fb979054b429c973e36a9273130",
      "422dcfb7402e47bcb705cbe9692fbe04",
      "a778b3411c3f4024bc5cec858eef5d4a",
      "d974ad0c78b548fcbc429b401cd99a8c",
      "6a0cb0870a754291b6194481e9298a43",
      "dfb54cec97214af5bf26b4fa65d6ffe8",
      "2b88aa2f2098414c94cf062a20268abf",
      "0418209539824da2a56822c16cd43b7f",
      "2de89f08500947f09d40b127276169cc",
      "922b234748c2425190b3cdd0d5fc1664"
     ]
    },
    "id": "CeenTX4Y1w59",
    "outputId": "834e8f2f-95ee-4683-9a63-992b9fe8ff46"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a3e97ae56854cccbee68d0c5b063e7c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "README.md:   0%|          | 0.00/1.30k [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5cd790662b764bc69e016da4123a3b11",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "train-00000-of-00001.parquet:   0%|          | 0.00/44.5M [00:00<?, ?B/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2c01baa14f4d463cbcdbaca200baf5e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Generating train split:   0%|          | 0/85089 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "!pip install datasets -q\n",
    "\n",
    "import textattack\n",
    "import pandas as pd\n",
    "from datasets import load_dataset\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "dataset = load_dataset('AiresPucrs/sentiment-analysis', split='train')\n",
    "df = dataset.to_pandas()\n",
    "\n",
    "df_positive = df[df['label'] == 1]\n",
    "df_negative = df[df['label'] == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TqiKYlvc1w59"
   },
   "source": [
    "Below, we are creating two datasets, separated by the original class label, containing several successful adversarial examples against our model. This step takes a while, so if you want to skip this part, you can directly download our adversarial dataset on the following cells. ALL models and datasets can be found on the Hugging Face Hub. 🤗"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 423
    },
    "id": "ElNuuH3SnWW4",
    "outputId": "74ddbdbb-2e56-44aa-b5ed-e73bd99b9503"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "summary": "{\n  \"name\": \"display(adversarial_dataset\",\n  \"rows\": 13441,\n  \"fields\": [\n    {\n      \"column\": \"review\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 13425,\n        \"samples\": [\n          \"nWce voice understand hindi english useful even control compatible devices voiceloving\",\n          \"uBited its ridic you cant transfer credit you have my 400 amp youre taking 200 of it for cancelled flightling atleast let me keep the credit damn\",\n          \"americannair eah rebooked for tomorrow morning but extremely dPsappointed to mfiss a eedding\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"sentiment\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0,\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
       "type": "dataframe"
      },
      "text/html": [
       "\n",
       "  <div id=\"df-dc9ba6bd-9cce-44e1-977e-06a6c8825c60\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>it would be Mreat if there will be custom vide...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>the app is fine but the actual delivery is aYf...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>loev being abOle to stay in otuch with firends...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>uited line is just getting longer and moving s...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>usairways knows customer service htank you for...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13436</th>\n",
       "      <td>unHted sadly this wasnt just due to mother nat...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13437</th>\n",
       "      <td>Sunited i was on ua3782 and it was cZncelled f...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13438</th>\n",
       "      <td>southwestair stewardess really funny now i cou...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13439</th>\n",
       "      <td>sotuhwestair looks like you are up and running...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13440</th>\n",
       "      <td>americanair dfwairport me too lVove living so ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13441 rows × 2 columns</p>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-dc9ba6bd-9cce-44e1-977e-06a6c8825c60')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-dc9ba6bd-9cce-44e1-977e-06a6c8825c60 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-dc9ba6bd-9cce-44e1-977e-06a6c8825c60');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "\n",
       "<div id=\"df-90cb549a-4b12-46e9-ae88-67dc3ffb739d\">\n",
       "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-90cb549a-4b12-46e9-ae88-67dc3ffb739d')\"\n",
       "            title=\"Suggest charts\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
       "     width=\"24px\">\n",
       "    <g>\n",
       "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
       "    </g>\n",
       "</svg>\n",
       "  </button>\n",
       "\n",
       "<style>\n",
       "  .colab-df-quickchart {\n",
       "      --bg-color: #E8F0FE;\n",
       "      --fill-color: #1967D2;\n",
       "      --hover-bg-color: #E2EBFA;\n",
       "      --hover-fill-color: #174EA6;\n",
       "      --disabled-fill-color: #AAA;\n",
       "      --disabled-bg-color: #DDD;\n",
       "  }\n",
       "\n",
       "  [theme=dark] .colab-df-quickchart {\n",
       "      --bg-color: #3B4455;\n",
       "      --fill-color: #D2E3FC;\n",
       "      --hover-bg-color: #434B5C;\n",
       "      --hover-fill-color: #FFFFFF;\n",
       "      --disabled-bg-color: #3B4455;\n",
       "      --disabled-fill-color: #666;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart {\n",
       "    background-color: var(--bg-color);\n",
       "    border: none;\n",
       "    border-radius: 50%;\n",
       "    cursor: pointer;\n",
       "    display: none;\n",
       "    fill: var(--fill-color);\n",
       "    height: 32px;\n",
       "    padding: 0;\n",
       "    width: 32px;\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart:hover {\n",
       "    background-color: var(--hover-bg-color);\n",
       "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "    fill: var(--button-hover-fill-color);\n",
       "  }\n",
       "\n",
       "  .colab-df-quickchart-complete:disabled,\n",
       "  .colab-df-quickchart-complete:disabled:hover {\n",
       "    background-color: var(--disabled-bg-color);\n",
       "    fill: var(--disabled-fill-color);\n",
       "    box-shadow: none;\n",
       "  }\n",
       "\n",
       "  .colab-df-spinner {\n",
       "    border: 2px solid var(--fill-color);\n",
       "    border-color: transparent;\n",
       "    border-bottom-color: var(--fill-color);\n",
       "    animation:\n",
       "      spin 1s steps(1) infinite;\n",
       "  }\n",
       "\n",
       "  @keyframes spin {\n",
       "    0% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "      border-left-color: var(--fill-color);\n",
       "    }\n",
       "    20% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    30% {\n",
       "      border-color: transparent;\n",
       "      border-left-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    40% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-top-color: var(--fill-color);\n",
       "    }\n",
       "    60% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "    }\n",
       "    80% {\n",
       "      border-color: transparent;\n",
       "      border-right-color: var(--fill-color);\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "    90% {\n",
       "      border-color: transparent;\n",
       "      border-bottom-color: var(--fill-color);\n",
       "    }\n",
       "  }\n",
       "</style>\n",
       "\n",
       "  <script>\n",
       "    async function quickchart(key) {\n",
       "      const quickchartButtonEl =\n",
       "        document.querySelector('#' + key + ' button');\n",
       "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
       "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
       "      try {\n",
       "        const charts = await google.colab.kernel.invokeFunction(\n",
       "            'suggestCharts', [key], {});\n",
       "      } catch (error) {\n",
       "        console.error('Error during call to suggestCharts:', error);\n",
       "      }\n",
       "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
       "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
       "    }\n",
       "    (() => {\n",
       "      let quickchartButtonEl =\n",
       "        document.querySelector('#df-90cb549a-4b12-46e9-ae88-67dc3ffb739d button');\n",
       "      quickchartButtonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "    })();\n",
       "  </script>\n",
       "</div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "text/plain": [
       "                                                  review  sentiment\n",
       "0      it would be Mreat if there will be custom vide...          1\n",
       "1      the app is fine but the actual delivery is aYf...          0\n",
       "2      loev being abOle to stay in otuch with firends...          1\n",
       "3      uited line is just getting longer and moving s...          0\n",
       "4      usairways knows customer service htank you for...          1\n",
       "...                                                  ...        ...\n",
       "13436  unHted sadly this wasnt just due to mother nat...          0\n",
       "13437  Sunited i was on ua3782 and it was cZncelled f...          0\n",
       "13438  southwestair stewardess really funny now i cou...          1\n",
       "13439  sotuhwestair looks like you are up and running...          0\n",
       "13440  americanair dfwairport me too lVove living so ...          1\n",
       "\n",
       "[13441 rows x 2 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from textattack.attack_recipes import DeepWordBugGao2018\n",
    "from textattack import Attacker\n",
    "import pandas as pd\n",
    "\n",
    "# Wraped model\n",
    "wraped_model = ModelWrapper(model)\n",
    "\n",
    "# Create tuples for the `textattack.datasets.Dataset`\n",
    "x = list(df_positive.text)\n",
    "y = np.array(list(df_positive.label)).astype(int)\n",
    "data=[(x[i], int(y[i])) for i in range(len(df_positive)) if len(x[i]) < 256]\n",
    "dataset = textattack.datasets.Dataset(data)\n",
    "\n",
    "# Perfrom the `DeepWordBugGao2018` on every sample\n",
    "attack = DeepWordBugGao2018.build(wraped_model)\n",
    "attack_args = textattack.AttackArgs(\n",
    "    num_examples=len(data),\n",
    "    log_to_csv =\"adversarial_text_positive.csv\",\n",
    "    silent = True,\n",
    "    disable_stdout=True\n",
    "    )\n",
    "attacker = Attacker(attack, dataset, attack_args)\n",
    "attacker.attack_dataset()\n",
    "\n",
    "# The same for the negative sentiment portion of the dataset ...\n",
    "x = list(df_negative.review)\n",
    "y = np.array(list(df_negative.sentiment)).astype(int)\n",
    "\n",
    "data=[(x[i], int(y[i])) for i in range(len(df_negative)) if len(x[i]) < 256]\n",
    "\n",
    "dataset = textattack.datasets.Dataset(data)\n",
    "attack = DeepWordBugGao2018.build(wraped_model)\n",
    "attack_args = textattack.AttackArgs(\n",
    "    num_examples=len(data),\n",
    "    log_to_csv =\"adversarial_text_negative.csv\",\n",
    "    silent = True,\n",
    "    disable_stdout=True\n",
    "    )\n",
    "attacker = Attacker(attack, dataset, attack_args)\n",
    "attacker.attack_dataset()\n",
    "\n",
    "# Concatenate both datasets into a single adversarial training dataset\n",
    "df_positive = pd.read_csv('adversarial_text_positive.csv')\n",
    "df_negative = pd.read_csv('adversarial_text_negative.csv')\n",
    "\n",
    "df_positive = df_positive[ df_positive['result_type'] == 'Successful']\n",
    "df_negative = df_negative[ df_negative['result_type'] == 'Successful']\n",
    "\n",
    "negative_samples = [s.replace('[[', '').replace(']]', '') for s in df_negative.perturbed_text]\n",
    "positive_samples = [s.replace('[[', '').replace(']]', '') for s in df_positive.perturbed_text]\n",
    "\n",
    "df_positive_adversarial_samples = pd.DataFrame(positive_samples, columns=['text'])\n",
    "df_negative_adversarial_samples = pd.DataFrame(negative_samples, columns=['text'])\n",
    "\n",
    "df_positive_adversarial_samples['label'] = 1\n",
    "df_negative_adversarial_samples['label'] = 0\n",
    "\n",
    "adversarial_data = pd.concat(\n",
    "    [\n",
    "        df_positive_adversarial_samples,\n",
    "        df_negative_adversarial_samples\n",
    "    ],\n",
    "    ignore_index=True).sample(frac=1).reset_index(drop=True)\n",
    "\n",
    "display(adversarial_data)\n",
    "\n",
    "# Or simply download it from the Hub!\n",
    "# from datasets import load_dataset\n",
    "# adversarial_dataset = load_dataset('AiresPucrs/adversarial-sentiment-analysis', split=\"train\")\n",
    "#display(adversarial_dataset.to_pandas())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OuR5lZcJ1w5_"
   },
   "source": [
    "Now, we will train a new model using a mix of the original dataset and our new adversarial dataset, hoping it will be more resilient to the attack we introduce while creating our adversarial dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "fh4Cfjhdqm7-"
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "from sklearn.model_selection import train_test_split\n",
    "import pandas as pd\n",
    "\n",
    "dataset = load_dataset('AiresPucrs/sentiment-analysis', split='train')\n",
    "df = dataset.to_pandas()\n",
    "adversarial_dataset = load_dataset('AiresPucrs/adversarial-sentiment-analysis', split=\"train\")\n",
    "adversarial_df = adversarial_dataset.to_pandas()\n",
    "\n",
    "adversarial_training = adversarial_df.head(10000)\n",
    "adversarial_test = adversarial_df.tail(3441)\n",
    "\n",
    "# Concatenate both the original `sentiment-analysis` dataset ad the `adversarial_training` portion\n",
    "data = pd.concat([adversarial_training, df]).sample(frac=1).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4aRnt-UE1w6A"
   },
   "source": [
    "We will now train the same model (i.e., [same architecture and hyperparemeters](https://github.com/Nkluge-correa/TeenyTinyCastle/blob/master/ML-Explainability/NLP/model_maker.ipynb)) with this adversarial dataset. To avoid training this model, skip this cell and go straight to the evaluation comparison. ALL models and datasets can be found on the Hugging Face Hub. 🤗"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "z0imlU_h1w6A",
    "outputId": "94ef3ff7-3664-47c9-d562-e498112cac0c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version:  2.10.1\n",
      "Eager mode:  True\n",
      "GPU is NOT AVAILABLE\n",
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, None)]            0         \n",
      "                                                                 \n",
      " embedding (Embedding)       (None, None, 128)         640000    \n",
      "                                                                 \n",
      " bidirectional (Bidirectiona  (None, None, 128)        98816     \n",
      " l)                                                              \n",
      "                                                                 \n",
      " bidirectional_1 (Bidirectio  (None, 128)              98816     \n",
      " nal)                                                            \n",
      "                                                                 \n",
      " dense (Dense)               (None, 1)                 129       \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 837,761\n",
      "Trainable params: 837,761\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/20\n",
      "1902/1902 [==============================] - 738s 385ms/step - loss: 0.4667 - accuracy: 0.7825 - val_loss: 0.3873 - val_accuracy: 0.8317\n",
      "Epoch 2/20\n",
      "1902/1902 [==============================] - 735s 386ms/step - loss: 0.3626 - accuracy: 0.8468 - val_loss: 0.3434 - val_accuracy: 0.8494\n",
      "Epoch 3/20\n",
      "1902/1902 [==============================] - 755s 397ms/step - loss: 0.2975 - accuracy: 0.8750 - val_loss: 0.3449 - val_accuracy: 0.8468\n",
      "Epoch 4/20\n",
      "1902/1902 [==============================] - 759s 399ms/step - loss: 0.2486 - accuracy: 0.9001 - val_loss: 0.2993 - val_accuracy: 0.8780\n",
      "Epoch 5/20\n",
      "1902/1902 [==============================] - 751s 395ms/step - loss: 0.2104 - accuracy: 0.9170 - val_loss: 0.3033 - val_accuracy: 0.8752\n",
      "Epoch 6/20\n",
      "1902/1902 [==============================] - 753s 396ms/step - loss: 0.1749 - accuracy: 0.9330 - val_loss: 0.3263 - val_accuracy: 0.8722\n",
      "Epoch 7/20\n",
      "1902/1902 [==============================] - ETA: 0s - loss: 0.1454 - accuracy: 0.9458Restoring model weights from the end of the best epoch: 4.\n",
      "1902/1902 [==============================] - 756s 398ms/step - loss: 0.1454 - accuracy: 0.9458 - val_loss: 0.3562 - val_accuracy: 0.8655\n",
      "Epoch 7: early stopping\n",
      "595/595 [==============================] - 57s 96ms/step - loss: 0.3028 - accuracy: 0.8741\n",
      "Final Loss: 0.3.\n",
      "Final Performance: 87.41 %.\n"
     ]
    }
   ],
   "source": [
    "import io\n",
    "import json\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Define language model dimensions (same as the original model)\n",
    "vocab_size = 5000\n",
    "embed_size = 128\n",
    "sequence_length = 250\n",
    "\n",
    "# Train and save the new tokenizer\n",
    "tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=vocab_size,\n",
    "                      filters='!\"#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\\t\\n',\n",
    "                      lower=True,\n",
    "                      split=\" \",\n",
    "                      oov_token=\"<OOV>\")\n",
    "\n",
    "tokenizer.fit_on_texts(data.text)\n",
    "tokenizer_json = tokenizer.to_json()\n",
    "\n",
    "with io.open('tokenizer-BiLSTM-sentiment-classifier-adversarial.json', 'w', encoding='utf-8') as fp:\n",
    "    fp.write(json.dumps(tokenizer_json, ensure_ascii=False))\n",
    "    fp.close()\n",
    "\n",
    "# Split the data\n",
    "x_train, x_test, y_train, y_test = train_test_split(\n",
    "    data.text, data.label, test_size=0.2, random_state=42)\n",
    "\n",
    "# Tokenize and pad sequences\n",
    "x_train = tf.keras.utils.pad_sequences(\n",
    "    tokenizer.texts_to_sequences(x_train),\n",
    "    maxlen=sequence_length,\n",
    "    truncating='post')\n",
    "x_test = tf.keras.utils.pad_sequences(\n",
    "    tokenizer.texts_to_sequences(x_test),\n",
    "    maxlen=sequence_length,\n",
    "    truncating='post')\n",
    "y_train = np.array(y_train).astype(float)\n",
    "y_test = np.array(y_test).astype(float)\n",
    "\n",
    "# Define the BI-LSTM Network\n",
    "inputs = tf.keras.Input(shape=(None,), dtype=\"int32\")\n",
    "x = tf.keras.layers.Embedding(input_dim=vocab_size,\n",
    "                              output_dim=embed_size,\n",
    "                              input_length=sequence_length)(inputs)\n",
    "\n",
    "x = tf.keras.layers.Bidirectional(\n",
    "    tf.keras.layers.LSTM(64, return_sequences=True))(x)\n",
    "x = tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64))(x)\n",
    "\n",
    "outputs = tf.keras.layers.Dense(1, activation=\"sigmoid\")(x)\n",
    "model = tf.keras.Model(inputs, outputs)\n",
    "\n",
    "# Compile ...\n",
    "model.compile(loss=tf.losses.BinaryCrossentropy(),\n",
    "              optimizer='adam',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print(\"Version: \", tf.__version__)\n",
    "print(\"Eager mode: \", tf.executing_eagerly())\n",
    "print(\"GPU is\", \"available\" if tf.config.list_physical_devices('GPU') else \"NOT AVAILABLE\")\n",
    "model.summary()\n",
    "\n",
    "callbacks = [tf.keras.callbacks.ModelCheckpoint(\"BiLSTM-sentiment-classifier-adversarial.h5\",\n",
    "                                                save_best_only=True),\n",
    "            tf.keras.callbacks.EarlyStopping(monitor=\"val_loss\",\n",
    "                                            patience=3,\n",
    "                                            verbose=1,\n",
    "                                            mode=\"auto\",\n",
    "                                            baseline=None,\n",
    "                                            restore_best_weights=True)]\n",
    "# Train!\n",
    "model.fit(x_train,\n",
    "          y_train,\n",
    "          epochs=20,\n",
    "          validation_split=0.2,\n",
    "          callbacks=callbacks,\n",
    "          verbose=1)\n",
    "\n",
    "# Evaluate!\n",
    "test_loss_score, test_acc_score = model.evaluate(x_test, y_test)\n",
    "\n",
    "print(f'Final Loss: {test_loss_score:.2f}.')\n",
    "print(f'Final Performance: {test_acc_score * 100:.2f} %.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SDpmfkm_1w6A"
   },
   "source": [
    "Now, let us compare the robustness score of our models as the accuracy against the test portion of our adversarial dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-cXi9BFa6CMe",
    "outputId": "31060d78-c424-4ab1-cc9c-a736e3eff19f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Original model evaluation against adversaries\n",
      "\n",
      "108/108 [==============================] - 12s 99ms/step - loss: 1.1278 - accuracy: 0.3179\n",
      "Robustness Score: 31.79 %\n",
      "\n",
      "Adversarial-Training model evaluation against adversaries\n",
      "\n",
      "108/108 [==============================] - 12s 97ms/step - loss: 0.3114 - accuracy: 0.8637\n",
      "Robustness Score: 86.37 %\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from huggingface_hub import hf_hub_download\n",
    "\n",
    "def download_model_and_tokenizer(repo_id, model_file, tokenizer_file, local_dir=\"./\"):\n",
    "    # Download model and tokenizer\n",
    "    hf_hub_download(repo_id=repo_id, filename=model_file, local_dir=local_dir, repo_type=\"model\")\n",
    "    hf_hub_download(repo_id=repo_id, filename=tokenizer_file, local_dir=local_dir, repo_type=\"model\")\n",
    "\n",
    "    # Load the model\n",
    "    model = tf.keras.models.load_model(f'{local_dir}/{model_file}')\n",
    "\n",
    "    # Load the tokenizer\n",
    "    with open(f'{local_dir}/{tokenizer_file}') as fp:\n",
    "        tokenizer_data = json.load(fp)\n",
    "        tokenizer = tf.keras.preprocessing.text.tokenizer_from_json(tokenizer_data)\n",
    "\n",
    "    return model, tokenizer\n",
    "\n",
    "def evaluate_model(model, tokenizer, test_data, max_len=250):\n",
    "    # Prepare input data\n",
    "    x_test = tf.keras.utils.pad_sequences(\n",
    "        tokenizer.texts_to_sequences(test_data.text), maxlen=max_len, truncating='post')\n",
    "    y_test = np.array(test_data.label).astype(float)\n",
    "\n",
    "    # Evaluate the model\n",
    "    _, robustness_score = model.evaluate(x_test, y_test)\n",
    "    return robustness_score\n",
    "\n",
    "# Download and load the original and adversarial models with their tokenizers\n",
    "original_model, original_tokenizer = download_model_and_tokenizer(\n",
    "    repo_id=\"AiresPucrs/BiLSTM-sentiment-classifier\",\n",
    "    model_file=\"BiLSTM-sentiment-classifier.h5\",\n",
    "    tokenizer_file=\"tokenizer-BiLSTM-sentiment-classifier.json\"\n",
    ")\n",
    "\n",
    "adversarial_model, adversarial_tokenizer = download_model_and_tokenizer(\n",
    "    repo_id=\"AiresPucrs/BiLSTM-sentiment-classifier-adversarial\",\n",
    "    model_file=\"BiLSTM-sentiment-classifier-adversarial.h5\",\n",
    "    tokenizer_file=\"tokenizer-BiLSTM-sentiment-classifier-adversarial.json\"\n",
    ")\n",
    "\n",
    "# Evaluate both models against adversarial test data\n",
    "print('\\nOriginal model evaluation against adversaries\\n')\n",
    "original_robustness = evaluate_model(original_model, original_tokenizer, adversarial_test)\n",
    "print(f'Robustness Score: {original_robustness * 100:.2f} %')\n",
    "\n",
    "print('\\nAdversarial-Training model evaluation against adversaries\\n')\n",
    "adversarial_robustness = evaluate_model(adversarial_model, adversarial_tokenizer, adversarial_test)\n",
    "print(f'Robustness Score: {adversarial_robustness * 100:.2f} %')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-kjsoy1a1w6B"
   },
   "source": [
    "The model trained with a mix of standard and perturbed data achieves a robustness score of around 85%. Through adversarial training, models can learn to adapt and become more resilient to subtle input perturbations by intentionally being exposed to carefully crafted adversarial examples during training. This deliberate exposure helps the model better generalize and defend against adversarial attacks.\n",
    "\n",
    "---\n",
    "\n",
    "Return to the [castle](https://github.com/Nkluge-correa/teeny-tiny_castle)."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "gpuType": "A100",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  },
  "vscode": {
   "interpreter": {
    "hash": "aca09746cf57686f00a55ae76e987247ecfb5dd0b3b2e2474d8dbbf0c5e3377e"
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0418209539824da2a56822c16cd43b7f": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "12f6000e28e14db88785688ae20d03c9": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9dbadda0deb14110970f68736f5071cd",
      "placeholder": "​",
      "style": "IPY_MODEL_dbe9d443a7534b198c287f9e3643833b",
      "value": "(…)kenizer-BiLSTM-sentiment-classifier.json: 100%"
     }
    },
    "200847dc62fd40abbc0db0f9cce29d5a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_76302a5601014b9cb1a144ff15e7a7d0",
      "placeholder": "​",
      "style": "IPY_MODEL_82fbf497b6fa41cf8d66a6e6f169dc48",
      "value": "train-00000-of-00001.parquet: 100%"
     }
    },
    "2b88aa2f2098414c94cf062a20268abf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2c01baa14f4d463cbcdbaca200baf5e2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_99d10fb979054b429c973e36a9273130",
       "IPY_MODEL_422dcfb7402e47bcb705cbe9692fbe04",
       "IPY_MODEL_a778b3411c3f4024bc5cec858eef5d4a"
      ],
      "layout": "IPY_MODEL_d974ad0c78b548fcbc429b401cd99a8c"
     }
    },
    "2c7aac9423d743478cfa184e744114c4": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_ab155fcec67d43afb117028d5b2b3fc9",
      "max": 1300,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_ee11f687caf64e32bde665244e897df8",
      "value": 1300
     }
    },
    "2de89f08500947f09d40b127276169cc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3330e529cf7645f0b07ba9a2adac6700": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3a3e97ae56854cccbee68d0c5b063e7c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_cb35b4b8dafc40508a99a0f92d1eacfa",
       "IPY_MODEL_2c7aac9423d743478cfa184e744114c4",
       "IPY_MODEL_b6baba0c86f24721b71cf77e247dd2b6"
      ],
      "layout": "IPY_MODEL_9c12bbc9380a430f80d93faccdc34cf4"
     }
    },
    "3ffed0f2c9de45e792acfe1ca239ac79": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_50abb8019e0648fb86c9ccbed62e29d2",
      "placeholder": "​",
      "style": "IPY_MODEL_e6d5316cf65f4e218a357024672c88c1",
      "value": " 10.1M/10.1M [00:00&lt;00:00, 95.0MB/s]"
     }
    },
    "422dcfb7402e47bcb705cbe9692fbe04": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2b88aa2f2098414c94cf062a20268abf",
      "max": 85089,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_0418209539824da2a56822c16cd43b7f",
      "value": 85089
     }
    },
    "50abb8019e0648fb86c9ccbed62e29d2": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "551e06f61bfc403ea1d0785b3d38c37b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_12f6000e28e14db88785688ae20d03c9",
       "IPY_MODEL_b3917f6697684e8e9dfe7c6c03f8d471",
       "IPY_MODEL_680d5c4c5c0a47f7b0a3a00196fe5834"
      ],
      "layout": "IPY_MODEL_a3648f6d8b13440d8d4f837330fd9f4d"
     }
    },
    "5b340ee63dc14cc6a6a5b56770155aed": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5b5ac8675f544712b20d88c4740b2002": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5cd790662b764bc69e016da4123a3b11": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_200847dc62fd40abbc0db0f9cce29d5a",
       "IPY_MODEL_5d3f2952a9a940c6a93f4789af744bcb",
       "IPY_MODEL_630d14a7a13d4e2f8a8ca839f9d7d50c"
      ],
      "layout": "IPY_MODEL_5b5ac8675f544712b20d88c4740b2002"
     }
    },
    "5d3f2952a9a940c6a93f4789af744bcb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dc6771e5c44646de80f0be1cee9a3bea",
      "max": 44486982,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_da2e0232076d48dd9d0b7dc372a161c5",
      "value": 44486982
     }
    },
    "61b30e389e6a4c888204766c3e3777fa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "630d14a7a13d4e2f8a8ca839f9d7d50c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_af44d6cbf36f4a04aa900da10478269f",
      "placeholder": "​",
      "style": "IPY_MODEL_61b30e389e6a4c888204766c3e3777fa",
      "value": " 44.5M/44.5M [00:01&lt;00:00, 39.6MB/s]"
     }
    },
    "656d316d3517407eaed865011ce6cfdf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "67e849bb46bc40daadfa833d3612c1fb": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "680d5c4c5c0a47f7b0a3a00196fe5834": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6f00b32af7c846e8a5464fa9e3152fcf",
      "placeholder": "​",
      "style": "IPY_MODEL_c9bdf61d9bdb41178cb9d8f91da8a05b",
      "value": " 17.0M/17.0M [00:00&lt;00:00, 223MB/s]"
     }
    },
    "6a0cb0870a754291b6194481e9298a43": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6f00b32af7c846e8a5464fa9e3152fcf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "714f68d4e3474f8db092fde8e303d1f2": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_fd8f12c286c7492db55194454ef65f96",
      "placeholder": "​",
      "style": "IPY_MODEL_67e849bb46bc40daadfa833d3612c1fb",
      "value": "BiLSTM-sentiment-classifier.h5: 100%"
     }
    },
    "7556ea1c50d645468502cbe37b46917c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "76302a5601014b9cb1a144ff15e7a7d0": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "7adbcd21eb6e42c1b9ea8ef48d2f7a96": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "82fbf497b6fa41cf8d66a6e6f169dc48": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "922b234748c2425190b3cdd0d5fc1664": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "92be00d4ade547a3b191ee7e243c9b7b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "99915648be134032ba3239be15ffd082": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_714f68d4e3474f8db092fde8e303d1f2",
       "IPY_MODEL_cca69fd51f8f4ef4974813c0d1751515",
       "IPY_MODEL_3ffed0f2c9de45e792acfe1ca239ac79"
      ],
      "layout": "IPY_MODEL_656d316d3517407eaed865011ce6cfdf"
     }
    },
    "99d10fb979054b429c973e36a9273130": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6a0cb0870a754291b6194481e9298a43",
      "placeholder": "​",
      "style": "IPY_MODEL_dfb54cec97214af5bf26b4fa65d6ffe8",
      "value": "Generating train split: 100%"
     }
    },
    "9c12bbc9380a430f80d93faccdc34cf4": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9dbadda0deb14110970f68736f5071cd": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a3648f6d8b13440d8d4f837330fd9f4d": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a778b3411c3f4024bc5cec858eef5d4a": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_2de89f08500947f09d40b127276169cc",
      "placeholder": "​",
      "style": "IPY_MODEL_922b234748c2425190b3cdd0d5fc1664",
      "value": " 85089/85089 [00:00&lt;00:00, 149332.75 examples/s]"
     }
    },
    "ab155fcec67d43afb117028d5b2b3fc9": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "af44d6cbf36f4a04aa900da10478269f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "b3917f6697684e8e9dfe7c6c03f8d471": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5b340ee63dc14cc6a6a5b56770155aed",
      "max": 16996405,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_de70899085ba4f79831c0b48c3f8be3c",
      "value": 16996405
     }
    },
    "b6baba0c86f24721b71cf77e247dd2b6": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_db6cf093283649409d19541b7e7d4fdc",
      "placeholder": "​",
      "style": "IPY_MODEL_3330e529cf7645f0b07ba9a2adac6700",
      "value": " 1.30k/1.30k [00:00&lt;00:00, 78.1kB/s]"
     }
    },
    "c9bdf61d9bdb41178cb9d8f91da8a05b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "cb35b4b8dafc40508a99a0f92d1eacfa": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_e4ed34e4db164a31959e77b28abd03cf",
      "placeholder": "​",
      "style": "IPY_MODEL_7adbcd21eb6e42c1b9ea8ef48d2f7a96",
      "value": "README.md: 100%"
     }
    },
    "cca69fd51f8f4ef4974813c0d1751515": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7556ea1c50d645468502cbe37b46917c",
      "max": 10129544,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_92be00d4ade547a3b191ee7e243c9b7b",
      "value": 10129544
     }
    },
    "d974ad0c78b548fcbc429b401cd99a8c": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "da2e0232076d48dd9d0b7dc372a161c5": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "db6cf093283649409d19541b7e7d4fdc": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dbe9d443a7534b198c287f9e3643833b": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "dc6771e5c44646de80f0be1cee9a3bea": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "de70899085ba4f79831c0b48c3f8be3c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "dfb54cec97214af5bf26b4fa65d6ffe8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "e4ed34e4db164a31959e77b28abd03cf": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e6d5316cf65f4e218a357024672c88c1": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "ee11f687caf64e32bde665244e897df8": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "fd8f12c286c7492db55194454ef65f96": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}