HW05_dataset.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "colab": {
      "name": "HW05_dataset.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "uQRoo1jRP7B_",
        "9d7SrTIPP7B_",
        "VAmqcOvhP7CE",
        "N6VpA6kUP7CE",
        "D2RXhns3P7CE",
        "ChE3aHCYP7CF",
        "CrenFTn5P7CF",
        "Svvi0DJDP7CG",
        "nB99A7uaP7CG",
        "Jl16hCW-P7CH",
        "sK7YDlv-P7CH",
        "z4g7sZJeP7CI",
        "Ev0PmgCpP7CI",
        "-eJBl2qnP7CI",
        "zshbrr8tP7CJ",
        "9BMvu2X2P7CJ",
        "hRI-vU55P7CJ",
        "dvP5h5PUP7CJ",
        "dqUy6lnYP7CK",
        "hEjKIJ89P7CK",
        "2RIVIFGdP7CL",
        "ggnP4imDP7CM",
        "UmNKgmPdP7CM",
        "g5GDRX3uP7CM",
        "LtjcTpZNP7CN",
        "3O1DY1XqP7CN",
        "thHTQPU0P7CQ",
        "EYFQnjVWP7CQ",
        "P1URlNrIP7CR",
        "CJeaq7zyP7CR",
        "gJe1z6gpP7CR",
        "kTcfJYc1P7CR",
        "_jr2bQ4kP7CS",
        "bIHgJe7fP7CT"
      ],
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/dflkasjfh/RGB2LIDAR/blob/master/HW05_dataset.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0397aNiNP7B4"
      },
      "source": [
        "# **Homework 5 - Sequence-to-sequence**\n",
        "\n",
        "If you have any questions, feel free to email us at: ntu-ml-2021spring-ta@googlegroups.com"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q5H28f8_P7B6"
      },
      "source": [
        "### (4/21 Updates)\n",
        "1. Link to reference [training curves](https://wandb.ai/george0828zhang/hw5.seq2seq.new).\n",
        "\n",
        "### (4/14 Updates) \n",
        "1. Link to tutorial video [part 1](https://youtu.be/1pjS5_L5REI) [part 2](https://youtu.be/3XX9d0ymKgQ).\n",
        "2. Now defaults to load `\"avg_last_5_checkpoint.pt\"` to generate prediction.\n",
        "3. Expected run time on Colab with Tesla T4\n",
        "\n",
        "|Baseline|Details|Total Time|\n",
        "|-|:-:|:-:|\n",
        "|Simple|2m 15s $\\times$30 epochs|1hr 8m|\n",
        "|Medium|4m $\\times$30 epochs|2hr|\n",
        "|Strong|8m $\\times$30 epochs (backward)<br>+1hr (back-translation)<br>+15m $\\times$30 epochs (forward)|12hr 30m|"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TDEdF0EzP7B7"
      },
      "source": [
        "# Sequence-to-Sequence Introduction\n",
        "- Typical sequence-to-sequence (seq2seq) models are encoder-decoder models, which usually consists of two parts, the encoder and decoder, respectively. These two parts can be implemented with recurrent neural network (RNN) or transformer, primarily to deal with input/output sequences of dynamic length.\n",
        "- **Encoder** encodes a sequence of inputs, such as text, video or audio, into a single vector, which can be viewed as the abstractive representation of the inputs, containing information of the whole sequence.\n",
        "- **Decoder** decodes the vector output of encoder one step at a time, until the final output sequence is complete. Every decoding step is affected by previous step(s). Generally, one would add \"< BOS >\" at the begining of the sequence to indicate start of decoding, and \"< EOS >\" at the end to indicate end of decoding.\n",
        "\n",
        "![seq2seq](https://i.imgur.com/0zeDyuI.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8FtB7CRtP7B7"
      },
      "source": [
        "# Homework Description\n",
        "- English to Chinese (Traditional) Translation\n",
        "  - Input: an English sentence         (e.g.\t\ttom is a student .)\n",
        "  - Output: the Chinese translation  (e.g. \t\t湯姆 是 個 學生 。)\n",
        "\n",
        "- TODO\n",
        "    - Train a simple RNN seq2seq to acheive translation\n",
        "    - Switch to transformer model to boost performance\n",
        "    - Apply Back-translation to furthur boost performance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SdGt09goP7B8"
      },
      "source": [
        "# Download and import required packages"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Rp_dvPJP7B8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "4e4e6a2a-f97b-4b36-b8ac-4c111b32516e"
      },
      "source": [
        "!pip install 'torch>=1.6.0' editdistance matplotlib sacrebleu sacremoses sentencepiece tqdm wandb\n",
        "!pip install --upgrade jupyter ipywidgets"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: torch>=1.6.0 in /usr/local/lib/python3.7/dist-packages (1.9.0+cu102)\n",
            "Requirement already satisfied: editdistance in /usr/local/lib/python3.7/dist-packages (0.5.3)\n",
            "Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (3.2.2)\n",
            "Collecting sacrebleu\n",
            "  Downloading sacrebleu-2.0.0-py3-none-any.whl (90 kB)\n",
            "\u001b[K     |████████████████████████████████| 90 kB 8.2 MB/s \n",
            "\u001b[?25hCollecting sacremoses\n",
            "  Downloading sacremoses-0.0.45-py3-none-any.whl (895 kB)\n",
            "\u001b[K     |████████████████████████████████| 895 kB 19.3 MB/s \n",
            "\u001b[?25hCollecting sentencepiece\n",
            "  Downloading sentencepiece-0.1.96-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.2 MB 18.5 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (4.62.0)\n",
            "Collecting wandb\n",
            "  Downloading wandb-0.12.0-py2.py3-none-any.whl (1.6 MB)\n",
            "\u001b[K     |████████████████████████████████| 1.6 MB 64.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from torch>=1.6.0) (3.7.4.3)\n",
            "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.3.1)\n",
            "Requirement already satisfied: numpy>=1.11 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (1.19.5)\n",
            "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (0.10.0)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.8.2)\n",
            "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib) (2.4.7)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from cycler>=0.10->matplotlib) (1.15.0)\n",
            "Collecting colorama\n",
            "  Downloading colorama-0.4.4-py2.py3-none-any.whl (16 kB)\n",
            "Requirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from sacrebleu) (0.8.9)\n",
            "Collecting portalocker\n",
            "  Downloading portalocker-2.3.1-py2.py3-none-any.whl (15 kB)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from sacrebleu) (2019.12.20)\n",
            "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses) (7.1.2)\n",
            "Requirement already satisfied: joblib in /usr/local/lib/python3.7/dist-packages (from sacremoses) (1.0.1)\n",
            "Collecting pathtools\n",
            "  Downloading pathtools-0.1.2.tar.gz (11 kB)\n",
            "Collecting GitPython>=1.0.0\n",
            "  Downloading GitPython-3.1.18-py3-none-any.whl (170 kB)\n",
            "\u001b[K     |████████████████████████████████| 170 kB 50.1 MB/s \n",
            "\u001b[?25hCollecting sentry-sdk>=1.0.0\n",
            "  Downloading sentry_sdk-1.3.1-py2.py3-none-any.whl (133 kB)\n",
            "\u001b[K     |████████████████████████████████| 133 kB 56.3 MB/s \n",
            "\u001b[?25hCollecting configparser>=3.8.1\n",
            "  Downloading configparser-5.0.2-py3-none-any.whl (19 kB)\n",
            "Collecting shortuuid>=0.5.0\n",
            "  Downloading shortuuid-1.0.1-py3-none-any.whl (7.5 kB)\n",
            "Requirement already satisfied: protobuf>=3.12.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (3.17.3)\n",
            "Requirement already satisfied: requests<3,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (2.23.0)\n",
            "Requirement already satisfied: PyYAML in /usr/local/lib/python3.7/dist-packages (from wandb) (3.13)\n",
            "Collecting subprocess32>=3.5.3\n",
            "  Downloading subprocess32-3.5.4.tar.gz (97 kB)\n",
            "\u001b[K     |████████████████████████████████| 97 kB 6.4 MB/s \n",
            "\u001b[?25hCollecting docker-pycreds>=0.4.0\n",
            "  Downloading docker_pycreds-0.4.0-py2.py3-none-any.whl (9.0 kB)\n",
            "Requirement already satisfied: promise<3,>=2.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (2.3)\n",
            "Requirement already satisfied: psutil>=5.0.0 in /usr/local/lib/python3.7/dist-packages (from wandb) (5.4.8)\n",
            "Collecting gitdb<5,>=4.0.1\n",
            "  Downloading gitdb-4.0.7-py3-none-any.whl (63 kB)\n",
            "\u001b[K     |████████████████████████████████| 63 kB 1.9 MB/s \n",
            "\u001b[?25hCollecting smmap<5,>=3.0.1\n",
            "  Downloading smmap-4.0.0-py2.py3-none-any.whl (24 kB)\n",
            "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (1.24.3)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (2021.5.30)\n",
            "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (2.10)\n",
            "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.0.0->wandb) (3.0.4)\n",
            "Building wheels for collected packages: subprocess32, pathtools\n",
            "  Building wheel for subprocess32 (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for subprocess32: filename=subprocess32-3.5.4-py3-none-any.whl size=6502 sha256=3990a02012028176334f3672a06ab8f73cc22986e8840878f3997423544ba6c9\n",
            "  Stored in directory: /root/.cache/pip/wheels/50/ca/fa/8fca8d246e64f19488d07567547ddec8eb084e8c0d7a59226a\n",
            "  Building wheel for pathtools (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for pathtools: filename=pathtools-0.1.2-py3-none-any.whl size=8807 sha256=77b7be04212fdd8658074abbd76ed039ca4e8a3526d50a5aab23349aaa7b120b\n",
            "  Stored in directory: /root/.cache/pip/wheels/3e/31/09/fa59cef12cdcfecc627b3d24273699f390e71828921b2cbba2\n",
            "Successfully built subprocess32 pathtools\n",
            "Installing collected packages: smmap, gitdb, subprocess32, shortuuid, sentry-sdk, portalocker, pathtools, GitPython, docker-pycreds, configparser, colorama, wandb, sentencepiece, sacremoses, sacrebleu\n",
            "Successfully installed GitPython-3.1.18 colorama-0.4.4 configparser-5.0.2 docker-pycreds-0.4.0 gitdb-4.0.7 pathtools-0.1.2 portalocker-2.3.1 sacrebleu-2.0.0 sacremoses-0.0.45 sentencepiece-0.1.96 sentry-sdk-1.3.1 shortuuid-1.0.1 smmap-4.0.0 subprocess32-3.5.4 wandb-0.12.0\n",
            "Requirement already satisfied: jupyter in /usr/local/lib/python3.7/dist-packages (1.0.0)\n",
            "Requirement already satisfied: ipywidgets in /usr/local/lib/python3.7/dist-packages (7.6.3)\n",
            "Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from jupyter) (5.6.1)\n",
            "Requirement already satisfied: ipykernel in /usr/local/lib/python3.7/dist-packages (from jupyter) (4.10.1)\n",
            "Requirement already satisfied: jupyter-console in /usr/local/lib/python3.7/dist-packages (from jupyter) (5.2.0)\n",
            "Requirement already satisfied: qtconsole in /usr/local/lib/python3.7/dist-packages (from jupyter) (5.1.1)\n",
            "Requirement already satisfied: notebook in /usr/local/lib/python3.7/dist-packages (from jupyter) (5.3.1)\n",
            "Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.7/dist-packages (from ipywidgets) (5.0.5)\n",
            "Requirement already satisfied: widgetsnbextension~=3.5.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets) (3.5.1)\n",
            "Requirement already satisfied: nbformat>=4.2.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets) (5.1.3)\n",
            "Requirement already satisfied: jupyterlab-widgets>=1.0.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets) (1.0.0)\n",
            "Requirement already satisfied: ipython>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from ipywidgets) (5.5.0)\n",
            "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.7/dist-packages (from ipykernel->jupyter) (5.3.5)\n",
            "Requirement already satisfied: tornado>=4.0 in /usr/local/lib/python3.7/dist-packages (from ipykernel->jupyter) (5.1.1)\n",
            "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (57.4.0)\n",
            "Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (0.8.1)\n",
            "Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (4.8.0)\n",
            "Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (0.7.5)\n",
            "Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (2.6.1)\n",
            "Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (4.4.2)\n",
            "Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython>=4.0.0->ipywidgets) (1.0.18)\n",
            "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets) (0.2.0)\n",
            "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets) (4.7.1)\n",
            "Requirement already satisfied: jsonschema!=2.5.0,>=2.4 in /usr/local/lib/python3.7/dist-packages (from nbformat>=4.2.0->ipywidgets) (2.6.0)\n",
            "Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipywidgets) (0.2.5)\n",
            "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython>=4.0.0->ipywidgets) (1.15.0)\n",
            "Requirement already satisfied: terminado>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from notebook->jupyter) (0.11.0)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.7/dist-packages (from notebook->jupyter) (2.11.3)\n",
            "Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from notebook->jupyter) (1.8.0)\n",
            "Requirement already satisfied: pyzmq>=13 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel->jupyter) (22.2.1)\n",
            "Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from jupyter-client->ipykernel->jupyter) (2.8.2)\n",
            "Requirement already satisfied: ptyprocess in /usr/local/lib/python3.7/dist-packages (from terminado>=0.8.1->notebook->jupyter) (0.7.0)\n",
            "Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2->notebook->jupyter) (2.0.1)\n",
            "Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter) (4.0.0)\n",
            "Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter) (0.8.4)\n",
            "Requirement already satisfied: entrypoints>=0.2.2 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter) (0.3)\n",
            "Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter) (1.4.3)\n",
            "Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter) (0.5.0)\n",
            "Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter) (0.7.1)\n",
            "Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->jupyter) (0.5.1)\n",
            "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->jupyter) (21.0)\n",
            "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->bleach->nbconvert->jupyter) (2.4.7)\n",
            "Requirement already satisfied: qtpy in /usr/local/lib/python3.7/dist-packages (from qtconsole->jupyter) (1.10.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aBlVg887P7B8",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5d75e22a-73dc-4d0c-c194-c1469dc4b8d1"
      },
      "source": [
        "!git clone https://github.com/pytorch/fairseq.git\n",
        "!cd fairseq && git checkout 9a1c497\n",
        "!pip install --upgrade ./fairseq/"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'fairseq'...\n",
            "remote: Enumerating objects: 29010, done.\u001b[K\n",
            "remote: Counting objects: 100% (92/92), done.\u001b[K\n",
            "remote: Compressing objects: 100% (81/81), done.\u001b[K\n",
            "remote: Total 29010 (delta 15), reused 64 (delta 9), pack-reused 28918\u001b[K\n",
            "Receiving objects: 100% (29010/29010), 12.32 MiB | 22.66 MiB/s, done.\n",
            "Resolving deltas: 100% (21742/21742), done.\n",
            "Note: checking out '9a1c497'.\n",
            "\n",
            "You are in 'detached HEAD' state. You can look around, make experimental\n",
            "changes and commit them, and you can discard any commits you make in this\n",
            "state without impacting any branches by performing another checkout.\n",
            "\n",
            "If you want to create a new branch to retain commits you create, you may\n",
            "do so (now or later) by using -b with the checkout command again. Example:\n",
            "\n",
            "  git checkout -b <new-branch-name>\n",
            "\n",
            "HEAD is now at 9a1c4970 Make Hydra logging work with DDP (#1568)\n",
            "Processing ./fairseq\n",
            "\u001b[33m  DEPRECATION: A future pip version will change local packages to be built in-place without first copying to a temporary directory. We recommend you use --use-feature=in-tree-build to test your packages with this new behavior before it becomes the default.\n",
            "   pip 21.3 will remove support for this functionality. You can find discussion regarding this at https://github.com/pypa/pip/issues/7555.\u001b[0m\n",
            "  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "  Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n",
            "  Installing backend dependencies ... \u001b[?25l\u001b[?25hdone\n",
            "    Preparing wheel metadata ... \u001b[?25l\u001b[?25hdone\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (1.19.5)\n",
            "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (4.62.0)\n",
            "Requirement already satisfied: cffi in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (1.14.6)\n",
            "Collecting omegaconf<2.1\n",
            "  Downloading omegaconf-2.0.6-py3-none-any.whl (36 kB)\n",
            "Collecting hydra-core<1.1\n",
            "  Downloading hydra_core-1.0.7-py3-none-any.whl (123 kB)\n",
            "\u001b[K     |████████████████████████████████| 123 kB 43.8 MB/s \n",
            "\u001b[?25hRequirement already satisfied: cython in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (0.29.24)\n",
            "Requirement already satisfied: torch in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (1.9.0+cu102)\n",
            "Requirement already satisfied: regex in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (2019.12.20)\n",
            "Requirement already satisfied: sacrebleu>=1.4.12 in /usr/local/lib/python3.7/dist-packages (from fairseq==1.0.0a0+9a1c497) (2.0.0)\n",
            "Requirement already satisfied: importlib-resources in /usr/local/lib/python3.7/dist-packages (from hydra-core<1.1->fairseq==1.0.0a0+9a1c497) (5.2.2)\n",
            "Collecting antlr4-python3-runtime==4.8\n",
            "  Downloading antlr4-python3-runtime-4.8.tar.gz (112 kB)\n",
            "\u001b[K     |████████████████████████████████| 112 kB 48.6 MB/s \n",
            "\u001b[?25hRequirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from omegaconf<2.1->fairseq==1.0.0a0+9a1c497) (3.7.4.3)\n",
            "Collecting PyYAML>=5.1.*\n",
            "  Downloading PyYAML-5.4.1-cp37-cp37m-manylinux1_x86_64.whl (636 kB)\n",
            "\u001b[K     |████████████████████████████████| 636 kB 35.9 MB/s \n",
            "\u001b[?25hRequirement already satisfied: tabulate>=0.8.9 in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (0.8.9)\n",
            "Requirement already satisfied: colorama in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (0.4.4)\n",
            "Requirement already satisfied: portalocker in /usr/local/lib/python3.7/dist-packages (from sacrebleu>=1.4.12->fairseq==1.0.0a0+9a1c497) (2.3.1)\n",
            "Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi->fairseq==1.0.0a0+9a1c497) (2.20)\n",
            "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from importlib-resources->hydra-core<1.1->fairseq==1.0.0a0+9a1c497) (3.5.0)\n",
            "Building wheels for collected packages: fairseq, antlr4-python3-runtime\n",
            "  Building wheel for fairseq (PEP 517) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for fairseq: filename=fairseq-1.0.0a0+9a1c497-cp37-cp37m-linux_x86_64.whl size=2851168 sha256=a0fedeadd3d428609eaba8793d38d05c546c29503946d8b06d3d7e27c08742ee\n",
            "  Stored in directory: /tmp/pip-ephem-wheel-cache-6tffmg9v/wheels/7c/35/80/edbd520a1a7e615df007002aeea9f6bf5f3c8f9243e072f6ce\n",
            "  Building wheel for antlr4-python3-runtime (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.8-py3-none-any.whl size=141230 sha256=f420c580cea10d97860ebdd53047d8fd5aac81400e3e031fe1a537f79a71b8d4\n",
            "  Stored in directory: /root/.cache/pip/wheels/ca/33/b7/336836125fc9bb4ceaa4376d8abca10ca8bc84ddc824baea6c\n",
            "Successfully built fairseq antlr4-python3-runtime\n",
            "Installing collected packages: PyYAML, omegaconf, antlr4-python3-runtime, hydra-core, fairseq\n",
            "  Attempting uninstall: PyYAML\n",
            "    Found existing installation: PyYAML 3.13\n",
            "    Uninstalling PyYAML-3.13:\n",
            "      Successfully uninstalled PyYAML-3.13\n",
            "Successfully installed PyYAML-5.4.1 antlr4-python3-runtime-4.8 fairseq-1.0.0a0+9a1c497 hydra-core-1.0.7 omegaconf-2.0.6\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VYrA-A8HP7B9"
      },
      "source": [
        "import sys\n",
        "import pdb\n",
        "import pprint\n",
        "import logging\n",
        "import os\n",
        "import random\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "from torch.utils import data\n",
        "import numpy as np\n",
        "import tqdm.auto as tqdm\n",
        "from pathlib import Path\n",
        "from argparse import Namespace\n",
        "from fairseq import utils\n",
        "\n",
        "import matplotlib.pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sQ84ItXlP7B9"
      },
      "source": [
        "# Fix random seed"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HYsn-nsHP7B9"
      },
      "source": [
        "seed = 73\n",
        "random.seed(seed)\n",
        "torch.manual_seed(seed)\n",
        "if torch.cuda.is_available():\n",
        "    torch.cuda.manual_seed(seed)\n",
        "    torch.cuda.manual_seed_all(seed)  \n",
        "np.random.seed(seed)  \n",
        "torch.backends.cudnn.benchmark = False\n",
        "torch.backends.cudnn.deterministic = True"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zrOcDjiGP7B-"
      },
      "source": [
        "# Dataset Information\n",
        "\n",
        "## En-Zh Bilingual Parallel Corpus\n",
        "* [TED2020](#reimers-2020-multilingual-sentence-bert)\n",
        "    - Raw: 398,066 (sentences)   \n",
        "    - Processed: 393,980 (sentences)\n",
        "    \n",
        "\n",
        "## Testdata\n",
        "- Size: 4,000 (sentences)\n",
        "- **Chinese translation is undisclosed. The provided (.zh) file is psuedo translation, each line is a '。'**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SiNab78OP7B-"
      },
      "source": [
        "# Dataset Download"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T0--MLcQP7B-"
      },
      "source": [
        "## Install megatools (optional)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t92JKoHFP7B-"
      },
      "source": [
        "#!apt-get install megatools"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PrN8_US2P7B-"
      },
      "source": [
        "## Download and extract"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GWumMV0IP7B-",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "640abd57-84e9-4525-8f88-6a25b7781203"
      },
      "source": [
        "data_dir = './DATA/rawdata'\n",
        "dataset_name = 'ted2020'\n",
        "urls = (\n",
        "#    '\"https://onedrive.live.com/download?cid=3E549F3B24B238B4&resid=3E549F3B24B238B4%214989&authkey=AGgQ-DaR8eFSl1A\"', \n",
        "#    '\"https://onedrive.live.com/download?cid=3E549F3B24B238B4&resid=3E549F3B24B238B4%214987&authkey=AA4qP_azsicwZZM\"',\n",
        "# # If the above links die, use the following instead. \n",
        "     \"https://www.csie.ntu.edu.tw/~r09922057/ML2021-hw5/ted2020.tgz\",\n",
        "     \"https://www.csie.ntu.edu.tw/~r09922057/ML2021-hw5/test.tgz\",\n",
        "# # If the above links die, use the following instead. \n",
        "#     \"https://mega.nz/#!vEcTCISJ!3Rw0eHTZWPpdHBTbQEqBDikDEdFPr7fI8WxaXK9yZ9U\",\n",
        "#     \"https://mega.nz/#!zNcnGIoJ!oPJX9AvVVs11jc0SaK6vxP_lFUNTkEcK2WbxJpvjU5Y\",\n",
        ")\n",
        "file_names = (\n",
        "    'ted2020.tgz', # train & dev\n",
        "    'test.tgz', # test\n",
        ")\n",
        "prefix = Path(data_dir).absolute() / dataset_name\n",
        "\n",
        "prefix.mkdir(parents=True, exist_ok=True)\n",
        "for u, f in zip(urls, file_names):\n",
        "    path = prefix/f\n",
        "    if not path.exists():\n",
        "        if 'mega' in u:\n",
        "            !megadl {u} --path {path}\n",
        "        else:\n",
        "            !wget {u} -O {path}\n",
        "    if path.suffix == \".tgz\":\n",
        "        !tar -xvf {path} -C {prefix}\n",
        "    elif path.suffix == \".zip\":\n",
        "        !unzip -o {path} -d {prefix}\n",
        "!mv {prefix/'raw.en'} {prefix/'train_dev.raw.en'}\n",
        "!mv {prefix/'raw.zh'} {prefix/'train_dev.raw.zh'}\n",
        "!mv {prefix/'test.en'} {prefix/'test.raw.en'}\n",
        "!mv {prefix/'test.zh'} {prefix/'test.raw.zh'}"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2021-08-23 06:31:39--  https://www.csie.ntu.edu.tw/~r09922057/ML2021-hw5/ted2020.tgz\n",
            "Resolving www.csie.ntu.edu.tw (www.csie.ntu.edu.tw)... 140.112.30.26\n",
            "Connecting to www.csie.ntu.edu.tw (www.csie.ntu.edu.tw)|140.112.30.26|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 28601955 (27M) [application/x-gzip]\n",
            "Saving to: ‘/content/DATA/rawdata/ted2020/ted2020.tgz’\n",
            "\n",
            "/content/DATA/rawda 100%[===================>]  27.28M  9.54MB/s    in 2.9s    \n",
            "\n",
            "2021-08-23 06:31:43 (9.54 MB/s) - ‘/content/DATA/rawdata/ted2020/ted2020.tgz’ saved [28601955/28601955]\n",
            "\n",
            "raw.en\n",
            "raw.zh\n",
            "--2021-08-23 06:31:44--  https://www.csie.ntu.edu.tw/~r09922057/ML2021-hw5/test.tgz\n",
            "Resolving www.csie.ntu.edu.tw (www.csie.ntu.edu.tw)... 140.112.30.26\n",
            "Connecting to www.csie.ntu.edu.tw (www.csie.ntu.edu.tw)|140.112.30.26|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 150315 (147K) [application/x-gzip]\n",
            "Saving to: ‘/content/DATA/rawdata/ted2020/test.tgz’\n",
            "\n",
            "/content/DATA/rawda 100%[===================>] 146.79K   275KB/s    in 0.5s    \n",
            "\n",
            "2021-08-23 06:31:46 (275 KB/s) - ‘/content/DATA/rawdata/ted2020/test.tgz’ saved [150315/150315]\n",
            "\n",
            "test.en\n",
            "test.zh\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uQRoo1jRP7B_"
      },
      "source": [
        "## Language"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "asW_ib6WP7B_"
      },
      "source": [
        "src_lang = 'en'\n",
        "tgt_lang = 'zh'\n",
        "\n",
        "data_prefix = f'{prefix}/train_dev.raw'\n",
        "test_prefix = f'{prefix}/test.raw'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HvWIHmJjP7B_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d0b74205-8fff-4d70-8db1-ccc3464679c5"
      },
      "source": [
        "!head {data_prefix+'.'+src_lang} -n 5\n",
        "!head {data_prefix+'.'+tgt_lang} -n 5"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thank you so much, Chris.\n",
            "And it's truly a great honor to have the opportunity to come to this stage twice; I'm extremely grateful.\n",
            "I have been blown away by this conference, and I want to thank all of you for the many nice comments about what I had to say the other night.\n",
            "And I say that sincerely, partly because  I need that.\n",
            "Put yourselves in my position.\n",
            "非常謝謝你，克里斯。能有這個機會第二度踏上這個演講台\n",
            "真是一大榮幸。我非常感激。\n",
            "這個研討會給我留下了極為深刻的印象，我想感謝大家 對我之前演講的好評。\n",
            "我是由衷的想這麼說，有部份原因是因為 —— 我真的有需要!\n",
            "請你們設身處地為我想一想！\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9d7SrTIPP7B_"
      },
      "source": [
        "## Preprocess files"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z00uYuJXP7B_"
      },
      "source": [
        "import re\n",
        "\n",
        "def strQ2B(ustring):\n",
        "    \"\"\"Full width -> half width\"\"\"\n",
        "    # reference:https://ithelp.ithome.com.tw/articles/10233122\n",
        "    ss = []\n",
        "    for s in ustring:\n",
        "        rstring = \"\"\n",
        "        for uchar in s:\n",
        "            inside_code = ord(uchar)\n",
        "            if inside_code == 12288:  # Full width space: direct conversion\n",
        "                inside_code = 32\n",
        "            elif (inside_code >= 65281 and inside_code <= 65374):  # Full width chars (except space) conversion\n",
        "                inside_code -= 65248\n",
        "            rstring += chr(inside_code)\n",
        "        ss.append(rstring)\n",
        "    return ''.join(ss)\n",
        "                \n",
        "def clean_s(s, lang):\n",
        "    if lang == 'en':\n",
        "        s = re.sub(r\"\\([^()]*\\)\", \"\", s) # remove ([text])\n",
        "        s = s.replace('-', '') # remove '-'\n",
        "        s = re.sub('([.,;!?()\\\"])', r' \\1 ', s) # keep punctuation\n",
        "    elif lang == 'zh':\n",
        "        s = strQ2B(s) # Q2B\n",
        "        s = re.sub(r\"\\([^()]*\\)\", \"\", s) # remove ([text])\n",
        "        s = s.replace(' ', '')\n",
        "        s = s.replace('—', '')\n",
        "        s = s.replace('“', '\"')\n",
        "        s = s.replace('”', '\"')\n",
        "        s = s.replace('_', '')\n",
        "        s = re.sub('([。,;!?()\\\"~「」])', r' \\1 ', s) # keep punctuation\n",
        "    s = ' '.join(s.strip().split())\n",
        "    return s\n",
        "\n",
        "def len_s(s, lang):\n",
        "    if lang == 'zh':\n",
        "        return len(s)\n",
        "    return len(s.split())\n",
        "\n",
        "def clean_corpus(prefix, l1, l2, ratio=9, max_len=1000, min_len=1):\n",
        "    if Path(f'{prefix}.clean.{l1}').exists() and Path(f'{prefix}.clean.{l2}').exists():\n",
        "        print(f'{prefix}.clean.{l1} & {l2} exists. skipping clean.')\n",
        "        return\n",
        "    with open(f'{prefix}.{l1}', 'r') as l1_in_f:\n",
        "        with open(f'{prefix}.{l2}', 'r') as l2_in_f:\n",
        "            with open(f'{prefix}.clean.{l1}', 'w') as l1_out_f:\n",
        "                with open(f'{prefix}.clean.{l2}', 'w') as l2_out_f:\n",
        "                    for s1 in l1_in_f:\n",
        "                        s1 = s1.strip()\n",
        "                        s2 = l2_in_f.readline().strip()\n",
        "                        s1 = clean_s(s1, l1)\n",
        "                        s2 = clean_s(s2, l2)\n",
        "                        s1_len = len_s(s1, l1)\n",
        "                        s2_len = len_s(s2, l2)\n",
        "                        if min_len > 0: # remove short sentence\n",
        "                            if s1_len < min_len or s2_len < min_len:\n",
        "                                continue\n",
        "                        if max_len > 0: # remove long sentence\n",
        "                            if s1_len > max_len or s2_len > max_len:\n",
        "                                continue\n",
        "                        if ratio > 0: # remove by ratio of length\n",
        "                            if s1_len/s2_len > ratio or s2_len/s1_len > ratio:\n",
        "                                continue\n",
        "                        print(s1, file=l1_out_f)\n",
        "                        print(s2, file=l2_out_f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BHQJphF2P7CA"
      },
      "source": [
        "clean_corpus(data_prefix, src_lang, tgt_lang)\n",
        "clean_corpus(test_prefix, src_lang, tgt_lang, ratio=-1, min_len=-1, max_len=-1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dWHFqoEDP7CB",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "2858ac82-3bb3-4c21-a503-768440acdd57"
      },
      "source": [
        "!head {data_prefix+'.clean.'+src_lang} -n 5\n",
        "!head {data_prefix+'.clean.'+tgt_lang} -n 5"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Thank you so much , Chris .\n",
            "And it's truly a great honor to have the opportunity to come to this stage twice ; I'm extremely grateful .\n",
            "I have been blown away by this conference , and I want to thank all of you for the many nice comments about what I had to say the other night .\n",
            "And I say that sincerely , partly because I need that .\n",
            "Put yourselves in my position .\n",
            "非常謝謝你 , 克里斯 。 能有這個機會第二度踏上這個演講台\n",
            "真是一大榮幸 。 我非常感激 。\n",
            "這個研討會給我留下了極為深刻的印象 , 我想感謝大家對我之前演講的好評 。\n",
            "我是由衷的想這麼說 , 有部份原因是因為我真的有需要 !\n",
            "請你們設身處地為我想一想 !\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KZBJmx5P7CB"
      },
      "source": [
        "## Split into train/valid"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E7gMQJukP7CB"
      },
      "source": [
        "valid_ratio = 0.01 # 3000~4000 would suffice\n",
        "train_ratio = 1 - valid_ratio"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-vE9oKk5P7CB"
      },
      "source": [
        "if (prefix/f'train.clean.{src_lang}').exists() \\\n",
        "and (prefix/f'train.clean.{tgt_lang}').exists() \\\n",
        "and (prefix/f'valid.clean.{src_lang}').exists() \\\n",
        "and (prefix/f'valid.clean.{tgt_lang}').exists():\n",
        "    print(f'train/valid splits exists. skipping split.')\n",
        "else:\n",
        "    line_num = sum(1 for line in open(f'{data_prefix}.clean.{src_lang}'))\n",
        "    labels = list(range(line_num))\n",
        "    random.shuffle(labels)\n",
        "    for lang in [src_lang, tgt_lang]:\n",
        "        train_f = open(os.path.join(data_dir, dataset_name, f'train.clean.{lang}'), 'w')\n",
        "        valid_f = open(os.path.join(data_dir, dataset_name, f'valid.clean.{lang}'), 'w')\n",
        "        count = 0\n",
        "        for line in open(f'{data_prefix}.clean.{lang}', 'r'):\n",
        "            if labels[count]/line_num < train_ratio:\n",
        "                train_f.write(line)\n",
        "            else:\n",
        "                valid_f.write(line)\n",
        "            count += 1\n",
        "        train_f.close()\n",
        "        valid_f.close()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5m_qu5_4P7CB"
      },
      "source": [
        "## Subword Units \n",
        "Out of vocabulary (OOV) has been a major problem in machine translation. This can be alleviated by using subword units.\n",
        "- We will use the [sentencepiece](#kudo-richardson-2018-sentencepiece) package\n",
        "- select 'unigram' or 'byte-pair encoding (BPE)' algorithm"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "605KH4egP7CC"
      },
      "source": [
        "import sentencepiece as spm\n",
        "vocab_size = 8000\n",
        "if (prefix/f'spm{vocab_size}.model').exists():\n",
        "    print(f'{prefix}/spm{vocab_size}.model exists. skipping spm_train.')\n",
        "else:\n",
        "    spm.SentencePieceTrainer.train(\n",
        "        input=','.join([f'{prefix}/train.clean.{src_lang}',\n",
        "                        f'{prefix}/valid.clean.{src_lang}',\n",
        "                        f'{prefix}/train.clean.{tgt_lang}',\n",
        "                        f'{prefix}/valid.clean.{tgt_lang}']),\n",
        "        model_prefix=prefix/f'spm{vocab_size}',\n",
        "        vocab_size=vocab_size,\n",
        "        character_coverage=1,\n",
        "        model_type='unigram', # 'bpe' works as well\n",
        "        input_sentence_size=1e6,\n",
        "        shuffle_input_sentence=True,\n",
        "        normalization_rule_name='nmt_nfkc_cf',\n",
        "    )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DKGz-RhvP7CC"
      },
      "source": [
        "spm_model = spm.SentencePieceProcessor(model_file=str(prefix/f'spm{vocab_size}.model'))\n",
        "in_tag = {\n",
        "    'train': 'train.clean',\n",
        "    'valid': 'valid.clean',\n",
        "    'test': 'test.raw.clean',\n",
        "}\n",
        "for split in ['train', 'valid', 'test']:\n",
        "    for lang in [src_lang, tgt_lang]:\n",
        "        out_path = prefix/f'{split}.{lang}'\n",
        "        if out_path.exists():\n",
        "            print(f\"{out_path} exists. skipping spm_encode.\")\n",
        "        else:\n",
        "            with open(prefix/f'{split}.{lang}', 'w') as out_f:\n",
        "                with open(prefix/f'{in_tag[split]}.{lang}', 'r') as in_f:\n",
        "                    for line in in_f:\n",
        "                        line = line.strip()\n",
        "                        tok = spm_model.encode(line, out_type=str)\n",
        "                        print(' '.join(tok), file=out_f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1UWsxS3fP7CC",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f9ec5832-8522-446d-a5a9-31acecac3514"
      },
      "source": [
        "!head {data_dir+'/'+dataset_name+'/train.'+src_lang} -n 5\n",
        "!head {data_dir+'/'+dataset_name+'/train.'+tgt_lang} -n 5"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "▁thank ▁you ▁so ▁much ▁, ▁chris ▁.\n",
            "▁and ▁it ' s ▁ t ru ly ▁a ▁great ▁ho n or ▁to ▁have ▁the ▁ op port un ity ▁to ▁come ▁to ▁this ▁st age ▁ t wi ce ▁; ▁i ' m ▁ex t re me ly ▁gr ate ful ▁.\n",
            "▁i ▁have ▁been ▁ bl ow n ▁away ▁by ▁this ▁con fer ence ▁, ▁and ▁i ▁want ▁to ▁thank ▁all ▁of ▁you ▁for ▁the ▁many ▁ ni ce ▁ com ment s ▁about ▁what ▁i ▁had ▁to ▁say ▁the ▁other ▁night ▁.\n",
            "▁and ▁i ▁say ▁that ▁since re ly ▁, ▁part ly ▁because ▁i ▁need ▁that ▁.\n",
            "▁put ▁your s el ve s ▁in ▁my ▁po s ition ▁.\n",
            "▁ 非常 謝 謝 你 ▁, ▁ 克 里 斯 ▁。 ▁ 能 有 這個 機會 第二 度 踏 上 這個 演講 台\n",
            "▁ 真 是 一 大 榮 幸 ▁。 ▁我 非常 感 激 ▁。\n",
            "▁這個 研 討 會 給我 留 下 了 極 為 深 刻 的 印 象 ▁, ▁我想 感 謝 大家 對我 之前 演講 的 好 評 ▁。\n",
            "▁我 是由 衷 的 想 這麼 說 ▁, ▁有 部份 原因 是因為 我 真的 有 需要 ▁!\n",
            "▁ 請 你們 設 身 處 地 為 我想 一 想 ▁!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "04pMpQ6OP7CC"
      },
      "source": [
        "## Binarize the data with fairseq"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J4pXlkiuP7CD",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "d633861f-76c7-4a0f-bcff-e43d95497edd"
      },
      "source": [
        "binpath = Path('./DATA/data-bin', dataset_name)\n",
        "if binpath.exists():\n",
        "    print(binpath, \"exists, will not overwrite!\")\n",
        "else:\n",
        "    !python -m fairseq_cli.preprocess \\\n",
        "        --source-lang {src_lang}\\\n",
        "        --target-lang {tgt_lang}\\\n",
        "        --trainpref {prefix/'train'}\\\n",
        "        --validpref {prefix/'valid'}\\\n",
        "        --testpref {prefix/'test'}\\\n",
        "        --destdir {binpath}\\\n",
        "        --joined-dictionary\\\n",
        "        --workers 2"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-08-23 06:46:06 | INFO | fairseq_cli.preprocess | Namespace(align_suffix=None, alignfile=None, all_gather_list_size=16384, azureml_logging=False, bf16=False, bpe=None, cpu=False, criterion='cross_entropy', dataset_impl='mmap', destdir='DATA/data-bin/ted2020', empty_cache_freq=0, fp16=False, fp16_init_scale=128, fp16_no_flatten_grads=False, fp16_scale_tolerance=0.0, fp16_scale_window=None, joined_dictionary=True, log_format=None, log_interval=100, lr_scheduler='fixed', memory_efficient_bf16=False, memory_efficient_fp16=False, min_loss_scale=0.0001, model_parallel_size=1, no_progress_bar=False, nwordssrc=-1, nwordstgt=-1, only_source=False, optimizer=None, padding_factor=8, profile=False, quantization_config_path=None, reset_logging=False, scoring='bleu', seed=1, source_lang='en', srcdict=None, suppress_crashes=False, target_lang='zh', task='translation', tensorboard_logdir=None, testpref='/content/DATA/rawdata/ted2020/test', tgtdict=None, threshold_loss_scale=None, thresholdsrc=0, thresholdtgt=0, tokenizer=None, tpu=False, trainpref='/content/DATA/rawdata/ted2020/train', user_dir=None, validpref='/content/DATA/rawdata/ted2020/valid', wandb_project=None, workers=2)\n",
            "2021-08-23 06:47:27 | INFO | fairseq_cli.preprocess | [en] Dictionary: 8000 types\n",
            "2021-08-23 06:48:32 | INFO | fairseq_cli.preprocess | [en] /content/DATA/rawdata/ted2020/train.en: 390041 sents, 12215608 tokens, 0.0% replaced by <unk>\n",
            "2021-08-23 06:48:32 | INFO | fairseq_cli.preprocess | [en] Dictionary: 8000 types\n",
            "2021-08-23 06:48:33 | INFO | fairseq_cli.preprocess | [en] /content/DATA/rawdata/ted2020/valid.en: 3939 sents, 122560 tokens, 0.0% replaced by <unk>\n",
            "2021-08-23 06:48:33 | INFO | fairseq_cli.preprocess | [en] Dictionary: 8000 types\n",
            "2021-08-23 06:48:33 | INFO | fairseq_cli.preprocess | [en] /content/DATA/rawdata/ted2020/test.en: 4000 sents, 122894 tokens, 0.0% replaced by <unk>\n",
            "2021-08-23 06:48:33 | INFO | fairseq_cli.preprocess | [zh] Dictionary: 8000 types\n",
            "2021-08-23 06:49:31 | INFO | fairseq_cli.preprocess | [zh] /content/DATA/rawdata/ted2020/train.zh: 390041 sents, 9533185 tokens, 0.0% replaced by <unk>\n",
            "2021-08-23 06:49:31 | INFO | fairseq_cli.preprocess | [zh] Dictionary: 8000 types\n",
            "2021-08-23 06:49:32 | INFO | fairseq_cli.preprocess | [zh] /content/DATA/rawdata/ted2020/valid.zh: 3939 sents, 96124 tokens, 0.0052% replaced by <unk>\n",
            "2021-08-23 06:49:32 | INFO | fairseq_cli.preprocess | [zh] Dictionary: 8000 types\n",
            "2021-08-23 06:49:32 | INFO | fairseq_cli.preprocess | [zh] /content/DATA/rawdata/ted2020/test.zh: 4000 sents, 8000 tokens, 0.0% replaced by <unk>\n",
            "2021-08-23 06:49:32 | INFO | fairseq_cli.preprocess | Wrote preprocessed data to DATA/data-bin/ted2020\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "I__CmJLeP7CD"
      },
      "source": [
        "# Configuration for Experiments"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-xUrlDRVP7CD"
      },
      "source": [
        "config = Namespace(\n",
        "    datadir = \"./DATA/data-bin/ted2020\",\n",
        "    savedir = \"./checkpoints/rnn\",\n",
        "    source_lang = \"en\",\n",
        "    target_lang = \"zh\",\n",
        "    \n",
        "    # cpu threads when fetching & processing data.\n",
        "    num_workers=2,  \n",
        "    # batch size in terms of tokens. gradient accumulation increases the effective batchsize.\n",
        "    max_tokens=8192,\n",
        "    accum_steps=2,\n",
        "    \n",
        "    # the lr s calculated from Noam lr scheduler. you can tune the maximum lr by this factor.\n",
        "    lr_factor=2.,\n",
        "    lr_warmup=4000,\n",
        "    \n",
        "    # clipping gradient norm helps alleviate gradient exploding\n",
        "    clip_norm=1.0,\n",
        "    \n",
        "    # maximum epochs for training\n",
        "    max_epoch=30,\n",
        "    start_epoch=1,\n",
        "    \n",
        "    # beam size for beam search\n",
        "    beam=5, \n",
        "    # generate sequences of maximum length ax + b, where x is the source length\n",
        "    max_len_a=1.2, \n",
        "    max_len_b=10, \n",
        "    # when decoding, post process sentence by removing sentencepiece symbols and jieba tokenization.\n",
        "    post_process = \"sentencepiece\",\n",
        "    \n",
        "    # checkpoints\n",
        "    keep_last_epochs=5,\n",
        "    resume=None, # if resume from checkpoint name (under config.savedir)\n",
        "    \n",
        "    # logging\n",
        "    use_wandb=False,\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IRiqIHT2P7CD"
      },
      "source": [
        "# Logging\n",
        "- logging package logs ordinary messages\n",
        "- wandb logs the loss, bleu, etc. in the training process"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "wZoKp6QFP7CD"
      },
      "source": [
        "logging.basicConfig(\n",
        "    format=\"%(asctime)s | %(levelname)s | %(name)s | %(message)s\",\n",
        "    datefmt=\"%Y-%m-%d %H:%M:%S\",\n",
        "    level=\"INFO\", # \"DEBUG\" \"WARNING\" \"ERROR\"\n",
        "    stream=sys.stdout,\n",
        ")\n",
        "proj = \"hw5.seq2seq\"\n",
        "logger = logging.getLogger(proj)\n",
        "if config.use_wandb:\n",
        "    import wandb\n",
        "    wandb.init(project=proj, name=Path(config.savedir).stem, config=config)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sZCwT2Zpy9Hd"
      },
      "source": [
        "#test github's behavier on saving"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yaBXvBNCzx2N"
      },
      "source": [
        "#test2 github's behavier on saving"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5tq5N0xjP7CD"
      },
      "source": [
        "# CUDA Environment"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MlXgGu8aP7CE",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 338
        },
        "outputId": "883d8cab-9365-4989-8c9a-2bcf32a9a801"
      },
      "source": [
        "cuda_env = utils.CudaEnvironment()\n",
        "utils.CudaEnvironment.pretty_print_cuda_env_list([cuda_env])\n",
        "device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-21-2b2eec4ba29f>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcuda_env\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mCudaEnvironment\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mutils\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mCudaEnvironment\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpretty_print_cuda_env_list\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mcuda_env\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0mdevice\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdevice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'cuda:0'\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_available\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0;34m'cpu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/fairseq/utils.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    679\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0mCudaEnvironment\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mobject\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    680\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 681\u001b[0;31m         \u001b[0mcur_device\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcurrent_device\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    682\u001b[0m         \u001b[0mprop\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcuda\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_device_properties\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"cuda:{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcur_device\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    683\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mprop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/cuda/__init__.py\u001b[0m in \u001b[0;36mcurrent_device\u001b[0;34m()\u001b[0m\n\u001b[1;32m    430\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0mcurrent_device\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    431\u001b[0m     \u001b[0;34mr\"\"\"Returns the index of a currently selected device.\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 432\u001b[0;31m     \u001b[0m_lazy_init\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    433\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cuda_getDevice\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    434\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/cuda/__init__.py\u001b[0m in \u001b[0;36m_lazy_init\u001b[0;34m()\u001b[0m\n\u001b[1;32m    170\u001b[0m         \u001b[0;31m# This function throws if there's a driver initialization error, no GPUs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    171\u001b[0m         \u001b[0;31m# are found or any other error occurs\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 172\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_C\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_cuda_init\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    173\u001b[0m         \u001b[0;31m# Some of the queued calls may reentrantly call _lazy_init();\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    174\u001b[0m         \u001b[0;31m# we need to just return without initializing in that case.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mRuntimeError\u001b[0m: No CUDA GPUs are available"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VAmqcOvhP7CE"
      },
      "source": [
        "# Dataloading"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N6VpA6kUP7CE"
      },
      "source": [
        "## We borrow the TranslationTask from fairseq\n",
        "* used to load the binarized data created above\n",
        "* well-implemented data iterator (dataloader)\n",
        "* built-in task.source_dictionary and task.target_dictionary are also handy\n",
        "* well-implemented beach search decoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t7cBbTuxP7CE"
      },
      "source": [
        "from fairseq.tasks.translation import TranslationConfig, TranslationTask\n",
        "\n",
        "## setup task\n",
        "task_cfg = TranslationConfig(\n",
        "    data=config.datadir,\n",
        "    source_lang=config.source_lang,\n",
        "    target_lang=config.target_lang,\n",
        "    train_subset=\"train\",\n",
        "    required_seq_len_multiple=8,\n",
        "    dataset_impl=\"mmap\",\n",
        "    upsample_primary=1,\n",
        ")\n",
        "task = TranslationTask.setup_task(task_cfg)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yuLIOHvpP7CE"
      },
      "source": [
        "logger.info(\"loading data for epoch 1\")\n",
        "task.load_dataset(split=\"train\", epoch=1, combine=True) # combine if you have back-translation data.\n",
        "task.load_dataset(split=\"valid\", epoch=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QKzwRATIP7CE"
      },
      "source": [
        "sample = task.dataset(\"valid\")[1]\n",
        "pprint.pprint(sample)\n",
        "pprint.pprint(\n",
        "    \"Source: \" + \\\n",
        "    task.source_dictionary.string(\n",
        "        sample['source'],\n",
        "        config.post_process,\n",
        "    )\n",
        ")\n",
        "pprint.pprint(\n",
        "    \"Target: \" + \\\n",
        "    task.target_dictionary.string(\n",
        "        sample['target'],\n",
        "        config.post_process,\n",
        "    )\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D2RXhns3P7CE"
      },
      "source": [
        "## Dataset Iterator"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HcWIl5hlP7CF"
      },
      "source": [
        "* Controls every batch to contain no more than N tokens, which optimizes GPU memory efficiency\n",
        "* Shuffles the training set for every epoch\n",
        "* Ignore sentences exceeding maximum length\n",
        "* Pad all sentences in a batch to the same length, which enables parallel computing by GPU\n",
        "* Add eos and shift one token\n",
        "    - teacher forcing: to train the model to predict the next token based on prefix, we feed the right shifted target sequence as the decoder input.\n",
        "    - generally, prepending bos to the target would do the job (as shown below)\n",
        "![seq2seq](https://i.imgur.com/0zeDyuI.png)\n",
        "    - in fairseq however, this is done by moving the eos token to the begining. Empirically, this has the same effect. For instance:\n",
        "    ```\n",
        "    # output target (target) and Decoder input (prev_output_tokens): \n",
        "                   eos = 2\n",
        "                target = 419,  711,  238,  888,  792,   60,  968,    8,    2\n",
        "    prev_output_tokens = 2,  419,  711,  238,  888,  792,   60,  968,    8\n",
        "    ```\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Zsuj9QUHP7CF"
      },
      "source": [
        "def load_data_iterator(task, split, epoch=1, max_tokens=4000, num_workers=1, cached=True):\n",
        "    batch_iterator = task.get_batch_iterator(\n",
        "        dataset=task.dataset(split),\n",
        "        max_tokens=max_tokens,\n",
        "        max_sentences=None,\n",
        "        max_positions=utils.resolve_max_positions(\n",
        "            task.max_positions(),\n",
        "            max_tokens,\n",
        "        ),\n",
        "        ignore_invalid_inputs=True,\n",
        "        seed=seed,\n",
        "        num_workers=num_workers,\n",
        "        epoch=epoch,\n",
        "        disable_iterator_cache=not cached,\n",
        "        # Set this to False to speed up. However, if set to False, changing max_tokens beyond \n",
        "        # first call of this method has no effect. \n",
        "    )\n",
        "    return batch_iterator\n",
        "\n",
        "demo_epoch_obj = load_data_iterator(task, \"valid\", epoch=1, max_tokens=20, num_workers=1, cached=False)\n",
        "demo_iter = demo_epoch_obj.next_epoch_itr(shuffle=True)\n",
        "sample = next(demo_iter)\n",
        "sample"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sMy8S7OKP7CF"
      },
      "source": [
        "* each batch is a python dict, with string key and Tensor value. Contents are described below:\n",
        "```python\n",
        "batch = {\n",
        "    \"id\": id, # id for each example \n",
        "    \"nsentences\": len(samples), # batch size (sentences)\n",
        "    \"ntokens\": ntokens, # batch size (tokens)\n",
        "    \"net_input\": {\n",
        "        \"src_tokens\": src_tokens, # sequence in source language\n",
        "        \"src_lengths\": src_lengths, # sequence length of each example before padding\n",
        "        \"prev_output_tokens\": prev_output_tokens, # right shifted target, as mentioned above.\n",
        "    },\n",
        "    \"target\": target, # target sequence\n",
        "}\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ChE3aHCYP7CF"
      },
      "source": [
        "# Model Architecture\n",
        "* We again inherit fairseq's encoder, decoder and model, so that in the testing phase we can directly leverage fairseq's beam search decoder."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QyLbkvn8P7CF"
      },
      "source": [
        "from fairseq.models import (\n",
        "    FairseqEncoder, \n",
        "    FairseqIncrementalDecoder,\n",
        "    FairseqEncoderDecoderModel\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CrenFTn5P7CF"
      },
      "source": [
        "## Encoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aryPT8jMP7CF"
      },
      "source": [
        "- The Encoder is a RNN or Transformer Encoder. The following description is for RNN. For every input token, Encoder will generate a output vector and a hidden states vector, and the hidden states vector is passed on to the next step. In other words, the Encoder sequentially reads in the input sequence, and outputs a single vector at each timestep, then finally outputs the final hidden states, or content vector, at the last timestep.\n",
        "- Parameters:\n",
        "  - *args*\n",
        "      - encoder_embed_dim: the dimension of embeddings, this compresses the one-hot vector into fixed dimensions, which achieves dimension reduction\n",
        "      - encoder_ffn_embed_dim is the dimension of hidden states and output vectors\n",
        "      - encoder_layers is the number of layers for Encoder RNN\n",
        "      - dropout determines the probability of a neuron's activation being set to 0, in order to prevent overfitting. Generally this is applied in training, and removed in testing.\n",
        "  - *dictionary*: the dictionary provided by fairseq. it's used to obtain the padding index, and in turn the encoder padding mask. \n",
        "  - *embed_tokens*: an instance of token embeddings (nn.Embedding)\n",
        "\n",
        "- Inputs: \n",
        "    - *src_tokens*: integer sequence representing english e.g. 1, 28, 29, 205, 2 \n",
        "- Outputs: \n",
        "    - *outputs*: the output of RNN at each timestep, can be furthur processed by Attention\n",
        "    - *final_hiddens*: the hidden states of each timestep, will be passed to decoder for decoding\n",
        "    - *encoder_padding_mask*: this tells the decoder which position to ignore\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a9MUo6b7P7CG"
      },
      "source": [
        "class RNNEncoder(FairseqEncoder):\n",
        "    def __init__(self, args, dictionary, embed_tokens):\n",
        "        super().__init__(dictionary)\n",
        "        self.embed_tokens = embed_tokens\n",
        "        \n",
        "        self.embed_dim = args.encoder_embed_dim\n",
        "        self.hidden_dim = args.encoder_ffn_embed_dim\n",
        "        self.num_layers = args.encoder_layers\n",
        "        \n",
        "        self.dropout_in_module = nn.Dropout(args.dropout)\n",
        "        self.rnn = nn.GRU(\n",
        "            self.embed_dim, \n",
        "            self.hidden_dim, \n",
        "            self.num_layers, \n",
        "            dropout=args.dropout, \n",
        "            batch_first=False, \n",
        "            bidirectional=True\n",
        "        )\n",
        "        self.dropout_out_module = nn.Dropout(args.dropout)\n",
        "        \n",
        "        self.padding_idx = dictionary.pad()\n",
        "        \n",
        "    def combine_bidir(self, outs, bsz: int):\n",
        "        out = outs.view(self.num_layers, 2, bsz, -1).transpose(1, 2).contiguous()\n",
        "        return out.view(self.num_layers, bsz, -1)\n",
        "\n",
        "    def forward(self, src_tokens, **unused):\n",
        "        bsz, seqlen = src_tokens.size()\n",
        "        \n",
        "        # get embeddings\n",
        "        x = self.embed_tokens(src_tokens)\n",
        "        x = self.dropout_in_module(x)\n",
        "\n",
        "        # B x T x C -> T x B x C\n",
        "        x = x.transpose(0, 1)\n",
        "        \n",
        "        # pass thru bidirectional RNN\n",
        "        h0 = x.new_zeros(2 * self.num_layers, bsz, self.hidden_dim)\n",
        "        x, final_hiddens = self.rnn(x, h0)\n",
        "        outputs = self.dropout_out_module(x)\n",
        "        # outputs = [sequence len, batch size, hid dim * directions]\n",
        "        # hidden =  [num_layers * directions, batch size  , hid dim]\n",
        "        \n",
        "        # Since Encoder is bidirectional, we need to concatenate the hidden states of two directions\n",
        "        final_hiddens = self.combine_bidir(final_hiddens, bsz)\n",
        "        # hidden =  [num_layers x batch x num_directions*hidden]\n",
        "        \n",
        "        encoder_padding_mask = src_tokens.eq(self.padding_idx).t()\n",
        "        return tuple(\n",
        "            (\n",
        "                outputs,  # seq_len x batch x hidden\n",
        "                final_hiddens,  # num_layers x batch x num_directions*hidden\n",
        "                encoder_padding_mask,  # seq_len x batch\n",
        "            )\n",
        "        )\n",
        "    \n",
        "    def reorder_encoder_out(self, encoder_out, new_order):\n",
        "        # This is used by fairseq's beam search. How and why is not particularly important here.\n",
        "        return tuple(\n",
        "            (\n",
        "                encoder_out[0].index_select(1, new_order),\n",
        "                encoder_out[1].index_select(1, new_order),\n",
        "                encoder_out[2].index_select(1, new_order),\n",
        "            )\n",
        "        )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Svvi0DJDP7CG"
      },
      "source": [
        "## Attention"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0D_WMN_oP7CG"
      },
      "source": [
        "- When the input sequence is long, \"content vector\" alone cannot accurately represent the whole sequence, attention mechanism can provide the Decoder more information.\n",
        "- According to the **Decoder embeddings** of the current timestep, match the **Encoder outputs** with decoder embeddings to determine correlation, and then sum the Encoder outputs weighted by the correlation as the input to **Decoder** RNN.\n",
        "- Common attention implementations use neural network / dot product as the correlation between **query** (decoder embeddings) and **key** (Encoder outputs), followed by **softmax**  to obtain a distribution, and finally **values** (Encoder outputs) is **weighted sum**-ed by said distribution.\n",
        "\n",
        "- Parameters:\n",
        "  - *input_embed_dim*: dimensionality of key, should be that of the vector in decoder to attend others\n",
        "  - *source_embed_dim*: dimensionality of query, should be that of the vector to be attended to (encoder outputs)\n",
        "  - *output_embed_dim*: dimensionality of value, should be that of the vector after attention, expected by the next layer\n",
        "\n",
        "- Inputs: \n",
        "    - *inputs*: is the key, the vector to attend to others\n",
        "    - *encoder_outputs*:  is the query/value, the vector to be attended to\n",
        "    - *encoder_padding_mask*: this tells the decoder which position to ignore\n",
        "- Outputs: \n",
        "    - *output*: the context vector after attention\n",
        "    - *attention score*: the attention distribution\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-MZcOWguP7CG"
      },
      "source": [
        "class AttentionLayer(nn.Module):\n",
        "    def __init__(self, input_embed_dim, source_embed_dim, output_embed_dim, bias=False):\n",
        "        super().__init__()\n",
        "\n",
        "        self.input_proj = nn.Linear(input_embed_dim, source_embed_dim, bias=bias)\n",
        "        self.output_proj = nn.Linear(\n",
        "            input_embed_dim + source_embed_dim, output_embed_dim, bias=bias\n",
        "        )\n",
        "\n",
        "    def forward(self, inputs, encoder_outputs, encoder_padding_mask):\n",
        "        # inputs: T, B, dim\n",
        "        # encoder_outputs: S x B x dim\n",
        "        # padding mask:  S x B\n",
        "        \n",
        "        # convert all to batch first\n",
        "        inputs = inputs.transpose(1,0) # B, T, dim\n",
        "        encoder_outputs = encoder_outputs.transpose(1,0) # B, S, dim\n",
        "        encoder_padding_mask = encoder_padding_mask.transpose(1,0) # B, S\n",
        "        \n",
        "        # project to the dimensionality of encoder_outputs\n",
        "        x = self.input_proj(inputs)\n",
        "\n",
        "        # compute attention\n",
        "        # (B, T, dim) x (B, dim, S) = (B, T, S)\n",
        "        attn_scores = torch.bmm(x, encoder_outputs.transpose(1,2))\n",
        "\n",
        "        # cancel the attention at positions corresponding to padding\n",
        "        if encoder_padding_mask is not None:\n",
        "            # leveraging broadcast  B, S -> (B, 1, S)\n",
        "            encoder_padding_mask = encoder_padding_mask.unsqueeze(1)\n",
        "            attn_scores = (\n",
        "                attn_scores.float()\n",
        "                .masked_fill_(encoder_padding_mask, float(\"-inf\"))\n",
        "                .type_as(attn_scores)\n",
        "            )  # FP16 support: cast to float and back\n",
        "\n",
        "        # softmax on the dimension corresponding to source sequence\n",
        "        attn_scores = F.softmax(attn_scores, dim=-1)\n",
        "\n",
        "        # shape (B, T, S) x (B, S, dim) = (B, T, dim) weighted sum\n",
        "        x = torch.bmm(attn_scores, encoder_outputs)\n",
        "\n",
        "        # (B, T, dim)\n",
        "        x = torch.cat((x, inputs), dim=-1)\n",
        "        x = torch.tanh(self.output_proj(x)) # concat + linear + tanh\n",
        "        \n",
        "        # restore shape (B, T, dim) -> (T, B, dim)\n",
        "        return x.transpose(1,0), attn_scores"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nB99A7uaP7CG"
      },
      "source": [
        "## Decoder"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "roXAwF9sP7CG"
      },
      "source": [
        "* The hidden states of **Decoder** will be initialized by the final hidden states of **Encoder** (the content vector)\n",
        "* At the same time, **Decoder** will change its hidden states based on the input of the current timestep (the outputs of previous timesteps), and generates an output\n",
        "* Attention improves the performance\n",
        "* The seq2seq steps are implemented in decoder, so that later the Seq2Seq class can accept RNN and Transformer, without furthur modification.\n",
        "- Parameters:\n",
        "  - *args*\n",
        "      - decoder_embed_dim: is the dimensionality of the decoder embeddings, similar to encoder_embed_dim，\n",
        "      - decoder_ffn_embed_dim: is the dimensionality of the decoder RNN hidden states, similar to encoder_ffn_embed_dim\n",
        "      - decoder_layers: number of layers of RNN decoder\n",
        "      - share_decoder_input_output_embed: usually, the projection matrix of the decoder will share weights with the decoder input embeddings\n",
        "  - *dictionary*: the dictionary provided by fairseq\n",
        "  - *embed_tokens*: an instance of token embeddings (nn.Embedding)\n",
        "- Inputs: \n",
        "    - *prev_output_tokens*: integer sequence representing the right-shifted target e.g. 1, 28, 29, 205, 2 \n",
        "    - *encoder_out*: encoder's output.\n",
        "    - *incremental_state*: in order to speed up decoding during test time, we will save the hidden state of each timestep. see forward() for details.\n",
        "- Outputs: \n",
        "    - *outputs*: the logits (before softmax) output of decoder for each timesteps\n",
        "    - *extra*: unsused"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XkG01sDDP7CH"
      },
      "source": [
        "class RNNDecoder(FairseqIncrementalDecoder):\n",
        "    def __init__(self, args, dictionary, embed_tokens):\n",
        "        super().__init__(dictionary)\n",
        "        self.embed_tokens = embed_tokens\n",
        "        \n",
        "        assert args.decoder_layers == args.encoder_layers, f\"\"\"seq2seq rnn requires that encoder \n",
        "        and decoder have same layers of rnn. got: {args.encoder_layers, args.decoder_layers}\"\"\"\n",
        "        assert args.decoder_ffn_embed_dim == args.encoder_ffn_embed_dim*2, f\"\"\"seq2seq-rnn requires \n",
        "        that decoder hidden to be 2*encoder hidden dim. got: {args.decoder_ffn_embed_dim, args.encoder_ffn_embed_dim*2}\"\"\"\n",
        "        \n",
        "        self.embed_dim = args.decoder_embed_dim\n",
        "        self.hidden_dim = args.decoder_ffn_embed_dim\n",
        "        self.num_layers = args.decoder_layers\n",
        "        \n",
        "        \n",
        "        self.dropout_in_module = nn.Dropout(args.dropout)\n",
        "        self.rnn = nn.GRU(\n",
        "            self.embed_dim, \n",
        "            self.hidden_dim, \n",
        "            self.num_layers, \n",
        "            dropout=args.dropout, \n",
        "            batch_first=False, \n",
        "            bidirectional=False\n",
        "        )\n",
        "        self.attention = AttentionLayer(\n",
        "            self.embed_dim, self.hidden_dim, self.embed_dim, bias=False\n",
        "        ) \n",
        "        # self.attention = None\n",
        "        self.dropout_out_module = nn.Dropout(args.dropout)\n",
        "        \n",
        "        if self.hidden_dim != self.embed_dim:\n",
        "            self.project_out_dim = nn.Linear(self.hidden_dim, self.embed_dim)\n",
        "        else:\n",
        "            self.project_out_dim = None\n",
        "        \n",
        "        if args.share_decoder_input_output_embed:\n",
        "            self.output_projection = nn.Linear(\n",
        "                self.embed_tokens.weight.shape[1],\n",
        "                self.embed_tokens.weight.shape[0],\n",
        "                bias=False,\n",
        "            )\n",
        "            self.output_projection.weight = self.embed_tokens.weight\n",
        "        else:\n",
        "            self.output_projection = nn.Linear(\n",
        "                self.output_embed_dim, len(dictionary), bias=False\n",
        "            )\n",
        "            nn.init.normal_(\n",
        "                self.output_projection.weight, mean=0, std=self.output_embed_dim ** -0.5\n",
        "            )\n",
        "        \n",
        "    def forward(self, prev_output_tokens, encoder_out, incremental_state=None, **unused):\n",
        "        # extract the outputs from encoder\n",
        "        encoder_outputs, encoder_hiddens, encoder_padding_mask = encoder_out\n",
        "        # outputs:          seq_len x batch x num_directions*hidden\n",
        "        # encoder_hiddens:  num_layers x batch x num_directions*encoder_hidden\n",
        "        # padding_mask:     seq_len x batch\n",
        "        \n",
        "        if incremental_state is not None and len(incremental_state) > 0:\n",
        "            # if the information from last timestep is retained, we can continue from there instead of starting from bos\n",
        "            prev_output_tokens = prev_output_tokens[:, -1:]\n",
        "            cache_state = self.get_incremental_state(incremental_state, \"cached_state\")\n",
        "            prev_hiddens = cache_state[\"prev_hiddens\"]\n",
        "        else:\n",
        "            # incremental state does not exist, either this is training time, or the first timestep of test time\n",
        "            # prepare for seq2seq: pass the encoder_hidden to the decoder hidden states\n",
        "            prev_hiddens = encoder_hiddens\n",
        "        \n",
        "        bsz, seqlen = prev_output_tokens.size()\n",
        "        \n",
        "        # embed tokens\n",
        "        x = self.embed_tokens(prev_output_tokens)\n",
        "        x = self.dropout_in_module(x)\n",
        "\n",
        "        # B x T x C -> T x B x C\n",
        "        x = x.transpose(0, 1)\n",
        "                \n",
        "        # decoder-to-encoder attention\n",
        "        if self.attention is not None:\n",
        "            x, attn = self.attention(x, encoder_outputs, encoder_padding_mask)\n",
        "                        \n",
        "        # pass thru unidirectional RNN\n",
        "        x, final_hiddens = self.rnn(x, prev_hiddens)\n",
        "        # outputs = [sequence len, batch size, hid dim]\n",
        "        # hidden =  [num_layers * directions, batch size  , hid dim]\n",
        "        x = self.dropout_out_module(x)\n",
        "                \n",
        "        # project to embedding size (if hidden differs from embed size, and share_embedding is True, \n",
        "        # we need to do an extra projection)\n",
        "        if self.project_out_dim != None:\n",
        "            x = self.project_out_dim(x)\n",
        "        \n",
        "        # project to vocab size\n",
        "        x = self.output_projection(x)\n",
        "        \n",
        "        # T x B x C -> B x T x C\n",
        "        x = x.transpose(1, 0)\n",
        "        \n",
        "        # if incremental, record the hidden states of current timestep, which will be restored in the next timestep\n",
        "        cache_state = {\n",
        "            \"prev_hiddens\": final_hiddens,\n",
        "        }\n",
        "        self.set_incremental_state(incremental_state, \"cached_state\", cache_state)\n",
        "        \n",
        "        return x, None\n",
        "    \n",
        "    def reorder_incremental_state(\n",
        "        self,\n",
        "        incremental_state,\n",
        "        new_order,\n",
        "    ):\n",
        "        # This is used by fairseq's beam search. How and why is not particularly important here.\n",
        "        cache_state = self.get_incremental_state(incremental_state, \"cached_state\")\n",
        "        prev_hiddens = cache_state[\"prev_hiddens\"]\n",
        "        prev_hiddens = [p.index_select(0, new_order) for p in prev_hiddens]\n",
        "        cache_state = {\n",
        "            \"prev_hiddens\": torch.stack(prev_hiddens),\n",
        "        }\n",
        "        self.set_incremental_state(incremental_state, \"cached_state\", cache_state)\n",
        "        return"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jl16hCW-P7CH"
      },
      "source": [
        "## Seq2Seq\n",
        "- Composed of **Encoder** and **Decoder**\n",
        "- Recieves inputs and pass to **Encoder** \n",
        "- Pass the outputs from **Encoder** to **Decoder**\n",
        "- **Decoder** will decode according to outputs of previous timesteps as well as **Encoder** outputs  \n",
        "- Once done decoding, return the **Decoder** outputs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "COzYYH5wP7CH"
      },
      "source": [
        "class Seq2Seq(FairseqEncoderDecoderModel):\n",
        "    def __init__(self, args, encoder, decoder):\n",
        "        super().__init__(encoder, decoder)\n",
        "        self.args = args\n",
        "    \n",
        "    def forward(\n",
        "        self,\n",
        "        src_tokens,\n",
        "        src_lengths,\n",
        "        prev_output_tokens,\n",
        "        return_all_hiddens: bool = True,\n",
        "    ):\n",
        "        \"\"\"\n",
        "        Run the forward pass for an encoder-decoder model.\n",
        "        \"\"\"\n",
        "        encoder_out = self.encoder(\n",
        "            src_tokens, src_lengths=src_lengths, return_all_hiddens=return_all_hiddens\n",
        "        )\n",
        "        logits, extra = self.decoder(\n",
        "            prev_output_tokens,\n",
        "            encoder_out=encoder_out,\n",
        "            src_lengths=src_lengths,\n",
        "            return_all_hiddens=return_all_hiddens,\n",
        "        )\n",
        "        return logits, extra"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sK7YDlv-P7CH"
      },
      "source": [
        "# Model Initialization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wUC6Fg_uP7CH"
      },
      "source": [
        "# # HINT: transformer architecture\n",
        "# from fairseq.models.transformer import (\n",
        "#     TransformerEncoder, \n",
        "#     TransformerDecoder,\n",
        "# )\n",
        "\n",
        "def build_model(args, task):\n",
        "    \"\"\" build a model instance based on hyperparameters \"\"\"\n",
        "    src_dict, tgt_dict = task.source_dictionary, task.target_dictionary\n",
        "\n",
        "    # token embeddings\n",
        "    encoder_embed_tokens = nn.Embedding(len(src_dict), args.encoder_embed_dim, src_dict.pad())\n",
        "    decoder_embed_tokens = nn.Embedding(len(tgt_dict), args.decoder_embed_dim, tgt_dict.pad())\n",
        "    \n",
        "    # encoder decoder\n",
        "    # HINT: TODO: switch to TransformerEncoder & TransformerDecoder\n",
        "    encoder = RNNEncoder(args, src_dict, encoder_embed_tokens)\n",
        "    decoder = RNNDecoder(args, tgt_dict, decoder_embed_tokens)\n",
        "    \n",
        "    # sequence to sequence model\n",
        "    model = Seq2Seq(args, encoder, decoder)\n",
        "    \n",
        "    # initialization for seq2seq model is important, requires extra handling\n",
        "    def init_params(module):\n",
        "        from fairseq.modules import MultiheadAttention\n",
        "        if isinstance(module, nn.Linear):\n",
        "            module.weight.data.normal_(mean=0.0, std=0.02)\n",
        "            if module.bias is not None:\n",
        "                module.bias.data.zero_()\n",
        "        if isinstance(module, nn.Embedding):\n",
        "            module.weight.data.normal_(mean=0.0, std=0.02)\n",
        "            if module.padding_idx is not None:\n",
        "                module.weight.data[module.padding_idx].zero_()\n",
        "        if isinstance(module, MultiheadAttention):\n",
        "            module.q_proj.weight.data.normal_(mean=0.0, std=0.02)\n",
        "            module.k_proj.weight.data.normal_(mean=0.0, std=0.02)\n",
        "            module.v_proj.weight.data.normal_(mean=0.0, std=0.02)\n",
        "        if isinstance(module, nn.RNNBase):\n",
        "            for name, param in module.named_parameters():\n",
        "                if \"weight\" in name or \"bias\" in name:\n",
        "                    param.data.uniform_(-0.1, 0.1)\n",
        "            \n",
        "    # weight initialization\n",
        "    model.apply(init_params)\n",
        "    return model"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z4g7sZJeP7CI"
      },
      "source": [
        "## Architecture Related Configuration\n",
        "reference implementation\n",
        "\n",
        "|model|embedding dim|encoder ffn|encoder layers|decoder ffn|decoder layers|\n",
        "|-|-|-|-|-|-|\n",
        "|RNN|256|512|1|1024|1|\n",
        "|Transformer|256|1024|4|1024|4|\n",
        "\n",
        "For strong baseline, please refer to the hyperparameters for *transformer-base* in Table 3 in [Attention is all you need](#vaswani2017)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pCxEjMVjP7CI"
      },
      "source": [
        "arch_args = Namespace(\n",
        "    encoder_embed_dim=256,\n",
        "    encoder_ffn_embed_dim=512,\n",
        "    encoder_layers=1,\n",
        "    decoder_embed_dim=256,\n",
        "    decoder_ffn_embed_dim=1024,\n",
        "    decoder_layers=1,\n",
        "    share_decoder_input_output_embed=True,\n",
        "    dropout=0.3,\n",
        ")\n",
        "\n",
        "# # HINT: these patches on parameters for Transformer\n",
        "# def add_transformer_args(args):\n",
        "#     args.encoder_attention_heads=4\n",
        "#     args.encoder_normalize_before=True\n",
        "    \n",
        "#     args.decoder_attention_heads=4\n",
        "#     args.decoder_normalize_before=True\n",
        "    \n",
        "#     args.activation_fn=\"relu\"\n",
        "#     args.max_source_positions=1024\n",
        "#     args.max_target_positions=1024\n",
        "    \n",
        "#     # patches on default parameters for Transformer (those not set above)\n",
        "#     from fairseq.models.transformer import base_architecture\n",
        "#     base_architecture(arch_args)\n",
        "\n",
        "# add_transformer_args(arch_args)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yhXW-nUqP7CI"
      },
      "source": [
        "if config.use_wandb:\n",
        "    wandb.config.update(vars(arch_args))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pc-eB4BKP7CI"
      },
      "source": [
        "model = build_model(arch_args, task)\n",
        "logger.info(model)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ev0PmgCpP7CI"
      },
      "source": [
        "# Optimization"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-eJBl2qnP7CI"
      },
      "source": [
        "## Loss: Label Smoothing Regularization\n",
        "* let the model learn to generate less concentrated distribution, and prevent over-confidence\n",
        "* sometimes the ground truth may not be the only answer. thus, when calculating loss, we reserve some probability for incorrect labels\n",
        "* avoids overfitting\n",
        "\n",
        "code [source](https://fairseq.readthedocs.io/en/latest/_modules/fairseq/criterions/label_smoothed_cross_entropy.html)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jeq3CuEXP7CI"
      },
      "source": [
        "class LabelSmoothedCrossEntropyCriterion(nn.Module):\n",
        "    def __init__(self, smoothing, ignore_index=None, reduce=True):\n",
        "        super().__init__()\n",
        "        self.smoothing = smoothing\n",
        "        self.ignore_index = ignore_index\n",
        "        self.reduce = reduce\n",
        "    \n",
        "    def forward(self, lprobs, target):\n",
        "        if target.dim() == lprobs.dim() - 1:\n",
        "            target = target.unsqueeze(-1)\n",
        "        # nll: Negative log likelihood，the cross-entropy when target is one-hot. following line is same as F.nll_loss\n",
        "        nll_loss = -lprobs.gather(dim=-1, index=target)\n",
        "        #  reserve some probability for other labels. thus when calculating cross-entropy, \n",
        "        # equivalent to summing the log probs of all labels\n",
        "        smooth_loss = -lprobs.sum(dim=-1, keepdim=True)\n",
        "        if self.ignore_index is not None:\n",
        "            pad_mask = target.eq(self.ignore_index)\n",
        "            nll_loss.masked_fill_(pad_mask, 0.0)\n",
        "            smooth_loss.masked_fill_(pad_mask, 0.0)\n",
        "        else:\n",
        "            nll_loss = nll_loss.squeeze(-1)\n",
        "            smooth_loss = smooth_loss.squeeze(-1)\n",
        "        if self.reduce:\n",
        "            nll_loss = nll_loss.sum()\n",
        "            smooth_loss = smooth_loss.sum()\n",
        "        # when calculating cross-entropy, add the loss of other labels\n",
        "        eps_i = self.smoothing / lprobs.size(-1)\n",
        "        loss = (1.0 - self.smoothing) * nll_loss + eps_i * smooth_loss\n",
        "        return loss\n",
        "\n",
        "# generally, 0.1 is good enough\n",
        "criterion = LabelSmoothedCrossEntropyCriterion(\n",
        "    smoothing=0.1,\n",
        "    ignore_index=task.target_dictionary.pad(),\n",
        ")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zshbrr8tP7CJ"
      },
      "source": [
        "## Optimizer: Adam + lr scheduling\n",
        "Inverse square root scheduling is important to the stability when training Transformer. It's later used on RNN as well.\n",
        "Update the learning rate according to the following equation. Linearly increase the first stage, then decay proportionally to the inverse square root of timestep.\n",
        "$$lrate = d_{\\text{model}}^{-0.5}\\cdot\\min({step\\_num}^{-0.5},{step\\_num}\\cdot{warmup\\_steps}^{-1.5})$$\n",
        "code [source](https://nlp.seas.harvard.edu/2018/04/03/attention.html)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "R5mOsZaRP7CJ"
      },
      "source": [
        "class NoamOpt:\n",
        "    \"Optim wrapper that implements rate.\"\n",
        "    def __init__(self, model_size, factor, warmup, optimizer):\n",
        "        self.optimizer = optimizer\n",
        "        self._step = 0\n",
        "        self.warmup = warmup\n",
        "        self.factor = factor\n",
        "        self.model_size = model_size\n",
        "        self._rate = 0\n",
        "    \n",
        "    @property\n",
        "    def param_groups(self):\n",
        "        return self.optimizer.param_groups\n",
        "        \n",
        "    def multiply_grads(self, c):\n",
        "        \"\"\"Multiplies grads by a constant *c*.\"\"\"                \n",
        "        for group in self.param_groups:\n",
        "            for p in group['params']:\n",
        "                if p.grad is not None:\n",
        "                    p.grad.data.mul_(c)\n",
        "        \n",
        "    def step(self):\n",
        "        \"Update parameters and rate\"\n",
        "        self._step += 1\n",
        "        rate = self.rate()\n",
        "        for p in self.param_groups:\n",
        "            p['lr'] = rate\n",
        "        self._rate = rate\n",
        "        self.optimizer.step()\n",
        "        \n",
        "    def rate(self, step = None):\n",
        "        \"Implement `lrate` above\"\n",
        "        if step is None:\n",
        "            step = self._step\n",
        "        return 0 if not step else self.factor * \\\n",
        "            (self.model_size ** (-0.5) *\n",
        "            min(step ** (-0.5), step * self.warmup ** (-1.5)))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9BMvu2X2P7CJ"
      },
      "source": [
        "## Scheduling Visualized"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BfJnRqdGP7CJ"
      },
      "source": [
        "optimizer = NoamOpt(\n",
        "    model_size=arch_args.encoder_embed_dim, \n",
        "    factor=config.lr_factor, \n",
        "    warmup=config.lr_warmup, \n",
        "    optimizer=torch.optim.AdamW(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9, weight_decay=0.0001))\n",
        "plt.plot(np.arange(1, 100000), [optimizer.rate(i) for i in range(1, 100000)])\n",
        "plt.legend([f\"{optimizer.model_size}:{optimizer.warmup}\"])\n",
        "None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hRI-vU55P7CJ"
      },
      "source": [
        "# Training Procedure"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvP5h5PUP7CJ"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s5-iLNeLP7CJ"
      },
      "source": [
        "from fairseq.data import iterators\n",
        "from torch.cuda.amp import GradScaler, autocast\n",
        "\n",
        "def train_one_epoch(epoch_itr, model, task, criterion, optimizer, accum_steps=1):\n",
        "    itr = epoch_itr.next_epoch_itr(shuffle=True)\n",
        "    itr = iterators.GroupedIterator(itr, accum_steps) # gradient accumulation: update every accum_steps samples\n",
        "    \n",
        "    stats = {\"loss\": []}\n",
        "    scaler = GradScaler() # automatic mixed precision (amp) \n",
        "    \n",
        "    model.train()\n",
        "    progress = tqdm.tqdm(itr, desc=f\"train epoch {epoch_itr.epoch}\", leave=False)\n",
        "    for samples in progress:\n",
        "        model.zero_grad()\n",
        "        accum_loss = 0\n",
        "        sample_size = 0\n",
        "        # gradient accumulation: update every accum_steps samples\n",
        "        for i, sample in enumerate(samples):\n",
        "            if i == 1:\n",
        "                # emptying the CUDA cache after the first step can reduce the chance of OOM\n",
        "                torch.cuda.empty_cache()\n",
        "\n",
        "            sample = utils.move_to_cuda(sample, device=device)\n",
        "            target = sample[\"target\"]\n",
        "            sample_size_i = sample[\"ntokens\"]\n",
        "            sample_size += sample_size_i\n",
        "            \n",
        "            # mixed precision training\n",
        "            with autocast():\n",
        "                net_output = model.forward(**sample[\"net_input\"])\n",
        "                lprobs = F.log_softmax(net_output[0], -1)            \n",
        "                loss = criterion(lprobs.view(-1, lprobs.size(-1)), target.view(-1))\n",
        "                \n",
        "                # logging\n",
        "                accum_loss += loss.item()\n",
        "                # back-prop\n",
        "                scaler.scale(loss).backward()                \n",
        "        \n",
        "        scaler.unscale_(optimizer)\n",
        "        optimizer.multiply_grads(1 / (sample_size or 1.0)) # (sample_size or 1.0) handles the case of a zero gradient\n",
        "        gnorm = nn.utils.clip_grad_norm_(model.parameters(), config.clip_norm) # grad norm clipping prevents gradient exploding\n",
        "        \n",
        "        scaler.step(optimizer)\n",
        "        scaler.update()\n",
        "        \n",
        "        # logging\n",
        "        loss_print = accum_loss/sample_size\n",
        "        stats[\"loss\"].append(loss_print)\n",
        "        progress.set_postfix(loss=loss_print)\n",
        "        if config.use_wandb:\n",
        "            wandb.log({\n",
        "                \"train/loss\": loss_print,\n",
        "                \"train/grad_norm\": gnorm.item(),\n",
        "                \"train/lr\": optimizer.rate(),\n",
        "                \"train/sample_size\": sample_size,\n",
        "            })\n",
        "        \n",
        "    loss_print = np.mean(stats[\"loss\"])\n",
        "    logger.info(f\"training loss: {loss_print:.4f}\")\n",
        "    return stats"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dqUy6lnYP7CK"
      },
      "source": [
        "## Validation & Inference\n",
        "To prevent overfitting, validation is required every epoch to validate the performance on unseen data.\n",
        "- the procedure is essensially same as training, with the addition of inference step\n",
        "- after validation we can save the model weights\n",
        "\n",
        "Validation loss alone cannot describe the actual performance of the model\n",
        "- Directly produce translation hypotheses based on current model, then calculate BLEU with the reference translation\n",
        "- We can also manually examine the hypotheses' quality\n",
        "- We use fairseq's sequence generator for beam search to generate translation hypotheses"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5_B2g5CQP7CK"
      },
      "source": [
        "# fairseq's beam search generator\n",
        "# given model and input seqeunce, produce translation hypotheses by beam search\n",
        "sequence_generator = task.build_generator([model], config)\n",
        "\n",
        "def decode(toks, dictionary):\n",
        "    # convert from Tensor to human readable sentence\n",
        "    s = dictionary.string(\n",
        "        toks.int().cpu(),\n",
        "        config.post_process,\n",
        "    )\n",
        "    return s if s else \"<unk>\"\n",
        "\n",
        "def inference_step(sample, model):\n",
        "    gen_out = sequence_generator.generate([model], sample)\n",
        "    srcs = []\n",
        "    hyps = []\n",
        "    refs = []\n",
        "    for i in range(len(gen_out)):\n",
        "        # for each sample, collect the input, hypothesis and reference, later be used to calculate BLEU\n",
        "        srcs.append(decode(\n",
        "            utils.strip_pad(sample[\"net_input\"][\"src_tokens\"][i], task.source_dictionary.pad()), \n",
        "            task.source_dictionary,\n",
        "        ))\n",
        "        hyps.append(decode(\n",
        "            gen_out[i][0][\"tokens\"], # 0 indicates using the top hypothesis in beam\n",
        "            task.target_dictionary,\n",
        "        ))\n",
        "        refs.append(decode(\n",
        "            utils.strip_pad(sample[\"target\"][i], task.target_dictionary.pad()), \n",
        "            task.target_dictionary,\n",
        "        ))\n",
        "    return srcs, hyps, refs"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z0Dkv8IjP7CK"
      },
      "source": [
        "import shutil\n",
        "import sacrebleu\n",
        "\n",
        "def validate(model, task, criterion, log_to_wandb=True):\n",
        "    logger.info('begin validation')\n",
        "    itr = load_data_iterator(task, \"valid\", 1, config.max_tokens, config.num_workers).next_epoch_itr(shuffle=False)\n",
        "    \n",
        "    stats = {\"loss\":[], \"bleu\": 0, \"srcs\":[], \"hyps\":[], \"refs\":[]}\n",
        "    srcs = []\n",
        "    hyps = []\n",
        "    refs = []\n",
        "    \n",
        "    model.eval()\n",
        "    progress = tqdm.tqdm(itr, desc=f\"validation\", leave=False)\n",
        "    with torch.no_grad():\n",
        "        for i, sample in enumerate(progress):\n",
        "            # validation loss\n",
        "            sample = utils.move_to_cuda(sample, device=device)\n",
        "            net_output = model.forward(**sample[\"net_input\"])\n",
        "\n",
        "            lprobs = F.log_softmax(net_output[0], -1)\n",
        "            target = sample[\"target\"]\n",
        "            sample_size = sample[\"ntokens\"]\n",
        "            loss = criterion(lprobs.view(-1, lprobs.size(-1)), target.view(-1)) / sample_size\n",
        "            progress.set_postfix(valid_loss=loss.item())\n",
        "            stats[\"loss\"].append(loss)\n",
        "            \n",
        "            # do inference\n",
        "            s, h, r = inference_step(sample, model)\n",
        "            srcs.extend(s)\n",
        "            hyps.extend(h)\n",
        "            refs.extend(r)\n",
        "            \n",
        "    tok = 'zh' if task.cfg.target_lang == 'zh' else '13a'\n",
        "    stats[\"loss\"] = torch.stack(stats[\"loss\"]).mean().item()\n",
        "    stats[\"bleu\"] = sacrebleu.corpus_bleu(hyps, [refs], tokenize=tok) # 計算BLEU score\n",
        "    stats[\"srcs\"] = srcs\n",
        "    stats[\"hyps\"] = hyps\n",
        "    stats[\"refs\"] = refs\n",
        "    \n",
        "    if config.use_wandb and log_to_wandb:\n",
        "        wandb.log({\n",
        "            \"valid/loss\": stats[\"loss\"],\n",
        "            \"valid/bleu\": stats[\"bleu\"].score,\n",
        "        }, commit=False)\n",
        "    \n",
        "    showid = np.random.randint(len(hyps))\n",
        "    logger.info(\"example source: \" + srcs[showid])\n",
        "    logger.info(\"example hypothesis: \" + hyps[showid])\n",
        "    logger.info(\"example reference: \" + refs[showid])\n",
        "    \n",
        "    # show bleu results\n",
        "    logger.info(f\"validation loss:\\t{stats['loss']:.4f}\")\n",
        "    logger.info(stats[\"bleu\"].format())\n",
        "    return stats"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hEjKIJ89P7CK"
      },
      "source": [
        "# Save and Load Model Weights"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yAIF1192P7CL"
      },
      "source": [
        "def validate_and_save(model, task, criterion, optimizer, epoch, save=True):   \n",
        "    stats = validate(model, task, criterion)\n",
        "    bleu = stats['bleu']\n",
        "    loss = stats['loss']\n",
        "    if save:\n",
        "        # save epoch checkpoints\n",
        "        savedir = Path(config.savedir).absolute()\n",
        "        savedir.mkdir(parents=True, exist_ok=True)\n",
        "        \n",
        "        check = {\n",
        "            \"model\": model.state_dict(),\n",
        "            \"stats\": {\"bleu\": bleu.score, \"loss\": loss},\n",
        "            \"optim\": {\"step\": optimizer._step}\n",
        "        }\n",
        "        torch.save(check, savedir/f\"checkpoint{epoch}.pt\")\n",
        "        shutil.copy(savedir/f\"checkpoint{epoch}.pt\", savedir/f\"checkpoint_last.pt\")\n",
        "        logger.info(f\"saved epoch checkpoint: {savedir}/checkpoint{epoch}.pt\")\n",
        "    \n",
        "        # save epoch samples\n",
        "        with open(savedir/f\"samples{epoch}.{config.source_lang}-{config.target_lang}.txt\", \"w\") as f:\n",
        "            for s, h in zip(stats[\"srcs\"], stats[\"hyps\"]):\n",
        "                f.write(f\"{s}\\t{h}\\n\")\n",
        "\n",
        "        # get best valid bleu    \n",
        "        if getattr(validate_and_save, \"best_bleu\", 0) < bleu.score:\n",
        "            validate_and_save.best_bleu = bleu.score\n",
        "            torch.save(check, savedir/f\"checkpoint_best.pt\")\n",
        "            \n",
        "        del_file = savedir / f\"checkpoint{epoch - config.keep_last_epochs}.pt\"\n",
        "        if del_file.exists():\n",
        "            del_file.unlink()\n",
        "    \n",
        "    return stats\n",
        "\n",
        "def try_load_checkpoint(model, optimizer=None, name=None):\n",
        "    name = name if name else \"checkpoint_last.pt\"\n",
        "    checkpath = Path(config.savedir)/name\n",
        "    if checkpath.exists():\n",
        "        check = torch.load(checkpath)\n",
        "        model.load_state_dict(check[\"model\"])\n",
        "        stats = check[\"stats\"]\n",
        "        step = \"unknown\"\n",
        "        if optimizer != None:\n",
        "            optimizer._step = step = check[\"optim\"][\"step\"]\n",
        "        logger.info(f\"loaded checkpoint {checkpath}: step={step} loss={stats['loss']} bleu={stats['bleu']}\")\n",
        "    else:\n",
        "        logger.info(f\"no checkpoints found at {checkpath}!\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2RIVIFGdP7CL"
      },
      "source": [
        "# Main\n",
        "## Training loop"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V3Z5YVGZP7CL"
      },
      "source": [
        "model = model.to(device=device)\n",
        "criterion = criterion.to(device=device)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JS-2grzDP7CL"
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CqilULoUP7CL"
      },
      "source": [
        "logger.info(\"task: {}\".format(task.__class__.__name__))\n",
        "logger.info(\"encoder: {}\".format(model.encoder.__class__.__name__))\n",
        "logger.info(\"decoder: {}\".format(model.decoder.__class__.__name__))\n",
        "logger.info(\"criterion: {}\".format(criterion.__class__.__name__))\n",
        "logger.info(\"optimizer: {}\".format(optimizer.__class__.__name__))\n",
        "logger.info(\n",
        "    \"num. model params: {:,} (num. trained: {:,})\".format(\n",
        "        sum(p.numel() for p in model.parameters()),\n",
        "        sum(p.numel() for p in model.parameters() if p.requires_grad),\n",
        "    )\n",
        ")\n",
        "logger.info(f\"max tokens per batch = {config.max_tokens}, accumulate steps = {config.accum_steps}\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "i2a5Wr77P7CL"
      },
      "source": [
        "epoch_itr = load_data_iterator(task, \"train\", config.start_epoch, config.max_tokens, config.num_workers)\n",
        "try_load_checkpoint(model, optimizer, name=config.resume)\n",
        "while epoch_itr.next_epoch_idx <= config.max_epoch:\n",
        "    # train for one epoch\n",
        "    train_one_epoch(epoch_itr, model, task, criterion, optimizer, config.accum_steps)\n",
        "    stats = validate_and_save(model, task, criterion, optimizer, epoch=epoch_itr.epoch)\n",
        "    logger.info(\"end of epoch {}\".format(epoch_itr.epoch))    \n",
        "    epoch_itr = load_data_iterator(task, \"train\", epoch_itr.next_epoch_idx, config.max_tokens, config.num_workers)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ggnP4imDP7CM"
      },
      "source": [
        "# Submission"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TjUddYLBP7CM"
      },
      "source": [
        "# averaging a few checkpoints can have a similar effect to ensemble\n",
        "checkdir=config.savedir\n",
        "!python ./fairseq/scripts/average_checkpoints.py \\\n",
        "--inputs {checkdir} \\\n",
        "--num-epoch-checkpoints 5 \\\n",
        "--output {checkdir}/avg_last_5_checkpoint.pt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UmNKgmPdP7CM"
      },
      "source": [
        "## Confirm model weights used to generate submission"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_fOc2CtZP7CM"
      },
      "source": [
        "# checkpoint_last.pt : latest epoch\n",
        "# checkpoint_best.pt : highest validation bleu\n",
        "# avg_last_5_checkpoint.pt:　the average of last 5 epochs\n",
        "try_load_checkpoint(model, name=\"avg_last_5_checkpoint.pt\")\n",
        "validate(model, task, criterion, log_to_wandb=False)\n",
        "None"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g5GDRX3uP7CM"
      },
      "source": [
        "## Generate Prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iAx5bCP4P7CM"
      },
      "source": [
        "def generate_prediction(model, task, split=\"test\", outfile=\"./prediction.txt\"):    \n",
        "    task.load_dataset(split=split, epoch=1)\n",
        "    itr = load_data_iterator(task, split, 1, config.max_tokens, config.num_workers).next_epoch_itr(shuffle=False)\n",
        "    \n",
        "    idxs = []\n",
        "    hyps = []\n",
        "\n",
        "    model.eval()\n",
        "    progress = tqdm.tqdm(itr, desc=f\"prediction\")\n",
        "    with torch.no_grad():\n",
        "        for i, sample in enumerate(progress):\n",
        "            # validation loss\n",
        "            sample = utils.move_to_cuda(sample, device=device)\n",
        "\n",
        "            # do inference\n",
        "            s, h, r = inference_step(sample, model)\n",
        "            \n",
        "            hyps.extend(h)\n",
        "            idxs.extend(list(sample['id']))\n",
        "            \n",
        "    # sort based on the order before preprocess\n",
        "    hyps = [x for _,x in sorted(zip(idxs,hyps))]\n",
        "    \n",
        "    with open(outfile, \"w\") as f:\n",
        "        for h in hyps:\n",
        "            f.write(h+\"\\n\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "534wB6lpP7CN"
      },
      "source": [
        "generate_prediction(model, task)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6w8VBFujP7CN"
      },
      "source": [
        "raise"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LtjcTpZNP7CN"
      },
      "source": [
        "# Back-translation"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3O1DY1XqP7CN"
      },
      "source": [
        "## Train a backward translation model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "58VncMtTP7CN"
      },
      "source": [
        "1. Switch the source_lang and target_lang in **config** \n",
        "2. Change the savedir in **config** (eg. \"./checkpoints/transformer-back\")\n",
        "3. Train model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "thHTQPU0P7CQ"
      },
      "source": [
        "## Generate synthetic data with backward model "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EYFQnjVWP7CQ"
      },
      "source": [
        "### Download monolingual data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vJmUBehkP7CQ"
      },
      "source": [
        "mono_dataset_name = 'mono'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t20PNrLbP7CQ"
      },
      "source": [
        "mono_prefix = Path(data_dir).absolute() / mono_dataset_name\n",
        "mono_prefix.mkdir(parents=True, exist_ok=True)\n",
        "\n",
        "urls = (\n",
        "    '\"https://onedrive.live.com/download?cid=3E549F3B24B238B4&resid=3E549F3B24B238B4%214986&authkey=AANUKbGfZx0kM80\"',\n",
        "# # If the above links die, use the following instead. \n",
        "#     \"https://www.csie.ntu.edu.tw/~r09922057/ML2021-hw5/ted_zh_corpus.deduped.gz\",\n",
        "# # If the above links die, use the following instead. \n",
        "#     \"https://mega.nz/#!vMNnDShR!4eHDxzlpzIpdpeQTD-htatU_C7QwcBTwGDaSeBqH534\",\n",
        ")\n",
        "file_names = (\n",
        "    'ted_zh_corpus.deduped.gz',\n",
        ")\n",
        "\n",
        "for u, f in zip(urls, file_names):\n",
        "    path = mono_prefix/f\n",
        "    if not path.exists():\n",
        "        if 'mega' in u:\n",
        "            !megadl {u} --path {path}\n",
        "        else:\n",
        "            !wget {u} -O {path}\n",
        "    else:\n",
        "        print(f'{f} is exist, skip downloading')\n",
        "    if path.suffix == \".tgz\":\n",
        "        !tar -xvf {path} -C {prefix}\n",
        "    elif path.suffix == \".zip\":\n",
        "        !unzip -o {path} -d {prefix}\n",
        "    elif path.suffix == \".gz\":\n",
        "        !gzip -fkd {path}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P1URlNrIP7CR"
      },
      "source": [
        "### TODO: clean corpus\n",
        "\n",
        "1. remove sentences that are too long or too short\n",
        "2. unify punctuation\n",
        "\n",
        "hint: you can use clean_s() defined above to do this"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wERsa5P1P7CR"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CJeaq7zyP7CR"
      },
      "source": [
        "### TODO: Subword Units\n",
        "\n",
        "Use the spm model of the backward model to tokenize the data into subword units\n",
        "\n",
        "hint: spm model is located at DATA/raw-data/\\[dataset\\]/spm\\[vocab_num\\].model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KiOg13IYP7CR"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gJe1z6gpP7CR"
      },
      "source": [
        "### Binarize\n",
        "\n",
        "use fairseq to binarize data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mUwf7LElP7CR"
      },
      "source": [
        "binpath = Path('./DATA/data-bin', mono_dataset_name)\n",
        "src_dict_file = './DATA/data-bin/ted2020/dict.en.txt'\n",
        "tgt_dict_file = src_dict_file\n",
        "monopref = str(mono_prefix/\"mono.tok\") # whatever filepath you get after applying subword tokenization\n",
        "if binpath.exists():\n",
        "    print(binpath, \"exists, will not overwrite!\")\n",
        "else:\n",
        "    !python -m fairseq_cli.preprocess\\\n",
        "        --source-lang 'zh'\\\n",
        "        --target-lang 'en'\\\n",
        "        --trainpref {monopref}\\\n",
        "        --destdir {binpath}\\\n",
        "        --srcdict {src_dict_file}\\\n",
        "        --tgtdict {tgt_dict_file}\\\n",
        "        --workers 2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kTcfJYc1P7CR"
      },
      "source": [
        "### TODO: Generate synthetic data with backward model\n",
        "\n",
        "Add binarized monolingual data to the original data directory, and name it with \"split_name\"\n",
        "\n",
        "ex. ./DATA/data-bin/ted2020/\\[split_name\\].zh-en.\\[\"en\", \"zh\"\\].\\[\"bin\", \"idx\"\\]\n",
        "\n",
        "then you can use 'generate_prediction(model, task, split=\"split_name\")' to generate translation prediction"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "__G4lyuDP7CS"
      },
      "source": [
        "# Add binarized monolingual data to the original data directory, and name it with \"split_name\"\n",
        "# ex. ./DATA/data-bin/ted2020/\\[split_name\\].zh-en.\\[\"en\", \"zh\"\\].\\[\"bin\", \"idx\"\\]\n",
        "!cp ./DATA/data-bin/mono/train.zh-en.zh.bin ./DATA/data-bin/ted2020/mono.zh-en.zh.bin\n",
        "!cp ./DATA/data-bin/mono/train.zh-en.zh.idx ./DATA/data-bin/ted2020/mono.zh-en.zh.idx\n",
        "!cp ./DATA/data-bin/mono/train.zh-en.en.bin ./DATA/data-bin/ted2020/mono.zh-en.en.bin\n",
        "!cp ./DATA/data-bin/mono/train.zh-en.en.idx ./DATA/data-bin/ted2020/mono.zh-en.en.idx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DpuwC35FP7CS"
      },
      "source": [
        "# hint: do prediction on split='mono' to create prediction_file\n",
        "# generate_prediction( ... ,split=... ,outfile=... )"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_jr2bQ4kP7CS"
      },
      "source": [
        "### TODO: Create new dataset\n",
        "\n",
        "1. Combine the prediction data with monolingual data\n",
        "2. Use the original spm model to tokenize data into Subword Units\n",
        "3. Binarize data with fairseq"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UNJ6j9TnP7CS"
      },
      "source": [
        "# Combine prediction_file (.en) and mono.zh (.zh) into a new dataset.\n",
        "# \n",
        "# hint: tokenize prediction_file with the spm model\n",
        "# spm_model.encode(line, out_type=str)\n",
        "# output: ./DATA/rawdata/mono/mono.tok.en & mono.tok.zh\n",
        "#\n",
        "# hint: use fairseq to binarize these two files again\n",
        "# binpath = Path('./DATA/data-bin/synthetic')\n",
        "# src_dict_file = './DATA/data-bin/ted2020/dict.en.txt'\n",
        "# tgt_dict_file = src_dict_file\n",
        "# monopref = ./DATA/rawdata/mono/mono.tok # or whatever path after applying subword tokenization, w/o the suffix (.zh/.en)\n",
        "# if binpath.exists():\n",
        "#     print(binpath, \"exists, will not overwrite!\")\n",
        "# else:\n",
        "#     !python -m fairseq_cli.preprocess\\\n",
        "#         --source-lang 'zh'\\\n",
        "#         --target-lang 'en'\\\n",
        "#         --trainpref {monopref}\\\n",
        "#         --destdir {binpath}\\\n",
        "#         --srcdict {src_dict_file}\\\n",
        "#         --tgtdict {tgt_dict_file}\\\n",
        "#         --workers 2"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J25am1-AP7CS"
      },
      "source": [
        "# create a new dataset from all the files prepared above\n",
        "!cp -r ./DATA/data-bin/ted2020/ ./DATA/data-bin/ted2020_with_mono/\n",
        "\n",
        "!cp ./DATA/data-bin/synthetic/train.zh-en.zh.bin ./DATA/data-bin/ted2020_with_mono/train1.en-zh.zh.bin\n",
        "!cp ./DATA/data-bin/synthetic/train.zh-en.zh.idx ./DATA/data-bin/ted2020_with_mono/train1.en-zh.zh.idx\n",
        "!cp ./DATA/data-bin/synthetic/train.zh-en.en.bin ./DATA/data-bin/ted2020_with_mono/train1.en-zh.en.bin\n",
        "!cp ./DATA/data-bin/synthetic/train.zh-en.en.idx ./DATA/data-bin/ted2020_with_mono/train1.en-zh.en.idx"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3k6K73ABP7CS"
      },
      "source": [
        "Created new dataset \"ted2020_with_mono\"\n",
        "\n",
        "1. Change the datadir in **config** (\"./DATA/data-bin/ted2020_with_mono\")\n",
        "2. Switch back the source_lang and target_lang in **config** (\"en\", \"zh\")\n",
        "2. Change the savedir in **config** (eg. \"./checkpoints/transformer-bt\")\n",
        "3. Train model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bIHgJe7fP7CT"
      },
      "source": [
        "# References"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7KRgN3IOP7CT"
      },
      "source": [
        "1. <a name=ott2019fairseq></a>Ott, M., Edunov, S., Baevski, A., Fan, A., Gross, S., Ng, N., ... & Auli, M. (2019, June). fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations) (pp. 48-53).\n",
        "2. <a name=vaswani2017></a>Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017, December). Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems (pp. 6000-6010).\n",
        "3. <a name=reimers-2020-multilingual-sentence-bert></a>Reimers, N., & Gurevych, I. (2020, November). Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 4512-4525).\n",
        "4. <a name=tiedemann2012parallel></a>Tiedemann, J. (2012, May). Parallel Data, Tools and Interfaces in OPUS. In Lrec (Vol. 2012, pp. 2214-2218).\n",
        "5. <a name=kudo-richardson-2018-sentencepiece></a>Kudo, T., & Richardson, J. (2018, November). SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 66-71).\n",
        "6. <a name=sennrich-etal-2016-improving></a>Sennrich, R., Haddow, B., & Birch, A. (2016, August). Improving Neural Machine Translation Models with Monolingual Data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 86-96).\n",
        "7. <a name=edunov-etal-2018-understanding></a>Edunov, S., Ott, M., Auli, M., & Grangier, D. (2018). Understanding Back-Translation at Scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (pp. 489-500).\n",
        "8. https://github.com/ajinkyakulkarni14/TED-Multilingual-Parallel-Corpus\n",
        "9. https://ithelp.ithome.com.tw/articles/10233122\n",
        "10. https://nlp.seas.harvard.edu/2018/04/03/attention.html"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bBZ7f9oTP7CT"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}