10_transformers-from-scratch.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF3bIp-z3HEy"
      },
      "source": [
        "# 밑바닥부터 트랜스포머 훈련하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n7L8mO363HEy"
      },
      "source": [
        "> **Note:** 이 장에서는 분산 인프라에서 대규모 언어 모델을 훈련하기 위한 대용량 데이터셋과 스크립트를 만듭니다. 따라서 이 노트북의 모든 단계가 코랩이나 캐글 같은 플랫폼에서 실행 가능한 것은 아닙니다. 중요 지점에서 단계를 축소하거나 분산 훈련 스크립트를 만들 때 참고용으로 이 노트북을 사용하세요. \n",
        "\n",
        "> **Note:** 이 책의 다른 코드와 달리 이 장의 훈련 코드는 다중 GPU에서 스크립트로 실행하도록 만들어졌습니다. CodeParrot을 직접 훈련하려면  트랜스포머스 저장소([https://oreil.ly/ZyPPR](https://oreil.ly/ZyPPR))에 있는 스크립트를 사용하는 것이 좋습니다."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Serozi53HEy"
      },
      "source": [
        "## 대규모 데이터셋 수집하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TXT32egM3HEz"
      },
      "source": [
        "### 대규모 말뭉치 구축의 어려움"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KlldrP543HEz"
      },
      "outputs": [],
      "source": [
        "from transformers import pipeline, set_seed\n",
        "\n",
        "generation_gpt = pipeline(\"text-generation\", model=\"openai-gpt\")\n",
        "generation_gpt2 = pipeline(\"text-generation\", model=\"gpt2\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YY0HKWc43HE0",
        "outputId": "bdc7df6f-067a-415e-8311-77380ef67f38"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GPT  size: 116.5M parameters\n",
            "GPT2 size: 124.4M parameters\n"
          ]
        }
      ],
      "source": [
        "def model_size(model):\n",
        "    return sum(t.numel() for t in model.parameters())\n",
        "\n",
        "print(f\"GPT  크기: {model_size(generation_gpt.model)/1000**2:.1f}M parameters\")\n",
        "print(f\"GPT2 크기: {model_size(generation_gpt2.model)/1000**2:.1f}M parameters\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rIlZ09Kh3HE1"
      },
      "outputs": [],
      "source": [
        "set_seed(1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "nXoO1vXv3HE1",
        "outputId": "08286bdc-d456-4336-a000-216eabc5bcd9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GPT completions:\n",
            "1.\n",
            "When they came back.\n",
            " \" we need all we can get, \" jason said once they had settled into the back of\n",
            "the truck without anyone stopping them. \" after getting out here, it 'll be up\n",
            "to us what to find. for now\n",
            "2.\n",
            "When they came back.\n",
            " his gaze swept over her body. he 'd dressed her, too, in the borrowed clothes\n",
            "that she 'd worn for the journey.\n",
            " \" i thought it would be easier to just leave you there. \" a woman like\n",
            "3.\n",
            "When they came back to the house and she was sitting there with the little boy.\n",
            " \" don't be afraid, \" he told her. she nodded slowly, her eyes wide. she was so\n",
            "lost in whatever she discovered that tom knew her mistake\n",
            "\n",
            "GPT-2 completions:\n",
            "1.\n",
            "When they came back we had a big dinner and the other guys went to see what\n",
            "their opinion was on her. I did an hour and they were happy with it.\n",
            "2.\n",
            "When they came back to this island there had been another massacre, but he could\n",
            "not help but feel pity for the helpless victim who had been left to die, and\n",
            "that they had failed that day. And so was very, very grateful indeed.\n",
            "3.\n",
            "When they came back to our house after the morning, I asked if she was sure. She\n",
            "said, \"Nope.\" The two kids were gone that morning. I thought they were back to\n",
            "being a good friend.\n",
            "\n",
            "When Dost\n"
          ]
        }
      ],
      "source": [
        "def enum_pipeline_ouputs(pipe, prompt, num_return_sequences):\n",
        "    out = pipe(prompt, num_return_sequences=num_return_sequences,\n",
        "               clean_up_tokenization_spaces=True)\n",
        "    return \"\\n\".join(f\"{i+1}.\" + s[\"generated_text\"] for i, s in enumerate(out))\n",
        "\n",
        "prompt = \"\\nWhen they came back\"\n",
        "print(\"GPT 자동 완성:\\n\" + enum_pipeline_ouputs(generation_gpt, prompt, 3))\n",
        "print(\"\")\n",
        "print(\"GPT-2 자동 완성:\\n\" + enum_pipeline_ouputs(generation_gpt2, prompt, 3))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3cq-ipeF3HE2"
      },
      "source": [
        "### 사용자 정의 코드 데이터셋 만들기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4P05J3Dv3HE2"
      },
      "source": [
        "#### 구글 빅쿼리로 데이터셋 만들기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XBdD9t4Z3HE3"
      },
      "source": [
        "##### 사이드바: 잡음 필터링 여부"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wnVu1kWa3HE3"
      },
      "source": [
        "### 대용량 데이터셋 다루기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CaEUY1043HE3"
      },
      "source": [
        "#### 메모리 매핑"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dibZ4Y_n3HE3"
      },
      "source": [
        "> **Note:** 다음 코드 블록은 빅쿼리 데이터셋을 `codeparrot` 폴더에 다운로드했다고 가정합니다. 압축 파일을 풀면 ~180GB 디스크 공간이 필요하기 때문에 이 단계를 건너 뛰는 것이 좋습니다. 이 코드는 예시 목적으로 쓴 것입니다. 대신 디스크 공간을 많이 사용하지 않는 스트리밍 데이터셋을 사용할 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_yld2PG33HE4"
      },
      "outputs": [],
      "source": [
        "from datasets import load_dataset, DownloadConfig\n",
        "\n",
        "download_config = DownloadConfig(delete_extracted=True)\n",
        "dataset = load_dataset(\"./codeparrot\", split=\"train\",\n",
        "                       download_config=download_config)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cD4_GFdw3HE4",
        "outputId": "86a4bb6f-9ef3-418e-bd21-5ca148ccb02c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Number of python files code in dataset : 18695559\n",
            "Dataset size (cache file) : 183.68 GB\n",
            "RAM memory used: 4924 MB\n"
          ]
        }
      ],
      "source": [
        "import psutil, os\n",
        "\n",
        "print(f\"Number of python files code in dataset : {len(dataset)}\")\n",
        "ds_size = sum(os.stat(f[\"filename\"]).st_size for f in dataset.cache_files)\n",
        "# os.stat.st_size는 바이트 단위이므로 GB로 바꿉니다\n",
        "print(f\"Dataset size (cache file) : {ds_size / 2**30:.2f} GB\")\n",
        "# Process.memory_info는 바이트 단위이므로 MB로 바꿉니다\n",
        "print(f\"RAM used: {psutil.Process(os.getpid()).memory_info().rss >> 20} MB\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tGAfWuZN3HE5"
      },
      "source": [
        "#### 스트리밍"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "efYWTfyv3HE5",
        "outputId": "9a5615c4-62e8-4fce-c179-4b780386b5df"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Using custom data configuration default-cae7a1d2f0dbde67\n"
          ]
        }
      ],
      "source": [
        "streamed_dataset = load_dataset('./codeparrot', split=\"train\", streaming=True)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-4UVd3JT3HE5",
        "outputId": "3b494d2b-20f4-40b3-d7d2-c3df64b869fc"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "True\n",
            "True\n"
          ]
        }
      ],
      "source": [
        "iterator = iter(streamed_dataset)\n",
        "\n",
        "print(dataset[0] == next(iterator))\n",
        "print(dataset[1] == next(iterator))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qA9HQ37Q3HE5"
      },
      "outputs": [],
      "source": [
        "remote_dataset = load_dataset('transformersbook/codeparrot', split=\"train\",\n",
        "                              streaming=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUsleNkV3HE6"
      },
      "source": [
        "### 허깅 페이스 허브에 데이터셋 추가하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qP2-iVZP3HE6"
      },
      "source": [
        "## 토크나이저 구축하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FMt-olj-3HE6",
        "outputId": "aa4eaebb-5cd2-4ffb-c090-5fd0bc3a941e",
        "colab": {
          "referenced_widgets": [
            "29ced71e91434126970160a03cc006a5",
            "6f437f06babc4f01b5f02ed2e11a274f",
            "086fad17475145a2960cf393d6da4da5",
            "83b90218ddd54563b5abc524ed820741",
            "8fb62d059f0a4ab397767ec6497ab5ed",
            "b1fd604f95db44748a1b53a31be9ff5a"
          ]
        }
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "29ced71e91434126970160a03cc006a5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.17k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "6f437f06babc4f01b5f02ed2e11a274f",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/773k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "086fad17475145a2960cf393d6da4da5",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.32M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "83b90218ddd54563b5abc524ed820741",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/508 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "8fb62d059f0a4ab397767ec6497ab5ed",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/792k [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "b1fd604f95db44748a1b53a31be9ff5a",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/1.33M [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "def tok_list(tokenizer, string):\n",
        "    input_ids = tokenizer(string, add_special_tokens=False)[\"input_ids\"]\n",
        "    return [tokenizer.decode(tok) for tok in input_ids]\n",
        "\n",
        "tokenizer_T5 = AutoTokenizer.from_pretrained(\"t5-base\")\n",
        "tokenizer_camembert = AutoTokenizer.from_pretrained(\"camembert-base\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HdITSlKa3HE6",
        "outputId": "a1c5547f-6bdd-4c55-8377-8a7bf808e59e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "T5 tokens for \"sex\": ['', 's', 'ex']\n",
            "CamemBERT tokens for \"being\": ['be', 'ing']\n"
          ]
        }
      ],
      "source": [
        "print(f'\"sex\"에 대한 T5 토큰: {tok_list(tokenizer_T5,\"sex\")}')\n",
        "print(f'\"being\"에 대한 CamemBERT 토큰: {tok_list(tokenizer_camembert,\"being\")}')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0KYFqOSe3HE7"
      },
      "source": [
        "### 토크나이저 모델"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7PZSEzGz3HE7"
      },
      "source": [
        "### 토크나이저 성능 측정하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "56aJyajh3HE7"
      },
      "source": [
        "### 파이썬 코드를 위한 토크나이저"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iUX0Ff-L3HE7",
        "outputId": "70e271eb-7ce1-4c68-bc38-6a03e20662d3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['def', 'Ġsay', '_', 'hello', '():', 'Ċ', 'Ġ', 'Ġ', 'Ġ', 'Ġprint', '(\"',\n",
            "'Hello', ',', 'ĠWorld', '!\"', ')', 'Ġ#', 'ĠPrint', 'Ġit', 'Ċ', 'Ċ', 'say', '_',\n",
            "'hello', '()', 'Ċ']\n"
          ]
        }
      ],
      "source": [
        "from transformers import AutoTokenizer\n",
        "\n",
        "python_code = r\"\"\"def say_hello():\n",
        "    print(\"Hello, World!\")\n",
        "# Print it\n",
        "say_hello()\n",
        "\"\"\"\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n",
        "print(tokenizer(python_code).tokens())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "IPEXeWLq3HE7",
        "outputId": "62f08192-d82c-4d6a-a7b5-370c1832cdcd"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "None\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer.backend_tokenizer.normalizer)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hAVFbJQY3HE8",
        "outputId": "11b32594-aa31-4414-95af-19e8068da716"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[('def', (0, 3)), ('Ġsay', (3, 7)), ('_', (7, 8)), ('hello', (8, 13)), ('():',\n",
            "(13, 16)), ('ĊĠĠĠ', (16, 20)), ('Ġprint', (20, 26)), ('(\"', (26, 28)), ('Hello',\n",
            "(28, 33)), (',', (33, 34)), ('ĠWorld', (34, 40)), ('!\")', (40, 43)), ('Ġ#', (43,\n",
            "45)), ('ĠPrint', (45, 51)), ('Ġit', (51, 54)), ('Ċ', (54, 55)), ('Ċ', (55, 56)),\n",
            "('say', (56, 59)), ('_', (59, 60)), ('hello', (60, 65)), ('()', (65, 67)), ('Ċ',\n",
            "(67, 68))]\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(python_code))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CafKUOOw3HE8",
        "outputId": "ae4c3919-04fc-4d30-f192-a5aeaf0f2d4a"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "`a` is encoded as `b'a'` with a single byte: 97\n",
            "`€` is encoded as `b'\\xe2\\x82\\xac'` with three bytes: [226, 130, 172]\n"
          ]
        }
      ],
      "source": [
        "a, e = u\"a\", u\"€\"\n",
        "byte = ord(a.encode(\"utf-8\"))\n",
        "print(f'`{a}`는 단일 바이트 `{a.encode(\"utf-8\")}`로 인코딩됩니다: {byte}')\n",
        "byte = [ord(chr(i)) for i in e.encode(\"utf-8\")]\n",
        "print(f'`{e}`는 세 바이트 `{e.encode(\"utf-8\")}`로 인코딩됩니다: {byte}')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ySJyuwae3HE8",
        "outputId": "a12d6783-aaae-4cc5-a2ec-4527c4f02ad7"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Size of our base vocabulary: 256\n",
            "First element: `!`, last element: `Ń`\n"
          ]
        }
      ],
      "source": [
        "from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode\n",
        "\n",
        "byte_to_unicode_map = bytes_to_unicode()\n",
        "unicode_to_byte_map = dict((v, k) for k, v in byte_to_unicode_map.items())\n",
        "base_vocab = list(unicode_to_byte_map.keys())\n",
        "\n",
        "print(f'기본 어휘 사전 크기: {len(base_vocab)}')\n",
        "print(f'첫 번째 원소: `{base_vocab[0]}`, last element: `{base_vocab[-1]}`')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iASnuocX3HE9",
        "outputId": "337f3be9-cda0-4340-9f5c-5a7b934f06bd"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th>Description</th>\n",
              "      <th>Character</th>\n",
              "      <th>Bytes</th>\n",
              "      <th>Mapped bytes</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <td>Regular characters</td>\n",
              "      <td>`a` and `?`</td>\n",
              "      <td>97 and 63</td>\n",
              "      <td>`a` and `?`</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>Non-printable control character (CARRIAGE RETURN)</td>\n",
              "      <td>`U+000D`</td>\n",
              "      <td>13</td>\n",
              "      <td>`č`</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>A space</td>\n",
              "      <td>` `</td>\n",
              "      <td>32</td>\n",
              "      <td>`Ġ`</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>A non-breakable space</td>\n",
              "      <td>`\\xa0`</td>\n",
              "      <td>160</td>\n",
              "      <td>`ł`</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <td>A newline character</td>\n",
              "      <td>`\\n`</td>\n",
              "      <td>10</td>\n",
              "      <td>`Ċ`</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "# BPE 문자 매핑의 예\n",
        "import pandas as pd\n",
        "from transformers.models.gpt2.tokenization_gpt2 import bytes_to_unicode\n",
        "\n",
        "byte_to_unicode_map = bytes_to_unicode()\n",
        "unicode_to_byte_map = dict((v, k) for k, v in byte_to_unicode_map.items())\n",
        "base_vocab = list(unicode_to_byte_map.keys())\n",
        "\n",
        "examples = [\n",
        "    ['Regular characters', '`a` and `?`', f'{ord(\"a\")} and {ord(\"?\")}' , f'`{byte_to_unicode_map[ord(\"a\")]}` and `{byte_to_unicode_map[ord(\"?\")]}`'],\n",
        "    ['Nonprintable control character (carriage return)', '`U+000D`', f'13', f'`{byte_to_unicode_map[13]}`'],\n",
        "    ['A space', '` `', f'{ord(\" \")}', f'`{byte_to_unicode_map[ord(\" \")]}`'],\n",
        "    ['A nonbreakable space', '`\\\\xa0`', '160', f'`{byte_to_unicode_map[ord(chr(160))]}`'],\n",
        "    ['A newline character', '`\\\\n`', '10', f'`{byte_to_unicode_map[ord(chr(10))]}`'],\n",
        "]\n",
        "\n",
        "pd.DataFrame(examples, columns = ['Description', 'Character', 'Bytes', 'Mapped bytes'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ur6AbQal3HE9",
        "outputId": "7531aa56-5a2c-4b0b-aeaa-2dd97578eae0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[('def', (0, 3)), ('Ġsay', (3, 7)), ('_', (7, 8)), ('hello', (8, 13)), ('():',\n",
            "(13, 16)), ('ĊĠĠĠ', (16, 20)), ('Ġprint', (20, 26)), ('(\"', (26, 28)), ('Hello',\n",
            "(28, 33)), (',', (33, 34)), ('ĠWorld', (34, 40)), ('!\")', (40, 43)), ('Ġ#', (43,\n",
            "45)), ('ĠPrint', (45, 51)), ('Ġit', (51, 54)), ('Ċ', (54, 55)), ('Ċ', (55, 56)),\n",
            "('say', (56, 59)), ('_', (59, 60)), ('hello', (60, 65)), ('()', (65, 67)), ('Ċ',\n",
            "(67, 68))]\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(python_code))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "u1nTnnel3HE9",
        "outputId": "45e29f0e-bda9-4eba-d7d6-eac374be552f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Size of the vocabulary: 50257\n"
          ]
        }
      ],
      "source": [
        "print(f\"어휘 사전의 크기: {len(tokenizer)}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z43lB4ba3HE9",
        "outputId": "95536e37-b30c-43c3-dbb5-a7021839d473"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['def', 'Ġsay', '_', 'hello', '():', 'Ċ', 'Ġ', 'Ġ', 'Ġ', 'Ġprint', '(\"',\n",
            "'Hello', ',', 'ĠWorld', '!\"', ')', 'Ġ#', 'ĠPrint', 'Ġit', 'Ċ', 'Ċ', 'say', '_',\n",
            "'hello', '()', 'Ċ']\n"
          ]
        }
      ],
      "source": [
        "print(tokenizer(python_code).tokens())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6VEFnRmG3HE-"
      },
      "source": [
        "### 토크나이저 훈련하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MyBOTDJ-3HE-",
        "outputId": "b37a8fb6-795a-45f1-c5a8-b546a533c508"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ', '\n",
            "=================================================================', '\n",
            "----------------------------------------------------------------',\n",
            "'................................................................',\n",
            "'ÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂÃÂ',\n",
            "'----------------------------------------------------------------',\n",
            "'================================================================',\n",
            "'________________________________________________________________']\n"
          ]
        }
      ],
      "source": [
        "tokens = sorted(tokenizer.vocab.items(), key=lambda x: len(x[0]), reverse=True)\n",
        "print([f'{tokenizer.convert_tokens_to_string(t)}' for t, _ in tokens[:8]]);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vR0okIJz3HE-",
        "outputId": "7a7d6f8c-4f8c-4c5e-854c-bac58dfb5a17"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['<|endoftext|>', ' gazed', ' informants', ' Collider', ' regress', 'ominated',\n",
            "' amplification', 'Compar', '….\"', ' (/', 'Commission', ' Hitman']\n"
          ]
        }
      ],
      "source": [
        "tokens = sorted(tokenizer.vocab.items(), key=lambda x: x[1], reverse=True)\n",
        "print([f'{tokenizer.convert_tokens_to_string(t)}' for t, _ in tokens[:12]]);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8Q87FkFv3HE-",
        "outputId": "ea8f044e-7e91-4e39-83d9-987b5ad240db",
        "colab": {
          "referenced_widgets": [
            "743bca69d71649908db9ca5760af61d2"
          ]
        }
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "743bca69d71649908db9ca5760af61d2",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Check remote data files:   0%|          | 0/183 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Using custom data configuration codeparrot-train-99775fd6743284b5\n"
          ]
        }
      ],
      "source": [
        "from tqdm.auto import tqdm\n",
        "\n",
        "length = 100000\n",
        "dataset_name = 'transformersbook/codeparrot-train'\n",
        "dataset = load_dataset(dataset_name, split=\"train\", streaming=True)\n",
        "iter_dataset = iter(dataset)\n",
        "\n",
        "def batch_iterator(batch_size=10):\n",
        "    for _ in tqdm(range(0, length, batch_size)):\n",
        "        yield [next(iter_dataset)['content'] for _ in range(batch_size)]\n",
        "\n",
        "new_tokenizer = tokenizer.train_new_from_iterator(batch_iterator(), \n",
        "                                                  vocab_size=12500,\n",
        "                                                  initial_alphabet=base_vocab)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KD0S3-TG3HE_",
        "outputId": "7150914a-8d3e-40ac-9e9a-58c23bc63360"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['  ', '    ', '   ', '        ', 'se', 'in', '       ', 're', 'on', 'te', '\\n\n",
            "', '\\n        ', 'or', 'st', 'de', '\\n   ', 'th', 'le', ' =', 'lf', 'self',\n",
            "'me', 'al']\n"
          ]
        }
      ],
      "source": [
        "tokens = sorted(new_tokenizer.vocab.items(), key=lambda x: x[1], reverse=False)\n",
        "print([f'{tokenizer.convert_tokens_to_string(t)}' for t, _ in tokens[257:280]]);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fE2BoOss3HE_",
        "outputId": "6b76cac8-8dc1-490d-ff60-b3b1684e390b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[' capt', ' embedded', ' regarding', 'Bundle', '355', ' recv', ' dmp', ' vault',\n",
            "' Mongo', ' possibly', 'implementation', 'Matches']\n"
          ]
        }
      ],
      "source": [
        "print([f'{new_tokenizer.convert_tokens_to_string(t)}' for t,_ in tokens[-12:]]);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o0Seft4I3HE_",
        "outputId": "65729b6e-30e1-4d8b-b3a4-c1552be14005"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['def', 'Ġs', 'ay', '_', 'hello', '():', 'ĊĠĠĠ', 'Ġprint', '(\"', 'Hello', ',',\n",
            "'ĠWor', 'ld', '!\")', 'Ġ#', 'ĠPrint', 'Ġit', 'Ċ', 'Ċ', 's', 'ay', '_', 'hello',\n",
            "'()', 'Ċ']\n"
          ]
        }
      ],
      "source": [
        "print(new_tokenizer(python_code).tokens())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FTzZ0lNy3HE_",
        "outputId": "aa767cca-a59c-4c50-e880-762e69585434"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "There are in total 35 Python keywords.\n",
            "No, keyword `await` is not in the vocabulary\n",
            "No, keyword `finally` is not in the vocabulary\n",
            "No, keyword `nonlocal` is not in the vocabulary\n"
          ]
        }
      ],
      "source": [
        "import keyword\n",
        "\n",
        "print(f'파이썬 전체 예약어 개수: {len(keyword.kwlist)}')\n",
        "for keyw in keyword.kwlist:\n",
        "    if keyw not in new_tokenizer.vocab:\n",
        "        print(f'예약어 `{keyw}`는 어휘 사전에 없습니다.')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RRNPUf4D3HFA",
        "outputId": "cd6175e6-b3c1-4ba7-c018-fecc2b125c5d"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "100%|██████████| 200/200 [05:08<00:00,  1.54s/it]\n"
          ]
        }
      ],
      "source": [
        "length = 200000\n",
        "new_tokenizer_larger = tokenizer.train_new_from_iterator(batch_iterator(),\n",
        "    vocab_size=32768, initial_alphabet=base_vocab)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ny6ntObw3HFA",
        "outputId": "a9ff2645-be5a-4764-d107-627009c780c2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['lineEdit', 'spik', ' BC', 'pective', 'OTA', 'theus', 'FLUSH', ' excutils',\n",
            "'00000002', ' DIVISION', 'CursorPosition', ' InfoBar']\n"
          ]
        }
      ],
      "source": [
        "tokens = sorted(new_tokenizer_larger.vocab.items(), key=lambda x: x[1],\n",
        "                reverse=False)\n",
        "print([f'{tokenizer.convert_tokens_to_string(t)}' for t, _ in tokens[-12:]]);"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Ayx7ZB9g3HFA",
        "outputId": "8b971075-3e20-4853-f521-bb1e83d5a930"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['def', 'Ġsay', '_', 'hello', '():', 'ĊĠĠĠ', 'Ġprint', '(\"', 'Hello', ',',\n",
            "'ĠWorld', '!\")', 'Ġ#', 'ĠPrint', 'Ġit', 'Ċ', 'Ċ', 'say', '_', 'hello', '()',\n",
            "'Ċ']\n"
          ]
        }
      ],
      "source": [
        "print(new_tokenizer_larger(python_code).tokens())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CzKLb3Ia3HFA",
        "outputId": "80e2f7c1-b8a6-46dc-e6b2-0c9566f94292"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "No, keyword `nonlocal` is not in the vocabulary\n"
          ]
        }
      ],
      "source": [
        "for keyw in keyword.kwlist:\n",
        "    if keyw not in new_tokenizer_larger.vocab:\n",
        "        print(f'예약어 `{keyw}`는 어휘 사전에 없습니다.')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D9PUn27x3HFA"
      },
      "source": [
        "### 허브에 사용자 정의 토크나이저 저장하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gT-bYZQj3HFB",
        "outputId": "5c923433-0da1-4794-f1f1-cc8830ec2727"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Cloning https://huggingface.co/transformersbook/codeparrot into local empty directory.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'https://huggingface.co/transformersbook/codeparrot/commit/1c284adaa3cc9f8635ae7e3377bd3739f48bc09a'"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "model_ckpt = \"codeparrot\"\n",
        "org = \"transformersbook\"\n",
        "new_tokenizer_larger.push_to_hub(model_ckpt, organization=org)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7BLIV6_63HFB",
        "outputId": "2ad8ed0b-d44e-4567-a75f-7a4c72fef398"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "['def', 'Ġsay', '_', 'hello', '():', 'ĊĠĠĠ', 'Ġprint', '(\"', 'Hello', ',',\n",
            "'ĠWorld', '!\")', 'Ġ#', 'ĠPrint', 'Ġit', 'Ċ', 'Ċ', 'say', '_', 'hello', '()',\n",
            "'Ċ']\n"
          ]
        }
      ],
      "source": [
        "reloaded_tokenizer = AutoTokenizer.from_pretrained(org + \"/\" + model_ckpt)\n",
        "print(reloaded_tokenizer(python_code).tokens())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1mxyF6DA3HFB",
        "outputId": "30bf5f9c-a0c1-44dc-d833-339e4ceec0c8"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Cloning https://huggingface.co/transformersbook/codeparrot-small-vocabulary into local empty directory.\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "'https://huggingface.co/transformersbook/codeparrot-small-vocabulary/commit/0b37bed9956d95d0b79ada169f6a281e15c63381'"
            ]
          },
          "execution_count": null,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "new_tokenizer.push_to_hub(model_ckpt+ \"-small-vocabulary\", organization=org)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qEzxWNYB3HFB"
      },
      "source": [
        "## 밑바닥부터 모델을 훈련하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yCQa09PH3HFB"
      },
      "source": [
        "### 사전 훈련 목표"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-5UhiqVA3HFC"
      },
      "source": [
        "<img alt=\"Code snippet\" caption=\"An example of a Python function that could be found in our dataset\" src=\"https://github.com/rickiepark/nlp-with-transformers/blob/main/images/chapter10_code-snippet.png?raw=1\" id=\"code-snippet\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nfAe1YQ33HFC"
      },
      "source": [
        "#### 코잘 언어 모델링"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aEKLptnL3HFC"
      },
      "source": [
        "<img alt=\"CLM pretraining\" caption=\"In causal language modeling, the future tokens are masked and the model has to predict them; typically a decoder model such as GPT is used for such a task\" src=\"https://github.com/rickiepark/nlp-with-transformers/blob/main/images/chapter10_pretraining-clm.png?raw=1\" id=\"pretraining-clm\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylSDUdQZ3HFC"
      },
      "source": [
        "#### 마스크드 언어 모델링"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RZIRVjYO3HFC"
      },
      "source": [
        "<img alt=\"MLM pretraining\" caption=\"In masked language modeling some of the input tokens are either masked or replaced, and the model's task is to predict the original tokens; this is the architecture underlying the encoder branch of transformer models\" src=\"https://github.com/rickiepark/nlp-with-transformers/blob/main/images/chapter10_pretraining-mlm.png?raw=1\" id=\"pretraining-mlm\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9cL69lMQ3HFC"
      },
      "source": [
        "#### 시퀀스-투-시퀀스 훈련"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "isERZf7W3HFC"
      },
      "source": [
        "<img alt=\"Seq2seq pretraining\" caption=\"Using an encoder-decoder architecture for a sequence-to-sequence task where the inputs are split into comment/code pairs using heuristics: the model gets one element as input and needs to generate the other one\" src=\"https://github.com/rickiepark/nlp-with-transformers/blob/main/images/chapter10_pretraining-seq2seq.png?raw=1\" id=\"pretraining-seq2seq\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "esdE4Jl83HFD"
      },
      "source": [
        "### 모델 초기화"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_QAzZ0-m3HFD"
      },
      "source": [
        "> **NOTE**: 다음 코드 블록에서 대용량 GPT-2 체크포인트를 메모리에 로드합니다. 코랩이나 캐글 같은 플랫폼에서는 램이나 GPU 메모리가 부족하기 때문에 인스턴스가 종료될 수 있습니다. 이런 경우 `config = AutoConfig.from_pretrained(\"gpt2\", vocab_size=len(tokenizer))`로 설정을 바꾸어 작은 체크포인트를 사용하면 이 예제를 실행할 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZAzW0ZtB3HFD",
        "outputId": "ccf3fe32-c4d6-4054-c7f8-1936b2312fec",
        "colab": {
          "referenced_widgets": [
            "be84ca77ca144954af8ae4820ec6685b"
          ]
        }
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "be84ca77ca144954af8ae4820ec6685b",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Downloading:   0%|          | 0.00/787 [00:00<?, ?B/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "from transformers import AutoConfig, AutoModelForCausalLM, AutoTokenizer\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(org + \"/\" + model_ckpt)\n",
        "config = AutoConfig.from_pretrained(\"gpt2-xl\", vocab_size=len(tokenizer))\n",
        "model = AutoModelForCausalLM.from_config(config)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CSUdPXEO3HFD",
        "outputId": "508bb729-45f3-4b5a-c376-177abe836b62"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GPT-2 (xl) size: 1529.6M parameters\n"
          ]
        }
      ],
      "source": [
        "print(f'GPT-2 (xl) 크기: {model_size(model)/1000**2:.1f}M parameters')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LSLICfsG3HFD"
      },
      "outputs": [],
      "source": [
        "model.save_pretrained(\"models/\" + model_ckpt, push_to_hub=True,\n",
        "                      organization=org)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jQODx9VX3HFD"
      },
      "outputs": [],
      "source": [
        "tokenizer = AutoTokenizer.from_pretrained(model_ckpt)\n",
        "config_small = AutoConfig.from_pretrained(\"gpt2\", vocab_size=len(tokenizer))\n",
        "model_small = AutoModelForCausalLM.from_config(config_small)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mJeWHtQk3HFE",
        "outputId": "362299a7-77c9-4000-d6f9-8d07098e3cc8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GPT-2 size: 111.0M parameters\n"
          ]
        }
      ],
      "source": [
        "print(f'GPT-2 크기: {model_size(model_small)/1000**2:.1f}M parameters')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ia_TivsQ3HFE",
        "outputId": "c838c74b-163d-435c-d8b5-3b26a61aa7b8"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Cloning https://huggingface.co/transformersbook/codeparrot-small into local empty directory.\n"
          ]
        }
      ],
      "source": [
        "model_small.save_pretrained(\"models/\" + model_ckpt + \"-small\", push_to_hub=True,\n",
        "                            organization=org)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XaNci8GM3HFE"
      },
      "source": [
        "### 데이터로더 구축하기"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d1B7Uw-z3HFE"
      },
      "source": [
        "<img alt=\"Preprocessing for CLM\" caption=\"Preparing sequences of varying length for causal language modeling by concatenating several tokenized examples with an EOS token  before chunking them\" src=\"https://github.com/rickiepark/nlp-with-transformers/blob/main/images/chapter10_preprocessing-clm.png?raw=1\" id=\"preprocessing-clm\"/>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fqNbxWLi3HFE",
        "outputId": "e0c953e6-06db-46df-ed86-ed1703a4d567"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "  0%|          | 1/500 [00:00<01:16,  6.54it/s]Token indices sequence length is longer than the specified maximum sequence length for this model (2605 > 1024). Running this sequence through the model will result in indexing errors\n",
            "100%|██████████| 500/500 [00:04<00:00, 122.59it/s]\n"
          ]
        }
      ],
      "source": [
        "examples, total_characters, total_tokens = 500, 0, 0\n",
        "dataset = load_dataset('transformersbook/codeparrot-train', split='train',\n",
        "                       streaming=True)\n",
        "\n",
        "for _, example in tqdm(zip(range(examples), iter(dataset)), total=examples):\n",
        "    total_characters += len(example['content'])\n",
        "    total_tokens += len(tokenizer(example['content']).tokens())\n",
        "\n",
        "characters_per_token = total_characters / total_tokens"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3-JYsTcR3HFE",
        "outputId": "63624bf9-fc55-4e60-822f-ab8b9e00ec8f"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "3.6233025034779565\n"
          ]
        }
      ],
      "source": [
        "print(characters_per_token)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "K3LetK_p3HFF"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "from torch.utils.data import IterableDataset\n",
        "\n",
        "class ConstantLengthDataset(IterableDataset):\n",
        "    \n",
        "    def __init__(self, tokenizer, dataset, seq_length=1024,\n",
        "                 num_of_sequences=1024, chars_per_token=3.6):\n",
        "        self.tokenizer = tokenizer\n",
        "        self.concat_token_id = tokenizer.eos_token_id\n",
        "        self.dataset = dataset\n",
        "        self.seq_length = seq_length\n",
        "        self.input_characters = seq_length * chars_per_token * num_of_sequences\n",
        "    \n",
        "    def __iter__(self):\n",
        "        iterator = iter(self.dataset)\n",
        "        more_examples = True\n",
        "        while more_examples:\n",
        "            buffer, buffer_len = [], 0\n",
        "            while True:\n",
        "                if buffer_len >= self.input_characters:\n",
        "                    m=f\"Buffer full: {buffer_len}>={self.input_characters:.0f}\"\n",
        "                    print(m)\n",
        "                    break\n",
        "                try:\n",
        "                    m=f\"Fill buffer: {buffer_len}<{self.input_characters:.0f}\"\n",
        "                    print(m)\n",
        "                    buffer.append(next(iterator)[\"content\"])\n",
        "                    buffer_len += len(buffer[-1])\n",
        "                except StopIteration:\n",
        "                    iterator = iter(self.dataset)\n",
        "\n",
        "            all_token_ids = []\n",
        "            tokenized_inputs = self.tokenizer(buffer, truncation=False)\n",
        "            for tokenized_input in tokenized_inputs['input_ids']:\n",
        "                all_token_ids.extend(tokenized_input + [self.concat_token_id])\n",
        "            \n",
        "            for i in range(0, len(all_token_ids), self.seq_length):\n",
        "                input_ids = all_token_ids[i : i + self.seq_length]\n",
        "                if len(input_ids) == self.seq_length:\n",
        "                    yield torch.tensor(input_ids)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OtD6gUq03HFF",
        "outputId": "75e259f5-a13c-4ada-d202-cfcd91e18da3"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Fill buffer: 0<36864\n",
            "Fill buffer: 3311<36864\n",
            "Fill buffer: 9590<36864\n",
            "Fill buffer: 22177<36864\n",
            "Fill buffer: 25530<36864\n",
            "Fill buffer: 31098<36864\n",
            "Fill buffer: 32232<36864\n",
            "Fill buffer: 33867<36864\n",
            "Buffer full: 41172>=36864\n",
            "Lengths of the sequences: [1024, 1024, 1024, 1024, 1024]\n"
          ]
        }
      ],
      "source": [
        "shuffled_dataset = dataset.shuffle(buffer_size=100)\n",
        "constant_length_dataset = ConstantLengthDataset(tokenizer, shuffled_dataset,\n",
        "                                                num_of_sequences=10)\n",
        "dataset_iterator = iter(constant_length_dataset)\n",
        "\n",
        "lengths = [len(b) for _, b in zip(range(5), dataset_iterator)]\n",
        "print(f\"시퀀스 길이: {lengths}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AFFKp6dY3HFF"
      },
      "source": [
        "### 훈련 루프 정의하기"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NO4WkQu03HFF"
      },
      "outputs": [],
      "source": [
        "from argparse import Namespace\n",
        "\n",
        "# 작은 모델에 해당하는 파라미터\n",
        "config = {\"train_batch_size\": 2, # 12\n",
        "          \"valid_batch_size\": 2, # 12\n",
        "          \"weight_decay\": 0.1,\n",
        "          \"shuffle_buffer\": 1000,\n",
        "          \"learning_rate\": 2e-4, # 5e-4\n",
        "          \"lr_scheduler_type\": \"cosine\",\n",
        "          \"num_warmup_steps\": 750, # 2000\n",
        "          \"gradient_accumulation_steps\": 16, # 1\n",
        "          \"max_train_steps\": 50000, # 150000\n",
        "          \"max_eval_steps\": -1,\n",
        "          \"seq_length\": 1024,\n",
        "          \"seed\": 1,\n",
        "          \"save_checkpoint_steps\": 50000} # 15000\n",
        "\n",
        "args = Namespace(**config)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wqMdvIOH3HFF"
      },
      "outputs": [],
      "source": [
        "from torch.utils.tensorboard import SummaryWriter\n",
        "import logging\n",
        "import wandb\n",
        "\n",
        "def setup_logging(project_name):\n",
        "    logger = logging.getLogger(__name__)\n",
        "    logging.basicConfig(\n",
        "        format=\"%(asctime)s - %(levelname)s - %(name)s - %(message)s\",\n",
        "        datefmt=\"%m/%d/%Y %H:%M:%S\", level=logging.INFO, handlers=[\n",
        "        logging.FileHandler(f\"log/debug_{accelerator.process_index}.log\"),\n",
        "        logging.StreamHandler()])\n",
        "    if accelerator.is_main_process: # 로깅을 한 번만 설정합니다.\n",
        "        wandb.init(project=project_name, config=args)\n",
        "        run_name = wandb.run.name\n",
        "        tb_writer = SummaryWriter()\n",
        "        tb_writer.add_hparams(vars(args), {'0': 0})\n",
        "        logger.setLevel(logging.INFO)\n",
        "        datasets.utils.logging.set_verbosity_debug()\n",
        "        transformers.utils.logging.set_verbosity_info()\n",
        "    else:\n",
        "        tb_writer = None\n",
        "        run_name = ''\n",
        "        logger.setLevel(logging.ERROR)\n",
        "        datasets.utils.logging.set_verbosity_error()\n",
        "        transformers.utils.logging.set_verbosity_error()\n",
        "    return logger, tb_writer, run_name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lpTBjKt-3HFG"
      },
      "outputs": [],
      "source": [
        "def log_metrics(step, metrics):\n",
        "    logger.info(f\"Step {step}: {metrics}\")\n",
        "    if accelerator.is_main_process:\n",
        "        wandb.log(metrics)\n",
        "        [tb_writer.add_scalar(k, v, step) for k, v in metrics.items()]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8pHS7dGA3HFG",
        "outputId": "7e2cc073-ce0c-4151-fccd-9711f82d2997",
        "colab": {
          "referenced_widgets": [
            "328dc6d7d05c452e8d8e2cab5b4b9c4e"
          ]
        }
      },
      "outputs": [
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "328dc6d7d05c452e8d8e2cab5b4b9c4e",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "Check remote data files:   0%|          | 0/183 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        },
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Using custom data configuration codeparrot-train-938ce362e6f661b1\n",
            "Using custom data configuration codeparrot-valid-29167601d8e69487\n"
          ]
        }
      ],
      "source": [
        "from torch.utils.data.dataloader import DataLoader\n",
        "\n",
        "def create_dataloaders(dataset_name):\n",
        "    train_data = load_dataset(dataset_name+'-train', split=\"train\",\n",
        "                              streaming=True)\n",
        "    train_data = train_data.shuffle(buffer_size=args.shuffle_buffer,\n",
        "                                    seed=args.seed)\n",
        "    valid_data = load_dataset(dataset_name+'-valid', split=\"validation\",\n",
        "                              streaming=True)\n",
        "    \n",
        "    train_dataset = ConstantLengthDataset(tokenizer, train_data,\n",
        "                                          seq_length=args.seq_length)\n",
        "    valid_dataset = ConstantLengthDataset(tokenizer, valid_data,\n",
        "                                          seq_length=args.seq_length)\n",
        "    \n",
        "    train_dataloader=DataLoader(train_dataset, batch_size=args.train_batch_size)\n",
        "    eval_dataloader=DataLoader(valid_dataset, batch_size=args.valid_batch_size)\n",
        "    return train_dataloader, eval_dataloader"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bXF9-E0t3HFG"
      },
      "outputs": [],
      "source": [
        "def get_grouped_params(model, no_decay=[\"bias\", \"LayerNorm.weight\"]):\n",
        "    params_with_wd, params_without_wd = [], []\n",
        "    for n, p in model.named_parameters():\n",
        "        if any(nd in n for nd in no_decay):\n",
        "            params_without_wd.append(p)\n",
        "        else:\n",
        "            params_with_wd.append(p)\n",
        "    return [{'params': params_with_wd, 'weight_decay': args.weight_decay},\n",
        "            {'params': params_without_wd, 'weight_decay': 0.0}]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "I1erm1D83HFG"
      },
      "outputs": [],
      "source": [
        "def evaluate():\n",
        "    model.eval()\n",
        "    losses = []\n",
        "    for step, batch in enumerate(eval_dataloader):\n",
        "        with torch.no_grad():\n",
        "            outputs = model(batch, labels=batch)\n",
        "        loss = outputs.loss.repeat(args.valid_batch_size)\n",
        "        losses.append(accelerator.gather(loss))\n",
        "        if args.max_eval_steps > 0 and step >= args.max_eval_steps: break\n",
        "    loss = torch.mean(torch.cat(losses))\n",
        "    try:\n",
        "\t\tperplexity = torch.exp(loss)\n",
        "\texcept OverflowError:\n",
        "\t\tperplexity = torch.tensor(float(\"inf\"))\n",
        "    return loss.item(), perplexity.item()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jzKOeMhe3HFG"
      },
      "outputs": [],
      "source": [
        "set_seed(args.seed)\n",
        "\n",
        "# 엑셀러레이트\n",
        "accelerator = Accelerator()\n",
        "samples_per_step = accelerator.state.num_processes * args.train_batch_size\n",
        "\n",
        "# 로깅\n",
        "logger, tb_writer, run_name = setup_logging(project_name.split(\"/\")[1])\n",
        "logger.info(accelerator.state)\n",
        "\n",
        "# 모델과 토크나이저를 로드합니다\n",
        "if accelerator.is_main_process:\n",
        "    hf_repo = Repository(\"./\", clone_from=project_name, revision=run_name)\n",
        "model = AutoModelForCausalLM.from_pretrained(\"./\", gradient_checkpointing=True)\n",
        "tokenizer = AutoTokenizer.from_pretrained(\"./\")\n",
        "\n",
        "# 데이터셋과 데이터로더를 로드합니다\n",
        "train_dataloader, eval_dataloader = create_dataloaders(dataset_name)\n",
        "\n",
        "# 옵티마이저와 학습률 스케줄러를 준비합니다\n",
        "optimizer = AdamW(get_grouped_params(model), lr=args.learning_rate)\n",
        "lr_scheduler = get_scheduler(name=args.lr_scheduler_type, optimizer=optimizer,\n",
        "                             num_warmup_steps=args.num_warmup_steps,\n",
        "                             num_training_steps=args.max_train_steps,)\n",
        "def get_lr():\n",
        "    return optimizer.param_groups[0]['lr']\n",
        "\n",
        "# `accelerator`로 모든 것을 준비합니다(매개변수 순서는 중요하지 않습니다)\n",
        "model, optimizer, train_dataloader, eval_dataloader = accelerator.prepare(\n",
        "    model, optimizer, train_dataloader, eval_dataloader)\n",
        "\n",
        "# 모델을 훈련합니다\n",
        "model.train()\n",
        "completed_steps = 0\n",
        "for step, batch in enumerate(train_dataloader, start=1):\n",
        "    loss = model(batch, labels=batch).loss\n",
        "    log_metrics(step, {'lr': get_lr(), 'samples': step*samples_per_step,\n",
        "                       'steps': completed_steps, 'loss/train': loss.item()})\n",
        "    loss = loss / args.gradient_accumulation_steps\n",
        "    accelerator.backward(loss)\n",
        "    if step % args.gradient_accumulation_steps == 0:\n",
        "        optimizer.step()\n",
        "        lr_scheduler.step()\n",
        "        optimizer.zero_grad()\n",
        "        completed_steps += 1\n",
        "    if step % args.save_checkpoint_steps == 0:\n",
        "        logger.info('Evaluating and saving model checkpoint')\n",
        "        eval_loss, perplexity = evaluate()\n",
        "        log_metrics(step, {'loss/eval': eval_loss, 'perplexity': perplexity})\n",
        "        accelerator.wait_for_everyone()\n",
        "        unwrapped_model = accelerator.unwrap_model(model)\n",
        "        if accelerator.is_main_process:\n",
        "            unwrapped_model.save_pretrained(\"./\")\n",
        "            hf_repo.push_to_hub(commit_message=f'step {step}')\n",
        "        model.train()\n",
        "    if completed_steps >= args.max_train_steps:\n",
        "        break\n",
        "\n",
        "# 마지막 체크포인트를 평가하고 저장합니다\n",
        "logger.info('Evaluating and saving model after training')\n",
        "eval_loss, perplexity = evaluate()\n",
        "log_metrics(step, {'loss/eval': eval_loss, 'perplexity': perplexity})\n",
        "accelerator.wait_for_everyone()\n",
        "unwrapped_model = accelerator.unwrap_model(model)\n",
        "if accelerator.is_main_process:\n",
        "    unwrapped_model.save_pretrained(\"./\")\n",
        "    hf_repo.push_to_hub(commit_message=f'final model')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mzWQfR5M3HFH"
      },
      "source": [
        "<img alt=\"DDP\" caption=\"Illustration of the processing steps in DDP with four GPUs\" src=\"https://github.com/rickiepark/nlp-with-transformers/blob/main/images/chapter10_ddp.png?raw=1\" id=\"ddp\"/>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cN8eRTnr3HFH"
      },
      "source": [
        "### 훈련 실행"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vE7_MuBS3HFH"
      },
      "source": [
        "## 결과 및 분석"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Oy8cCX3b3HFH",
        "outputId": "3f2f75ba-4734-4b1e-a622-e366e3af824d"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "2021-10-20 18:29:01.107727: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory\n",
            "2021-10-20 18:29:01.107759: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.\n"
          ]
        }
      ],
      "source": [
        "from transformers import pipeline, set_seed\n",
        "\n",
        "model_ckpt = 'transformersbook/codeparrot-small'\n",
        "generation = pipeline('text-generation', model=model_ckpt, device=0)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OzUMGRLK3HFH"
      },
      "outputs": [],
      "source": [
        "import re\n",
        "from transformers import set_seed \n",
        "\n",
        "def first_block(string):\n",
        "    return re.split('\\nclass|\\ndef|\\n#|\\n@|\\nprint|\\nif', string)[0].rstrip()\n",
        "\n",
        "def complete_code(pipe, prompt, max_length=64, num_completions=4, seed=1):\n",
        "    set_seed(seed)\n",
        "    gen_kwargs = {\"temperature\":0.4, \"top_p\":0.95, \"top_k\":0, \"num_beams\":1,\n",
        "                  \"do_sample\":True,}\n",
        "    code_gens = generation(prompt, num_return_sequences=num_completions, \n",
        "                            max_length=max_length, **gen_kwargs)\n",
        "    code_strings = []\n",
        "    for code_gen in code_gens:\n",
        "        generated_code = first_block(code_gen['generated_text'][len(prompt):])\n",
        "        code_strings.append(generated_code)\n",
        "    print(('\\n'+'='*80 + '\\n').join(code_strings))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a5tNJ4df3HFH",
        "outputId": "f9ca3b29-e043-45d1-b6ff-5b3dd34c325c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "    return math.sqrt(a * b)\n",
            "================================================================================\n",
            "\n",
            "    return a * b / 2.0\n",
            "================================================================================\n",
            "\n",
            "    return a * b\n",
            "================================================================================\n",
            "\n",
            "    return a * b / a\n"
          ]
        }
      ],
      "source": [
        "prompt = '''def area_of_rectangle(a: float, b: float):\n",
        "    \"\"\"Return the area of the rectangle.\"\"\"'''\n",
        "complete_code(generation, prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Pokz65AK3HFI",
        "outputId": "9624808d-c13f-47e5-8d66-173ce1f11ea6"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "    if not html:\n",
            "        return []\n",
            "    return [url for url in re.findall(r'<a href=\"(/[^/]+/[^\"]+?)\">', html)]\n",
            "================================================================================\n",
            "\n",
            "    return [url for url in re.findall(r'<a href=\"(.*?)\"', html)\n",
            "            if url]\n",
            "================================================================================\n",
            "\n",
            "    return [url for url in re.findall(r'<a href=\"(/.*)\",', html)]\n",
            "================================================================================\n",
            "\n",
            "    return re.findall(r'<a href=\"(.*?)\" class=\"url\"[^>]*>', html)\n"
          ]
        }
      ],
      "source": [
        "prompt = '''def get_urls_from_html(html):\n",
        "    \"\"\"Get all embedded URLs in a HTML string.\"\"\"'''\n",
        "complete_code(generation, prompt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-_In2gdU3HFI",
        "outputId": "0f1b21d0-068a-48c3-c980-4b880984d402"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "https://github.com/huggingface/transformers | /allenai | /facebook |\n",
            "/asteroid-team | /google | /amazon | /speechbrain | /microsoft | /grammarly |\n",
            "/models | /inference-api | /distilbert-base-uncased |\n",
            "/dbmdz/bert-large-cased-finetuned-conll03-english |\n",
            "https://huggingface.co/transformers | https://arxiv.org/abs/1811.06031 |\n",
            "https://arxiv.org/abs/1803.10631 | https://transformer.huggingface.co/ | /coref\n",
            "| https://medium.com/huggingface/distilbert-8cf3380435b5\n"
          ]
        }
      ],
      "source": [
        "import requests\n",
        "\n",
        "def get_urls_from_html(html):\n",
        "    return [url for url in re.findall(r'<a href=\"(.*?)\"', html) if url]\n",
        "\n",
        "print(\" | \".join(get_urls_from_html(requests.get('https://hf.co/').text)))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vv5brIIS3HFI"
      },
      "source": [
        "> **NOTE**: 다음 코드 블록에서 대용량 GPT-2 체크포인트를 메모리에 로드합니다. 코랩이나 캐글 같은 플랫폼에서는 램이나 GPU 메모리가 부족하기 때문에 인스턴스가 종료될 수 있습니다. 이런 경우 `model_ckpt = \"transformersbook/codeparrot-small\"`로 바꾸어 작은 체크포인트를 사용하면 이 예제를 실행할 수 있습니다."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4pCQqqFc3HFI",
        "outputId": "c2fcef6c-bf6c-47ac-b6d8-3fe7f1f19368"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "    return np.mean(a)\n",
            "================================================================================\n",
            "\n",
            "    return np.mean(a)\n",
            "================================================================================\n",
            "\n",
            "    return np.mean(a)\n",
            "================================================================================\n",
            "\n",
            "    return np.mean(a)\n"
          ]
        }
      ],
      "source": [
        "model_ckpt = 'transformersbook/codeparrot'\n",
        "generation = pipeline('text-generation', model=model_ckpt, device=0)\n",
        "\n",
        "prompt = '''# a function in native python:\n",
        "def mean(a):\n",
        "    return sum(a)/len(a)\n",
        "\n",
        "# the same function using numpy:\n",
        "import numpy as np\n",
        "def mean(a):'''\n",
        "complete_code(generation, prompt, max_length=64)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "o2llT-e63HFI",
        "outputId": "6c166766-ddef-47fc-b5d8-ae386d1d9d48"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "Setting `pad_token_id` to `eos_token_id`:0 for open-end generation.\n"
          ]
        },
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "reg = DummyRegressor()\n",
            "\n",
            "forest = RandomForestClassifier(n_estimators=20)\n",
            "\n",
            "forest.fit(X, y)\n",
            "================================================================================\n",
            "\n",
            "clf = ExtraTreesClassifier(n_estimators=100, max_features='sqrt')\n",
            "clf.fit(X, y)\n",
            "================================================================================\n",
            "\n",
            "clf = RandomForestClassifier(n_estimators=20, n_jobs=n_jobs, random_state=1)\n",
            "clf.fit(X, y)\n",
            "================================================================================\n",
            "\n",
            "clf = RandomForestClassifier(n_estimators=20)\n",
            "clf.fit(X, y)\n"
          ]
        }
      ],
      "source": [
        "prompt = '''X = np.random.randn(100, 100)\n",
        "y = np.random.randint(0, 1, 100)\n",
        "\n",
        "# fit random forest classifier with 20 estimators'''\n",
        "complete_code(generation, prompt, max_length=96)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "__N3KpfJ3HFJ"
      },
      "source": [
        "## 결론"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}