3_deep_learning_for_images.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 細胞画像から細胞状態を直接予測する"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "今までは、細胞の大きさなどを観測した後に、芽の大きさのグループを分けることを考えていましたが、ここでは、画像から直接細胞状態を推定する予測器を作成します。手間のかかる特徴量の抽出を無くせるという利点がある一方で、どの特徴が重要で分類できたのかは分かりにくくなるので、目的によって、適切に使い分けてください。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "手書き文字（数字）認識のMNISTや画像分類のImageNetなど、機械学習でよく用いられているデータセットを用いた解析は、いずれの深層学習のフレームワークでもドキュメントやサンプルコードが豊富に存在しています。その一方で、データ形式が少し変化したりすると、とたんにドキュメントが少なくなります。この現状を踏まえ、本講義では、生命科学ではよくあるだろうシチュエーションのデータ形式を考えつつ、サンプルを用意します。難解だと思ったら、PyTorchのチュートリアルなどを見て、再度このドキュメント・プログラムを見てもらえればと思います。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ライブラリの読み込み"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "「2. 酵母画像解析（特徴量からの深層学習）」と同様に、利用するライブラリ一群を読み込みます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.optim as optim\n",
    "from torch.optim import lr_scheduler\n",
    "from torchvision import transforms\n",
    "import torch.utils.data as data\n",
    "import torch.nn.functional as F\n",
    "from PIL import Image\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import os\n",
    "import copy\n",
    "import argparse\n",
    "import time\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## データの読み込み"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "酵母の細胞の特徴量を解析した前節では、学習開始前に全てのデータを読み込み、学習時には、そこからバッチ毎にデータを読み込んでいました。画像の場合も同様ですが、ここでは、大規模なデータに対応できる工夫を導入します。\n",
    "\n",
    "画像や動画の場合には、枚数が多くなるとメモリに乗らない分量の大規模なデータになることがあります。このような場合には、学習前に全てのデータを読み込むことはできません。そこで、学習前にはファイル名（あるいは、ファイル名を知るために必要な値）だけを読み込み、バッチ毎に、バッチ内にあるサンプルの画像や動画を読み込みます。これにより、バッチ対象となるデータが読み込みさえすれば、学習を進めることができるので、全データ分のメモリが用意できなくても学習が可能です。\n",
    "\n",
    "<img src=\"img/dl_onthefly.png\">\n",
    "\n",
    "さらに、このバッチごとの読み込みには複数の利点があります。\n",
    "\n",
    "* 深層学習では、高速化のためCPUではなくGPUを利用した計算が行われますが、GPUの欠点として、メモリが少ないことが挙げられます。たとえば、現在よく使われるNVIDIA社のGTX1080で8GB、サーバ用途で利用されるNVIDIA社のTESLA V100で16GB〜32GBと、CPUから利用できるメモリに比べると少々少なくなっています。バッチごとのデータ読み込みが可能になることで、メモリ量が限られた環境でも、効率良い学習が可能になります。\n",
    "* データの擬似的な拡張(水増し：Augmentation)との相性が良いです。例えば、画像解析を考えた場合に、上下左右の反転をしたり、回転をしても、同一のクラスとして認識して欲しい場合が多くあります（カメラを傾けても、ネコはネコなので）。この場合、画像を適当な角度で回転して、学習しても構わないことになります。予め、様々な角度の画像を用意しておくことも、一つの作戦ですが、ただでさえ枚数が多い画像が更に多くなって、ハードディスク容量の圧迫に繋がる可能性があります。そこで、バッチ毎に画像を読み込む際に、乱数を発生させて、適当な角度に回転したり、上下左右を入れ替えたりすることで、あらたなデータを予め用意することなく、水増しが可能になります。\n",
    "\n",
    "以下のプログラムでは、make_dataset 関数が、特徴量同様に学習前に呼び出される関数です。この時、特徴量ではなく、画像のファイル名を作成し、酵母画像のファイル名とクラスの対応表を作成しています。\n",
    "画像は、PhotoID列をX, CellID列をYとすると、```data/images/C_yor202w_0_0_X_Y.png``` に入っています。よって、各細胞に対して、このファイル名と、クラスを割り当てます。クラスが\"no\", \"small\",\"medium\",\"large\" をそれぞれ別の次元とした4次元で表すのは、前節の事例と同じです。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_dataset(dir):\n",
    "    images = []\n",
    "    labels = []\n",
    "    dataset = pd.read_csv(os.path.join(dir, \"yeast_his3.csv\"))\n",
    "    for _, row in dataset[[\"Cgroup\",\"PhotoID\", \"CellID\"]].iterrows():\n",
    "        filename = \"C_yor202w_0_0_%d_%d\" % (row[\"PhotoID\"], row[\"CellID\"])\n",
    "        image_path = os.path.join(dir, \"images\", filename + \".png\")\n",
    "        y = [0, 0, 0, 0]\n",
    "        if row[\"Cgroup\"] == \"no\":\n",
    "            y = [1, 0, 0, 0]\n",
    "        elif row[\"Cgroup\"] == \"small\":\n",
    "            y = [0, 1, 0, 0]\n",
    "        elif row[\"Cgroup\"] == \"medium\":\n",
    "            y = [0, 0, 1, 0]\n",
    "        elif row[\"Cgroup\"] == \"large\":\n",
    "            y = [0, 0, 0, 1]\n",
    "        images.append(image_path)\n",
    "        labels.append(np.array(y))\n",
    "    return images, labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前節同様に、訓練データ、バリデーションデータ、テストデータで分割します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 全体を、training, valid, testに分ける。ここでは、3:1:1 に分割。\n",
    "# training + valid が、機械学習の training data に相当。\n",
    "datadir = \"data\"\n",
    "X, y = make_dataset(datadir)\n",
    "X_tmp, X_test, y_tmp, y_test = train_test_split(\n",
    "    X, y, test_size = 0.20)\n",
    "X_train, X_val, y_train, y_val = train_test_split(\n",
    "    X_tmp, y_tmp, test_size = 0.25\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "前節では、各バッチでは、DatasetFolderの``__getitem__``関数を呼び出すことで値を読み出していました（2節の「学習中のデータの読み込み（バッチごとの読み込み）」参照）。ここでも同様ですが、前節では、値を読み出すときには、PyTorchのTensor型に変換をしていたところを\n",
    "\n",
    "* 指定したパスの画像情報を読み出す (pil_loader関数)\n",
    "* 必要に応じて、画像の大きさを変換する (後述)。もしくは、水増しを行う。\n",
    "* 色の正規化を行う（後述）\n",
    "\n",
    "などの操作を追加します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class DatasetFolder(data.Dataset):\n",
    "    def __init__(self, X, y, loader, transform=None, target_transform=None):\n",
    "        self.loader = loader\n",
    "        self.samples = X\n",
    "        self.targets = y\n",
    "        self.transform = transform\n",
    "        self.target_transform = target_transform\n",
    "\n",
    "    def __getitem__(self, index):\n",
    "        path = self.samples[index]\n",
    "        target = self.targets[index]\n",
    "        sample = self.loader(path)\n",
    "        if self.transform is not None:\n",
    "            sample = self.transform(sample)\n",
    "        if self.target_transform is not None:\n",
    "            target = self.target_transform(target)\n",
    "        return sample, target\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.samples)\n",
    "\n",
    "\n",
    "def pil_loader(path):\n",
    "    with open(path, 'rb') as f:\n",
    "        img = Image.open(f)\n",
    "        return img.convert('RGB')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画像を読み込む際に、画像の大きさの変換や色の正規化を実施します。画像の大きさは、機械学習のベンチマークでよく使われる画像と同じ224x224に拡大しています。\n",
    "\n",
    "前節の事例では、特徴量も、クラスも、また、訓練、バリデーション、テストのいずれでも、一律に変換をしていました。ここでも訓練、バリデーション、テストのいずれでも基本的に一律の変換を実施しますが、特徴量は画像に変わっているため、画像特有の変換が実施できるように、それぞれ独立して定義を行います(data_transform内)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 画像の輝度値を補正するための関数を設定\n",
    "# ResNet等のPre-trained model 学習時に利用されていた値を利用\n",
    "normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
    "                                 std=[0.229, 0.224, 0.225])\n",
    "\n",
    "# 変換後の画像の幅と高さ\n",
    "WIDTH = 224\n",
    "HEIGHT = 224\n",
    "\n",
    "# training (validation, testも同様）時に、画像に対して変換を加える場合は、\n",
    "# ここに記述する。ResizeやFlipなど。\n",
    "# 参照：https://pytorch.org/docs/stable/torchvision/transforms.html\n",
    "# 変換のあと、pytorchで扱うために、Tensor型に変換してあげる必要あり。\n",
    "# normalize(上記の関数)は、Tensor型に変換したあと、実施\n",
    "data_transforms = {\n",
    "    # training data用。必要ならaugmentation(Flipや切り出し)を行う\n",
    "    # 今は、特段の加工は行わない。\n",
    "    'train': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        #normalize　# 後で画像を表示するために、一旦コメントアウトしておく。\n",
    "    ]),\n",
    "    # validation用。通常はFlip等は行わない。\n",
    "    'val': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ]),\n",
    "    # test用。こちらもFlip等は実施しない\n",
    "    'test': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ])\n",
    "}\n",
    "\n",
    "# クラスの変換。今回はPytorchのTensor型に変換するだけ\n",
    "# 他に必要な変換がある場合には、画像同様に記載可能。\n",
    "class ToTensorOfTarget(object):\n",
    "    def __call__(self, target):\n",
    "        return torch.from_numpy(target)\n",
    "\n",
    "target_transforms = transforms.Compose([\n",
    "        ToTensorOfTarget()\n",
    "])\n",
    "\n",
    "# 画像とクラスの読み込み用の関数を定義\n",
    "image_datasets = {\n",
    "    'train':DatasetFolder(X_train, y_train, pil_loader,\n",
    "                              data_transforms['train'],\n",
    "                              target_transforms),\n",
    "    'val':DatasetFolder(X_val, y_val, pil_loader,\n",
    "                             data_transforms['val'],\n",
    "                             target_transforms),\n",
    "    'test': DatasetFolder(X_test, y_test, pil_loader,\n",
    "                             data_transforms['test'],\n",
    "                             target_transforms)\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以上の準備が正しくできていることを確認するため、image_datasetsを呼び出して、帰ってくる画像を表示してみます。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQUAAAD8CAYAAAB+fLH0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztfX+sdllV3rO++2OSCgmgglMYO0DQRE07IkETWmpLVSCmI0200ERRiQMJJDWxiQOaltiYUCsaTRvaIRCgofxoECGGtlJSoyYFGRD54QgMiDLMZEbEAilkvu/eu/vH+677Pe+6a6299znnve+59ztPcnJ+7bP3Pvvs9exnrX3e80opBQsWLFiguLLrCixYsGBeWEhhwYIFG1hIYcGCBRtYSGHBggUbWEhhwYIFG1hIYcGCBRvYGimIyLNF5JMicq+I3LmtchYsWDAtZBvvKYjIHoBPAfgBAPcB+CCAF5RS/nTywhYsWDAptqUUng7g3lLKZ0spVwG8FcDtWyprwYIFE2J/S/k+HsDnaf8+AN8bJRYRV66ISJR+Y/+838qM6qVorY+Xjx6z52pltpbfm29PuVNeC/j3MfRZt17Xm3+UPsunpQyv7cb281LKF0sp31xLty1S8HrDxh2JyB0A7jhzoQhEBFeuXDnd5oXTaCOVUk6X08Jo2xpa1lk5T15HdbFledd68PLT+7YLp7f198ritrD1b2nXqK1aSNrm1ZOPrXe22HuvGYzXN6J8oucX5dFSb6+8DK2k0ENKDz/88F9UC8b2SOE+ALfQ/hMA3M8JSil3AbgLiJXCggULzh/bIoUPAniKiDwRwBcAPB/Av9hSWQsqiFTJ3JTCLrD8IPAstkIKpZQjEXkZgP8JYA/A60spnxiZZ3jMcx+GQt2SHt87SuvlY9N57kOrwdbuwStrCvcha5uhpJBJ+Zoc96738m9xGzxXoIbIla25GN79R/t6LCPmljxasC2lgFLKewC8Z4J8AMTBxejBeUaRwRqyl2cUGOSHxdfa2IN3rWc8bLg1tJJYjRQiI54LKdhjXloPXhpdn5ycuGXpca1rLTbE53U/izPY62yf4+dpn61nD17fG4OtkcI2wI1nG2AqpTAErSOtd51nRFE+Xp62LWoG2qIUasY8hhT4WDbSt5KBTWvzi67ndrOEcOXKlbQvZSRs87XlZ2rHK6eWpoYh186GFGznsZ08UwO2ofk6m5ZJpWXk9+oXGX5kZNk92zzGug/ZNUOVQo3ozoMU+JhnLJEBZQapiuDk5OTMcWvYGTxSjkiBVYitZ3T/tiw9b9tXj/H9XGhSiOCNgHw8unF7LDMkPu+pkCwPfig6faj7LaTAae11tTpreiaxmsEPUQoZCXr34t2/lxaIpwd1u6YQavnw2hoLP2dWCTZvbuPIyLx74j6lZKAqhMnBq4vtj1HevJ3VrwezJwVFdLPRyNFiUB5qSiG6hhe9Xg0wu85uM7mMUQoRAbQohW2SglUKYw29JW3kLihJcHpVCXxfmaF55+0xSzzAihx028sjGumjZ5/l0YsLQwpzRcuonF1rtzOD1G1WJMBZMtRy9eWnbZNCZvy7JAW7WKWg+7o+Pj7eIHbP5Wg1VE6jCoQJZ4oRfVuYDSm0jIIKrzP0lsWds6Xc1lHRjvitiiXKKytDj3MbMBGMJYWaYdt6RPX1rgVilWf3M387WtuZhZOTkw1S4H1di8iZdJ6iqM1O2Lprmlaytejpp1NgdqQQ7Ss8EvB8sagM73wLIWWjujU6e25Inl7+kVvBfqaIYG9vr4sUeoKPLR27hVAUduQdohQidQDAJQImA973SIFnI9gNsbB90N5X1g49pMDHWvrtEMyGFBQZK3KD87EoTTa6esd4mw0tq19mBFMpBUsKTA72eu+3E0NJwb4vUSOLljSWzKZWCnaUt25CtHikwLKf87J9MKpb9Hy9NrPPM0ILcYwli9mQwtRsN6Ts3odiDSgywN66eAbp5R+Rwt7eXkoMERkMdS3mTAothKCLRwp2etLWm+vDxOI9V7seQgrngdmQgqLWMDW5mV0fddiWclvzzAzXu7aljpZ0stkJJgImh4wU7LplaVERNVIA2qaUa6Rg/X32+z3jPz4+Pk1zfHy8YfwabNS+xEqBiSMihVLKmWlHT+EOxZSKIMIsSKFmPOdRPq9b0up2i+H01iUaxSPjtdcpGYwlhYwkhs5inCcpRApBDd2qA61fpBT4HqI4BpOBvVevPbx0u8YsSAHw4wdDrmsdvYY+lB5SqJFdVPfIYCN3AMBGuqnch0wN1EghexZ879FzHkoKbMA6amekYLf39vbCQCSriihu4bkzraTAAU3v3noxlGRmSwpj8ulZuOwWA7aG6BmtNZzoPr3yo0Dh3t7eGWPX9FqXjBRs2/A1UZv0Bhpro+BQUrDHsrVnpF6coCXwyC4Gk4KuvTzY/Tg+Pt6oO7u32r42TpHFK7x79trL62M9mA0pWLTejG2MIaTglRft94yWtgybT1Rnz6iZFHRtSSFbbFlZe3nnovTRvfC+bU/e1rplz7VmCB4psKF5MwgZOVhyURLQfDz1oWSg4BehbHt4sS8ROTOzYft2RJRR+qGYDSnYmxjrPowlhaicMcTj5QNsjthRTEBJQRdNa6/1Ao2Rkbbutxh/tK6RQtTutTgD73ukwFLcvmsQqQYd3dnQ1OCPj4+xt7eH4+PjU1Jg5SAiG9dyWVG7WWgduW089cD3z2nsdUOJYTApiMgtAN4E4FsAnAC4q5TyGyLySgA/A+Cv1klfUVbfVqjlN7QqZ/KJ8hpDCtH1tQdt8/c6iOeCWGWwt7eH/f191z3wCMGe99qmZvj2mlqajAAyUvDarsd9sMfswoTguRRMCnt7eyEpKCFcuXLldNsSAl/PwUs1Ur7fSDHY+7JTnHxdNHjWzmcYoxSOAPxcKeXDIvJIAB8Skfeuz/16KeVXR+Q9Gr1G25OfF63Pjun1vI6utarAkkIWW/DIoIUUuF58ridNDb2dM0vf+yztiMsGqmVZxaBQleEtnovAdffKsvna9Hqudr9DlXQLBpNCKeUBAA+st78qIvdg9Wn3raNFCdj92ujeMtK3xhDYSJkEvDrZYOJQUrB52XgCl60dKrt/b8SKwPlFsjU6FhFLi/sQnbN+duRzs1zn58NpM/fMI2FL9Koi7HsQ3L78ToWWOWSEnwqTxBRE5FYA3w3gAwCeAeBlIvITAO7GSk38zRTlrMs602CZjB2rFloIxhpi1Dmya3hh42cS0OOZIvDW9l6A+otenL4l4j0EvaSQIatbTWLzfXr10HZk9yGK33jtblUFz2hw21oysORdu4+pMJoUROQRAN4B4GdLKV8RkdcA+LcAynr9agA/7Vx3+r8PrTfVImNbSKFHKdRIoeZKRCSRdawhpODVAcij+7XO1mJYnjF6xO1d30sKPZ2/1f2IyuW2YUKI3AmOHwCbysO6GTpVyfECL3bQev9Tq4pRpCAiB1gRwptLKb8FAKWUB+n8awH8jndtof99uHLlyqC7sg+31fhbR8go7xalUBvJrXGzq9BKChEp1dQR+7qeEffIdG8dHbPnMlJoQUbkdtT30kbnuU5qrEoMShI9pOA9G20HS8zbeDW6F2NmHwTA6wDcU0r5NTp+8zreAADPA/DxcVU8U67rPrSQwtCO16IO7HmPFGz8QKcWLQFEMQRLEl4dgOt+MNff87F533ZO7pReZN9e7xFEq4RvfS5ZOu8ZewNCdE3Ud7z3E1pIwebj1YGVgQY4Oa+L6D48A8CPA/iYiHxkfewVAF4gIrdh5T58DsCLR9XQQdTwraQQdaBou6YSMpcgWzxjZzKISMFTCtF9e8ZeM3RFZPzetUPynVIpeEboqaWojaJ2VD/fvtPQSgoWqja8NJr32HYZizGzD38IuP8ZOfq/HnoQGas9p/veurbtGX80+vMxfa8gIoQaKVhlYANawGZ03NY/M+oWksiWLG2UL9eF29a2efasa+cycrTGX0vvleG1rSoKrx72uWts4tq1axtKgZWBbaPeNhnrcszmjcaWG8mMWveHMGwm+bwO5ZGA3eeXjawxe68sRy6EF1iMfFTtWJH01/UQQ982KfQ8p9ozzEhB28bbjtrVK4sDuNHA4w0cXMbR0dHpNisIdiNsnp5LwfX0XMFezIYUhiJTCXzeHquteTsK6mXugBcY9NwA7xi/uejFD7yYgYVnoLw9F1Lg5zgGY0hBDdwjBs3H1tmSgqdAVBlwfnaf87bt2EKal1op1NA6mtcM3svXpuHIsWV8SwaZ0XsxgRZSsMesUrGdxXbaocbemm4qUsieSy/GkILWU+MHtp3taMwBR77e9hf9rQT3maOjozP1zNozuk9bJwuv3VtxYUghQmQkfI6P1c575OJJQC/wZxXCwcHBGdeghRQs2Xidm4lA62kNr3fpuS5La8/pPtdtavSSAhMA19UbELz89Dq+V9tXTk5OTkndcx9sO3EA07oO2X0zudl+MASzIQV7I15jZMqAj9vra+oh60QZKXiBwyxgGBGEt/biB/Y+vdHEGqL3QZCLSgot12VGHBkQuwI2H73WHvOMj6cVlTR0NkHLV6XA9+R9BKbVjbHbrCCGtvNsSGEMIlb31rVrLZtHJOCN+NmUYhRA5HVUlq2/99CtYUbfBxiqIGwZXpktpOARWSuGkoJeG5GCt1hCYEURDTycr27bH1hZA44+8sKKwbsfvqch7ZThwpBCTRb2kIJu21HYk3hZINEatyWDg4ODkAiUBGryEjhr/J70jAhhLClw+TWi8FRBK1quac2XR1NdrHFFpMCxAu/52/wVPIpbEvAGk5OTE+zv7288I4ZHHva5R4qlNhDWMHtS8G5sKlLwRgM2VH6gns8fLR4peHlYl8TeA+C/SlwbqfkDIGPdBy7blh/t22syjB3VsjzYcLyRnEnBG53t89e0Sg5ef2PwOS8fJQav/fm+7AtNHolM0Y6K2ZOChW18Tz1456K15zZ4sQNP8kfuwf7+Pvb3909JIYoZeGVzvRjZqJ2phIwYonz4OJdp6+LV6zzh1cNLkw0gPOp6SkGDhUwGQKxcvbrwM9Z8SikbhBARMIDTD7zwMUvGWRv0Ypak4BlFTR30koIdCTyZ57kN9n2CLJbApOAFEb37snUH2n15fg2XlUIrKfC+bvPaStXadkYs2fURhpJTZMBs+KwQLGGXshmM1P2s73FZPFOhx7l9ovZX2O8/ap1t2kwd92CWpMAY6j7wtT0KgY1XXYkoJpCRglUK2YwCr6OOzx0gMnL7IdGh7oNXtvcsMkNvHbWmGN08Q+K8vX4REYNuW1LgeIKO9F7/0fztmkmFVQnX3SPdUsrpjIXXN/hXlVNh9qSg2IZSaFUJtdiBt9Ylcxui+/RG6kwZeKQwxH3gMu22tx9hSAcdoxQsKXjpon5hicHrT5FUZ5fPmynQtSoFSxqRerBgUrCDg42B2LKH4MKQgiKTgWOUQovL0LswKVgSilSBZ7DRR0YzpbANUmjtZDVZ3+peRPkOqUd0vmegYcONBhRWDnqNZ/xe4JHbhJf9/f0z7aXIPsxy4UnBk3d83DvWm79HChkhRGohUwieW8Edxdaj1iHsB0WjYxGBZIYREQSf74WncmxeLaSQPeMs79o1tozoWksYADa+2MzuRkQKHDvy6sqBx6g+IhIqBa6nvSY614ILRwpj3AebjzcDkM04eAafqQTbWYDNH9JEhlIjgBop8FrzjJSCVxdbnxqyvIaSwtAO3QJv5M4MjUdjrbNn/Haf01pwn+N9jxTsLIWmsW6Dl/8QYp8NKbRirPvQ4jq0xBU4ZlB7g7GmcjzjtcbdSwotiy3T1oXXGTIS2IZSiAw5qoNeo/ve9d498XOzbWxJgfsSuwSe4lAisITAAwbX++Dg4LRcDSKXcvbn1dxmY0h1ig+3fg7AVwEcAzgqpTxNRB4D4G0AbsXq60s/VpIvOnvGvA2lMEQlDFEL9jzXQbc9GRiRQWtMYduk0EMQPaTA5duObY9Ngeg+LBHYOlhC8KYylRC8mSyrDHi2g5UF37MqBe4HXEamFIZiKqXwj0opX6T9OwG8r5TyKhG5c73/8xOVNQg1leAdqxl+q1JQMHvrtmegHiG0kgKn0TwjRATB1523UmCjsB3b7rfUjdNmxOPlq3XwRmOrEnRbjVUNV/PjKU2+H168e1elwCpBYxoZIYwh1G25D7cD+P719hsB/B4qpLBtpVBbhrgQtYUDTUAsZ/WcNeoxpBCNxi1pua7nQQqRwW4LvWTi9SnvRScbOLTP276r4OVroUqB/8/SG8y864ZiClIoAH5XRAqA/1xWn25/XFl/0bmU8oCIPNZeJB3/+9BLDN51nrvQM6twcHCw8f6B97No+5AiaQ4gnT7MphxbSIHLytJcFPTWt4fQWtK1KhZ+TrbPsntgicYrjwcqr59qOaoatEyrEIc86ylI4RmllPvXhv9eEfmzlosK/e/D3t6eW3NPMfDxFlLwFIHI2b9osy8e8bZdlCA8v9EqAy9yDfik0KoUekd/Ljvbb4VNmymFobCjbE+evXWw7oTn4lkjt+mZDNgttEThEbPXZzlmYfupvq3K27afMUH0YjQplFLuX68fEpF3Ang6gAdl/f8PInIzgIfGlBE9kNb0TAbZC0o8+mdkoIs3DVVTCpnh87EplQKv7fZFgHUtavUfqiy0DC6vNqrb61uUQg2eO2L7KQcdNdag1/J09BCM/YeobwBwpaz+YPYbAPwggF8C8G4ALwTwqvX6XR15usbfoxS8B+LFDbIAYrYoOUQqRMtmpRAZeRQriJRCTTVwWYxIOUTH+Ljdbj0fHW/1d3vJoFaXCC1KAfA/p2/LtIqAZyisC2HLjpRt5EIwKXBAm+sxBGOVwuMAvHNd+D6A/1pK+R8i8kEAbxeRFwH4SwA/OiTzyMh7ScEjBsu+NULwYgqee2M7mF1nht+qFGoLlxnVpXY8Q5TH0PyiMnrJIKqfIjKSWv7eAJUpMqsUIkLQ9F7+7Ip6/VSJgEmB68BuTC9GkUIp5bMA/p5z/K8BPKsnr8jwvWPRYoOJkTrQ9cHBweliDd8jAXuM7jdce7GCFlKoHc8IgevgPJtqmlZs2yXZljIYUg/PwKzC4vOW0LhPcuzB5qvXMSEcHBxslOvNaGkd9WfW9l+tezCbNxp7SSE6nr1lZhm3lxTsJ9sVNqDjdRYvHjA0kFhDZqw196DFfail9fajem7boMfCU2DRsew8v2fA/de+1ajp9RhwXSmwIvBmvvhFqsgNb8EsSIEbR/cj4+c09lwUP2BC0AChGn4rKXjxB33wKg2BzVHFk/8RKXhKIiKEFsKYSgVMRQC1snrJbmg5Hqzh2JHbM3BOp+e9587n1WA1KMj9xrMBXSsh8EtRlgy431ti6MUsSAGIX0n2zkdpuWG40aJXkJkUmBy8xebDPqKnFGpk4KmGKL3tYDVsQymMKSe7PkrXQwCtaVtjCjYuxBKf4wP2ek2r23yNZ6Q6oGR9mvsz9wvbF5kQLDH0YjakwCwJxMZv4RGDJQQ71cgvJPWQgl3sgwfOGopHBJFiaFEI54XzVApzh1UB3vx/TSnY2QAbS+AYgnUvuC9rP/GUcORC9GI2pFAjgJaYgxdUzF5K6iEFzzXhN8k8drYGH8UQoheXel2HXgOek1KYq/ug5yOl4KkK3bZKwZKJNxDqNawgetwH7qcXmhSsP6XH7H6kFKzUap1qZDI4PDxMg4tRsFPhzQt7hj4kjmCN/jywKIXrsPcXvSnoxR90bd9X4fNR7Mz2Z+4HUaDxUimFDJ5BeuogemW5hRRaftPAdfHQYtS9Mw27IoUFm8ja3usPnvugaTkvT/FaN8NTwdmSDV4tmA0pRO6Cl87KpNa3Ea3xR66D92tHWycrG+1IGikETx14bz1GOC/3YaxS6CGwbVzvYUigUc+zW2DT8XkvPQf/snpon7Z9gAOIdkoymoEYSgjAjEghQ6QQPHchIoFs2pHX3tSjjTZH8FRBr1Lw8lmUwu4xRikAm+5lRgpq/HzMkoXXR/f2Vn97PwUxzIYUWpRCLX5Qcw08VyFyHThflny8tqMDb3vk0PpGYkYCre7FRSCQSN1k6ceW56EWaOQ0nlLwrrFuQtZftD/bvmH7ulUPl959yNDqa0XE4BGCRwYHBwen+ahboo3cYnSeYXvvJHikwXn2kMSC84Fn4HosUgp6TlWCfabah/UtR+4jCg7C87Wt7sOFJoVMKUTBmGz60Xtb0e57U5Ue8wJnjdyLKWRGX1MKmg+vvbxr5fF1HqI0UR2iOkV51sqv1Wmq6z2MiSnUyvdUABOB9/y4T+sr0DzLwPWxs3OtSmEIZkMKNWQR2Cx4eHh4GJKCXhvNNjAp6KupihYS6Fk0T2+9YD6I3IcsbaT4+P0F+0tKrwwmB++dnKmIYZak4CkDe/Oee3B4eOgukcvAcQM79Qj4sQFe7J+4trx7MDW8/IeOEAu2h9ogki16PcPaRRRvswqjBbMkBUUUUOTAolUFutx0001nSCFyFTI/LHpoSggeMUTuwRRoyXdRGPNHRg52UOH0jIgELFH0YjApiMi3Y/XfDoonAfjXAB4F4GcA/NX6+CtKKe/pyDecgsxmGiwR8L4XO/DeUrSs6j0wSwb6ld2I5Wu+v53r9tZcH4uaUsgi5LXnYH1lr2xOa6/PrhmCsWQXXV8zHPuM9Ji3eOVl7iL3L3UfbAzKtm+mCiL3tweDSaGU8kkAt60ruQfgCwDeCeCnAPx6KeVXh+a9zjN0HaLfL1hCsKTg/bJMy+K19QXtQxqiEpbRe4FFj1KwiALv3nYvpnIfngXgM6WUv5jSn42IgacQmRg8QlD3wb5/4KkCb9/z+2qkMAf3YYkrzBtZjKG172RqYUygsV9b+Hg+gLfQ/stE5KMi8noReXRvZt7N6rZ1HzyVEJEDp1NC8aK2iiwQFAUZa67D1DiPMhaMx5hAYxZPyKYmvan1FowmBRE5BPBPAfy39aHXAHgyVq7FAwBeHVx3h4jcLSJ368iW+UrZG4sZMSgB8EyF9+KH5w96D09jCFEsYUrjXAz98qE2gnt9rhaTqgUaexXDFO7DcwB8uJTy4PqmHqQKvxbA73gXFfozmMPDw8I+kjcHG72YZN0Gjwy8AAzV48wbZ70sHhFCizFH13juTE0ReA9+V4RyGYmsJ3hqg71ZsNLmY/udTa/PX49PpRAUU5DCC0Cug6z/BGa9+zwAH+/JrCW4WHMbrKvARGBVgc4DM1oJwQaFdK15DIUliUUtXAxksRxr/NHo7SlTNn5bRosb0YuxfwbztwD8AIAX0+FfEZHbABSs/ob+xc6lXl4AfNaL4gitbgM/AMu2wOYv2DxjjGIKnszz5pd5rfAM3R7zVIQ9z23H+/Zcdn1Uz6FpM3j37KFH7raW3WKwrbBtrsfUaNnwbRpbplWpXn8TEbefem72TkmhlPI1AN9ojv34kLys0fa8sGSJgY/rN/O9aUcrw+gemtSCJ+mnGNmnzGvB+aFVKeh+i1JgUvAMfHZKYUrYmIJHDPbbBx4h2MCi/SMNBb8sYn0+XUezCi2kwHll6CGAWt5LTGG7qMUUIoXmqQVPMXh9j7+RoOmtG6H7l4oUPHkfuRD20+wREfASSfkrV65sSDKFNdKWqaJFKSzoVQpRHl6/U8PXNJzPDaMUPDKIPpcWvbHIDBv56pHhT+US1FyM1sXeQ9R+rUrDa5PoWG/aDJeJ4Gr3khGBHQRbpsQt6TAhAGfjcb1TkYrZkYJ9USmKK3hfS7IvI3lTPnZkj0ig9rqpJ/ss7PHM6GvKo1YW18sr97xxmYy/BZ5K8FwJ3feMtUYMfEyfs6rdKd5PUMyOFDKlUPtSc/QlmlbjnFIheOXpdlYH7/yCi4GIBBitBMF5cp/w1AbHFDgmd6lIoRZTiJSC5zp400IZIbQGEb262/yjslpdhYwUog7IHcdTGXN0H3YdIM1UYJaulh8/B5X4NsDY4zpk9WSbscTQi1mQgg00Whei5j54P3iKGDMz1h4ymNp9aCEFT/V4bemVe9640RSOHcX1GLBd94HzilyIXsyCFIBcKfQGGi1jRsZZI4Epfu1YI4Cay9Ja5kUNNA7ptHNBrX0iMrDGmr2vkP2exlMKnL99e7cVsyOFaFqlRSlkkdfMKFteYd42xhDPghsX3izGUIWgmA0pKKxSsKO/pxIy2RSRQM8XkzQfW8/MbehxFabAQiaXG5HRR6QwBrMhBe/marMQ2bsJkUro/ZGTZ8Tsz2X+ewsZTEEO7M9eRMUx1PWYqpwoFlBLN7Rs7j96LsrbM/IaKUTXtWKWpJC5DrWF8wLOEkIpZUMhWMXA6bx3FDJisMGgDFEam2/twWaxh4tADtE99tS7N+5Su741XU/MxfZJ7jeR6rQxgygWEamH7F4yzIIUlAgAuL914Fea9WtK3leV9vf3z3QyJoTj42McHR3h6OgI165dw7Vr1073dbG/hPRcCW58r6MMWcYgqsNFwaIU6kohWmpphmAWpAD0vafgvdas7kPE2D3uQ/SuQi/OI5aw4MZDjRTG4sKSQhRwzOS0Ysxo7hHNGBXQ+xBr97Vg/mh1V/TcGAUwhCSafkIlqw+wPiQiH6djjxGR94rIp9frR6+Pi4j8pojcK6uPtz61tfJMCHYGwntnwRKCrjkP2zCtRj0lvDyzAFLWCbIgqr3HWp5zWGpt1vJMxpJz77Mfck9Z/t65Wvt4Lk/P9Rlaf1f5BgDPNsfuBPC+UspTALxvvQ+svtn4lPVyB1Yfck1hCcG+fNSjFFre6BqqFLYRL5jKiPi+FswbU/SPVgy5pokUSim/D+BL5vDtAN643n4jgB+h428qK7wfwKNE5OZaGfamrevAa/tPT977DNwQuj21QbciYnBbvxZC4LVVB1YF1dq4tawIXhu3jHBeGS3tPvYZtV7vnffaJNvm/ew5RceiMhm27aLrezEmpvC4sv5AaynlARF57Pr44wF8ntLdtz72ABJwA7T+fNpTCgBOv1ZTSjzdNZQg7IPsvY7vN6pfBr2m1qFsu0bnI7TUjw3Gu7/smui6IXVtTdtSpyydPc/to9u1MrK2igihx0Xxru/FNgKrlqUqAAAgAElEQVSNXi3OPDURuQMr9+L046rr46FSyNwIPQesGke/qOSxtmXu6PhUpBB1Ws8v1ONeh7Pp7LW1UaKng1jyyervkUfLPU+lxHYFbp/Wtm1pT96PFptPpgx7MebPYB5Ut2C9fmh9/D4At1C6JwC4315cSrmrlPK0UsrT9vf3q3K2ZTYi+sm0Z5hDjHjbqMn5BQvOA2NI4d0AXrjefiGAd9Hxn5AVvg/Al8v1/4Fw0WIMliCYKDxiAK4rhujV5agOHixp9H5zwSsrwlD14eU7haLZBS4bMdYGvSEDwraeZ5P7ICJvAfD9AL5JRO4D8G8AvArA20XkRQD+EsCPrpO/B8BzAdwL4GtY/Qt1FfwRiqghasSwv79/5tv4nhE33G/qYnCe0UtOmZTP/MuWcr20Q6SsB6/+24ZX34j0zqv8DNkzaC0vU8TegGjLj1zdrK6taCKFUsoLglPPctIWAC/trYiNKbQskRtxdHR0aiQtH2LNlMnJyYne16AR1zN864tbQ/QIJiIdzsP6m7VOEm23pK+hx9BaCWAsUfUYrVc2t7OH6Dnb/jVEKXCf4fK2oRSm+tfp0YgaIUoXxRj4pSUmhcx14LxbcB7S2xKCV77FRZfYNwJaiYHT9GCKfjiL15xbGyJzHXpiCi31qcn4ni8mjXEfPMXgpbXuw9DOsbgPdWzTfYhUqy1/5+7DeSBiUD5vXYfoPYYWpeBJvYyQhroP2b1mkj0imsxoI4lp865tzwHbcB92iV6XodYfgR0HGucOqyB6GnbBgouMbRDDhSeFTG5FwR2LbcYGFswHYwKNLdcPGe2HwOurCyk4GCPDFiwYiszF3TamcGU9zIYUslmHLA5g114+Ud6ev24X+6C3rSSiYGIWT6jtR7DBSb72PO6Vy9wlaW8zgBkRRRZPstfrce6TUb+4VKQwFotSWLALLEphhlhiCgtasY2YQkYKS0xhBlhUwoIbDTe8Uqjd8BAXYsr53QU3Fjx1OjSf3r7LfbX2AeJezJoUWgzVO84Nmf37rie/vHOL2pg3xj6blqnGKG2L+xCVkfWrzC1WtBJDL2ZNChEikohiCLWH7kVyFyw4b0T9Neu/N4z7EN3MFO6DnXpryZOn6ez0pM1vQT/GjvQ9U7Bj0tVUQi0/LwDO+y2ug/a7xX0Y4D7YL0Iv7sPlxY3sPvCP/W4IUrAYohR6P/e+YEErzjPQaNH6jsKQPl39noL4fwTz70Xkz2T1Zy/vFJFHrY/fKiJfF5GPrJf/1F2jNXpuejHoBRcRre5ujRha/waxFS0fWXkDzv4RzHsBfFcp5e8C+BSAl9O5z5RSblsvL2mtSGTgNf+pVzm0lL0QzIJtIHIrWNV6i+eCcExs6v9CrZJCcf4IppTyu6WUo/Xu+7H6YvMkiFSAvVHrS7UQA+9zeVG5CxZMgYgEMje3NqBZOzg+Pj79l/QoztCKKWIKPw3gbbT/RBH5YwBfAfCLpZQ/8C4S+t+Hw8PDM0ZYI4eMDHRWoCVQY9eWMGqBJJu3d0zz4fU2wWVkvq43e2LrnKXl8ry8x2AbbZQF9cZcz+db+kiUtqYUAP8HUDWl0ItRpCAivwDgCMCb14ceAPCtpZS/FpHvAfDbIvKdpZSv2GtLKXcBuAsAHvGIRxQ6XlUIrSphXcfUdbDl8nULFkyNHjLgTwsC8af6PDLYCSmIyAsB/DCAZ5V1yaWUhwE8vN7+kIh8BsC3Abg7y8ur/BD3Qa+jOto6u+6D13B2pFywYCiiganVfWBYYohcBT7Wi0GkICLPBvDzAP5hKeVrdPybAXyplHIsIk/C6p+nP9uTd+Q2DAk0ruu0KIUFs0EWM4iUgjfgZTaxdfdB/D+CeTmAmwC8d2087y+rmYZnAvglETkCcAzgJaUU+2/VLryAn71JDaZES09jLEa/4DzRSwScppcMWv7OIEOVFIr/RzCvC9K+A8A7umsBnMocz/iPjo5w7dq10+Xq1at4+OGHcXBwgMPDQ1y9evX03NHR0SlBeAGxmjxrQWugkc9FwcYoeLfgcsBzEby/Joj+RV2VgiUDax+62EHy3AONU8KLqPaohSjIwjMRgP/JtigCb+Exd4Zs9iEq0/M7bXpbB8/vbCGbjJy8MqL2idy2GuwIOBcMmaWwbRX1s0ghZLEFzT9zpz2lfO6BxinBlbc3bG+c2bHXfWgd2WtYlMKCVniEkLkRGTEAuX1Eg2IvZkEKjIwJ1fA9MvAkUy3wuMQVFpwHPKWaxRSymYda4P3SKAVgM6aQLdaHsr6UbZwICyEsOC+0EkCLOqgFGMfOPAAzIoXa7IPnNkTbWcN4AZ+pApALLh5a3MhWVzNzCTK3YQoSsAphZ280TgkmBe8FDEsM0aLpbJwC6P+Jqo0Z2LhALaYwVaDxRkBL8HJMftEx73iL8StaZ7g8tRAFJDXfWqA9elnpUrgPXPnWWYeIFGqBlhZS8B5UFChswbYDjUOJo3WWYluoGdd51qFHKWTk0hNI9IjBC7q3KgWPEC4sKQBnlUJGDBEhqELQ+EQ2C5ERwoIFFi1Kzm5n7kL2QWFFRAhWJUw58wDMiBTUkPVljdq7CUoCTAiqFBSZ+xD5eozFfTg/XGb3wfa1aMrRcx+uXLkSkkHmRlwqpZDNOrQEGq2RWbS4D9n1Cy4XWt0Hz7WM0tViCbXpRzbk1pjCpZuS9PyoLAKbBRqzh1cjBE8ZLLgcyAi/Zb+HFFoIwCMCq2xVUdbe4o1iCReaFCw8ItD92o+i+MEA/uu62QNjRPuZqzBWbVi34rJjW4qs1X1ojSVFpGC3M1ch+nCKwhscAYS/beCX9TxyGIrZkEKmFFpfYDo6Otr4MAV/oALon5K8UQzzRsB5KYXaB1NqLoPG1tjAbdwseyfnUimFFvehphI4pqDY29sDsMn00Q9R7G/Xd+lCtI5gEYbWm5WK7bSRihkz2kf1HKsgplIKXjDQDjY2bY0MvE+s6dqO+tZdrrkQlyamALR9TyF7X0H3gc0HbV2HKADED37XhLBgegxVCp5KmEopAGdnyGpBdu7vLYQwpA8P/d+HV4rIF+T6/zs8l869XETuFZFPisgPddcI9dc8PTLwGss2TOTbzXG5UbDrdm59DtH5FhKIiMGC+7rnIvcGG4eiRSm8AcB/APAmc/zXSym/ygdE5DsAPB/AdwL42wD+l4h8WynluFZIzX2w5BApBZ3XjUb61k6wqIQFQP6im56vpW3pZ9zPW97ebY0nbMV9KKX8vojc2pjf7QDeWlYfcP1zEbkXwNMB/J9KGd2k4J3jNPzFGsXQEXohiO1janXk5Zcd6zV07/qacrDXR7GEse8lbN19SPAyWf1t3OtF5NHrY48H8HlKc9/62BmIyB0icreI3K0RVw8tsxFZIwVld5HDecjSGxXndf+RodtztTymfv6KTA23uAxTYigpvAbAkwHchtV/Pbx6fdxrXbfGpZS7SilPK6U8jaO5lj35uKcIPELQtYceY12wAIiDjbUliltpPgpvwPNIYJvBRcag2YdSyoPUYK8F8Dvr3fsA3EJJnwDg/sY8T9eROuhRCrax7ANpmTu2Mm9K2LxtZ+kdxRR6r5z3FBDp+0Zja57e9hTYtvsQXZdNQdqZB23TLKBeUwsZMZyr+yAiN9Pu8wDozMS7ATxfRG4SkSdi9b8PfzSkDEWv69Aiqxb3YT64zO5Dy+vNCk8peIrBiy1MPWgN/d+H7xeR27ByDT4H4MXrG/uEiLwdwJ9i9XdyL22ZeVhfe7quBRpb3Qcv0MjbtRFgm0phwSYuulKwRGAVQ6RGgdg1bokn7Gr2ofl/H9bpfxnAL3fXpBFeo+k0zbVr104bPws6RoyusQ12OTR9tm6tN+c9d7TcG7sT531PtfoNrVOk5Gx/qZFCRgBA/VukvYp45zGFbcAGFb3znnJQUtA/idEHoX+ooT+Ssh1YH6D+6Yb+IYeysULfedBrgevffGBD5zSe6vHusYUohoygNq+sk0T5t3QsVlK1eo7tqFH+Wbm1e8vcRN5XQ+c/aLFp9RinyV5njoy859ePGRlsVSnMCV6jskq4evXq6cNTI1ejtuzPD47/lUfP2/KAzQ/A6D7nn5GZzW/uamGMnN/2vXnGbMsf4/pZY48WPc8EoaTgKQV+7q0KoIcYuIwxfWx2pOCNqvZ8phSYEJR1+WfUuo6IwSvHdjpWCrXAkXdfHknYe7ajsKd0xrQjl2G3WxG5D97oGI2Y3vVZ3WoxAD5m247L1O1IHWQxgixWsLe3d8bN8Iy2JWYwRClE/a4HsyOFDJY9uTGVFPb397G/v3/6ZWdP/keksL+/75ZlFYLCCzxFdb5IKgGYv1KIiMKSUG9dWghBDZ+JwZtl4PiWN5htQy1cKqVQYzePaT2lcHBwgIODg433w/UBlVLOyD7rPlh5p9fpNVnDe3Kx5Z5bR/NW2PyiUXxo/jVEymmKPKM4QFRuTaF4eVkF0KIWPJfBGyxq7yF4M2xjSOCGUQqe+7C3t3fmn6fZffAMw3MfNG89zkrDYlEKPs5TKdRIYRtKofbbBs6L13ZAGzqzcEMoBc9obANaA8wYN3u5g1UDxx88X4yPMXvreVsn7kzA9S8/2fuzboge70WNrLw0PUpmKGqE0qJWplQwtXZqVQUtb8ECZ1WvN8NQe1ux5l5skxhmQQoMSwjqJ1piiKZwomMcbNSHf3Jygr29vTONDCA8ZtUD58eGz+RgiYCPe/cfoWY8NqiWqRyu+5C6tNQtOt/ictn4wBCSsPfmxZbGEIJHDJ4itO5uTSHU3l70iOZSkkI0SutNA5vSzlMFUYN7SkFENgjBlqvqQc/rQ88Cl5lS4HtjghgzWnv18OoT4TzcmBYVkI3k26qDHuf+4AUUe1UCkMe/amqhZYBblMJaFXiKocV94PxsJ8hIwSMVmw9w1gfVdLbTeAFL794jKWpHToXnKmSkYAnSbvO9evVj9BjtEAOP7lfP2Tax6b00nI8lhJpi8NRBTSnwwDLGfYjyutSk4DWm3nwk2VvdB0+ualCR303g8j1mZkLivFqUgiU27/6t/Lfw5HAvKXhlcp7eM+lFizqYIs+Wa/gePVLIlELNfeB8FF7/ndp9uGGUglZeDeb4ePOfo3lKUY/rA9LfPFy9ehVXr17F4eHh6RQlv9CkjanX7e/vnxKEno86wbVr1047gH5KnhGNHN6D1M7Kx7UNWqQ1d3a7ZMa9TXfBG815Ozpv6zR1Xfk5RUqBl/39/Y1X36P+YPOMiCD6nJpOo7d8h5H/TT1zJaLtXsyOFHS7ZfHkmDY2L/qg7VuOalg8slsj9eqp10f1V7C60fK4bFuOp5ay/GtKYFeoKRAPNZXUiohUbVtlcQSPDGruWKQwa65CyzImnnBhScEzAGv8wNkHbBvfsjC/5cikcHx8fGY2wuvE3ijG5XnyPYox6D0o8SjsrIW9f933jMbroGM6Qy/YjbEuGp+L0npuT1RGDyKl4j3zLKho31y0isz2lRZCyP4xvUYGN6z7oNuWDDK1EDW6VQvc2Nwx+cHbhmQ3gMuaSikwWTAJ8L2zu1AjBlsnSy4R6XnqqNah2OCtS+CRgXeOjTZSQ159PAXQEl+J4gjsMkS/hozaulXBRv/yxMf4uowQasQwBi0fWXk9gB8G8FAp5bvWx94G4NvXSR4F4P+WUm4TkVsB3APgk+tz7y+lvKSlIh4p6HamFNRAuWGZCKJXn9U4rYSMRjG9xv41nVd/BSsFzziz/axNLiIypTA1PEKwBh7FCjzFYPO09+WRQaQWvP8qsW/gemvOv4UItq0U3gDzvw+llH9OD+DVAL5M6T9TSrmttyKeFLOjSMTI3PhMDPv7++6rz14Qk19z5lFBiUKv0x9beSNINCox7D6rhogUrEHxcTayKUeLFniGHiFTCrVRuAf2WdipRk8l1NwHrwxv4Mr6ZKQahroNO1UKJfnfB1m12I8B+MejaoHhSoHVQqQUdDYiUwr8K0nuSEdHR6ekoL+xmFIpeA/TWy9KoR2Ru5BNPUYzDpH7wseyQaoWS7BuRTRN2UIEUxHE2JjCPwDwYCnl03TsiSLyxwC+AuAXSyl/0JtpJq2jtDbgeO3atdMHfHBwgP39/Y21kgpwPQjFnZWnqE5OTs64H1pm9p6EvQdWPxGi4Fvt4UbtlCkVS8TRsSGI7rOmKlqJIlJqnpvQQgqZutP6an+xUj6LIWSKIHtnYWpD78FYUngBgLfQ/gMAvrWU8tci8j0AfltEvrOU8hV7oYjcAeAO4Owbfq0BHbswIezv758+bJ590LV2Tu0E9qfTesy6D2zs3sPlGId2IlYmnvG1ICIHT2FEbZgZpJd/CxF5ZbGL4OUVuVtemZyPt+0pgsxVqB2LyNgL9HmDQYsiyIjBDnIREWyTJAaTgojsA/hnAL5Hj5XV38U9vN7+kIh8BsC3AbjbXl9KuQvAXQCwv79fzLkupWAfyt7eHq5evXr9JtcfXlGloIQBXI8naJxhfW9npKM3a+E9ZO5cx8ebH7JWYsgUw1il4OXn+fNRnlk7t6bNRvttKYXMTaiRhRd/sPcZEUEU08pcBxtUtANOS+xgmxijFP4JgD8rpdynB0TkmwF8qZRyLCJPwup/Hz7bklk0ykUjIZ9jKX98fHz6hqOCfyLNa1UDShTWnWAcHh6eqaMGHTmYxEFIjUfY0dd2vMiY7Vqv9QhTl5rPvk1SaO2sdpTn4y2KMSIE6xq0qILMddD6eC5qbaahRSHUph0zMoj2pyCQQf/7UEp5HVb/Lv0Wk/yZAH5JRI4AHAN4SSnlS72VqjVAlF6nDPXh6jE1fKsUNGZwcHBw+iCiTmbroIHNvb2902nK4+PjjU7FH23Relpjz7a9/awNeO3l0ZL3EFKwJOS5IDUyiuqUtZln0FG8wCMEJn9Whl4drOFHhNAaUKy5D0wOUXtvC0P/9wGllJ90jr0DwDuGVCQadbijZYs+GIYes0qB920A0XuphQ1bO5N+6YlfdlFCUujvKfRHV1rPbESM3AUmOttOnnG2kk6GGkFE6+waWxdvhOZrPD/fG+Wz6caau5DVhe/NqgPPDciIwFMMPQFGrUdPuw/BbN5ojNCqEvShMbShPaWg+3a6Uq+znUW/r6DH7BQWdzJNx9dwnewDtiOClsHrCJZEPEIYSgqto/sYZKTgkQFfE5FCpBSAs59k99rZjs5e7CDa7nEfemcctq0QFLMlhR5/iY1L5PqbjnqcP+zKakHJgUlDR3dLAqWUjRda9Dp1IXhbO6EqB6siRGTjLTWFF4D0DIHbR6/P3ryLtqfuZD2EA8A1WoXnNmSk4LkPUZpICehz5ufuvRTnvY6s64gIWmcYWkjhhlEKUYdqbSRPLei+vtnIP6O26kFfhT44ODiTP7/pyJ+Bty+7WNVw7dq103TeiG3rGnVYTW+Nw74cFbVj1LZjCcLzvVvL1nP2PYGsnBalELkKEcFwvT3DY0NXQrh27VroSkQxgxZlMBZT5TMbUlB4fjMrgIz1dbrPKgerEtSAlQSYFOzD4k7EKoCJgJUCd0T9BkPU6SOj5FkQawh6n3p/HATtHakt6UTn7HOxLkuURwS+Dzuye+l6lYLdj+oWkQHvW1fAKgWPGCKlMGS2YYhSGEs0syMFRosL4Z3TDqwdgd0H7TSsFKxKUHAn4w+yKJEoEegUqP3LMH1Xgo2IO4E9xp2ZlUOkApgQvE7jteN5wCO7SMFwu9l0HiF6g4JHAjX1EfUdVpullDOGbpVCpBZ64we2LrvEbEhhyMPLSIHBswR6zioFJQjuxKwKOJagAcpIKShuuumm07wyF4cJwBJDrX1s3pxn1p42z127D/Z/PDPJnykF61pFbgK3Lz8XXYDrs1fejIJHCFYt2LjD4j6MhCeRaqTA7xrwOU+iekrh8PAQwOYow/8LoZ338PAQN9100wYpMHEorl696tbfCzbpqB+NgIqMBKxS4vbL2ngq9LovnvuQqQPezhauC6uuSIJbQmBZ76mESClYtRC5DL3Tj5ESbHnGQzBbUlDUZJ4nobmR+IHz7IC6Ew8//PBGXMH6eQz7AyubhuWwKhGuP8tSL9YgsvnLT6saIplZI07bHrw9ZYdqIQVOwyqL/5hV0w0hiKguGSFHbyjWlAITR0QKERnMGbMlBa/zWr/bQgnC5mM7gM4MMDHw59q8h61KQdWDvsHI6djI9G1Je5zlrY2Ss+Tk/KNRxWsr7qy8740q2fEWcKyE9+157xxwXa2xysrcB+94pAy8+7SkEE0pahp2Bfgjq5nBtygDW6cplYK9dghmSwrApr/tEQPD+uAeOfAIbN9d0I7JHYJ9RzZe/vozGyHPGigpRBIvCo4xIYjI6TrqLJ485nqp+rBt6m3rOnI/PMPmtJERR+c818sz+FreUd2sSrNrT+rbeIF9R4HTcFvXiCEi4BoyUujNqxWzIAXuAFEDMBnoNl/PxmvBnYBHZX53QUdur7OoStDOq4FJricbHpOCd5963r64Y4/z/Q4hBUugUR68b8mV29MjNruOSMBb88xDKynottbVe9bes/cM2JslYCVgj1ml0EIKtnzb9tGI3jLS39BKgfd5NFaw321hjUVfZVaj5WlEAG5nsKSggUmvLF1r/pH/y2UyKdh3HSwp2I5kDSSLSXCbZlK1FZ7htvj/ih6lYNvYIzA9ztseUXpKQLdVDVjSyGIKUf4eGdg61nDDKgUP3o0zIdjtaISzboeSwtHREa5evXqm80WkwG9BHh4enhq899acls8ferFk4LkPwPVfX2oeLaTAYIXAxOBdHxFM1AFtWRkpRAQxVClY2DrrMb4H+/zZeCOFkP3MOVIKHinwOY9wa8S8KIUAUUNaqW47AsMqBT7OHVHTeB0FwMZvJJQUvB/f8Kif1Y1Ji+/NOx4ZNedjlU7UPj0L18NDZvi9pJBNSXqwAwUfr6kDL16gz1qVQhRvsG6lJQV7zGvLRSk0oFXG2gcObHY2jinw6Guv01FUpbr9BoOm04evBMKGX0o5M9Ixyeg1ANwRyAtA2fthkmklBYbeZzRtuw1S8O7Bq6e91pbDBMnHvGfqEYESvEcGGSl4MQW7HxFATSF465pSGDPiD0XLR1Zuwerz7t8C4ATAXaWU3xCRxwB4G4BbAXwOwI+VUv5GVk/yNwA8F8DXAPxkKeXDtXJshJzXDK8R1XA5sGaNRtNpMJGJgTufN8oAmx/hAFaG7n2ngYlDXYyoQ9rOkxnVUFLoUQm2HPsM7POI3CY+5iknThs98+ycRwSWzPkZei8SWXeg9nUku23LtiTP57L7aEV2TUSsQ9GiFI4A/Fwp5cMi8kgAHxKR9wL4SQDvK6W8SkTuBHAngJ8H8BysPsP2FADfC+A163WKoQ3EhKBGr+e4sfQaVhjqc3NeXiBK03Pao6OjjW8zMDnwm49qlFEQKrpvT4ZnpKCwvxaM4hGto1PtuUSkoNus2FpdAruf1dWO0DWXwZKCJYTaD5ksmXvqwLZ5dH/ePc4BLV9eegCrrzSjlPJVEbkHwOMB3I7VZ9oA4I0Afg8rUrgdwJvK6i7fLyKPEpGb1/lk5WxsRx3E7ishAP6vC71Gt0FHTx0cHx/j4ODgdFTga09OTjZIwX68RQni5ORk48tNXjn2Xq1K8NolIwVWHaVs/ku3l8dYpWDrYA0/IwVuDwsmdY7ee8YYKYNWUsgCjRHRZKTgtavXfl47txA0P/tIeUVltqArpiAitwL4bgAfAPA4NfRSygMi8th1sscD+Dxddt/6WEgKUeNlzKrbTAQs9bNRR18O4rJLOeuDHh8fb7yVyOf0T2aYFHSf32PQdxXsPXlys+Y+0HMIDdR2rmi7hyRqyEghO6dtGZVj68jkz3EDJYUaGXjPt0UpWOOPVIGnEGqk0IrseezCfdCCH4HV9xd/tpTylUQGeifO1Fbofx+4k2c3xmmsnNaHxXKdyUHJAMBGOn0zUQNy3Dn4K892hGFSODw8xMHBwen/TeiPq5io9D4VtrNxGlYJnvuQtVMUnGshhWjf5lUrM1IKXlqrArx+FRlmzfhbScEjCJs+MvxW98xDNBBGeZ0XmkhBRA6wIoQ3l1J+a334QXULRORmAA+tj98H4Ba6/AkA7rd5Fvrfh729vRIZj+57LOuNIDza2ga17oVNG8lCW666D0dHR6cffuW1vicP4PRDK7Y8lr12tONRr3fUtojI28ZbxpBCVmaN5K0k9+CRQhQArK35+iiYaK/LSMAjBb7v8zDoaCAYipbZBwHwOgD3lFJ+jU69G8ALAbxqvX4XHX+ZiLwVqwDjl0slngDU/dVsBGSj9q6LjtnOa89z7IHfNNT03JF4nls//3ZycnL6F3W2/EiSRiOPradVF63t2ttpPUK2eVmlx/vZ9dpuUXxFYYnDG+1bFQK7HRG5eK5dzwjeSuSZUoiQuWZ6/Vh3okUpPAPAjwP4mIh8ZH3sFViRwdtF5EUA/hLAj67PvQer6ch7sZqS/KlaAV5DRGSg6b3RqIUUvMazaCER2zH1IyscfDw5OQlfdeZ79hSBRwRcP+/hZ53BGmumADyybCUXazQZYXi+u1cnjxTYmJVYso+fsApjUrDpewihhTBaDDN6XjVyyAa+6Lm1oGX24Q/hxwkA4FlO+gLgpb0VsdK+lHLmxRc7+nidyI7yXmfmRtRpST7O05Ccp17PnUjdCP23KP7BlB63pGKDh9ZoLCwhcRt57WPvN9r22sdTIZaAakSQpeN1phS4DnZ2YSgpRMrMLjbNVKSQyfzWAcEicoXtc+vBbN5oVEP0Xnf1ZhP0Ol74IfJUXDTic4NxEJIX/caCGjy/xaivOtu1vsikn2zTe9Cy7W8duG6c3l7r/W7C3ofXRoyoY0ayNMrHdvzM1+ZgIl9rDTkiFRvw0+u86USPFDLDtfVsceV6yCAbqWuk4F3rDQ7aLzQ9D7BbUQrnhYgp7SjJaXmfCSAjgciYot9c6hMAAAdLSURBVMZjVaLqQw1W4wyWLHTNH4q1pOCRgMjZP5mxr1bb6/k+PMWQjVRTIDIya1xR2l5S0GUoKXj1sHWaUiG0GOUQw7WDg+3jQ/JUzIYUolHPa/AWDLkG2Pxug9bn5OT6S0jWdbHKhI0XQJUU7DH7q0FLDENJIfM/7baXR23xlIIlBZuXJQWOK9RIwZs1aIkJ6DMeY/Q1QjgveAQ0RT1mQQoe0wG5n2Vv3I76arw2+BgZk72Gy2eDt28IMmFompOT669bZ+6DRxD83QbrSnhk4t1P1Cms6hpDCva4Z2jR6M/G7ikFL10WaPQIwcvTUyxTEITXbnadIUsTqTseeGwbt5YbYRakAPif5AaG32T20Dyp5UmuyPA5X73OjuRMJJ4iiJZMGWRKwSNTizGk4LVpj3Fxet1mNyC7vhYY9Egji3Fki6dWvDTe/YwxxCHgslnBjq3HbEjBUwrAWVJg4/WO6XEr81vBD9y6BN5fy+s5VQdstBkp8L1651sDjZ5i2CYp6LrF+L1OmpFCZLiRwXtE0UIALQTRc59RG1m09EVOwwOWLS9yFW3dIqWRYZakoPA6U7QfdWrrPti1VQsATl951mOsFHSt13tkoNt6feY+6Nq6FFkwcggpeKQ7pVLI9jkf3revFVujVYK2wULP+FuUQesxW98eUrCwRl5LY9F6Ta0ePZgFKWQjWEYMEcY0TGRQSgicRuvNsxJsoBy0bFEKfCyKNWRkkI0KU5GCXWek4LVnRgqeAogMHvB/PdmyZGQSISIFPhfdZ3ZsDDIiGlPWLEgB6Jtrb8WYhuGIeeTSWMO0JMBKITPiFlLoUQoezosUomu9fRto7CWFaMSOrmlxLyxqSmEOmLoesyGFXSFTBrxtO5uei0b9saTgLZeRFLJfIvaQAucfXZOpBq+u9h7nQgLbxmxIYVcN7hmSGjNvW+WQGfwYUuB9jxSy85mRZ2XVsC1S4JmDTNZHpMB52rKz+EGPUqjdz2XEbEhhV2hVCsD1jtYyWotsfi1aj0WkwGV76S4rKbQQQRQMzJRCpgqyJbr/RSksOAPugK2ynwnEpmFkSiHK+zK4D60GnM0QcD5R3bK8vfu40TELUtjlA2l1H7w61lRA6w+YIpLI8r4MSoHzsPn1koJX157Fq1vWFpcZsyCFXcJ7yJ77wEuLCojS8JrL8/ZvBFKw51qlfpSnV6dWMqgZ/I1ACMBCCoPQqmwuSyfaBSm0GLOX52Vp813iSj3JggULbiQspLBgwYINzMJ9KKV88etf//r/A/DFXddlBL4JF7v+wMW/h4tef2C79/B3WhLJXHwwEbm7lPK0XddjKC56/YGLfw8Xvf7APO5hcR8WLFiwgYUUFixYsIE5kcJdu67ASFz0+gMX/x4uev2BGdzDbGIKCxYsmAfmpBQWLFgwA+ycFETk2SLySRG5V0Tu3HV9WiEinxORj4nIR0Tk7vWxx4jIe0Xk0+v1o3ddT4aIvF5EHhKRj9Mxt86ywm+un8tHReSpu6v5aV29+r9SRL6wfg4fEZHn0rmXr+v/SRH5od3U+jpE5BYR+d8ico+IfEJE/uX6+LyeQe8PR6ZcAOwB+AyAJwE4BPAnAL5jl3XqqPvnAHyTOfYrAO5cb98J4N/tup6mfs8E8FQAH6/VGav/A/3vAATA9wH4wEzr/0oA/8pJ+x3r/nQTgCeu+9nejut/M4CnrrcfCeBT63rO6hnsWik8HcC9pZTPllKuAngrgNt3XKcxuB3AG9fbbwTwIzusyxmUUn4fwJfM4ajOtwN4U1nh/QAeJSI3n09NfQT1j3A7gLeWUh4upfw5Vn94/PStVa4BpZQHSikfXm9/FcA9AB6PmT2DXZPC4wF8nvbvWx+7CCgAfldEPiQid6yPPa6U8gCw6gAAHruz2rUjqvNFejYvW8vr15PLNuv6i8itAL4bwAcws2ewa1Lwfrd7UaZDnlFKeSqA5wB4qYg8c9cVmhgX5dm8BsCTAdwG4AEAr14fn239ReQRAN4B4GdLKV/JkjrHtn4PuyaF+wDcQvtPAHD/jurShVLK/ev1QwDeiZU0fVDl3Xr90O5q2Iyozhfi2ZRSHiylHJdSTgC8FtddhFnWX0QOsCKEN5dSfmt9eFbPYNek8EEATxGRJ4rIIYDnA3j3jutUhYh8g4g8UrcB/CCAj2NV9xeuk70QwLt2U8MuRHV+N4CfWEfAvw/Al1XizgnGx34eVs8BWNX/+SJyk4g8EcBTAPzRedePIasv2rwOwD2llF+jU/N6BruMxlKE9VNYRYd/Ydf1aazzk7CKbP8JgE9ovQF8I4D3Afj0ev2YXdfV1PstWEnsa1iNQi+K6oyVdP2P6+fyMQBPm2n9/8u6fh/FyohupvS/sK7/JwE8Zwb1//tYyf+PAvjIennu3J7B8kbjggULNrBr92HBggUzw0IKCxYs2MBCCgsWLNjAQgoLFizYwEIKCxYs2MBCCgsWLNjAQgoLFizYwEIKCxYs2MD/BzwZGFCPt1KgAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sample_id = 3\n",
    "sample_img, _ = image_datasets['train'][sample_id]\n",
    "sample_img = transforms.ToPILImage('RGB')(sample_img) # Tensor を画像に変換（もとに戻す）\n",
    "plt.figure()\n",
    "plt.imshow(sample_img)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画像を表示しやすいように、標準化をコメントアウトしていたので、戻します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_transforms = {\n",
    "    # training data用。必要ならaugmentation(Flipや切り出し)を行う\n",
    "    # 今は、特段の加工は行わない。\n",
    "    'train': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ]),\n",
    "    # validation用。通常はFlip等は行わない。\n",
    "    'val': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ]),\n",
    "    # test用。こちらもFlip等は実施しない\n",
    "    'test': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ])\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2節と同様、バッチごとの読み込み用の設定をします。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# バッチサイズ分のデータを読み込む。\n",
    "# training はデータをシャッフルし、読み込み始める画像をランダムにする。\n",
    "# 他はシャッフルの必要なし。\n",
    "batch_size=64\n",
    "workers=0\n",
    "dataloaders = {\n",
    "    'train': torch.utils.data.DataLoader(\n",
    "        image_datasets['train'],\n",
    "        batch_size=batch_size,\n",
    "        shuffle=True,\n",
    "        num_workers=workers),\n",
    "    'val': torch.utils.data.DataLoader(\n",
    "        image_datasets['val'],\n",
    "        batch_size=batch_size,\n",
    "        shuffle=False,\n",
    "        num_workers=workers),\n",
    "    'test': torch.utils.data.DataLoader(\n",
    "        image_datasets['test'],\n",
    "        batch_size=batch_size,\n",
    "        shuffle=False,\n",
    "        num_workers=workers)\n",
    "}\n",
    "dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val', 'test']}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ネットワーク構造"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "画像解析で広く使われるResNetなどの固定したアーキテクチャもありますが、ここでは汎用性に重きをおいて、シンプルなCNNの記載からスタートします。\n",
    "\n",
    "ここでは、\n",
    "```入力画像 -> Conv2D -> ReLU -> MaxPooling -> Conv2D -> ReLU -> MaxPooling -> Linear -> ReLU -> Linear ```\n",
    "という形のネットワークを組みます。\n",
    "\n",
    "ここでは、Convolution(畳み込み)とMaxPoolingが行われていますが、詳細の説明は、こちらのページなどをご覧ください。\n",
    "* https://deepage.net/deep_learning/2016/11/07/convolutional_neural_network.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Net(nn.Module):\n",
    "    def __init__(self):\n",
    "        super(Net, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(3, 6, 3, padding=(1,1))\n",
    "        self.pool = nn.MaxPool2d(2, 2)\n",
    "        self.conv2 = nn.Conv2d(6, 16, 3, padding=(1,1))\n",
    "        self.fc1 = nn.Linear(16 * 56 * 56, 120) # channel_num * x * y\n",
    "        self.fc2 = nn.Linear(120, 4)\n",
    "\n",
    "    def forward(self, x):\n",
    "        # ** convolution layers **\n",
    "        # 224 x 224 -> 112 x 112 \n",
    "        x = self.pool(F.relu(self.conv1(x)))\n",
    "        # 112 x 112 -> 56 x 56\n",
    "        x = self.pool(F.relu(self.conv2(x)))\n",
    "        # convert to 1-dim\n",
    "        x = x.view(-1, 16 * 56 * 56) # channel_num * x * y\n",
    "        # ** classification layers **\n",
    "        x = F.relu(self.fc1(x))\n",
    "        x = self.fc2(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "conv1とconv2が、2次元（＝画像）のコンボリューションを実施する層になっています。poolはプーリング層、fc1とfc2は全結合層です。\n",
    "* nn.Conv2d(3, 6, 5, padding=(2,2)): 3チャンネル(=RGB)の入力を、6チャネルに拡張。\n",
    "* nn.MaxPool2d(2, 2): 2x2のマス目でMaxPooling を実施。\n",
    "* ... \n",
    "と、利用する層の種類を定義しています。\n",
    "\n",
    "上記の関数で定義した層を、どのように組み合わせるのか表したものが、forward 関数です。こちらでは、活性化関数としてReLUを利用して、\n",
    "* x = self.pool(F.relu(self.conv1(x))): 入力画像 -> Conv2D (conv1) -> ReLU -> MaxPooling (pool)\n",
    "* x = self.pool(F.relu(self.conv2(x))): -> Conv2D (conv2) -> ReLU -> MaxPooling (pool)\n",
    "* x.view(-1, 16 * 56 * 56): 平滑化\n",
    "* x = F.relu(self.fc1(x)): -> FC -> ReLU\n",
    "* x = self.fc2(x): 最後のclassification層\n",
    "\n",
    "という形で、ネットワークの構成を順番に記載しています。\n",
    "\n",
    "前節同様に、作成したネットワークは、指定するデバイスに転送します。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "device_name = \"cpu\"\n",
    "device = torch.device(device_name)\n",
    "model = Net()\n",
    "model = model.to(device_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 学習ステップの定義"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "学習ステップの詳細を定義します。現在までに、ネットワークアーキテクチャの変更と、バッチごとのデータ取得を画像を取得するよう変更しましたが、この学習ステップは、前節と全く同じ関数です。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_test_accuracy(model, criterion, optimizer, phase):\n",
    "    running_loss = 0.0\n",
    "    running_corrects = 0\n",
    "    model.train(False)\n",
    "\n",
    "    for inputs, labels in dataloaders[phase]:\n",
    "        labels = labels.float()\n",
    "        inputs = inputs.to(device)\n",
    "        labels = labels.to(device)\n",
    "\n",
    "        #optimizer.zero_grad()\n",
    "\n",
    "        # 訓練のときだけ履歴を保持する\n",
    "        with torch.set_grad_enabled(phase == 'train'):\n",
    "            outputs = model(inputs)\n",
    "            _, classnums = torch.max(labels, 1)\n",
    "            _, preds = torch.max(outputs, 1)\n",
    "            loss = criterion(outputs, classnums)\n",
    "\n",
    "        # 統計情報\n",
    "        running_loss += loss.item() * inputs.size(0)\n",
    "        running_corrects += torch.sum(preds == classnums)\n",
    "\n",
    "    # サンプル数で割って平均を求める\n",
    "    epoch_loss = running_loss / dataset_sizes[phase]\n",
    "    epoch_acc = running_corrects.double() / dataset_sizes[phase]\n",
    "    print('On Test:\\tLoss: {:.4f} Acc: {:.4f}'.format(epoch_loss, epoch_acc))\n",
    "\n",
    "\n",
    "def train_model(model, criterion, optimizer, scheduler, num_epochs=25):\n",
    "    since = time.time()\n",
    "    # 途中経過でモデル保存するための初期化\n",
    "    best_model_wts = copy.deepcopy(model.state_dict())\n",
    "    best_acc = 0.0\n",
    "    # 時間計測用\n",
    "    end = time.time()\n",
    "\n",
    "    print(model)\n",
    "    print()\n",
    "\n",
    "    for epoch in range(num_epochs):\n",
    "        print('Epoch:{}/{}'.format(epoch, num_epochs - 1), end=\"\")\n",
    "\n",
    "        # 各エポックで訓練+バリデーションを実行\n",
    "        for phase in ['train', 'val']:\n",
    "            if phase == 'train':\n",
    "                scheduler.step()\n",
    "                model.train(True)\n",
    "            else:\n",
    "                model.train(False)\n",
    "\n",
    "            running_loss = 0.0\n",
    "            running_corrects = 0\n",
    "\n",
    "            for inputs, labels in dataloaders[phase]:\n",
    "                labels = labels.float()\n",
    "                inputs = inputs.to(device)\n",
    "                labels = labels.to(device)\n",
    "\n",
    "                optimizer.zero_grad()\n",
    "\n",
    "                # 訓練のときだけ履歴を保持する\n",
    "                with torch.set_grad_enabled(phase == 'train'):\n",
    "                    outputs = model(inputs)\n",
    "                    _, classnums = torch.max(labels, 1)\n",
    "                    _, preds = torch.max(outputs, 1)\n",
    "                    loss = criterion(outputs, classnums)\n",
    "                    if phase == 'train':\n",
    "                        loss.backward()\n",
    "                        optimizer.step()\n",
    "\n",
    "                # 統計情報\n",
    "                running_loss += loss.item() * inputs.size(0)\n",
    "                running_corrects += torch.sum(preds == classnums)\n",
    "\n",
    "            # サンプル数で割って平均を求める\n",
    "            epoch_loss = running_loss / dataset_sizes[phase]\n",
    "            epoch_acc = running_corrects.double() / dataset_sizes[phase]\n",
    "\n",
    "            print('\\t{} Loss: {:.4f} Acc: {:.4f} Time: {:.4f}'.format(phase, epoch_loss, epoch_acc, time.time()-end), end=\"\")\n",
    "\n",
    "            # 精度が改善したらモデルを保存する\n",
    "            if phase == 'val' and epoch_acc > best_acc:\n",
    "                best_acc = epoch_acc\n",
    "                best_model_wts = copy.deepcopy(model.state_dict())\n",
    "            end = time.time()\n",
    "\n",
    "        print()\n",
    "\n",
    "    time_elapsed = time.time() - since\n",
    "    print()\n",
    "    print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))\n",
    "    print('Best val acc: {:.4f}'.format(best_acc))\n",
    "\n",
    "    model.load_state_dict(best_model_wts)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 深層学習の実行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "実行も、前節と同様です。つまり、深層学習を用いて画像の分類をする場合には、一見長く見えるプログラムの部分は変更の必要は無く、モデルやデータの読み込みを変更すればよいとわかります。\n",
    "\n",
    "各エポック、１０秒以上かかりますので、表示が出なくても焦らずに待ってください。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (fc1): Linear(in_features=50176, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=4, bias=True)\n",
      ")\n",
      "\n",
      "Epoch:0/19\ttrain Loss: 1.3828 Acc: 0.2566 Time: 6.5385\tval Loss: 1.3967 Acc: 0.2500 Time: 1.0999\n",
      "Epoch:1/19\ttrain Loss: 1.3621 Acc: 0.3363 Time: 5.8226\tval Loss: 1.4580 Acc: 0.2500 Time: 1.3437\n",
      "Epoch:2/19\ttrain Loss: 1.3628 Acc: 0.3363 Time: 5.9553\tval Loss: 1.4820 Acc: 0.2500 Time: 1.2344\n",
      "Epoch:3/19\ttrain Loss: 1.3552 Acc: 0.3363 Time: 5.9387\tval Loss: 1.4490 Acc: 0.2500 Time: 1.2921\n",
      "Epoch:4/19\ttrain Loss: 1.3435 Acc: 0.3363 Time: 5.9288\tval Loss: 1.4368 Acc: 0.2500 Time: 1.2397\n",
      "Epoch:5/19\ttrain Loss: 1.3340 Acc: 0.3363 Time: 5.9790\tval Loss: 1.4171 Acc: 0.2500 Time: 1.2628\n",
      "Epoch:6/19\ttrain Loss: 1.2993 Acc: 0.3363 Time: 5.7437\tval Loss: 1.3919 Acc: 0.2632 Time: 1.2565\n",
      "Epoch:7/19\ttrain Loss: 1.2183 Acc: 0.4425 Time: 5.8301\tval Loss: 1.3026 Acc: 0.3026 Time: 1.1480\n",
      "Epoch:8/19\ttrain Loss: 1.0392 Acc: 0.5885 Time: 6.1044\tval Loss: 1.1663 Acc: 0.4342 Time: 1.3841\n",
      "Epoch:9/19\ttrain Loss: 0.7955 Acc: 0.6593 Time: 6.1387\tval Loss: 1.1070 Acc: 0.5526 Time: 1.2910\n",
      "Epoch:10/19\ttrain Loss: 0.6267 Acc: 0.7345 Time: 6.0887\tval Loss: 1.0298 Acc: 0.5921 Time: 1.2691\n",
      "Epoch:11/19\ttrain Loss: 0.5860 Acc: 0.7743 Time: 6.0849\tval Loss: 1.2427 Acc: 0.5395 Time: 1.2655\n",
      "Epoch:12/19\ttrain Loss: 0.6464 Acc: 0.7345 Time: 5.8955\tval Loss: 1.4552 Acc: 0.4868 Time: 1.0862\n",
      "Epoch:13/19\ttrain Loss: 0.6402 Acc: 0.7743 Time: 6.0914\tval Loss: 1.8814 Acc: 0.4737 Time: 1.2774\n",
      "Epoch:14/19\ttrain Loss: 0.8720 Acc: 0.7434 Time: 6.1376\tval Loss: 1.1684 Acc: 0.5263 Time: 1.2636\n",
      "Epoch:15/19\ttrain Loss: 0.5762 Acc: 0.8053 Time: 6.0654\tval Loss: 0.9481 Acc: 0.5526 Time: 1.2746\n",
      "Epoch:16/19\ttrain Loss: 0.5840 Acc: 0.7611 Time: 6.1331\tval Loss: 1.1262 Acc: 0.5526 Time: 1.2476\n",
      "Epoch:17/19\ttrain Loss: 0.5563 Acc: 0.8097 Time: 6.0546\tval Loss: 1.3967 Acc: 0.5132 Time: 1.2955\n",
      "Epoch:18/19\ttrain Loss: 0.4588 Acc: 0.8230 Time: 6.0708\tval Loss: 1.3133 Acc: 0.5000 Time: 1.1768\n",
      "Epoch:19/19\ttrain Loss: 0.3736 Acc: 0.8673 Time: 5.9831\tval Loss: 1.6298 Acc: 0.5000 Time: 1.3433\n",
      "\n",
      "Training complete in 2m 26s\n",
      "Best val acc: 0.5921\n",
      "On Test:\tLoss: 1.1252 Acc: 0.4605\n"
     ]
    }
   ],
   "source": [
    "epochs = 20\n",
    "batch_size = 64\n",
    "lr = 0.01\n",
    "momentum = 0.9\n",
    "outdir = \".\"\n",
    "\n",
    "# モデルの初期化\n",
    "model = Net()\n",
    "model = model.to(device)\n",
    "\n",
    "# 損失関数、\n",
    "# パラメータの最適化方法、学習率の更新方法を定義。\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)\n",
    "exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.7)\n",
    "\n",
    "# 実際の学習を実施\n",
    "model = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=epochs)\n",
    "# テストデータでの精度を求める\n",
    "print_test_accuracy(model, criterion, optimizer, 'test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "私が実験した範囲では、強い初期値依存性があるので、何度か繰り返してみて、結果を確認してみてください。\n",
    "\n",
    "多くのケースで、train Accが0.8を超えるなど、訓練データでの精度が高い一方で、バリデーションデータやテストデータでは、Accが0.5程度と低い値となります。この様な状態は、過学習(overfit)である可能性が高いです。深層学習では、このようなシンプルなモデルでもパラメータ数がデータ数に比べて極端に多いので、頻繁に過学習が起こります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Augmentation (水増し)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "過学習を避ける一つの方法として、画像の水増しがあります。ここでは、データ読み込み時に水増しをしてみましょう。\n",
    "RandomHorizontalFlip()で左右反転、RandomRotationで回転を行い、向き依存性を解消すると同時に、サンプル画像の数を増やします。\n",
    "\n",
    "データの水増しは訓練データに足してのみ実施すればよく、バリデーションやテストフェーズでは、水増しを行う必要はありません（これらの画像の精度を判断したいので）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Net(\n",
      "  (conv1): Conv2d(3, 6, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)\n",
      "  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))\n",
      "  (fc1): Linear(in_features=50176, out_features=120, bias=True)\n",
      "  (fc2): Linear(in_features=120, out_features=4, bias=True)\n",
      ")\n",
      "\n",
      "Epoch:0/19\ttrain Loss: 1.3625 Acc: 0.3186 Time: 6.6071\tval Loss: 1.4350 Acc: 0.2500 Time: 1.2805\n",
      "Epoch:1/19\ttrain Loss: 1.2695 Acc: 0.4823 Time: 6.4406\tval Loss: 1.3482 Acc: 0.2763 Time: 1.2792\n",
      "Epoch:2/19\ttrain Loss: 0.9678 Acc: 0.5885 Time: 5.8816\tval Loss: 2.1244 Acc: 0.3553 Time: 1.1852\n",
      "Epoch:3/19\ttrain Loss: 1.1492 Acc: 0.5841 Time: 7.1153\tval Loss: 1.1461 Acc: 0.4342 Time: 1.3533\n",
      "Epoch:4/19\ttrain Loss: 0.8195 Acc: 0.6239 Time: 7.8085\tval Loss: 1.3828 Acc: 0.4211 Time: 1.5132\n",
      "Epoch:5/19\ttrain Loss: 0.6904 Acc: 0.6814 Time: 6.8857\tval Loss: 1.2263 Acc: 0.4737 Time: 1.5122\n",
      "Epoch:6/19\ttrain Loss: 0.6287 Acc: 0.6947 Time: 6.2648\tval Loss: 1.2252 Acc: 0.5132 Time: 1.4544\n",
      "Epoch:7/19\ttrain Loss: 0.6759 Acc: 0.6858 Time: 6.8727\tval Loss: 1.7746 Acc: 0.4342 Time: 1.7480\n",
      "Epoch:8/19\ttrain Loss: 0.7313 Acc: 0.7080 Time: 8.1955\tval Loss: 1.6897 Acc: 0.4079 Time: 1.8142\n",
      "Epoch:9/19\ttrain Loss: 0.6042 Acc: 0.7788 Time: 7.4462\tval Loss: 1.1754 Acc: 0.5789 Time: 1.4452\n",
      "Epoch:10/19\ttrain Loss: 0.4479 Acc: 0.8363 Time: 7.1308\tval Loss: 1.7357 Acc: 0.4079 Time: 1.3082\n",
      "Epoch:11/19\ttrain Loss: 0.3378 Acc: 0.8628 Time: 6.7540\tval Loss: 2.1761 Acc: 0.4737 Time: 1.4651\n",
      "Epoch:12/19\ttrain Loss: 0.2525 Acc: 0.9027 Time: 7.0139\tval Loss: 2.1187 Acc: 0.4605 Time: 1.4604\n",
      "Epoch:13/19\ttrain Loss: 0.2720 Acc: 0.9027 Time: 7.8092\tval Loss: 3.0896 Acc: 0.4474 Time: 1.5510\n",
      "Epoch:14/19\ttrain Loss: 0.2348 Acc: 0.9248 Time: 7.1889\tval Loss: 2.2062 Acc: 0.4605 Time: 1.3012\n",
      "Epoch:15/19\ttrain Loss: 0.3194 Acc: 0.8805 Time: 6.1278\tval Loss: 2.4479 Acc: 0.4605 Time: 1.3805\n",
      "Epoch:16/19\ttrain Loss: 0.1791 Acc: 0.9425 Time: 6.7836\tval Loss: 2.6497 Acc: 0.4474 Time: 1.5108\n",
      "Epoch:17/19\ttrain Loss: 0.1773 Acc: 0.9381 Time: 6.7176\tval Loss: 2.9788 Acc: 0.5000 Time: 1.1579\n",
      "Epoch:18/19\ttrain Loss: 0.1759 Acc: 0.9513 Time: 6.3921\tval Loss: 2.6102 Acc: 0.5000 Time: 1.5368\n",
      "Epoch:19/19\ttrain Loss: 0.0992 Acc: 0.9735 Time: 6.9636\tval Loss: 2.9552 Acc: 0.5263 Time: 1.4303\n",
      "\n",
      "Training complete in 2m 47s\n",
      "Best val acc: 0.5789\n",
      "On Test:\tLoss: 1.4974 Acc: 0.5132\n"
     ]
    }
   ],
   "source": [
    "data_transforms = {\n",
    "    # 訓練データ\n",
    "    # 水増し（左右反転、45度回転）を行う\n",
    "    'train': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.RandomHorizontalFlip(),\n",
    "        transforms.RandomRotation(45),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ]),\n",
    "    # バリデーションデータ\n",
    "    # 通常はFlip等は行わない。\n",
    "    'val': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ]),\n",
    "    # テストデータ\n",
    "    # 通常はFlip等は行わない。\n",
    "    'test': transforms.Compose([\n",
    "        transforms.Resize((WIDTH, HEIGHT)),\n",
    "        transforms.ToTensor(),\n",
    "        normalize\n",
    "    ])\n",
    "}\n",
    "\n",
    "# 以下は、水増し前と同じ。\n",
    "epochs = 20\n",
    "batch_size = 64\n",
    "lr = 0.03\n",
    "momentum = 0.9\n",
    "outdir = \".\"\n",
    "\n",
    "model = Net()\n",
    "model = model.to(device)\n",
    "\n",
    "# 損失関数、最適化方法、学習率の更新方法を定義\n",
    "criterion = nn.CrossEntropyLoss()\n",
    "optimizer = optim.SGD(model.parameters(), lr=lr, momentum=momentum)\n",
    "exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.7)\n",
    "\n",
    "# 深層学習の実行\n",
    "model = train_model(model, criterion, optimizer, exp_lr_scheduler, num_epochs=epochs)\n",
    "# テストデータでの精度を求める\n",
    "print_test_accuracy(model, criterion, optimizer, 'test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本質的に画像の枚数が不十分であることもあり、十分な精度向上は見込めませんが、手順として、このようなことが行われるということを認識して頂ければと思います。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 積み残しの課題\n",
    "\n",
    "何点か、このテキストで積み残している課題です。\n",
    "1. 最適化手法の説明。このテキストでは SGD (stochastic gradient descent)を利用していますが、Adamなど他の手法もある。モデルによっては、Adam等の手法の方が、的確に収束することがある\n",
    "2. ResNet等頻繁に利用されているモデルの利用。今回は単純なCNNを利用しましたが、画像処理のコンペで高い精度を達成しているものとして、VGG、ResNetなどのモデルがあります。これらのネットワークの導入を行っていません\n",
    "3. それらを用いた転移学習。VGG, ResNetなどは、ImageNetで学習した事前学習モデル（プレトレーニングモデル）が配布されています。これらを用いることで、画像の枚数が少ない場合は、エポック数が少なくても、学習が効率的に進むことがあります。（常に、ではありません）\n",
    "4. 核以外の染色画像の利用。配布してある画像には、アクチン画像（C_0_1で始まるもの）と核染色画像(C_0_2で始まるもの)が含まれています。これらは利用しておらず、細胞壁の蛍光画像のみ利用しています。\n",
    "\n",
    "学習に関して本質的に対処が必要な問題として\n",
    "1. 画像の枚数の確保\n",
    "2. 今回は顕微鏡画像から切り出した酵母の細胞画像を利用しているが、同一視野の画像は類似の傾向があると考えられるので、その補正（テスト画像選択時に、同一視野の画像を全て除くなど）\n",
    "\n",
    "などがあります。学習時に意図しないバイアスは頻繁に入りうるので、注意をする必要があります。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "今回の内容をまとめたソースコードは、```src/yeast_image_cnn.py``` になります。\n",
    "シェルから\n",
    "```bash\n",
    "$ python src/yeast_image_cnn.py data -b 64 --lr 0.1 --epoch 20 -j 0\n",
    "```\n",
    "の様な形で実行可能です。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}