03-modules-and-mlps.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read the data\n",
    "\n",
    "We will download the MNIST dataset for training a classifier. Torch provides a convenient function for that.\n",
    "\n",
    "The MNIST dataset is composed of images of digits that must be classified with labels from 0 to 9. The inputs are 28x28 matrices containing the grayscale intensity in each pixel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "import torchvision\n",
    "from torchvision import datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from IPython.core.debugger import set_trace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset\n",
    "It's easy to create your `Dataset`,\n",
    "but PyTorch comes with some\n",
    "[built-in datasets](https://pytorch.org/docs/stable/torchvision/datasets.html):\n",
    "\n",
    "- MNIST\n",
    "- Fashion-MNIST\n",
    "- KMNIST\n",
    "- EMNIST\n",
    "- FakeData\n",
    "- COCO\n",
    "  - Captions\n",
    "  - Detection\n",
    "- LSUN\n",
    "- ImageFolder\n",
    "- DatasetFolder\n",
    "- Imagenet-12\n",
    "- CIFAR\n",
    "- STL10\n",
    "- SVHN\n",
    "- PhotoTour\n",
    "- SBU\n",
    "- Flickr\n",
    "- VOC\n",
    "- Cityscapes\n",
    "\n",
    "`Dataset` gives you information about the number of samples (implement `__len__`) and gives you the sample at a given index (implement `__getitem__`.\n",
    "It's a nice and simple abstraction to work with data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "class Dataset(object):\n",
    "    def __getitem__(self, index):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def __len__(self):\n",
    "        raise NotImplementedError\n",
    "\n",
    "    def __add__(self, other):\n",
    "        return ConcatDataset([self, other])\n",
    "```\n",
    "\n",
    "For now, let's use MNIST. You'll have an example on how to use `Dataset` in your next homework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = datasets.MNIST('../data', train=True, download=True)\n",
    "test_data = datasets.MNIST('../data', train=False)\n",
    "\n",
    "train_x = train_data.data\n",
    "train_y = train_data.targets\n",
    "test_x = test_data.data\n",
    "test_y = test_data.targets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_train_examples = train_x.shape[0]\n",
    "n_test_examples = test_x.shape[0]\n",
    "\n",
    "print('%d training instances and %d test instances' % (n_train_examples, n_test_examples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check the shape of our training data to see how many input features there are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(train_x.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And what the image looks like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.imshow(train_x[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Formatting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each sample is a 28x28 matrix. But we want to represent them as vectors, since our model doesn't take any advantage of the 2-d nature of the data.\n",
    "\n",
    "So, we reshape the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_features = 28 * 28\n",
    "new_shape = [n_train_examples, num_features]\n",
    "train_x_vectors = train_x.reshape(new_shape)\n",
    "print(train_x_vectors.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we reshape an array (or torch tensor, for that matter), we don't need to specify all dimensions. We can leave one as -1, and it will be automatically determined from the size of the data. This is useful when we don't know a priori the shape of some array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x_vectors = train_x.view(-1, num_features)\n",
    "test_x_vectors = test_x.view(n_test_examples, -1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also, the values are integers in the range $[0, 255]$. It is better to work with float values in a smaller interval, such as $[0, 1]$ or $[-1, 1]$. There are some more elaborate normalization techniques, but for now let's just normalize it to $[0, 1]$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x_norm = train_x_vectors / 255\n",
    "test_x_norm = test_x_vectors / 255\n",
    "print(train_x_norm[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops! Notice that the arrays had integer values, but the result of the division would be floats. One way to change the `dtype` of a torch tensor is using `.to()`.\n",
    "\n",
    "Keep in mind that data type is a common source of errors!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x_vectors = train_x_vectors.to(torch.float)\n",
    "test_x_vectors = test_x_vectors.to(torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x_norm = train_x_vectors / 255\n",
    "test_x_norm = test_x_vectors / 255\n",
    "print(train_x_norm[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, check the labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(torch.unique(train_y))\n",
    "num_classes = len(torch.unique(train_y))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modules and MLPs\n",
    "\n",
    "We've seen how the internals of simple linear classifier work. However, we still had to set a lot of things manually. It's much better to have a higher-level API that encapsulates the classifier.\n",
    "\n",
    "We are going to see that now, with pytorch Module objects. Then, it will allow us to build more complex models, like a multilayer perceptron."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We begin by loading the data again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import numpy as np\n",
    "from torchvision import datasets\n",
    "from matplotlib import pyplot as pl\n",
    "\n",
    "train_dataset = datasets.MNIST('../data', train=True, download=True, transform=torchvision.transforms.ToTensor())\n",
    "test_dataset = datasets.MNIST('../data', train=False, transform=torchvision.transforms.ToTensor())\n",
    "\n",
    "train_x = train_dataset.data\n",
    "train_y = train_dataset.targets\n",
    "test_x = test_dataset.data\n",
    "test_y = test_dataset.targets\n",
    "\n",
    "num_features = 28 * 28\n",
    "num_classes = len(np.unique(train_y))\n",
    "new_shape = [-1, num_features]\n",
    "train_x_vectors = train_x.reshape(new_shape)\n",
    "test_x_vectors = test_x.reshape(new_shape)\n",
    "\n",
    "# shorten the names\n",
    "train_x = train_x_vectors.to(torch.float) / 255\n",
    "test_x = test_x_vectors.to(torch.float) / 255"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Modules\n",
    "\n",
    "Let's create a linear model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class LinearModel(nn.Module):\n",
    "    def __init__(self, n_features, n_classes):\n",
    "        super().__init__()\n",
    "        self.linear_layer = nn.Linear(n_features, n_classes)\n",
    "        \n",
    "    def forward(self, X):\n",
    "        return self.linear_layer(X)\n",
    "\n",
    "linear_model = LinearModel(num_features, num_classes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model can be called as function to compute an output. Let's see how it works:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = train_x[:2]\n",
    "\n",
    "answers = linear_model(batch)\n",
    "answers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Same as doing the forward method $$w^T x + b$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch @ linear_model.linear_layer.weight.t() + linear_model.linear_layer.bias"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_function = nn.CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.1\n",
    "\n",
    "# the optimizer needs to be told which are the parameters to optimize\n",
    "optimizer = torch.optim.SGD(linear_model.parameters(), lr=learning_rate, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Batching\n",
    "\n",
    "Batching can be boring to code. `DataLoader` helps!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "train_dataloader = DataLoader(train_dataset, batch_size=64, shuffle=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training loop\n",
    "\n",
    "Now we write the main training loop. This is the basic skeleton for training pytorch models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, train_dataloader, num_epochs, optimizer):\n",
    "    losses = []\n",
    "\n",
    "    for epoch in range(1, num_epochs+1):\n",
    "        print('Starting epoch %d' % epoch)\n",
    "        total_loss = 0\n",
    "        hits = 0\n",
    "\n",
    "        for batch_x, batch_y in train_dataloader:\n",
    "            optimizer.zero_grad()\n",
    "            # get the data for this batch\n",
    "            batch_x = batch_x.reshape(batch_x.shape[0], -1)\n",
    "            batch_x = batch_x.to(torch.float) / 255\n",
    "\n",
    "            # forward pass\n",
    "            logits = model(batch_x)\n",
    "\n",
    "            # compute the loss\n",
    "            loss = loss_function(logits, batch_y)\n",
    "            loss_value = loss.item()\n",
    "            total_loss += loss_value\n",
    "            losses.append(loss_value)\n",
    "\n",
    "            y_pred = logits.argmax(dim=1)\n",
    "\n",
    "            hits += torch.sum(y_pred == batch_y).item()\n",
    "\n",
    "            loss.backward()\n",
    "            # after determining the gradients, take a step toward their direction\n",
    "            optimizer.step()\n",
    "\n",
    "        avg_loss = total_loss / len(train_dataloader.dataset)\n",
    "        print('Epoch loss: %.4f' % avg_loss)\n",
    "        acc = hits / len(train_dataloader.dataset)\n",
    "        print('Epoch accuracy: %.4f' % acc)\n",
    "    \n",
    "    return np.array(losses)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "linear_losses = train_model(linear_model, train_dataloader, 10, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Graphics are good to understand the performance of a model. Let's plot the loss curve by batch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(linear_losses, \".\", label=\"linear\")\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Multilayer Perceptron\n",
    "\n",
    "We can now proceed to a more sofisticated classifier: a multilayer perceptron. Let's build one using the Sequential API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hidden_size = 200\n",
    "\n",
    "class MLP(nn.Module):\n",
    "    def __init__(self, n_features, hidden_size, n_classes):\n",
    "        super().__init__()\n",
    "        linear_layer1 = nn.Linear(n_features, hidden_size)\n",
    "        linear_layer2 = nn.Linear(hidden_size, hidden_size)\n",
    "        linear_layer3 = nn.Linear(hidden_size, n_classes)\n",
    "        self.feedforward = nn.Sequential(linear_layer1, \n",
    "                                  nn.Tanh(), \n",
    "                                  linear_layer2,\n",
    "                                  nn.Tanh(),\n",
    "                                  linear_layer3)\n",
    "\n",
    "    def forward(self, X):\n",
    "        return self.feedforward(X)\n",
    "\n",
    "mlp = MLP(num_features, hidden_size, num_classes)\n",
    "loss_function = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(mlp.parameters(), lr=0.1, momentum=0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the model. How do the loss and accuracy compare with the linear model?\n",
    "\n",
    "You probably also noticed a difference in running time!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp_losses = train_model(mlp, train_dataloader, 5, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the different concentration of dots in the MLP and Linear graphics!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(linear_losses, \".\", label=\"linear\")\n",
    "ax.plot(mlp_losses, \".\", label=\"mlp\")\n",
    "ax.legend()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Validation data\n",
    "\n",
    "Evaluating the performance on training data is important to understand if the model is actually learning, but if we want to know if our model has any usefulness, we should evaluate its performance on validation or test data.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate_model(model, test_x, test_y):\n",
    "    model.eval()\n",
    "    with torch.no_grad():\n",
    "        loss_function = torch.nn.CrossEntropyLoss()\n",
    "        logits = model(test_x)\n",
    "        loss = loss_function(logits, test_y)\n",
    "        \n",
    "        y_pred = logits.argmax(dim=1)\n",
    "        hits = torch.sum(y_pred == test_y).item()\n",
    "    model.train()\n",
    "    return loss / len(test_x), hits / len(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp, train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp, test_x, test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(linear_model, train_x, train_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(linear_model, test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How can we make our model better? There are two things to be done:\n",
    "\n",
    "1. **Hyperparameter search**. Do a grid search or random search on the hyperparameters (hidden size, learning rate, batch size, activation function, type of optimizer, ...)\n",
    "2. **Generalize better**. This include either finding some better feature representation or regularizing, i.e., add some kind of penalty to the model weights that encourages it to find a more general solution. Examples: L2-norm weight regularization, dropout.\n",
    "3. **Early stop**. Evaluate the model on validation data after each epoch or some number of batches; only save it when validation performance increases. This means detecting when the model achieved its performance peak."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dropout\n",
    "\n",
    "We could try dropout. It effectivelly deactivates some neural connections at random, forcing the network to avoid depending on specific inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLPDropout(nn.Module):\n",
    "    def __init__(self, n_features, hidden_size, n_classes, p_dropout):\n",
    "        super().__init__()\n",
    "        linear_layer1 = nn.Linear(n_features, hidden_size)\n",
    "        linear_layer2 = nn.Linear(hidden_size, n_classes)\n",
    "        self.feedforward = nn.Sequential(\n",
    "            linear_layer1,\n",
    "            nn.Tanh(),\n",
    "            nn.Dropout(p_dropout),\n",
    "            linear_layer2)\n",
    "\n",
    "    def forward(self, X):\n",
    "        return self.feedforward(X)\n",
    "\n",
    "hidden_size = 200\n",
    "p_dropout = 0.5\n",
    "mlp_dropout = MLPDropout(num_features, hidden_size, num_classes, p_dropout)\n",
    "loss_function = nn.CrossEntropyLoss()\n",
    "optimizer = torch.optim.SGD(mlp_dropout.parameters(), lr=0.1, momentum=0.9)  # weight_decay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "losses = train_model(mlp_dropout, train_dataloader, 3, optimizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training loss is a bit worse, as expected. After all, we are obstructing some connections.\n",
    "\n",
    "Now let's check validation performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp, test_x, test_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp_dropout, test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No improvement. Ideally, we should retrain our model with different hyperparamters (learning rates, layer sizes, number of layers, dropout rate) as well as some changes in the structure (different optimizers, activation functions, losses).\n",
    "\n",
    "However, data representation plays a key role. Do you think representing the input as independent pixels is a good idea for recognizing digits?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving\n",
    "\n",
    "Persisting the model after training is obviously important to reuse it later.\n",
    "\n",
    "In Pytorch, we can save the model calling `save()` and passing  the model's `state_dict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.save(mlp.state_dict(), 'mlp.model')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Later, recreate the model and load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlp2 = MLP(num_features, hidden_size, num_classes)\n",
    "mlp2.load_state_dict(torch.load('mlp.model'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check the performance to see if it's the same!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluate_model(mlp, test_x, test_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The End\n",
    "\n",
    "![https://twitter.com/karpathy/status/1013244313327681536](img/common_mistakes.png)\n",
    "https://twitter.com/karpathy/status/1013244313327681536"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}