PyTorch Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# An introduction to Pytorch\n",
    "\n",
    "Pytorch is a platform for deep learning in Python. \n",
    "\n",
    "It provides tools for efficiently creating, training, testing and analyzing neural networks:\n",
    "\n",
    "* Different types of layers (embedding, linear, convolutional, recurrent)\n",
    "* Activation functions (tanh, relu, sigmoid, etc.)\n",
    "* Gradient computation\n",
    "* Optimizer (adam, adagrad, RMSprop, SGD, etc.)\n",
    "* Implementations speed gains in GPU\n",
    "\n",
    "## Alternatives\n",
    "\n",
    "Other platforms for deep learning in Python exist, with different focuses: Tensorflow, Caffe, MXNet,...\n",
    "\n",
    "* Pytorch is comparetively simple to use \n",
    "* ... and also the only one besides Tensorflow I have experience with 🙂\n",
    "* Feel free to try the others!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors\n",
    "\n",
    "Let's start with some basics: tensors are similar to numpy arrays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 = torch.arange(10)\n",
    "v2 = torch.arange(10, 20)\n",
    "\n",
    "print(\"v1: %s\\n\" % v1)\n",
    "print(\"v2: %s\\n\" % v2)\n",
    "print(\"Dot product: %d\" % v1.dot(v2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Setting values manually or randomly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v3 = torch.tensor([2, 4, 6, 8])\n",
    "v4 = torch.rand(10)\n",
    "\n",
    "print(\"v3: %s\\n\" % v3)\n",
    "print(\"v4: %s\\n\" % v4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### You can also change a value inside the tensor manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v4[1] = 0.1\n",
    "print(v4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Accessing values (indexing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Individual tensor positions are scalars, or 0-dimension tensor:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(v1[0])\n",
    "print(v1[0].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`.item()` returns a Python number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number = v1[0].item()\n",
    "print(number)\n",
    "print(isinstance(number, int))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Elementwise operations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 + v2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 * v2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some caveats when working with integer values!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v1 / v2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.tensor(v1, dtype=torch.float)\n",
    "y = torch.tensor(v2, dtype=torch.float)\n",
    "x / y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Operations with constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x ** 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Matrices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m1 = torch.rand(5, 4)\n",
    "m2 = torch.rand(4, 5)\n",
    "\n",
    "print(\"m1: %s\\n\" % m1)\n",
    "print(\"m2: %s\\n\" % m2)\n",
    "print(m1.dot(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops... that can be misleading if you are used to numpy. Instead, call `matmul`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(m1.matmul(m2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Higher order tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t = torch.rand(3, 4, 5)\n",
    "t"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broadcasting\n",
    "\n",
    "Broadcasting means doing some arithmetic operation with tensors of different ranks, as if the smaller one were expanded, or broadcast, to match the larger.\n",
    "\n",
    "Let's experiment with a matrix (rank 2 tensor) and a vector (rank 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m = torch.rand(5, 4)\n",
    "v = torch.rand(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"m:\", m)\n",
    "print()\n",
    "print(\"v:\", v)\n",
    "print()\n",
    "\n",
    "m_plus_v = m + v\n",
    "print(\"m + v:\", m_plus_v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see row by row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"m[0] = %s\\n\" % m[0])\n",
    "print(\"v = %s\\n\" % v)\n",
    "\n",
    "row_sum = m[0] + v\n",
    "print(\"m[0] + v = %s\\n\" % row_sum)\n",
    "print(\"(m + v)[0] = %s\" % m_plus_v[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also reshape tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = v.view([2, 2])\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = v.view([4, 1])\n",
    "v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that shape `[4, 1]` is not broadcastable to match `[5, 4]`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m + v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... but `[1, 4]` is!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v = v.view([1, 4])\n",
    "m + v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Broadcasting can be tricky sometimes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = torch.rand(4, 1)\n",
    "u + v"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always take care with tensor shapes! It is a good practice to verify in the interpreter how some expression is evaluated before inserting into your model code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Useful Functions\n",
    "\n",
    "Pytorch (and other libraries) have many functions that operate on tensors. Let's try some of them and plot the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as pl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a vector x with values from -10 to 10, and intervals of 0.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.arange(-10, 10, 0.1, dtype=torch.float)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.numpy()` method converts Pytorch tensors to numpy array. It is necessary to plot with matplotlib."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.sin()\n",
    "pl.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hyperbolic tangent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = x.tanh()\n",
    "pl.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$e^x$ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = torch.exp(x)\n",
    "pl.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can implement softmax. Since it's a function of the whole array, the plot has a slightly different meaning (notice that the y-axis only goes until 0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "exps = torch.exp(x)\n",
    "z = exps.sum()\n",
    "softmax = exps / z\n",
    "pl.plot(x.numpy(), softmax.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Anyway, torch also provides an implementation of softmax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = torch.softmax(x, dim=0)\n",
    "pl.plot(x.numpy(), y.numpy())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the softmax with another x:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = torch.randn([100])\n",
    "y = torch.softmax(x, dim=0)\n",
    "pl.plot(x.numpy(), y.numpy(), '.')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}