NN02_MLP_Complete.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br>\n",
    "<div style=\"font-size:83px; font-weight:bold; text-align: center;\">Learn You a Neural Net! </div>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`whoami`  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "`stu`  \n",
    "Machine Learning Engineer @Opendoor  \n",
    "@mstewart141  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Goal: Puzzle through the mechanics of a fully connected neural net on our way to implementing one in a modular/reusable style"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Per Wikipedia:\n",
    "> An ANN is based on a collection of connected units or **neurons**... the output of each artificial neuron is calculated by a **non-linear function** of the **sum of its inputs**... Artificial neurons and edges typically have a **weight** that adjusts as learning proceeds. The weight increases or decreases the strength of the signal at a connection... Typically, neurons are organized in **layers**... Signals travel from the first (input), to the last (output) layer (...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![title](ann2.jpeg)\n",
    "[(_source_)](https://towardsdatascience.com/multi-layer-neural-networks-with-sigmoid-function-deep-learning-for-rookies-2-bf464f09eb7f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Artificial neurons and edges typically have a **weight** that adjusts as learning proceeds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Ok but how do we make learning *proceed*?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Almost always, we rely on a technique in the gradient descent family of algorithms!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Per Wikipedia:\n",
    "> Gradient descent is a first-order iterative optimization algorithm for finding the __minimum__ of a function. To find a local minimum of a function using gradient descent, one takes steps proportional to the negative of the gradient (or of the approximate gradient) of the function at the current point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# First, some data initialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "import enum\n",
    "from typing import List, Callable\n",
    "\n",
    "import numpy as np\n",
    "from sklearn import base, linear_model\n",
    "from sympy import var, plot, diff, solve, exp, tanh, functions, stats\n",
    "from toolz import pipe\n",
    "\n",
    "Tensor = np.array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "X = np.array([[0, 0, 1],  # XOR problem\n",
    "              [0, 1, 1],\n",
    "              [1, 0, 1],\n",
    "              [0, 1, 0],\n",
    "              [1, 0, 0],\n",
    "              [1, 1, 1]])\n",
    "Y = np.array([[0, 1, 1, 1, 1, 0]]).T\n",
    "\n",
    "sig = lambda z: 1 / (1 + np.exp(-z))\n",
    "sig_prime = lambda z: z * (1 - z)  # three lines of calculus later"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### _XOR_  is a bit wonky.\n",
    "\n",
    "### Linear model? No dice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0],\n",
       "       [0],\n",
       "       [0],\n",
       "       [1],\n",
       "       [1],\n",
       "       [0]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "linear_model.LogisticRegression(C=1e18).fit(X, Y.ravel()).predict(X).reshape(-1, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# What would a NN / MLP look like in a for loop?\n",
    "\n",
    "## _(a.k.a. gotta start somewhere)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "### Aside: we'll use `sigmoid` as a non-linear activation because it is widely known:\n",
    "\n",
    "# $$\\begin{equation} \\sigma(z) = 1\\space /\\space {(1+e^{-z})} \\end{equation}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[ 0.0016]\n",
      " [ 0.9989]\n",
      " [ 0.9985]\n",
      " [ 1.    ]\n",
      " [ 0.9999]\n",
      " [ 0.0014]]\n"
     ]
    }
   ],
   "source": [
    "betas0_1 = np.random.random(size=(3, 4)) - 0.5\n",
    "betas1_2 = np.random.random(size=(4, 1)) - 0.5\n",
    "\n",
    "lr = 16\n",
    "Z = np.matmul\n",
    "\n",
    "for i in range(30000):\n",
    "    # forward\n",
    "    layer0 = X\n",
    "    layer1 = sig(Z(layer0, betas0_1))\n",
    "    layer2 = sig(Z(layer1, betas1_2))  # hypothesis for NN\n",
    "    \n",
    "    # backward \n",
    "    layer2_loss = layer2 - Y\n",
    "    layer2_delta = layer2_loss * sig_prime(layer2)\n",
    "    grad1_2 = layer1.T @ layer2_delta  # find our gradients\n",
    "    \n",
    "    layer1_loss = layer2_delta @ betas1_2.T\n",
    "    layer1_delta = layer1_loss * sig_prime(layer1)\n",
    "    grad0_1 = layer0.T @ layer1_delta    \n",
    "    \n",
    "    betas0_1 = betas0_1 - lr * grad0_1  # update iteratively\n",
    "    betas1_2 = betas1_2 - lr * grad1_2\n",
    "\n",
    "else:\n",
    "    print(np.round(layer2, 4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Why is this so? Chain rule!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# $Cost = C(\\sigma(Z(X W)))$\n",
    "\n",
    "## $where \\space\\space C(\\hat{y}, y) = \\frac{1}{2}(\\hat{y} - y)^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "# $\\begin{split}C'(W) &= C'(\\sigma) \\cdot \\sigma'(Z) \\cdot Z'(W) \\\\\n",
    "      &= (\\hat{y} -y) \\cdot \\sigma'(Z) \\cdot X\\end{split}$\n",
    "      \n",
    "(_latex [source](https://ml-cheatsheet.readthedocs.io/en/latest/backpropagation.html)_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Ok, your turn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Now, what could go wrong with the above plan?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Ok! But, how can we generalize this??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class LayerKind(enum.Enum):\n",
    "    SIGMOID = (sig, sig_prime)  # activation functions\n",
    "    TANH = (np.tanh, lambda z: 1 - z ** 2)\n",
    "    def __init__(self, fn: Callable, fn_prime: Callable):\n",
    "        self.fn = fn\n",
    "        self.fn_prime = fn_prime\n",
    "\n",
    "K = LayerKind"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "class Layer:\n",
    "    def __init__(self, kind: LayerKind, shape: tuple):\n",
    "        self.fn = kind.fn\n",
    "        self.fn_prime = kind.fn_prime\n",
    "        self.betas = np.random.random(shape) - 0.5\n",
    "    def forward(self, layer_input: Tensor) -> Tensor:\n",
    "        self.input = layer_input\n",
    "        self.activated = self.fn(self.input @ self.betas)\n",
    "        return self.activated\n",
    "    def backward(self, this_layer_loss: Tensor) -> Tensor:\n",
    "        '''Backward propagate gradient info for a mXk layer where prior layer was mXn and next layer is mXj.\n",
    "\n",
    "        So, layer input is mXn, `this_layer_loss` is mXk, and betas are nXk. Grad best be nXk like our errors!\n",
    "\n",
    "        self.activated: mXk\n",
    "        this_layer_loss: mXk\n",
    "        error_signal: mXk\n",
    "        self.betas: nXk\n",
    "        grad: nXk\n",
    "        self.input: mXn\n",
    "        prev_layer_loss: mXn\n",
    "        '''\n",
    "        error_signal = this_layer_loss * self.fn_prime(self.activated)\n",
    "        grad = self.input.T @ error_signal\n",
    "        self.betas = self.betas - 0.4 * grad\n",
    "        prev_layer_loss = error_signal @ self.betas.T\n",
    "        return prev_layer_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "class NN(base.BaseEstimator):\n",
    "    \"\"\"NN uses `Layer` and basically does nothing.\"\"\"\n",
    "    \n",
    "    def __init__(self, layers: List[Layer]):\n",
    "        self.layers = layers\n",
    "    \n",
    "    def _forward(self, X: Tensor) -> Tensor:\n",
    "        return pipe(X, *(layer.forward for layer in self.layers))\n",
    "    \n",
    "    def _backward(self, last_layer_loss: Tensor) -> None:\n",
    "        pipe(last_layer_loss, *(layer.backward for layer in reversed(self.layers)))\n",
    "    \n",
    "    def fit(self, X: Tensor, Y: Tensor) -> 'NN':\n",
    "        for _ in range(60000):\n",
    "            hypothesis = self._forward(X)\n",
    "            self._backward(hypothesis - Y)\n",
    "        return self\n",
    "    \n",
    "    predict_proba = _forward\n",
    "    \n",
    "    def predict(self, X: Tensor) -> Tensor:\n",
    "        return pipe(X, self.predict_proba, np.round, lambda arr: arr.astype(int))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "y_hat = NN(layers=[\n",
    "    Layer(kind=K.SIGMOID, shape=(3, 4)),\n",
    "    Layer(kind=K.SIGMOID, shape=(4, 1)),\n",
    "]).fit(X, Y).predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all(y_hat == Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Ok, your turn!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Can we implement a better stopping criterion?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<br>\n",
    "<div style=\"font-size:80px; font-weight:bold; text-align: center;\"> Questions? </div>\n",
    "<br>\n",
    "  \n",
    "  \n",
    "_@[mstewart141](https://twitter.com/mstewart141) [twitter, github, linkedin]_"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}