contents/Machine_Learning/Neural_Networks_Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](https://upload-images.jianshu.io/upload_images/1194012-03b47f040418215a.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一. Cost Function and Backpropagation\n",
    "\n",
    "### 1. Cost Function\n",
    "\n",
    "![](https://img.halfrost.com/Blog/ArticleImage/72_3.png)\n",
    "\n",
    "\n",
    "假设训练集中有 m 个训练样本，$\\begin{Bmatrix} (x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}), \\cdots ,(x^{(m)},y^{(m)}) \\end{Bmatrix}$，L 表示神经网络的总层数 Layer，用 $S_{l}$ 表示第 L 层的单元数(神经元的数量)，但是不包括第 L 层的偏差单元(常数项)。令 K 为输出层的单元数目，即 最后一层的单元数。\n",
    "\n",
    "**符号约定**：\n",
    "\n",
    "$z_i^{(j)}$ =  第 $j$ 层的第 $i$ 个节点（神经元）的“计算值”     \n",
    "$a_i^{(j)}$ = 第 $j$ 层的第 $i$ 个节点（神经元）的“激活值”    \n",
    "$\\Theta^{(l)}_{i,j}$ = 映射第 $l$ 层到第 $l+1$ 层的权值矩阵的第 $i$ 行第 $j$ 列的分量     \n",
    "$L$ = 神经网络总层数（包括输入层、隐层和输出层）      \n",
    "$s_l$ = 第 $l$ 层节点（神经元）个数，不包括偏移量节点。      \n",
    "$K$ = 输出节点个数     \n",
    "$h_{\\theta}(x)_k$ = 第 $k$ 个预测输出结果     \n",
    "$x^{(i)}$ = 第 $i$ 个样本特征向量     \n",
    "$x^{(i)}_k$ = 第 $i$ 个样本的第 $k$ 个特征值    \n",
    "$y^{(i)}$ = 第 $i$ 个样本实际结果向量   \n",
    "$y^{(i)}_k$ = 第 $i$ 个样本结果向量的第 $k$ 个分量   \n",
    "\n",
    "\n",
    "\n",
    "之前讨论的逻辑回归中代价函数如下：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\rm{CostFunction} = \\rm{F}({\\theta}) &= -\\frac{1}{m}\\left [ \\sum_{i=1}^{m} y^{(i)}logh_{\\theta}(x^{(i)}) + (1-y^{(i)})log(1-h_{\\theta}(x^{(i)})) \\right ] +\\frac{\\lambda}{2m} \\sum_{j=1}^{n}\\theta_{j}^{2}  \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "扩展到神经网络中：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\rm{CostFunction} = \\rm{F}({\\Theta}) &= -\\frac{1}{m}\\left [ \\sum_{i=1}^{m} \\sum_{k=1}^{K} y^{(i)}_{k} log(h_{\\Theta}(x^{(i)}))_{k} + (1-y^{(i)}_{k})log(1-(h_{\\Theta}(x^{(i)}))_{k}) \\right ] +\\frac{\\lambda}{2m} \\sum_{l=1}^{L-1} \\sum_{i=1}^{S_{l}}\\sum_{j=1}^{S_{l} +1}(\\Theta_{j,i}^{(l)})^{2}  \\\\\n",
    "h_{\\Theta}(x) &\\in \\mathbb{R}^{K} \\;\\;\\;\\;\\;\\;\\;\\;\\; (h_{\\Theta}(x))_{i} = i^{th} \\;\\;output \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "$h_{\\Theta}(x)$ 是一个 K 维向量，$ i $ 表示选择输出神经网络输出向量中的第 i 个元素。\n",
    "\n",
    "神经网络的代价函数相比逻辑回归的代价函数，前一项的求和过程中多了一个 $ \\sum_{k=1}^{K} $ ,由于 K 代表了最后一层的单元数，所以这里就是累加了 k 个输出层的代价函数。\n",
    "\n",
    "\n",
    "后一项是正则化项，神经网络的正则化项看起来特别复杂，其实就是对 $ (\\Theta_{j,i}^{(l)})^{2} $ 项对所有的 i，j，l的值求和。正如在逻辑回归中的一样，这里要除去那些对应于偏差值的项，因为我们不对它们进行求和，即不对 $ (\\Theta_{j,0}^{(l)})^{2} \\;\\;\\;\\;(i=0) $ 项求和。\n",
    "\n",
    "### 2. Backpropagation Algorithm 反向传播算法\n",
    "\n",
    "\n",
    "令 $ \\delta_{j}^{(l)} $ 表示第 $l$ 层第 $j$ 个结点的误差。\n",
    "\n",
    "反向传播从最后一层开始往前推：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\delta_{j}^{(L)} &= a_{j}^{(L)} - y_{j} \\\\\n",
    "&=(h_{\\theta}(x))_{j} - y_{j} \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "往前计算几步：\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\delta^{(3)} &= (\\Theta^{(3)})^{T}\\delta^{(4)} . * g^{'}(z^{(3)}) \\\\\n",
    "\\delta^{(2)} &= (\\Theta^{(2)})^{T}\\delta^{(3)} . * g^{'}(z^{(2)}) \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "逻辑函数（Sigmoid函数）求导：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\sigma(x)'&=\\left(\\frac{1}{1+e^{-x}}\\right)'=\\frac{-(1+e^{-x})'}{(1+e^{-x})^2}=\\frac{-1'-(e^{-x})'}{(1+e^{-x})^2}=\\frac{0-(-x)'(e^{-x})}{(1+e^{-x})^2}=\\frac{-(-1)(e^{-x})}{(1+e^{-x})^2}=\\frac{e^{-x}}{(1+e^{-x})^2} \\newline &=\\left(\\frac{1}{1+e^{-x}}\\right)\\left(\\frac{e^{-x}}{1+e^{-x}}\\right)=\\sigma(x)\\left(\\frac{+1-1 + e^{-x}}{1+e^{-x}}\\right)=\\sigma(x)\\left(\\frac{1 + e^{-x}}{1+e^{-x}} - \\frac{1}{1+e^{-x}}\\right)\\\\\n",
    "&=\\sigma(x)(1 - \\sigma(x))\\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "可以算出 $g^{'}(z^{(3)}) = a^{(3)} . * (1-a^{(3)})$ ， $g^{'}(z^{(2)}) = a^{(2)} . * (1-a^{(2)})$。\n",
    "\n",
    "\n",
    "![](https://img.halfrost.com/Blog/ArticleImage/72_4.png)\n",
    "\n",
    "\n",
    "于是可以给出反向传播的算法步骤：\n",
    "\n",
    "首先有一个训练集 $\\begin{Bmatrix} (x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}), \\cdots ,(x^{(m)},y^{(m)}) \\end{Bmatrix}$，初始值对每一个 $(l,i,j)$ 都设置 $\\Delta^{(l)}_{i,j} := 0$ ，即初始矩阵是全零矩阵。\n",
    "\n",
    "针对 $1-m$ 训练集开始以下步骤的训练：\n",
    "\n",
    "### (1) 前向传播\n",
    "\n",
    "设置 $ a^{(1)} := x^{(t)} $，并按照前向传播的方法，计算出每一层的激励 $a^{(l)}$ 。\n",
    "\n",
    "![](https://img.halfrost.com/Blog/ArticleImage/72_5.png)\n",
    "\n",
    "\n",
    "### (2) 计算误差\n",
    "\n",
    "利用 $y^{(t)}$，计算 $\\delta^{(L)} = a^{(L)} - y^{t}$\n",
    "\n",
    "其中 $L$ 是我们的总层数，$a^{(L)}$ 是最后一层激活单元输出的向量。所以我们最后一层的“误差值”仅仅是我们在最后一层的实际结果和 y 中的正确输出的差异。为了获得最后一层之前的图层的增量值，我们可以使用下面步骤中的方程，让我们从右向左前进：\n",
    "\n",
    "### (3) 反向传播\n",
    "\n",
    "通过 $\\delta^{(l)} = ((\\Theta^{(l)})^{(T)}\\delta^{(l+1)}).* a^{(l)} .*(1-a^{(l)})$，计算 $\\delta^{(L-1)},\\delta^{(L-2)},\\cdots,\\delta^{(2)}$ 计算出每一层神经节点的误差。\n",
    "\n",
    "### (4) 计算偏导数\n",
    "\n",
    "最后利用 $\\Delta^{(l)}_{i,j} := \\Delta^{(l)}_{i,j} + a_{j}^{(l)}\\delta_{i}^{(l+1)}$，或者矢量表示为 $\\Delta^{(l)} := \\Delta^{(l)} + \\delta^{(l+1)}(a^{(l)})^{T}$。\n",
    "\n",
    "$$\n",
    "\\frac{\\partial }{\\partial \\Theta_{i,j}^{(l)} }F(\\Theta) = D_{i,j}^{(l)} := \\left\\{\\begin{matrix}\n",
    "\\frac{1}{m} \\left( \\Delta_{i,j}^{(l)} + \\lambda\\Theta_{i,j}^{(l)}  \\right) \\;\\;\\;\\;\\;\\;\\;\\; j\\neq 0\\\\ \n",
    "\\frac{1}{m}\\Delta_{i,j}^{(l)} \\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\;\\; j = 0\n",
    "\\end{matrix}\\right.$$\n",
    "\n",
    "### (5) 更新矩阵\n",
    "\n",
    "更新各层的权值矩阵 $\\Theta^{(l)}$ ，其中 $\\alpha$  为学习率：\n",
    "\n",
    "$$\\Theta^{(l)} = \\Theta^{(l)} - \\alpha D^{(l)}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "------------------------------------------------------\n",
    "\n",
    "## 二. 推导\n",
    "\n",
    "\n",
    "### 1. 目标\n",
    "\n",
    "求 $\\min_\\Theta F(\\Theta)$\n",
    "\n",
    "### 2. 思路\n",
    "\n",
    "类似梯度下降法，给定一个初值后，计算出所有节点的计算值和激活值，然后根据代价函数的变化不断调整参数值（权值），最终不断逼近最优结果，使代价函数值最小。\n",
    "\n",
    "### 3. 推导过程\n",
    "\n",
    "为了实现上述思路，我们必须首先计算代价函数的偏导数：\n",
    "\n",
    "$$\\dfrac{\\partial}{\\partial \\Theta_{i,j}^{(l)}}F(\\Theta)$$\n",
    "\n",
    "这个偏导并不好求，为了方便推导，我们假设只有一个样本（$m=1$，可忽略代价函数中的外部求和），并舍弃正规化部分，然后分为两种情况来求。\n",
    "\n",
    "### 情况1 隐藏层 → 输出层\n",
    "\n",
    "我们知道：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "h_\\Theta(x) &= a^{(j+1)} = g(z^{(j+1)}) \\\\\n",
    "z^{(j)} &= \\Theta^{(j-1)}a^{(j-1)} \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "另外，输出层即第$L$层。\n",
    "\n",
    "所以：\n",
    "\n",
    "$$\\dfrac{\\partial}{\\partial \\Theta_{i,j}^{(L)}}F(\\Theta)\n",
    "= \\dfrac{\\partial F(\\Theta)}{\\partial h_{\\Theta}(x)_i} \\dfrac{\\partial h_{\\Theta}(x)_i}{\\partial z_i^{(L)}} \\dfrac{\\partial z_i^{(L)}}{\\partial  \\Theta_{i,j}^{(L)}}\n",
    "= \\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(L)}} \\dfrac{\\partial a_i^{(L)}}{\\partial z_i^{(L)}} \\dfrac{\\partial z_i^{(L)}}{\\partial \\Theta_{i,j}^{(L)}}$$\n",
    "\n",
    "其中：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(L)}} &= \\dfrac{a_i^{(L)} - y_i}{(1 - a_i^{(L)})a_i^{(L)}} \\\\\n",
    "\\dfrac{\\partial a_i^{(L)}}{\\partial z_i^{(L)}} &= \\dfrac{\\partial g(z_i^{(L)})}{\\partial z_i^{(L)}} = \\dfrac{e^{z_i^{(L)}}}{(e^{z_i^{(L)}}+1)^2} = a_i^{(L)} (1 - a_i^{(L)}) \\\\\n",
    "\\dfrac{\\partial z_i^{(L)}}{\\partial \\Theta_{i,j}^{(L)}} &= \\dfrac{\\partial ( \\sum_{k=0}^{s_{(L-1)}}\\; \\Theta_{i,k}^{(L)} a_k^{(L-1)})}{\\partial  \\Theta_{i,j}^{(L)}} = a_j^{(L-1)} \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "综上：\n",
    "\n",
    "$$\n",
    "\\begin{split}\n",
    "\\dfrac{\\partial}{\\partial \\Theta_{i,j}^{(L)}}F(\\Theta)\n",
    "=& \\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(L)}} \\dfrac{\\partial a_i^{(L)}}{\\partial z_i^{(L)}} \\dfrac{\\partial z_i^{(L)}}{\\partial \\Theta_{i,j}^{(L)}} \\newline  \n",
    "=& \\dfrac{a_i^{(L)} - y_i}{(1 - a_i^{(L)})a_i^{(L)}} a_i^{(L)} (1 - a_i^{(L)}) a_j^{(L-1)} \\newline  \n",
    "=& (a_i^{(L)} - y_i)a_j^{(L-1)}\n",
    "\\end{split}\n",
    "$$\n",
    "\n",
    "### 情况2 隐藏层 / 输入层 → 隐藏层\n",
    "\n",
    "因为 $a^{(1)}=x$，所以可以将输入层和隐藏层同样对待。\n",
    "\n",
    "$$\\dfrac{\\partial}{\\partial \\Theta_{i,j}^{(l)}}F(\\Theta)\n",
    "=\\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(l)}} \\dfrac{\\partial a_i^{(l)}}{\\partial z_i^{(l)}} \\dfrac{\\partial z_i^{(l)}}{\\partial \\Theta_{i,j}^{(l)}}\\ (l = 1, 2, ..., L-1)$$\n",
    "\n",
    "其中后两部分偏导很容易根据前面所得类推出来：\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\dfrac{\\partial a_i^{(l)}}{\\partial z_i^{(l)}} &= \\dfrac{e^{z_i^{(l)}}}{(e^{z_i^{(l)}}+1)^2} = a_i^{(l)} (1 - a_i^{(l)}) \\\\\n",
    "\\dfrac{\\partial z_i^{(l)}}{\\partial \\Theta_{i,j}^{(l)}} &= a_j^{(l-1)} \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "第一部分偏导是不好求解的，或者说是没法直接求解的，我们可以得到一个递推式：\n",
    "\n",
    "$$\\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(l)}} \n",
    "= \\sum_{k=1}^{s_{(l+1)}} \\Bigg[\\dfrac{\\partial F(\\Theta)}{\\partial a_k^{(l+1)}} \\dfrac{\\partial a_k^{(l+1)}}{\\partial z_k^{(l+1)}} \\dfrac{\\partial z_k^{(l+1)}}{\\partial a_i^{(l)}}\\Bigg]$$\n",
    "\n",
    "\n",
    ">因为该层的激活值与下一层各节点都有关，链式法则求导时需一一求导，所以有上式中的求和。\n",
    "\n",
    "递推式中第一部分是递推项，后两部分同样易求：\n",
    "\n",
    "\n",
    "$$\n",
    "\\begin{align*}\n",
    "\\dfrac{\\partial a_k^{(l+1)}}{\\partial z_{k}^{(l+1)}} &= \\dfrac{e^{z_{k}^{(l+1)}}}{(e^{z_{k}^{(l+1)}}+1)^2} = a_k^{(l+1)} (1 - a_k^{(l+1)}) \\\\\n",
    "\\dfrac{\\partial z_k^{(l+1)}}{\\partial a_i^{(l)}} &= \\dfrac{\\partial ( \\sum_{j=0}^{s_l} \\Theta_{k,j}^{(l+1)} a_j^{(l)})}{\\partial a_i^{(l)}} = \\Theta_{k,i}^{(l+1)} \\\\\n",
    "\\end{align*}\n",
    "$$\n",
    "\n",
    "所以，递推式为：\n",
    "\n",
    "$$\n",
    "\\begin{split}\n",
    "\\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(l)}} \n",
    "=& \\sum_{k=1}^{s_{(l+1)}} \\Bigg[\\dfrac{\\partial F(\\Theta)}{\\partial a_k^{(l+1)}} \\dfrac{\\partial a_k^{(l+1)}}{\\partial z_k^{(l+1)}} \\dfrac{\\partial z_k^{(l+1)}}{\\partial a_i^{(l)}}\\Bigg] \\newline  \n",
    "=& \\sum_{k=1}^{s_{(l+1)}} \\Bigg[ \\dfrac{\\partial F(\\Theta)}{\\partial a_k^{(l+1)}} \\dfrac{\\partial a_k^{(l+1)}}{\\partial z_k^{(l+1)}} \\Theta_{k,i}^{(l+1)} \\Bigg] \\newline  \n",
    "=& \\sum_{k=1}^{s_{(l+1)}} \\Bigg[ \\dfrac{\\partial F(\\Theta)}{\\partial a_k^{(l+1)}} a_k^{(l+1)} (1 - a_k^{(l+1)}) \\Theta_{k,i}^{(l+1)} \\Bigg]\n",
    "\\end{split}\n",
    "$$\n",
    "\n",
    "为了简化表达式，定义第 $l$ 层第 $i$ 个节点的误差：\n",
    "\n",
    "$$\\begin{split}\n",
    "\\delta^{(l)}_i \n",
    "=& \\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(l)}} \\dfrac{\\partial a_i^{(l)}}{\\partial z_i^{(l)}} \\newline  \n",
    "=& \\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(l)}} a_i^{(l)} (1 - a_i^{(l)})  \\newline  \n",
    "=& \\sum_{k=1}^{s_{(l+1)}} \\Bigg[ \\dfrac{\\partial F(\\Theta)}{\\partial a_k^{(l+1)}} \\dfrac{\\partial a_k^{(l+1)}}{\\partial z_k^{(l+1)}} \\Theta_{k,i}^{(l+1)} \\Bigg] a_i^{(l)} (1 - a_i^{(l)}) \\newline  \n",
    "=& \\sum_{k=1}^{s_{(l+1)}} \\Big[\\delta^{(l+1)}_k \\Theta_{k,i}^{(l+1)} \\Big] a_i^{(l)} (1 - a_i^{(l)})\n",
    "\\end{split}$$\n",
    "\n",
    "\n",
    "可知，**情况1**的误差为：\n",
    "\n",
    "$$\\begin{split}\n",
    "\\delta^{(L)}_i \n",
    "=& \\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(L)}} \\dfrac{\\partial a_i^{(L)}}{\\partial z_i^{(L)}} \\newline  \n",
    "=& \\dfrac{a_i^{(L)} - y_i}{(1 - a_i^{(L)})a_i^{(L)}} a_i^{(L)} (1 - a_i^{(L)}) \\newline  \n",
    "=& a_i^{(L)} - y_i\n",
    "\\end{split}$$\n",
    "\n",
    "最终的代价函数的偏导为：\n",
    "\n",
    "$$\\begin{split}\n",
    "\\dfrac{\\partial}{\\partial \\Theta_{i,j}^{(l)}}F(\\Theta) \n",
    "=& \\dfrac{\\partial F(\\Theta)}{\\partial a_i^{(l)}} \\dfrac{\\partial a_i^{(l)}}{\\partial z_i^{(l)}} \\dfrac{\\partial z_i^{(l)}}{\\partial \\Theta_{i,j}^{(l)}} \\newline  \n",
    "=& \\delta^{(l)}_i \\dfrac{\\partial z_i^{(l)}}{\\partial \\Theta_{i,j}^{(l)}} \\newline  \n",
    "=& \\delta^{(l)}_i a_j^{(l-1)} \n",
    "\\end{split}$$\n",
    "\n",
    "\n",
    "我们发现，引入误差 $\\delta^{(l)}_i$ 后，这个公式可以通用于**情况1**和**情况2**。\n",
    "\n",
    "可以看出，当前层的代价函数偏导，需要依赖于后一层的计算结果。这也是为什么这个算法的名称叫做“反向传播算法”。\n",
    "\n",
    "\n",
    "### 4. 总结算法公式\n",
    "\n",
    "\n",
    "- 输出层误差\n",
    "\n",
    "$$\\delta^{(L)}_i = a_i^{(L)} - y_i$$\n",
    "\n",
    "\n",
    "- 隐藏层误差（反向传播计算）\n",
    "\n",
    "$$\\delta^{(l)}_i = \\sum_{k=1}^{s_{(l+1)}} \\Big[\\delta^{(l+1)}_k \\Theta_{k,i}^{(l+1)} \\Big] a_i^{(l)} (1 - a_i^{(l)})$$\n",
    "\n",
    "- 代价函数偏导计算（通用）\n",
    "\n",
    "$$\\dfrac{\\partial}{\\partial \\Theta_{i,j}^{(l)}}F(\\Theta) = \\delta^{(l)}_i a_j^{(l-1)}$$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "------------------------------------------------------\n",
    "\n",
    "## 三. Backpropagation Algorithm 反向传播算法过程\n",
    "\n",
    "\n",
    "![](https://img.halfrost.com/Blog/ArticleImage/72_2_.png)\n",
    "\n",
    "\n",
    "\n",
    "有了上述推导，我们描述一下算法具体的操作流程：\n",
    "\n",
    "- 输入：输入样本数据，初始化权值参数（建议随机生成较小的数）。\n",
    "- 前馈：计算各层（$l=2, 3, ..., L$）各节点的计算值（$z^{(l)}=\\Theta^{(l-1)}a^{(l-1)}$）和激活值（$a^{(l)}=g(z^{(l)})$）。\n",
    "- 输出层误差：计算输出层误差<script type=\"math/tex\">\\delta^{(L)}</script>（公式见前文）。\n",
    "- 反向传播误差：计算各层（$l=L-1, L-2, ..., 2$）的误差 $\\delta^{(l)}$（公式见前文）。\n",
    "- 输出：得到代价函数的梯度 $\\nabla F(\\Theta)$（参考前文偏导计算公式）。\n",
    "\n",
    "\n",
    "反向传播算法帮助我们得到了代价函数的梯度，我们就可以借助梯度下降法训练神经网络了。\n",
    "\n",
    "$$\\Theta := \\Theta - \\alpha  \\nabla F(\\Theta)$$\n",
    "\n",
    "$\\alpha $ 为学习速率。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "------------------------------------------------------\n",
    "\n",
    "## 四. Backpropagation Algorithm implementation 算法实现\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "以3层神经网络（输入层、隐层、输出层各一）为例。\n",
    "\n",
    "- X 为大小为样本数∗特征数的样本特征矩阵\n",
    "- Y 为大小为样本数∗输出节点数的样本类别（结果）矩阵\n",
    "- Theta1 为输入层→隐层的权值矩阵\n",
    "- Theta2 为隐藏层→输出层的权值矩阵\n",
    "- m 为样本数\n",
    "- K 为输出层节点数\n",
    "- H 为隐藏层节点数\n",
    "- sigmoid 函数即逻辑函数（S型函数，Sigmoid函数）\n",
    "- sigmoidGradient 函数即 Sigmoid 函数的导函数\n",
    "- 代码实现中，考虑了正规化，避免出现过拟合问题。\n",
    "\n",
    "### 1. 前馈阶段\n",
    "\n",
    "逐层计算各节点值和激活值。\n",
    "\n",
    "\n",
    "```c\n",
    "\n",
    "a1 = X;\n",
    "z2 = [ones(m, 1), a1] * Theta1';\n",
    "a2 = sigmoid(z2);\n",
    "z3 = [ones(m, 1), a2] * Theta2';\n",
    "a3 = sigmoid(z3);\n",
    "\n",
    "```\n",
    "\n",
    "### 2. 代价函数\n",
    "\n",
    "正规化部分需注意代价函数不惩罚偏移参数，即 $\\Theta_{i,0}$（代码表示为 $Theta(:,1)$）。\n",
    "\n",
    "```c\n",
    "\n",
    "F = 1 / m * sum((-log(a3) .* Y - log(1 .- a3) .* (1 - Y))(:)) + ... # 代价部分\n",
    " lambda / 2 / m * (sum((Theta1(:, 2:end) .^ 2)(:)) + sum((Theta2(:, 2:end) .^ 2)(:))); \n",
    " # 正规化部分，lambda为正规参数，需除去偏移参数Theta*(:,1)\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "### 3. 反向传播\n",
    "\n",
    "输出层误差和 $\\Theta^{(2)}$ 梯度计算，反向传播计算隐层误差和 $\\Theta^{(1)}$ 梯度。\n",
    "\n",
    "仍需注意正规化时排除偏移参数，另外注意为激活值补一个偏移量 $1$。\n",
    "\n",
    "\n",
    "```c\n",
    "\n",
    "function g = sigmoid(z)\n",
    "    g = 1.0 ./ (1.0 + exp(-z));\n",
    "end\n",
    "\n",
    "function g = sigmoidGradient(z)\n",
    "    g = sigmoid(z) .* (1 - sigmoid(z));\n",
    "end\n",
    "\n",
    "delta3 = a3 - Y;\n",
    "\n",
    "Theta2_grad = 1 / m * delta3' * [ones(m, 1), a2] + ...\n",
    "  lambda / m * [zeros(K, 1), Theta2(:, 2:end)]; # 正规化部分\n",
    "\n",
    "delta2 = (delta3 * Theta2 .* sigmoidGradient([ones(m, 1), z2]));\n",
    "delta2 = delta2(:, 2:end); # 反向计算多一个偏移参数误差，除去\n",
    "\n",
    "Theta1_grad = 1 / m *  delta2' * [ones(m, 1), a1] + ...\n",
    "  lambda / m * [zeros(H, 1), Theta1(:, 2:end)]; # 正规化部分\n",
    "\n",
    "```\n",
    "\n",
    "\n",
    "------------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "推荐阅读：\n",
    "\n",
    "[Principles of training multi-layer neural network using backpropagation\n",
    "](http://galaxy.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html)\n",
    "\n",
    "[如何直观地解释 back propagation 算法？](https://www.zhihu.com/question/27239198)\n",
    "\n",
    "------------------------------------------------------\n",
    "\n",
    "> GitHub Repo：[Halfrost-Field](https://github.com/halfrost/Halfrost-Field)\n",
    "> \n",
    "> Follow: [halfrost · GitHub](https://github.com/halfrost)\n",
    ">\n",
    "> Source: [https://github.com/halfrost/Halfrost-Field/blob/master/contents/Machine_Learning/Neural\\_Networks\\_Learning.ipynb](https://github.com/halfrost/Halfrost-Field/blob/master/contents/Machine_Learning/Neural_Networks_Learning.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}