MLP_PaperModel_Breast_cancer.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "The dependencies required for the program to run include\n",
    "1.Numpy - Performing Mathematical Calculations\n",
    "2.MatPlotLib - Library for Ploting Graphs\n",
    "3.Pandas - For handling the data. Pandas provide many read-made features for reading csv, and handling data as a DataFrame object \n",
    "4.Seaborn\n",
    "5.Random\n",
    "6.Math\n",
    "7.Sklearn\n",
    "\"\"\"\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "import seaborn as sns\n",
    "import random\n",
    "from math import sqrt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import MinMaxScaler\n",
    "\n",
    "np.random.seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Initializing the Parameters for our Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initializer for an L-layer Neural Network.\n",
    "- The function `parameter_initialization` is a helper function which takes in input a list(dimensions) which contain information about the Layer (both hidden ,input and output layer included) and returns a params dictionary with the requisite size of the params in the given layer using a randomizer function\n",
    "- Additionally assert is used to make sure that the given parameter matrix matches in size with the dimensions dictionary\n",
    "- Each layer in the parameter dictionary is represented as Wn and bn where n indicates the nth layer of the network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def parameter_initialization(dimensions):\n",
    "    \n",
    "    np.random.seed(4)\n",
    "    #Initializing the Parameters Dictionary\n",
    "    params = {}\n",
    "    #L is used to store the length of the total layers in the network\n",
    "    L = len(dimensions)            \n",
    "    #Next we iterate through each layer of the network , filling in the parameter of the network\n",
    "    for l in range(1, L):\n",
    "        #np.random.randn(x,y) creates a random array of size x,y drawn from normal distribution\n",
    "        #We further multiply this matrix with a value of 0.01 to keep the values of the Weights small in the initial iterations of the network\n",
    "        params['W' + str(l)] = np.random.randn(dimensions[l], dimensions[l-1])*0.01\n",
    "        params['b' + str(l)] = np.zeros((dimensions[l],1))\n",
    "        \n",
    "    return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Forward Propagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the params in place ,we now perform forward propagation . \n",
    "\n",
    "First we start of by defining `forward_prop_lin` module which computes the following equations:\n",
    "\n",
    "$$Z^{[l]} = W^{[l]}A^{[l-1]} +b^{[l]}$$\n",
    "\n",
    "Next we code `sigmoid` function which performs the $\\sigma(Z) = \\sigma(W A + b) = \\frac{1}{ 1 + e^{-(W A + b)}}$. operation on the individual elements of the matrix Z.\n",
    "\n",
    "Also for comparitive purpose we also code `relu` function doing $A = RELU(Z) = max(0, Z)$ . In the later sections of the notebook we ran the code using both relu and sigmoid and have drawn a comparitive study of both the methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_prop_lin(A, W, b):\n",
    "    #Z is the input of the activation function which is calculated from the given formula\n",
    "    Z = np.dot(W,A) + b\n",
    "    #temp_backprop is a python dictionary , which will be required later in the backward pass\n",
    "    temp_backprop= (A, W, b)\n",
    "    \n",
    "    return Z,temp_backprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(Z):\n",
    "    #Using Sigmoid Activator , the function has been described in detail in the markdown\n",
    "    A = 1/(1+np.exp(-Z))\n",
    "    temp_backprop= Z\n",
    "    return A,temp_backprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def relu(Z):\n",
    "    #Using Relu Activator, the function has been described in detail in the markdown\n",
    "    A = np.maximum(0,Z)\n",
    "    temp_backprop= Z \n",
    "    return A,temp_backprop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now with the linear forward model in place, we next code up the two helper functions `forward_prop_activation` and `forward_prop_full`.\n",
    "\n",
    "- `forward_prop_activation` takes in input Activations of the previous layer, Weights and bias and applies the requisite activation function i.e sigmoid or relu activator and returns the Activation for the next layer. Also in the process we store thelist_tempbecause they'd be required for future back propagations\n",
    "\n",
    "- `forward_prop_full` this is the main function , this function sums up the entire forward pass for a training example in the dataset taking in X and params for the entire network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_prop_activation(A_prev, W, b, activation):\n",
    "    #There are two choice of activations in this code namely sigmoid and relu\n",
    "    if activation == \"sigmoid\":\n",
    "        #If sigmoid is selected we first apply forward_prop_lin to find the pre-activation parameter Z and then apply the sigmoid activator\n",
    "        Z, linear_temp = forward_prop_lin(A_prev, W, b)\n",
    "        A, activation_temp = sigmoid(Z)\n",
    "    \n",
    "    elif activation == \"relu\":\n",
    "        #If relu is selected we first apply forward_prop_lin to find the pre-activation parameter Z and then apply the relu activator\n",
    "        Z, linear_temp = forward_prop_lin(A_prev, W, b)\n",
    "        A, activation_temp = relu(Z)\n",
    "\n",
    "    temp_backprop= (linear_temp, activation_temp)\n",
    "    #Finally we return the new Activation Parameter for the next layer alongside thelist_tempfor propagation\n",
    "    return A,temp_backprop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_prop_full(X, params):\n",
    "    #list_temp for the history of Activations\n",
    "    list_temp= []\n",
    "    #Activation A[0] is X in the Base Case\n",
    "    A = X\n",
    "    #Since params has 2 params for each Layers . Length of Parameter // 2  returns the Length of the Layer\n",
    "    L = len(params) // 2           \n",
    "    \n",
    "    #Iterating through the individual layers applying Relu Activators for L-1 Layers\n",
    "    for l in range(1, L):\n",
    "        #A prev = A[L-1]\n",
    "        A_prev = A \n",
    "        #Here activations of the individual layers are propagated in the iterative fasion using relu activators with the params as specified in the params dictionary\n",
    "        A,temp_backprop= forward_prop_activation(A_prev, params['W' + str(l)], params['b' + str(l)], \"sigmoid\")\n",
    "        list_temp.append(temp_backprop)\n",
    "        \n",
    "    #The last layer of the network is a sigmoid activator   \n",
    "    AL,temp_backprop= forward_prop_activation(A, params['W' + str(L)], params['b' + str(L)], \"relu\")\n",
    "    list_temp.append(temp_backprop)\n",
    "    #Finally we return the Last Level of Activation alongside the list_temp\n",
    "            \n",
    "    return AL, list_temp"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Cost Function\n",
    "\n",
    "Before Coding up BackPropagation we need to define some Cost Function to check if your model is actually learning  for this purpose we compute the cross-entropy cost $J$, using the following formula: $$-\\frac{1}{m} \\sum\\limits_{i = 1}^{m} (y^{(i)}\\log\\left(a^{[L] (i)}\\right) + (1-y^{(i)})\\log\\left(1- a^{[L](i)}\\right)) \\tag{7}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cost_function(AL, Y):\n",
    "    #Y.shape[1] returns the no of training example\n",
    "    m = Y.shape[1]\n",
    "    #np.multiply is used for the element wise multiplication of the vector\n",
    "    #longprobs vector is ensured that it has the same shape as Y as we have done element wise multiplication\n",
    "    mul_temp = np.multiply(np.log(AL),Y) + np.multiply(np.log(1-AL),1-Y)\n",
    "    #The cost across various examples are summed up to a single cost value\n",
    "    cost = (-1/m)*np.sum(mul_temp, axis = 1)\n",
    "    cost = np.squeeze(cost)      \n",
    "    \n",
    "    return cost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Backpropagation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have calculated the Cost function , using Cross Entropy Losses. We proceed with the Backpropagation of the Layers , with the updation of the params.\n",
    "\n",
    "Similiar to Forward Propagation we start of by defining `backward_prop_lin` module which computes the following equations:\n",
    "\n",
    "Assuming that we have already calculated $dZ$. The outputs $(dW^{[l]}, db^{[l]}, dA^{[l]})$ are computed using the input $dZ^{[l]}$.\n",
    "\n",
    "$$ dW^{[l]} = \\frac{\\partial \\mathcal{L} }{\\partial W^{[l]}} = \\frac{1}{m} dZ^{[l]} A^{[l-1] T} $$\n",
    "$$ db^{[l]} = \\frac{\\partial \\mathcal{L} }{\\partial b^{[l]}} = \\frac{1}{m} \\sum_{i = 1}^{m} dZ^{[l](i)}$$\n",
    "$$ dA^{[l-1]} = \\frac{\\partial \\mathcal{L} }{\\partial A^{[l-1]}} = W^{[l] T} dZ^{[l]} $$\n",
    "\n",
    "Next we code `derivative_sigmoid` function which performs the \n",
    "```python\n",
    "dZ = derivative_sigmoid(dA, activation_temp)\n",
    "```\n",
    "And also `derivative_relu` function which performs\n",
    "```python\n",
    "dZ = derivative_relu(dA, activation_temp)\n",
    "```\n",
    "If $f(.)$ is the activation function, \n",
    "`derivative_sigmoid` and `derivative_relu` compute $$dZ^{[l]} = dA^{[l]} * f'(Z^{[l]}) \\tag{11}$$.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward_prop_lin(dZ,temp_backprop):\n",
    "    #The values of the Previous Activators , Weights and Bias are recovered fromtemp_backprop\n",
    "    A_prev, W, b =temp_backprop\n",
    "    m = A_prev.shape[1]\n",
    "\n",
    "    #The three outputs are calculated using the above formulas\n",
    "    dW = (1/m)*np.dot(dZ,A_prev.T)\n",
    "    db = (1/m)*np.sum(dZ, axis=1, keepdims = True)\n",
    "    dA_prev = np.dot(W.T,dZ)\n",
    "\n",
    "    return dA_prev, dW, db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def derivative_relu(dA,temp_backprop):\n",
    "    #Z recovered fromtemp_backprop\n",
    "    Z =temp_backprop\n",
    "    dZ = np.array(dA, copy=True) \n",
    "    #DZ for ReLU is straight forward , giving zero for Z<=0 and dA for Z>=0\n",
    "    dZ[Z <= 0] = 0\n",
    "    \n",
    "    return dZ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def derivative_sigmoid(dA,temp_backprop):\n",
    "    #Z recovered fromtemp_backprop\n",
    "    Z =temp_backprop\n",
    "    #DZ for Sigmoid \n",
    "    s = 1/(1+np.exp(-Z))\n",
    "    dZ = dA * s * (1-s)\n",
    "    \n",
    "    return dZ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similiar to the Linear Forward Model we now code up the two helper functions `backward_prop_activation` and `backward_prop_full`.\n",
    "\n",
    "- `backward_prop_activation` takes in input change in activations of the previous layer,temp_backpropand activation  and applies the requisite activation_backward function by first computing dZ followed by backward_prop_lin function\n",
    "\n",
    "- `backward_prop_full` this is the main function , this function sums up the entire backward pass for a training example in the dataset taking in X and params for the entire network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward_prop_activation(dA,temp_backprop, activation):\n",
    "    #Retrieving the linear_temp and activation_temp from thetemp_backprop\n",
    "    linear_temp, activation_temp =temp_backprop\n",
    "    #Depending upon the activation function speified in the activation we perform a backward pass on the data\n",
    "    if activation == \"relu\":\n",
    "        #if the activator used was relu we first compute Dz from derivative_relu pass and then use backward_prop_lin on the Dz Parameter thus obtained\n",
    "        dZ = derivative_relu(dA, activation_temp)\n",
    "        dA_prev, dW, db = backward_prop_lin(dZ, linear_temp)\n",
    "    \n",
    "    elif activation == \"sigmoid\":\n",
    "        #if the activator used was sigmoid we first compute Dz from derivative_sigmoid pass and then use backward_prop_lin on the Dz Parameter thus obtained\n",
    "        dZ = derivative_sigmoid(dA, activation_temp)\n",
    "        dA_prev, dW, db = backward_prop_lin(dZ, linear_temp)\n",
    " \n",
    "    return dA_prev, dW, db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def backward_prop_full(AL, Y, list_temp):\n",
    "    \"\"\"\n",
    "    The final function that sums up the enter backpropagation model\n",
    "    1. gradients = a dictionary with the gradients which will be used for parameter updation\n",
    "    2. L = no of layers in the network\n",
    "    \"\"\"\n",
    "    gradients = {}\n",
    "    L = len(list_temp) \n",
    "    m = AL.shape[1]\n",
    "    Y = Y.reshape(AL.shape) \n",
    "    #To compute DL/DA for rhe first Back Propagation\n",
    "    dAL = - (np.divide(Y, AL) - np.divide(1 - Y, 1 - AL))\n",
    "\n",
    "    current_temp = list_temp[L-1]\n",
    "    #Updation of the gradient of the final layer which happens to have a sigmoid activator\n",
    "    gradients[\"dA\" + str(L-1)], gradients[\"dW\" + str(L)], gradients[\"db\" + str(L)] = backward_prop_activation(dAL, current_temp, \"relu\")\n",
    "\n",
    "    \n",
    "    for l in reversed(range(L-1)):\n",
    "        #Next we iterate through individual Layers of the network calculating the gradients at corresponding layers which are sigmoid activated \n",
    "        current_temp = list_temp[l]\n",
    "        dA_prev_temp, dW_temp, db_temp = backward_prop_activation(gradients[\"dA\" + str(l+1)], current_temp, \"sigmoid\")\n",
    "        gradients[\"dA\" + str(l)] = dA_prev_temp\n",
    "        gradients[\"dW\" + str(l + 1)] = dW_temp\n",
    "        gradients[\"db\" + str(l + 1)] = db_temp\n",
    "\n",
    "    #Here in this step the gradient is finally returned for parameter updation\n",
    "    return gradients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Parameter Updation\n",
    "With one pass of Forward and Backward Completed , we conclude the pass by Updating the params of each individual layer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def params_update(params, gradients, learning_rate):\n",
    "    L = len(params) // 2\n",
    "    # number of layers in the neural network\n",
    "    # Iterating through Individual params and updating them with the corresponding gradients multiplied by learning_rate\n",
    "    for l in range(L):\n",
    "        params[\"W\" + str(l + 1)] = params[\"W\" + str(l + 1)] - learning_rate * gradients[\"dW\" + str(l + 1)]\n",
    "        params[\"b\" + str(l + 1)] = params[\"b\" + str(l + 1)] - learning_rate * gradients[\"db\" + str(l + 1)]\n",
    "    \n",
    "    return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Final Batch Model \n",
    "Having coded all the precomponents of the Network, finally we assemble all of them to form a full fledged network. The function `neural_network_batch` provides the implementation for the batch modeling of the Network. In this, the batch_size is set by the user. The errors are updated only after the processing of a batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def neural_network_batch(X, Y, layers_dims, learning_rate = 0.1, num_iterations = 3000, print_cost=False,batch_size=32):\n",
    "    \n",
    "\n",
    "    np.random.seed(1)\n",
    "    costs, accu = [],[]                         \n",
    "    #keep track of cost\n",
    "    #batch_size=8\n",
    "    num_batches = X.shape[1]//batch_size\n",
    "    print(\"number of batches: \" + str(num_batches))\n",
    "    #Initialize the params for the network using the parameter_initialization helper function\n",
    "    params = parameter_initialization(layers_dims)\n",
    " \n",
    "    #Iterating throught the whole set num_iterations times, correcting gradients at each level and also collecting the cost to review them later \n",
    "    for i in range(0, num_iterations):\n",
    "        #For every batch of training example we run the algorithm\n",
    "        for k in range(0,num_batches):\n",
    "            #X_lite and Y_lite contains batch_size entries of X and Y respectively  for training purpose\n",
    "            X_lite = X[:,k*batch_size:(k+1)*batch_size]\n",
    "            Y_lite = Y[:,k*batch_size:(k+1)*batch_size]            \n",
    "             \n",
    "            #One pass of Forward Propagation\n",
    "            AL,list_temp= forward_prop_full(X_lite, params)\n",
    "            \n",
    "            if(k==0):\n",
    "                AL_lite = AL\n",
    "            else:\n",
    "                AL_lite = np.concatenate((AL_lite,AL),axis=1)\n",
    "          \n",
    "            #Computing the Cost function\n",
    "            cost = cost_function(AL, Y_lite)\n",
    "           \n",
    "        \n",
    "            #One pass of Backward Propagation\n",
    "            gradients = backward_prop_full(AL, Y_lite, list_temp)\n",
    "            \n",
    "            #Updation of params\n",
    "            params = params_update(params, gradients, learning_rate)\n",
    "        \n",
    "            \n",
    "\n",
    "        temp_array = AL_lite\n",
    "        p = np.zeros((1,temp_array.shape[1]))\n",
    "        for j in range(0, temp_array.shape[1]):\n",
    "            if temp_array[0,j] > 0.5:\n",
    "                p[0,j] = 1\n",
    "            else:\n",
    "                p[0,j] = 0\n",
    "        \n",
    "        Y_acc = Y[:,0:num_batches*batch_size]\n",
    "        \n",
    "        # Print the cost every 100 training example\n",
    "        if print_cost and i % 10 == 0:\n",
    "            print (\"Cost after iteration %i: %f\" %(i, cost))\n",
    "            acc1 = np.sum((p == Y_acc)/AL_lite.shape[1])\n",
    "            print (\"Accuracy after iteration %i: %f\" %(i, acc1))\n",
    "        if print_cost and i % 10 == 0:\n",
    "            costs.append(cost)\n",
    "            accu.append(acc1)\n",
    "    \n",
    "    print (\"Training Accuracy %f\" %( acc1))\n",
    "    # plot the cost\n",
    "    #plt.subplot(1, 2, 1)     \n",
    "    plt.plot(np.squeeze(costs),label='cost')\n",
    "    plt.plot(np.squeeze(accu),label='accuracy')\n",
    "    plt.ylabel('Metrics')\n",
    "    plt.xlabel('iterations (per tens)')\n",
    "    plt.title(\"Learning rate =\" + str(learning_rate))\n",
    "    plt.legend()\n",
    "    plt.show()\n",
    "    \n",
    "    return params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prediction\n",
    "With the trained model, we are in a position to start making predictions about the independent, unseen training data to assess the credibility of the model. The metric used for the same, is accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(X, y, params):\n",
    "    \"\"\"\n",
    "    To Predict the label , given the dataset X\n",
    "    \"\"\"\n",
    "    \n",
    "    m = X.shape[1] \n",
    "    n = len(params) // 2 # number of layers in the neural network\n",
    "    p = np.zeros((1,m))\n",
    "    \n",
    "    # With the model trained , we do one forward pass of the model and check the probability of the label\n",
    "    temp_array,list_temp= forward_prop_full(X, params)\n",
    "\n",
    "    \n",
    "    #If the probability of any label is greater than 0.5 our model predicts it as 1 or 0\n",
    "    for i in range(0, temp_array.shape[1]):\n",
    "        if temp_array[0,i] > 0.5:\n",
    "            p[0,i] = 1\n",
    "        else:\n",
    "            p[0,i] = 0\n",
    "    #As we already know the correct value of the output we can also calculate the accuracy of the model\n",
    "    print(\"Test Accuracy: \"  + str(np.sum((p == y)/m)))\n",
    "        \n",
    "    return p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loading the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "data1 = pd.read_csv(\"breast_cancer_preprocessed.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "y1 = data1['Class']\n",
    "X1 = data1.drop(['Class'], 1)\n",
    "\n",
    "X1_train, X1_test, y1_train, y1_test = train_test_split(X1, y1, test_size = 0.2, random_state = 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9, 546)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X1_train = X1_train.T\n",
    "X1_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "layers_dims = [X1_train.shape[0], 45, 45, 45, 45, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1, 546)\n",
      "number of batches: 546\n",
      "Cost after iteration 0: 0.785094\n",
      "Accuracy after iteration 0: 0.668498\n",
      "Cost after iteration 10: 0.786758\n",
      "Accuracy after iteration 10: 0.668498\n",
      "Cost after iteration 20: 0.788445\n",
      "Accuracy after iteration 20: 0.668498\n",
      "Cost after iteration 30: 0.790156\n",
      "Accuracy after iteration 30: 0.668498\n",
      "Cost after iteration 40: 0.791888\n",
      "Accuracy after iteration 40: 0.668498\n",
      "Cost after iteration 50: 0.793642\n",
      "Accuracy after iteration 50: 0.668498\n",
      "Cost after iteration 60: 0.795417\n",
      "Accuracy after iteration 60: 0.668498\n",
      "Cost after iteration 70: 0.797210\n",
      "Accuracy after iteration 70: 0.668498\n",
      "Cost after iteration 80: 0.799023\n",
      "Accuracy after iteration 80: 0.668498\n",
      "Cost after iteration 90: 0.800852\n",
      "Accuracy after iteration 90: 0.668498\n",
      "Training Accuracy 0.668498\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmYHWWZ9/Hvj17S6Sx0Z5ElISSOYVgTIg0IDIsgTpxRgRkZAw6SuDCMBh1G5Q3CiBP0vRgRHUFmNCgwuBAgKG9UJMIERCVAOhrEJATCImncYtLZE3rJ/f5RdTrVne6uTtInpzv9+1zXufrUU09V3XUCz131PLUoIjAzM+vOAaUOwMzM+j4nCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcThY2oEj6saRLSx2HWX/jZGH7hKRXJL2t1HFExDsi4n9KHQeApMckfWgfbGeQpNslbZT0B0n/mlP/yrTexnS5QZl54yU9KmmrpOey/6aSjpW0QNKfJfkGrv2Mk4XtNySVlzqGgr4UC/BZYCJwOPBW4CpJUzurKOmvgVnAOWn9NwL/nqlyN/ArYCRwDTBP0uh0XjNwL/DB3t8FKzUnCys5Se+UtFTSeklPSJqUmTdL0ouSNklaLumCzLzpkn4h6cuS1gKfTct+LumLkholvSzpHZll2o7me1B3gqTH020/IulWSd/uYh/OktQg6f9I+gNwh6RaST+UtCZd/w8ljU3rfx44HfiqpM2SvpqWHynpYUnrJK2U9A+98BNfClwfEY0RsQK4DZjeTd1vRsSyiGgEri/UlXQE8GbguojYFhH3A88Cfw8QESsj4pvAsl6I2foYJwsrKUlTgNuBfyI5Wv06MD/T9fEiSaN6IMkR7rclHZJZxcnAS8BBwOczZSuBUcAXgG9KUhchdFf3u8DTaVyfBS7J2Z2DgREkR+SXkfz/dUc6PQ7YBnwVICKuAX4GzIyIoRExU9IQ4OF0u28ApgH/JenozjYm6b/SBNvZ59dpnVrgEOCZzKLPAMd0sQ/HdFL3IEkj03kvRcSmHq7L9iNOFlZqlwFfj4inIqI1HU94HXgLQETcFxG/i4gdEXEP8AJwUmb530XELRHREhHb0rLfRsRtEdEK/A9JY3lQF9vvtK6kccCJwGcioikifg7Mz9mXHSRH3a+nR95rI+L+iNiaNrCfB87sZvl3Aq9ExB3p/vwKuB+4sLPKEfGRiKjp4lM4Oxua/t2QWXQDMKyLGIZ2Upe0fsd5eeuy/YiThZXa4cAnskfFwGHAoQCS3p/poloPHEtyFlCwupN1/qHwJSK2pl+HdlKvu7qHAusyZV1tK2tNRGwvTEiqlvR1Sb+VtBF4HKiRVNbF8ocDJ3f4Ld5Hcsaypzanf4dnyoYDmzqpW6jfsS5p/Y7z8tZl+xEnCyu11cDnOxwVV0fE3ZIOJ+lfnwmMjIga4DdAtkupWFfd/B4YIak6U3ZYzjIdY/kE8JfAyRExHDgjLVcX9VcDP+3wWwyNiH/ubGOSvpaOd3T2WQaQjjv8HpicWXQyXY8rLOuk7h8jYm06742ShnWY7zGKAcDJwvalCklVmU85STK4XNLJSgyR9LdpgzSEpEFdAyBpBsmZRdFFxG+BepJB80pJpwDv2s3VDCMZp1gvaQRwXYf5fyS52qjgh8ARki6RVJF+TpR0VBcxXp4mk84+2XGEu4Br0wH3I4EPA3d2EfNdwAclHS2pBri2UDcingeWAtel/34XAJNIuspI//2qgMp0uioz9mT9nJOF7UsPkjSehc9nI6KepPH6KtAIrCK9+iYilgM3AYtIGtbjgF/sw3jfB5wCrAU+B9xDMp7SU/8JDAb+DDwJPNRh/leA96RXSt2cjmu8nWRg+3ckXWT/Aextg3sdyYUCvwV+CtwYEQ8BSBqXnomMA0jLvwA8CryaLpNNctOAOpJ/qxuA90TEmnTe4ST/roUzjW0kFw/YfkB++ZFZz0i6B3guIjqeIZjt93xmYdaFtAvoLyQdoOQmtvOAB0odl1kp9KW7TM36moOB75HcZ9EA/HN6OavZgONuKDMzy+VuKDMzy7XfdEONGjUqxo8fX+owzMz6lSVLlvw5Ikbn1dtvksX48eOpr68vdRhmZv2KpN/2pJ67oczMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVmuoiYLSVPT9wivkjSrk/njJD0q6VeSfi3pbzLzrk6XW6nkJfJmZlYiRbvPIn0b2K3AuSTP1VksaX762OmCa4F7I+K/0/cMPwiMT79PI3m376HAI5KOSF99aWY24EQEG7e3sG5LE+u2vM66Lc2s2/I6a7c0UTO4kotPHlfU7RfzpryTgFUR8RKApLkkT+3MJotg52saDyR5hj9pvbkR8TrwsqRV6foWFTFeM7N9prl1B41bmli3tYl1m5tYu6WJxq1NrN2c/t2SlBe+N25pomVH58/ye/O4mn6dLMbQ/p3FDcDJHep8FviJpCtI3or2tsyyT3ZYdkxxwjQz2zsRwdam1vSoP/kUGvi1Hc4ECvM3bm/pcn011RWMGFLJiOpKxo2oZsq4GmqrKxkxpJKRQyupra5k5JBBjBia1Blc2dVr3XtPqR/3cRFwZ0TclL628luSevzaTEmXAZcBjBtX3KxqZgNH645g/damdo1/4QxgXcfy9PN6y45O11VRJkYMSRv4oZUcV1vDiOoKRqSN/cjMvBFDKqkZXEF5Wd+79qiYyeI12r/gfmxalvVBYCpARCxK3987qofLEhFzgDkAdXV1fta6me0iItjW3Nque6exk8a+kBAatzSxflszXb29Yeig8qTxH1LJQcOrOOqQ4clZQOFTXbkzCQypZNigciTt250ugmImi8XAREkTSBr6acDFHeq8CpwD3Jm+lL4KWAPMB74r6UskA9wTgaeLGKuZ9RO7c9Rf6Abq6qi//ABRW2jgh1Ry1MHD2xLByOzf9Mi/prqCQeXF7/Lpi4qWLCKiRdJMYAFQBtweEcskzQbqI2I+8AngNklXkgx2T4/kbUzLJN1LMhjeAnzUV0KZ7X8igi1NrW0NfWMnR/ltZwJ7cNR/5MHDM338SXn2LGB41f5x1L8v7Ddvyqurqws/otystJpadtC4tf1RfWPHo/2tTW2DvY1bmmlq7fqov9CoFwZ3C4lgRHUFI4YOajsjSMoH7lH/3pC0JCLq8uqVeoDbzPqoHTuCDdua2x3xt+/zb2433biliU2vd32Fz4GDK9oa9jE1gzluzPB23Twdu3/2l77+/YWThdkAEBFsfr2Fxi3Nab9+cilntntnXeYsoHFrM+u3NtHFZf0MKj+AkUOSgdza6krGj6xuG9zNNviFs4La6r55hY/1nJOFWT+0ram180a+rfFv7tD4N9Hc2nnLnx3krR1SwV8ePGyXPv7CkX9NdQUjhwzaJ9f1W9/iZGFWYtubW1m/dWfj3ri1fTdPZ8lge3Pn/fwSbUfyI4YkN3Qdf1hNJhlUMmJIRbtuH3f3WE84WZj1oqaWHcllnW2Ne/POxr/tb9L9Uyjf0tT1hX7DqsrTI/qdV/eMGFLRofHfeX3/8MEVlB3ght96n5OFWReaW3ewfmtzu6P6xg7ThQTQmCaA7gZ4hw0qp2ZIBSPSa/YnvmFoW2NfU13RrvGvrU7KKtzPb32Ek4UNCM2tO9Ij+aS7p3D039b9U+jqSQd2121pYlM3z+4ZUllGTaYrZ8KoIW1H+zWZ/v/CEX9NdSWV5W74rf9ysrB+5/WW1nZH/O0b/Ob2/f5bm1i/pbnbI/5sw19TXcH4kdVpv3/S4Lf172fqVFV4gNcGFicLK6ntza3tGv3GrR0a/Q5dP+u3NrO5m4Z/6KDytga+tnrnEX9toYsnHfytzXT1uOE3y+dkYb2i8NiGtoHb7CBu5nvhLKDQDdTVVT2Q9PHXZi7ffNPooekZwM7GvnD0764es+JysrBd7NgRbNzeTGOmYe+28d+a1OnqOn4JagZXtB3JH1pTxdGHDm93hF/byRG/B3fN+g4ni/1ctn+/cUvzzoZ/684ksL5DN9CGbc1d3rlbfoCoyTTs40dVM6W6Jm3kK5Ij/0xff60v5zTbLzhZ9BMRwabXW1i/ZecAblsS2Nq8y9F/Yd7Wbq7hr6o4ID2KT7p2DqkZnCSBTFlN2uAnV/lU+AYuswHKyaIEmlp2sH5b2qCnA7gbtmWO+Lfs7NopJIL1W5u7fP+uBMOrKtqO9g8aXtX2yIbC0X677+lRvwd2zaynnCz2wo4dwabtLTuP9LelR/iZ7p62skx5d3fsVpYdQE16dF87pIKJbxi6s9sn7csvzCskgQPdzWNmReZkkdrW1Mr6bbv262/YtvPof32m+6eQBLrq25eSRzLXDE4a9dFDB3HEG4ZR09bg72zsa6or2pJAdWWZu3nMrM8Z8MniT5u2c/p/PNrlaxcBBleUtevCOaRmcLurewpH+gcO3nkG4EFdM9ufDPhkceDgCi49dXxbo184Eyj06x842DdtmZkN+GQxqLyMT//NUaUOw8ysT/NdT2ZmlsvJwszMchU1WUiaKmmlpFWSZnUy/8uSlqaf5yWtz8z7gqRlklZIulm+RMjMrGSKNmYhqQy4FTgXaAAWS5ofEcsLdSLiykz9K4Ap6fdTgdOASensnwNnAo8VK14zM+taMc8sTgJWRcRLEdEEzAXO66b+RcDd6fcAqoBKYBBQAfyxiLGamVk3ipksxgCrM9MNadkuJB0OTAAWAkTEIuBR4PfpZ0FErOhkucsk1UuqX7NmTS+Hb2ZmBX1lgHsaMC8iWgEkvQk4ChhLkmDOlnR6x4UiYk5E1EVE3ejRo/dpwGZmA0kxk8VrwGGZ6bFpWWemsbMLCuAC4MmI2BwRm4EfA6cUJUozM8tVzGSxGJgoaYKkSpKEML9jJUlHArXAokzxq8CZksolVZAMbu/SDWVmZvtG0ZJFRLQAM4EFJA39vRGxTNJsSe/OVJ0GzI2I7CP55gEvAs8CzwDPRMQPihWrmZl1T+3b6P6rrq4u6uvrSx2GmVm/ImlJRNTl1esrA9xmZtaHOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLFdRk4WkqZJWSlolaVYn878saWn6eV7S+sy8cZJ+ImmFpOWSxhczVjMz61p5sVYsqQy4FTgXaAAWS5ofEcsLdSLiykz9K4ApmVXcBXw+Ih6WNBTYUaxYzcyse8U8szgJWBURL0VEEzAXOK+b+hcBdwNIOhooj4iHASJic0RsLWKsZmbWjWImizHA6sx0Q1q2C0mHAxOAhWnREcB6Sd+T9CtJN6ZnKh2Xu0xSvaT6NWvW9HL4ZmZW0FcGuKcB8yKiNZ0uB04HPgmcCLwRmN5xoYiYExF1EVE3evTofRWrmdmAU8xk8RpwWGZ6bFrWmWmkXVCpBmBp2oXVAjwAvLkoUZqZWa5iJovFwERJEyRVkiSE+R0rSToSqAUWdVi2RlLhdOFsYHnHZc3MbN8oWrJIzwhmAguAFcC9EbFM0mxJ785UnQbMjYjILNtK0gX1v5KeBQTcVqxYzcyse8q00f1aXV1d1NfXlzoMM7N+RdKSiKjLq9dXBrjNzKwPc7IwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsV3mpAzAz64nm5mYaGhrYvn17qUPpl6qqqhg7diwVFRV7tHyPkoWkjwN3AJuAbwBTgFkR8ZM92qqZ2W5qaGhg2LBhjB8/HkmlDqdfiQjWrl1LQ0MDEyZM2KN19LQb6gMRsRF4O1ALXALcsEdbNDPbA9u3b2fkyJFOFHtAEiNHjtyrs7KeJovCv87fAN+KiGWZMjOzfcKJYs/t7W/X02SxRNJPSJLFAknDgB17tWUzswFm6dKlPPjgg6UOY4/0NFl8EJgFnBgRW4FKYEbeQpKmSlopaZWkWZ3M/7KkpenneUnrO8wfLqlB0ld7GKeZWZ81EJLFecCLEVFozFuBN3a3gKQy4FbgHcDRwEWSjs7WiYgrI+L4iDgeuAX4XofVXA883sMYzcyK7q677mLSpElMnjyZSy65hFdeeYWzzz6bSZMmcc455/Dqq68CcN9993HssccyefJkzjjjDJqamvjMZz7DPffcw/HHH88999xT4j3ZPT29dPa6iPh+YSIi1ku6Dnigm2VOAlZFxEsAkuaSJJ3lXdS/CLiuMCHpBOAg4CGgrodxmtkA8O8/WMby323s1XUefehwrnvXMd3WWbZsGZ/73Od44oknGDVqFOvWrePSSy9t+9x+++187GMf44EHHmD27NksWLCAMWPGsH79eiorK5k9ezb19fV89av9r7Okp2cWndXLSzRjgNWZ6Ya0bBeSDgcmAAvT6QOAm4BPdrcBSZdJqpdUv2bNmpxwzMz2zsKFC7nwwgsZNWoUACNGjGDRokVcfPHFAFxyySX8/Oc/B+C0005j+vTp3HbbbbS2tpYs5t7S0zOLeklfIulWAvgosKQX45gGzIuIwi/6EeDBiGjobgQ/IuYAcwDq6uqiF+Mxsz4s7wygL/ja177GU089xY9+9CNOOOEElizpzSZz3+vpmcUVQBNwT/p5nSRhdOc14LDM9Ni0rDPTgLsz06cAMyW9AnwReL8k39dhZiV19tlnc99997F27VoA1q1bx6mnnsrcuXMB+M53vsPpp58OwIsvvsjJJ5/M7NmzGT16NKtXr2bYsGFs2rSpZPHvjR6dWUTEFpKroXbHYmCipAkkSWIacHHHSpKOJLnRb1Fme+/LzJ8O1EXE7m7fzKxXHXPMMVxzzTWceeaZlJWVMWXKFG655RZmzJjBjTfeyOjRo7njjjsA+NSnPsULL7xARHDOOecwefJkxo0bxw033MDxxx/P1VdfzXvf+94S71HPdZssJP1nRPyLpB8Au3TzRMS7u1o2IlokzQQWAGXA7RGxTNJsoD4i5qdVpwFzI8LdSGbW5xUGs7MWLly4S73vfa/jxZ3JGMfixYuLFlsx5Z1ZfCv9+8U9WXlEPAg82KHsMx2mP5uzjjuBO/dk+2Zm1ju6TRYRsSS9X+KybNeQmZkNLLkD3OkVSodLqtwH8ZiZWR/U00tnXwJ+IWk+sKVQGBFfKkpUZmbWp/Q0WbyYfg4AhqVlHpA2MxsgeposlkfEfdkCSRcWIR4zM+uDenpT3tU9LDMzs/1Q3n0W7yB5h8UYSTdnZg0HWooZmJnZQNXS0kJ5eU87fvaNvDOL3wH1wHaSZ0EVPvOBvy5uaGZmfc/555/PCSecwDHHHMOcOXMAeOihh3jzm9/M5MmTOeeccwDYvHkzM2bM4LjjjmPSpEncf//9AAwdOrRtXfPmzWP69OkATJ8+ncsvv5yTTz6Zq666iqeffppTTjmFKVOmcOqpp7Jy5UoAWltb+eQnP8mxxx7LpEmTuOWWW1i4cCHnn39+23offvhhLrjggl7d77z7LJ4BnpH03bTuuIhY2asRmJntrh/Pgj8827vrPPg4eEf+I+huv/12RowYwbZt2zjxxBM577zz+PCHP8zjjz/OhAkTWLduHQDXX389Bx54IM8+m8TZ2NiYu+6GhgaeeOIJysrK2LhxIz/72c8oLy/nkUce4dOf/jT3338/c+bM4ZVXXmHp0qWUl5ezbt06amtr+chHPsKaNWvaHjnygQ98YO9+jw56ep4zleQu7kpggqTjgdndPe7DzGx/dPPNN/P97yev91m9ejVz5szhjDPOYMKECUDySA+ARx55pO0BgwC1tbW5677wwgspKysDYMOGDVx66aW88MILSKK5ubltvZdffnlbN1Vhe5dccgnf/va3mTFjBosWLeKuu+7qpT1O9DRZfJbkZUaPAUTE0vQBgWZm+14PzgCK4bHHHuORRx5h0aJFVFdXc9ZZZ3H88cfz3HPP9Xgd2dcubN++vd28IUOGtH3/t3/7N9761rfy/e9/n1deeYWzzjqr2/XOmDGDd73rXVRVVXHhhRf2+phHT6+Gao6IDR3KfJ+FmQ0oGzZsoLa2lurqap577jmefPJJtm/fzuOPP87LL78M0NYNde6553Lrrbe2LVvohjrooINYsWIFO3bsaDtD6WpbY8Yk74u7884728rPPfdcvv71r9PS0tJue4ceeiiHHnoon/vc55gxY0bv7XSqp8limaSLgTJJEyXdAjzR69GYmfVhU6dOpaWlhaOOOopZs2bxlre8hdGjRzNnzhz+7u/+jsmTJ7c9dvzaa6+lsbGx7T3cjz76KAA33HAD73znOzn11FM55JBDutzWVVddxdVXX82UKVPaEgPAhz70IcaNG9f2HvDvfve7bfPe9773cdhhh3HUUUf1+r6rJ08Gl1QNXAO8HRDJY8evj4jt3S64D9XV1UV9fX2pwzCzIlmxYkVRGsH9ycyZM5kyZQof/OAHO53f2W8oaUlE1OWtu6cvP9pKkiyu6Ul9MzPbt0444QSGDBnCTTfdVJT1592UN7+7+b4aysysbyj2O77zzixOAVaTvB/7KZIuKDMzG2DyksXBwLnARSTvz/4RcHdELCt2YGZmHUVEu0tPref29s3V3V4NFRGtEfFQRFwKvAVYBTyWvlvbzGyfqaqqYu3atXvd6A1EEcHatWupqqra43XkDnBLGgT8LcnZxXjgZqDri4PNzIpg7NixNDQ0sGbNmlKH0i9VVVUxduzYPV4+b4D7LuBY4EHg3yPiN3u8JTOzvVBRUdH2SA3b9/JuyvtHYCLwceAJSRvTzyZJG/NWLmmqpJWSVkma1cn8L0tamn6el7Q+LT9e0iJJyyT9WtJ792TnzMysd+Q9dband3jvQlIZcCvJAHkDsFjS/IhYnln/lZn6VwBT0smtwPsj4gVJhwJLJC2IiPV7Go+Zme25PU4GPXASsCoiXoqIJmAucF439S8iuUSXiHg+Il5Iv/8O+BMwuoixmplZN4qZLMaQ3KNR0JCW7ULS4cAEYGEn804ieTT6i53Mu0xSvaR6D3qZmRVPMZPF7pgGzIuI1myhpEOAbwEzImJHx4UiYk5E1EVE3ejRPvEwMyuWYiaL14DDMtNj07LOTCPtgiqQNJzkJsBrIuLJokRoZmY9UsxksRiYKGmCpEqShLDLs6YkHQnUAosyZZUk93LcFRHzihijmZn1QNGSRUS0ADNJHme+Arg3IpZJmi0p+wDCacDcaH9b5j8AZwDTM5fWHl+sWM3MrHs9ep9Ff+D3WZiZ7b6evs+irwxwm5lZH+ZkYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS4nCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcRU0WkqZKWilplaRZncz/sqSl6ed5Sesz8y6V9EL6ubSYcZqZWffKi7ViSWXArcC5QAOwWNL8iFheqBMRV2bqXwFMSb+PAK4D6oAAlqTLNhYrXjMz61oxzyxOAlZFxEsR0QTMBc7rpv5FwN3p978GHo6IdWmCeBiYWsRYzcysG8VMFmOA1ZnphrRsF5IOByYAC3dnWUmXSaqXVL9mzZpeCdrMzHbVVwa4pwHzIqJ1dxaKiDkRURcRdaNHjy5SaGZmVsxk8RpwWGZ6bFrWmWns7ILa3WXNzKzIipksFgMTJU2QVEmSEOZ3rCTpSKAWWJQpXgC8XVKtpFrg7WmZmZmVQNGuhoqIFkkzSRr5MuD2iFgmaTZQHxGFxDENmBsRkVl2naTrSRIOwOyIWFesWM3MrHvKtNH9Wl1dXdTX15c6DDOzfkXSkoioy6vXVwa4zcysD3OyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVmuoiYLSVMlrZS0StKsLur8g6TlkpZJ+m6m/Atp2QpJN0tSMWM1M7OulRdrxZLKgFuBc4EGYLGk+RGxPFNnInA1cFpENEp6Q1p+KnAaMCmt+nPgTOCxYsVrZmZdK+aZxUnAqoh4KSKagLnAeR3qfBi4NSIaASLiT2l5AFVAJTAIqAD+WMRYzcysG8VMFmOA1ZnphrQs6wjgCEm/kPSkpKkAEbEIeBT4ffpZEBErOm5A0mWS6iXVr1mzpig7YWZmpR/gLgcmAmcBFwG3SaqR9CbgKGAsSYI5W9LpHReOiDkRURcRdaNHj96HYZuZDSzFTBavAYdlpsemZVkNwPyIaI6Il4HnSZLHBcCTEbE5IjYDPwZOKWKsZmbWjWImi8XAREkTJFUC04D5Heo8QHJWgaRRJN1SLwGvAmdKKpdUQTK4vUs3lJmZ7RtFSxYR0QLMBBaQNPT3RsQySbMlvTuttgBYK2k5yRjFpyJiLTAPeBF4FngGeCYiflCsWM3MrHuKiFLH0Cvq6uqivr6+1GGYmfUrkpZERF1evVIPcJuZWT/gZGFmZrmcLMzMLJeThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVmYmVkuJwszM8vlZGFmZrmK9qa8fuXHs+APz5Y6CjOzPXPwcfCOG4q6CZ9ZmJlZLp9ZQNEzsplZf+czCzMzy+VkYWZmuZwszMwsl5OFmZnlcrIwM7NcThZmZpbLycLMzHI5WZiZWS5FRKlj6BWS1gC/3YtVjAL+3Evh9Hf+Ldrz79Gef4+d9off4vCIGJ1Xab9JFntLUn1E1JU6jr7Av0V7/j3a8++x00D6LdwNZWZmuZwszMwsl5PFTnNKHUAf4t+iPf8e7fn32GnA/BYeszAzs1w+szAzs1xOFmZmlmvAJwtJUyWtlLRK0qxSx1NKkg6T9Kik5ZKWSfp4qWMqNUllkn4l6YeljqXUJNVImifpOUkrJJ11dWPbAAAGGElEQVRS6phKSdKV6f8nv5F0t6SqUsdUTAM6WUgqA24F3gEcDVwk6ejSRlVSLcAnIuJo4C3ARwf47wHwcWBFqYPoI74CPBQRRwKTGcC/i6QxwMeAuog4FigDppU2quIa0MkCOAlYFREvRUQTMBc4r8QxlUxE/D4ifpl+30TSGIwpbVSlI2ks8LfAN0odS6lJOhA4A/gmQEQ0RcT60kZVcuXAYEnlQDXwuxLHU1QDPVmMAVZnphsYwI1jlqTxwBTgqdJGUlL/CVwF7Ch1IH3ABGANcEfaLfcNSUNKHVSpRMRrwBeBV4HfAxsi4ieljaq4BnqysE5IGgrcD/xLRGwsdTylIOmdwJ8iYkmpY+kjyoE3A/8dEVOALcCAHeOTVEvSCzEBOBQYIukfSxtVcQ30ZPEacFhmemxaNmBJqiBJFN+JiO+VOp4SOg14t6RXSLonz5b07dKGVFINQENEFM4055Ekj4HqbcDLEbEmIpqB7wGnljimohroyWIxMFHSBEmVJANU80scU8lIEkmf9IqI+FKp4ymliLg6IsZGxHiS/y4WRsR+feTYnYj4A7Ba0l+mRecAy0sYUqm9CrxFUnX6/8057OcD/uWlDqCUIqJF0kxgAcnVDLdHxLISh1VKpwGXAM9KWpqWfToiHixhTNZ3XAF8Jz2wegmYUeJ4SiYinpI0D/glyVWEv2I/f/SHH/dhZma5Bno3lJmZ9YCThZmZ5XKyMDOzXE4WZmaWy8nCzMxyOVlYnyfpifTveEkX9/K6P93ZtopF0vmSPlOkdX86v9Zur/M4SXf29nqt//Gls9ZvSDoL+GREvHM3limPiJZu5m+OiKG9EV8P43kCeHdE/Hkv17PLfhVrXyQ9AnwgIl7t7XVb/+EzC+vzJG1Ov94AnC5pafougTJJN0paLOnXkv4prX+WpJ9Jmk96l7GkByQtSd8/cFladgPJU0OXSvpOdltK3Ji+q+BZSe/NrPuxzHsdvpPewYukG9J3gfxa0hc72Y8jgNcLiULSnZK+Jqle0vPp86gK79Do0X5l1t3ZvvyjpKfTsq+nj+RH0mZJn5f0jKQnJR2Ull+Y7u8zkh7PrP4H7OeP37YeiAh//OnTH2Bz+vcs4IeZ8suAa9Pvg4B6kge7nUXyoLsJmboj0r+Dgd8AI7Pr7mRbfw88THJn/0Ekj3c4JF33BpLniB0ALAL+ChgJrGTn2XpNJ/sxA7gpM30n8FC6nokkz1+q2p396iz29PtRJI18RTr9X8D70+8BvCv9/oXMtp4FxnSMn+TO/h+U+r8Df0r7GdCP+7B+7+3AJEnvSacPJGl0m4CnI+LlTN2PSbog/X5YWm9tN+v+K+DuiGgF/ijpp8CJwMZ03Q0A6WNRxgNPAtuBbyp5q15nb9Y7hOQx31n3RsQO4AVJLwFH7uZ+deUc4ARgcXriMxj4UzqvKRPfEuDc9PsvgDsl3UvyYLyCP5E8WdUGMCcL688EXBERC9oVJmMbWzpMvw04JSK2SnqM5Ah+T72e+d4KlEfynLGTSBrp9wAzgbM7LLeNpOHP6jhoGPRwv3II+J+IuLqTec0RUdhuK2k7EBGXSzqZ5IVPSySdEBFrSX6rbT3cru2nPGZh/ckmYFhmegHwz+lj1ZF0RBcv5DkQaEwTxZEkr4wtaC4s38HPgPem4wejSd4S93RXgSl5B8iBkTx08UqS1452tAJ4U4eyCyUdIOkvgDeSdGX1dL86yu7L/wLvkfSGdB0jJB3e3cKS/iIinoqIz5CcARUe338ESdedDWA+s7D+5NdAq6RnSPr7v0LSBfTLdJB5DXB+J8s9BFwuaQVJY/xkZt4c4NeSfhkR78uUfx84BXiG5Gj/qoj4Q5psOjMM+H+SqkiO6v+1kzqPAzdJUubI/lWSJDQcuDwitkv6Rg/3q6N2+yLpWuAnkg4AmoGPAr/tZvkbJU1M4//fdN8B3gr8qAfbt/2YL50124ckfYVksPiR9P6FH0bEvBKH1SVJg4CfAn8V3VyCbPs/d0OZ7Vv/F6gudRC7YRwwy4nCfGZhZma5fGZhZma5nCzMzCyXk4WZmeVysjAzs1xOFmZmluv/A8IpJWaWYho8AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X1_train1 = np.asarray(X1_train)\n",
    "y1_train1 = np.asarray(y1_train)\n",
    "y1_train1 = y1_train1.reshape(1, y1_train1.shape[0])\n",
    "print(y1_train1.shape)\n",
    "params = neural_network_batch(X1_train1, y1_train1, layers_dims, learning_rate = 0.001, num_iterations = 100, print_cost = True, batch_size = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "X1_test = X1_test.T\n",
    "X1_test = np.asarray(X1_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "y1_test = np.asarray(y1_test)\n",
    "y1_test = y1_test.reshape(1, y1_test.shape[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test Accuracy: 0.5766423357664232\n"
     ]
    }
   ],
   "source": [
    "pred_test1 = predict(X1_test, y1_test, params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "coursera": {
   "course_slug": "neural-networks-deep-learning",
   "graded_item_id": "c4HO0",
   "launcher_item_id": "lSYZM"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}