07-multiclass-classification.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%config IPCompleter.greedy=True\n",
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sklearn.preprocessing, sklearn.datasets, sklearn.model_selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will be dealing with multiclass classification. We will have finally model, that can distinguish between all the numbers from the MNIST dataset and we will not need to deal with 4 and 9 only. The proper way of handling this problem is to use *softmax* function. I will show different approaches before, so we can compare them."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, we need data. The template is still the same, so I will not describe it any more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data, target = sklearn.datasets.fetch_openml('mnist_784', version=1, return_X_y=True, as_frame=False)\n",
    "target = target.astype(int)\n",
    "data = data.reshape(-1, 784)\n",
    "data[data < 128] = 0\n",
    "data[data > 0] = 1\n",
    "data = np.hstack([data, np.ones((data.shape[0],1))])\n",
    "train_data, test_data, train_target, test_target = sklearn.model_selection.train_test_split(data, target.astype(int), test_size=0.3, random_state=47)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Perceptron\n",
    "\n",
    "If you remember, we dealt with this problem in one of the previous notebook, when we were talking about the perceptron algorithm. Just as a reminder, let's do it once again here, co we may compare the results. I moved it into a separate class, so I don't need to copy-paste it here once again. If you are interested, it is in the [src/perceptron.py](src/perceptron.py) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.9741020408163266, Test accuracy: 0.911\n"
     ]
    }
   ],
   "source": [
    "from src.perceptron_05 import multiclass_perceptron\n",
    "\n",
    "train_acc, test_acc = multiclass_perceptron(train_data, train_target, test_data, test_target, iters=500, random_state=42)\n",
    "\n",
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results don't tell much yet. The test accuracy is maybe too low compare to the train accuracy, but we will see how different models will behave."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One-vs-one\n",
    "\n",
    "We may use the same approach for the logistic regression as well. I will use the `Neuron` implementation from previous notebooks. You can recall it in [src/neuron_05.py](src/neuron_05.py) file. We have seen the `BCELoss` class as well, so I will just copy it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.neuron_05 import Neuron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BCELoss:\n",
    "    def __call__(self, target, predicted):\n",
    "        return np.sum(-target * np.log(np.maximum(predicted, 1e-15)) - (1 - target) * np.log(np.maximum(1 - predicted, 1e-15)), axis=0)\n",
    "    def gradient(self, target, predicted):\n",
    "        return - target / (np.maximum(predicted, 1e-15)) + (1 - target) / (np.maximum(1 - predicted, 1e-15))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's make train the model for each combination. We have seen the code before, so again, I will not comment on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:08 Time:  0:00:08\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:06 Time:  0:00:06\n",
      "100% (200 of 200) |######################| Elapsed Time: 0:00:07 Time:  0:00:07\n"
     ]
    }
   ],
   "source": [
    "# train models\n",
    "models = np.empty((10,10), dtype=object)\n",
    "for i in range(10):\n",
    "    for j in range(i):\n",
    "        models[i][j] = Neuron(BCELoss(), epochs=200, learning_rate=0.001, batch_size=128, random_state=42+i*10+j)\n",
    "        mask = np.logical_or(train_target == i, train_target == j)\n",
    "        current_X = train_data[mask]\n",
    "        current_y = (train_target[mask] - j) / (i - j)\n",
    "        models[i][j].fit(current_X, current_y, progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That took a while, but we have the models. We may now predict the class of each model and take simply the majority for the example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict\n",
    "train_predictions = np.zeros((train_target.shape[0], 10), dtype=int)\n",
    "test_predictions = np.zeros((test_target.shape[0], 10), dtype=int)\n",
    "for i in range(10):\n",
    "    for j in range(i):\n",
    "        prediction = np.around(models[i][j].predict(train_data))\n",
    "        train_predictions[prediction == 0, j] += 1\n",
    "        train_predictions[prediction == 1, i] += 1\n",
    "        prediction = np.around(models[i][j].predict(test_data))\n",
    "        test_predictions[prediction == 0, j] += 1\n",
    "        test_predictions[prediction == 1, i] += 1\n",
    "train_predictions = train_predictions.argmax(axis=1)\n",
    "test_predictions = test_predictions.argmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.9466122448979591, Test accuracy: 0.9168571428571428\n"
     ]
    }
   ],
   "source": [
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)\n",
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't seem bad. The accuracy is a bit higher (for the test set) compared to the simple perceptron. Although the training accuracy is lower than the perceptron one, this is not what we care about. When we have a model, we care about its generalization error. The generalization is better for the logistic regression, as the test accuracy is better and training accuracy is closer to the test accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we didn't use the full potential of the model. Remember, that the logistic regression returns probabilities, not classes. And as you may think, probabilities $45\\% : 55\\%$ should have lower impact on the result that accuracies $1\\% : 99\\%$. But in the previous case, we treat them same.\n",
    "\n",
    "What if we were adding the probabilities instead of classes?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# predict\n",
    "train_predictions = np.zeros((train_target.shape[0], 10), dtype=float)\n",
    "test_predictions = np.zeros((test_target.shape[0], 10), dtype=float)\n",
    "for i in range(10):\n",
    "    for j in range(i):\n",
    "        prediction = models[i][j].predict(train_data)\n",
    "        train_predictions[:, j] += 1-prediction\n",
    "        train_predictions[:, i] += prediction\n",
    "        prediction = models[i][j].predict(test_data)\n",
    "        test_predictions[:, j] += 1-prediction\n",
    "        test_predictions[:, i] += prediction\n",
    "train_predictions = train_predictions.argmax(axis=1)\n",
    "test_predictions = test_predictions.argmax(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.9476326530612245, Test accuracy: 0.9248095238095239\n"
     ]
    }
   ],
   "source": [
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)\n",
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, this is the best result so far. By using probabilities, we allowed the model to use its full potential."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tree-based approach\n",
    "\n",
    "> Note that this chapter is more playing around than what would be done in reality. However, I thought it may be interesting for some of you to see what can be done and how it performs. If you want to see only the mandatory parts, that you gonna need in the following notebooks, please skip to the next chapter.\n",
    "\n",
    "The previous approach, sometimes called *one-vs-one*, had good results, however, we needed to train too many models - in fact when we have $c$ classes, we need $\\frac{c\\cdot(c-1)}{2}$ models. In our case that is $45$ models in total. In reality, that's too much, especially if training of one model took hours or days. First what we may think about is build a tree.\n",
    "\n",
    "We may think about it as a decision tree. At each node, there is a logistic regression that tells us, which sides to continue. We put the resulting classes to the leaves. We still need only binary classification, so we may use the perceptron or logistic regression. We may visualize the tree.\n",
    "\n",
    "![tree in order](img/TreeInOrder_07.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try it. As the implementation is straightforward, but still quite long (I preferred simplicity in this case), I left the implementation itself in the [src/treebased_07.py](src/treebased_07.py) file. Fell free to look at it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from src.treebased_07 import TreeInOrder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (400 of 400) |######################| Elapsed Time: 0:01:07 Time:  0:01:07\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:35 Time:  0:00:35\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:33 Time:  0:00:33\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:21 Time:  0:00:21\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:19 Time:  0:00:19\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:15 Time:  0:00:15\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:13 Time:  0:00:13\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:12 Time:  0:00:12\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:13 Time:  0:00:13\n"
     ]
    }
   ],
   "source": [
    "tree = TreeInOrder(BCELoss(), epochs=400, learning_rate=0.001, batch_size=128, random_state=42)\n",
    "tree.fit(train_data, train_target, progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may notice how times decreases. More precisely, it decreases after every layer of the tree. This is quiet logical, as nodes at lower levels of tree uses less data for training (they need only about labels in their subtree). And of course, the less data we have, the faster the training is. Let's see the performance of this approach."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_predictions = tree.predict_probbased(train_data)\n",
    "test_predictions = tree.predict_probbased(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.8295102040816327, Test accuracy: 0.814952380952381\n"
     ]
    }
   ],
   "source": [
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That doesn't look good. We had almost $92.5\\%$ accuracy before, now we are down to $81\\%$. Why is that? We must understand that in the tree-based approach, the error is accumulated along the path to the node. Even if each model had only $5\\%$ error, it accumulates to $15\\%$ or $20\\%$ error along the way (from the root of the tree to the leaf). We didn't need to train 45 models and 9 was enough. In other words, we may spend more time train the model to achieve better accuracy compared to one-to-one approach."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During the inference (the prediction), I used probabilities and I computed the probability of each class. Finally, I picked up the class with the highest probability. Once again, we may get rid of the probabilities and round the value to either $0$ or $1$. That results in a binary search tree of some sort. It shouldn't be surprising, that the accuracy gets worse, as we are completely ignoring the cases, when the model knows its not sure about."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_predictions = tree.predict_direct(train_data)\n",
    "test_predictions = tree.predict_direct(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.818530612244898, Test accuracy: 0.8069047619047619\n"
     ]
    }
   ],
   "source": [
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the order of nodes is one of the hyperparameters - when we change it, the accuracy can increase or decrease. For example digits $1$ and $7$ are harder to distinguish than $6$ and $8$, so we may keep them together down in the tree. Let's take for example following tree.\n",
    "\n",
    "![tree with custom order](img/TreeSpecialOrder_07.svg)\n",
    "\n",
    "The order is just the first one, that I came. Let's see, how the accuracies change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (400 of 400) |######################| Elapsed Time: 0:01:07 Time:  0:01:07\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:34 Time:  0:00:34\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:34 Time:  0:00:34\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:21 Time:  0:00:21\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:13 Time:  0:00:13\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:12 Time:  0:00:12\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:19 Time:  0:00:19\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:14 Time:  0:00:14\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:00:13 Time:  0:00:13\n"
     ]
    }
   ],
   "source": [
    "from src.treebased_07 import TreeSpecialOrder\n",
    "tree = TreeSpecialOrder(BCELoss(), epochs=400, learning_rate=0.001, batch_size=128, random_state=42)\n",
    "tree.fit(train_data, train_target, progress=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.838265306122449, Test accuracy: 0.814952380952381\n"
     ]
    }
   ],
   "source": [
    "train_predictions = tree.predict_direct(train_data)\n",
    "test_predictions = tree.predict_direct(test_data)\n",
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)\n",
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.8449387755102041, Test accuracy: 0.8217619047619048\n"
     ]
    }
   ],
   "source": [
    "train_predictions = tree.predict_probbased(train_data)\n",
    "test_predictions = tree.predict_probbased(test_data)\n",
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)\n",
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you may see, both direct prediction (when no probabilities are used) and probability-based accuracies increased. Still not a very good, but we were able to increase the accuracy by $0.6\\%$ by simply changing the order of the leaves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As I said, this chapter was more playing around than what would be done in reality. Now let's see the proper way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# One-to-rest\n",
    "\n",
    "The final and the correct approach used in neural networks (we are slowly getting there, aren't we) is to something called one-vs-rest. We are going to train one model for each digit and it will return the probability of that digit. Complement is \"it is any other digit\". In the end, we are going to pick up the most probable digit as the prediction. First what we are going to need is to train the models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100% (400 of 400) |######################| Elapsed Time: 0:01:05 Time:  0:01:05\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:06 Time:  0:01:06\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:06 Time:  0:01:06\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:09 Time:  0:01:09\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:06 Time:  0:01:06\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:06 Time:  0:01:06\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:06 Time:  0:01:06\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:11 Time:  0:01:11\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:21 Time:  0:01:21\n",
      "100% (400 of 400) |######################| Elapsed Time: 0:01:14 Time:  0:01:14\n"
     ]
    }
   ],
   "source": [
    "models = np.empty((10,), dtype=object)\n",
    "for i in range(10):\n",
    "    current_target = train_target.copy() + 100     # copy the train target and shift it up by 100\n",
    "    current_target[current_target == 100 + i] = 1  # set the current training digit to equal 1\n",
    "    current_target[current_target >= 100] = 0      # all other digits set to 0\n",
    "    models[i] = Neuron(BCELoss(), epochs=400, learning_rate=0.001, batch_size=128, random_state=42+i)\n",
    "    models[i].fit(train_data, current_target, progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code should be straightforward. Variable `models` holds logistic regressions - at index $i$ (`models[i]`) is model, that predict probability of input being digit $i$. Now let's predict the labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_distribution = np.zeros((len(train_target), 10))\n",
    "test_distribution = np.zeros((len(test_target), 10))\n",
    "for i in range(10):\n",
    "    train_distribution[:,i] = models[i].predict(train_data)\n",
    "    test_distribution[:,i] = models[i].predict(test_data)\n",
    "train_predictions = np.argmax(train_distribution, axis=1)\n",
    "test_predictions = np.argmax(test_distribution, axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have our predictions. Notice that although `train_distribution` and `test_distribution` are called distributions, they are not yet. Remember, probabilities in distribution need's to accumulate to 1. To really obtain the distributions, we would need to divide it by the sum of probabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.    0.    0.003 0.979 0.    0.016 0.    0.    0.002 0.   ]\n",
      " [0.    0.953 0.003 0.018 0.    0.003 0.    0.005 0.006 0.011]\n",
      " [0.    0.787 0.    0.004 0.    0.018 0.003 0.    0.186 0.   ]\n",
      " [0.    0.802 0.161 0.001 0.005 0.01  0.    0.004 0.014 0.002]\n",
      " [0.    0.    0.011 0.    0.007 0.    0.982 0.    0.    0.   ]\n",
      " [0.    0.001 0.03  0.948 0.    0.015 0.    0.    0.006 0.   ]\n",
      " [0.002 0.    0.    0.    0.007 0.984 0.001 0.    0.003 0.002]\n",
      " [0.002 0.    0.    0.    0.933 0.002 0.    0.    0.025 0.038]\n",
      " [0.    0.    0.002 0.    0.    0.004 0.    0.    0.989 0.006]\n",
      " [0.    0.208 0.078 0.485 0.002 0.218 0.001 0.    0.008 0.001]]\n"
     ]
    }
   ],
   "source": [
    "train_distribution = train_distribution / np.sum(train_distribution, axis=1)[:,np.newaxis]\n",
    "test_distribution = test_distribution / np.sum(test_distribution, axis=1)[:,np.newaxis]\n",
    "with np.printoptions(precision=3, suppress=True):\n",
    "    print(test_distribution[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We usually want to have the distribution, so the model should return it instead of the unnormalize distribution. The function, that is responsible for that is called **softmax** and we will talk about it in the very next notebook. I don't want to implement it yet, as our model needs some refactoring, so we may plug it in.\n",
    "\n",
    "Let's see the accuracies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train accuracy: 0.9236938775510204, Test accuracy: 0.9060952380952381\n"
     ]
    }
   ],
   "source": [
    "train_acc = sklearn.metrics.accuracy_score(train_target, train_predictions)\n",
    "test_acc = sklearn.metrics.accuracy_score(test_target, test_predictions)\n",
    "print(f\"Train accuracy: {train_acc}, Test accuracy: {test_acc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks good. Although the accuracy is not as good as in the first one-vs-one approach. We trade fewer models to train for $2\\%$ of accuracy. That sound's bad right now, but this approach has one benefit - we can join all ten models into one and train them in parallel. That allows us to train model better, shorter time, and track the performance during training - we may monitor the progress and, for example, stop models with bad hyperparameters. Shorter time means we may test more hyperparameter combinations and we may, in fact, achieve better score by just modifying learning rate and other hyperparameters (we will have plenty of them through the following notebooks).\n",
    "\n",
    "Let's refactor the code and learn about softmax - the final stop before our first neural network."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}