use-case-2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "b289f509-0d9b-4238-b293-ad3b207c7cda",
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c9e59308-f76f-4f04-959f-4c943622f6eb",
   "metadata": {},
   "source": [
    "# Using LMI with a custom dataset\n",
    "\n",
    "## Use case 2 -- run and evaluate LMI with algorithms from [Learned metric index - proposition of learned indexing for unstructured data](https://www.sciencedirect.com/science/article/pii/S0306437921000326).\n",
    "\n",
    "### Necessary components:\n",
    "1. the `descriptors` file -- vector representation \n",
    "    - We assume that you have these descriptors ready\n",
    "2. the `labels` file -- file associating every object form `descriptors` to a node in the template index\n",
    "\n",
    "The supervised version of LMI assumes that you have a template indexing or clustering method that can provide the labels necessary for training for you.\n",
    "    \n",
    "#### Components that will be used but can be extracted from necessary components:\n",
    "5. the `knn-labels` file\n",
    "\n",
    "To learn how to **Create the ground-truth and queries files**, visit the `use-case-1.ipynb` notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df9804ba-4a4b-44ce-bd97-e43a12d4f309",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Example with a custom dataset\n",
    "\n",
    "Within the data folder, we have stored a `test/simple-data.csv` file, which contains a tiny example descriptor dataset on which we'll demonstrate the use with any custom dataset.\n",
    "\n",
    "## Steps:\n",
    "1. Load the configuration file\n",
    "2. Load the dataset using `SimpleDataLoader`\n",
    "3. Load the labels using `SimpleDataLoader`\n",
    "    - Create k-nn labels\n",
    "4. Train and search in the LMI\n",
    "5. Evaluate the results\n",
    "6. Train and search in the LMI using a Multilabel NN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e8a4096d-7e20-41e7-8c28-de2787cbe02a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "from lmi.utils import load_yaml, load_model_config\n",
    "from lmi.data.SimpleDataLoader import SimpleDataLoader\n",
    "from lmi.indexes.LearnedMetricIndex import LMI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "680b1c5f-5689-42bb-89f3-155aed6868a2",
   "metadata": {},
   "source": [
    "#### 1. Load the configuration file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "dc95c1b8-cc00-430b-9dd1-76fd5592a8ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'setup': 'lmi-test',\n",
       " 'data': {'data-dir': '/storage/brno12-cerit/home/tslaninakova/data/test',\n",
       "  'dataset-file': 'simple-data.csv',\n",
       "  'queries': 'simple-queries.txt',\n",
       "  'knn-gt': 'simple-knn.json',\n",
       "  'labels-dir': 'labels/',\n",
       "  'pivots-filename': 'pivots/M-tree.struct',\n",
       "  'normalize': False,\n",
       "  'shuffle': True},\n",
       " 'LMI': {'model-config': './supplementary-experiment-setups/data-driven/models/model-kmeans.yml',\n",
       "  'n_levels': 2,\n",
       "  'training-dataset-percentage': 1},\n",
       " 'experiment': {'output-dir': 'outputs',\n",
       "  'search-stop-conditions': [0.0005,\n",
       "   0.001,\n",
       "   0.003,\n",
       "   0.005,\n",
       "   0.01,\n",
       "   0.05,\n",
       "   0.1,\n",
       "   0.2,\n",
       "   0.3,\n",
       "   0.5],\n",
       "  'knns': 30}}"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config = load_yaml('./supplementary-experiment-setups/dummy-data-config.yml')\n",
    "config"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d456cc94-9a16-421f-80c1-be895ab733b5",
   "metadata": {},
   "source": [
    "#### 2. Load the dataset using `SimpleDataLoader`\n",
    "\n",
    "Note that if the loading method of `SimpleDataLoader` does not work with your dataset, you can easily modify it -- we use the Pandas API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "496dbfc5-74db-4c4f-b48f-d5420e376a67",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:lmi.data.SimpleDataLoader:Loading dataset from /storage/brno12-cerit/home/tslaninakova/data/test/simple-data.csv.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Sample</th>\n",
       "      <th>ecc</th>\n",
       "      <th>N</th>\n",
       "      <th>gammaG</th>\n",
       "      <th>Esoil</th>\n",
       "      <th>Econc</th>\n",
       "      <th>Dbot</th>\n",
       "      <th>H1</th>\n",
       "      <th>H2</th>\n",
       "      <th>H3</th>\n",
       "      <th>Mr_t</th>\n",
       "      <th>Mt_t</th>\n",
       "      <th>Mr_c</th>\n",
       "      <th>Mt_c</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>119.0</td>\n",
       "      <td>19.01165</td>\n",
       "      <td>2575.731201</td>\n",
       "      <td>0.93529</td>\n",
       "      <td>49.808781</td>\n",
       "      <td>30498.150391</td>\n",
       "      <td>21.289761</td>\n",
       "      <td>1.40301</td>\n",
       "      <td>1.08915</td>\n",
       "      <td>1.27428</td>\n",
       "      <td>-2.112034</td>\n",
       "      <td>-0.870522</td>\n",
       "      <td>2.437778</td>\n",
       "      <td>1.518358</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>873</th>\n",
       "      <td>874.0</td>\n",
       "      <td>14.27874</td>\n",
       "      <td>3356.581055</td>\n",
       "      <td>0.97378</td>\n",
       "      <td>106.582359</td>\n",
       "      <td>30839.126953</td>\n",
       "      <td>21.234859</td>\n",
       "      <td>1.43009</td>\n",
       "      <td>1.22072</td>\n",
       "      <td>0.97214</td>\n",
       "      <td>-1.536059</td>\n",
       "      <td>-0.744451</td>\n",
       "      <td>2.445121</td>\n",
       "      <td>1.569207</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Sample       ecc            N   gammaG       Esoil         Econc  \\\n",
       "118   119.0  19.01165  2575.731201  0.93529   49.808781  30498.150391   \n",
       "873   874.0  14.27874  3356.581055  0.97378  106.582359  30839.126953   \n",
       "\n",
       "          Dbot       H1       H2       H3      Mr_t      Mt_t      Mr_c  \\\n",
       "118  21.289761  1.40301  1.08915  1.27428 -2.112034 -0.870522  2.437778   \n",
       "873  21.234859  1.43009  1.22072  0.97214 -1.536059 -0.744451  2.445121   \n",
       "\n",
       "         Mt_c  \n",
       "118  1.518358  \n",
       "873  1.569207  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loader = SimpleDataLoader(config['data'])\n",
    "dataset = loader.load_descriptors()\n",
    "dataset.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e046b41-1481-48bc-ac6c-b511f3c9fcf2",
   "metadata": {},
   "source": [
    "#### 3. Load the labels using `SimpleDataLoader`\n",
    "\n",
    "The labels file, as specified in `config['data']` are located in the `labels/` subfolder with the number of files representing the depth of the template index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3997b65b-0483-4a21-9404-79468bf5a867",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "level-1.txt  level-2.txt\n"
     ]
    }
   ],
   "source": [
    "DIR=f\"{config['data']['data-dir']}/{config['data']['labels-dir']}\"\n",
    "%ls $DIR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "199e3b53-1efd-4c56-9b0d-947d98676bdc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9\t735\n",
      "3\t700\n",
      "4\t412\n",
      "9\t615\n",
      "8\t806\n"
     ]
    }
   ],
   "source": [
    "FILE=DIR+'/level-1.txt'\n",
    "!head -n 5 $FILE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "6842e1e5-b235-44af-a19e-faf67190dfd0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.7\t735\n",
      "3.1\t700\n",
      "4.5\t412\n",
      "9.5\t615\n",
      "8.8\t806\n"
     ]
    }
   ],
   "source": [
    "FILE=DIR+'/level-2.txt'\n",
    "!head -n 5 $FILE"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa9b7bcd-c77a-4ae3-a52a-ce65895b6856",
   "metadata": {},
   "source": [
    "This template index has 2 levels, the first captured in `level-1.txt` where every object (second column) has associated a node label (first column). In the second file, the location is specified as `\"first-node\".\"second-node\"`. Note that these labels represent a balanced tree, i.e., every object is present in every `level-*.txt` file. However, the tree can be unbalanced as well.\n",
    "\n",
    "The `load_labels` function expects the labels files in these forms and with these filenames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "09aa628f-51e6-4013-ba6a-25a8e904698b",
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = loader.load_labels()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "393b5cc6-a486-484e-b10e-39972977d4fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>L1</th>\n",
       "      <th>L2</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>object_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>735</th>\n",
       "      <td>9</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>736</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>737</th>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>738</th>\n",
       "      <td>9</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>739</th>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           L1  L2\n",
       "object_id        \n",
       "735         9   7\n",
       "736         1   9\n",
       "737         2   3\n",
       "738         9   5\n",
       "739         1   8"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels[735:].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d067d7-6294-48fb-b1ae-2619a95ef61c",
   "metadata": {},
   "source": [
    "#### 3.1 Create the k-NN labels\n",
    "\n",
    "If you're **interested in using the Multilabel-NN algorithm (if not, you can skip to 4.)** to train the LMI, you'll need to adjust the labels, such that each object has associated a vector of `k` (in our case 30) node identifiers as labels.\n",
    "\n",
    "As a precursor, we need to know the ground truth of every object -- `k` closest object to every object (as opposed to just a subset, which is what we did in `use-case-1.ipynb`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f68aac5b-a1aa-4111-91b1-35e6de9aba9e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 48s, sys: 262 ms, total: 3min 48s\n",
      "Wall time: 3min 53s\n"
     ]
    }
   ],
   "source": [
    "from lmi.distances.euclidean import get_euclidean_distance\n",
    "def compute_ground_truths(query_objects, dataset, k, metric):\n",
    "    ground_truths = {}\n",
    "    for query_object in query_objects:\n",
    "        distances_to_query_object = {}\n",
    "        for i, data_object in dataset.iterrows():\n",
    "            distances_to_query_object[str(i)] = float(metric(dataset.loc[query_object], data_object))\n",
    "        ground_truths[str(query_object)] = dict(\n",
    "            sorted(distances_to_query_object.items(), key=lambda item: item[1])[:k]\n",
    "        )\n",
    "    return ground_truths\n",
    "\n",
    "%time ground_truths = compute_ground_truths(dataset.index, dataset, k=2, metric=get_euclidean_distance)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ca814d75-04fb-4113-84d3-c95644180e2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# save the ground truths for later use\n",
    "from lmi.utils import save_json\n",
    "save_json(ground_truths, os.path.join(config['data']['data-dir'], 'simple-knn-all-objects.json'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "2355864b-058a-4f5e-9cc3-f1d83ab51eb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['L1', 'L2'], dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "cc324e39-f15c-417a-a4a9-6041c014d430",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 365 ms, sys: 0 ns, total: 365 ms\n",
      "Wall time: 383 ms\n"
     ]
    }
   ],
   "source": [
    "def get_knn_labels_from_ground_truth(ground_truths, labels):\n",
    "    all_vectors = {}\n",
    "    for obj, neighbors in ground_truths.items():\n",
    "        vector_per_object = []\n",
    "        for label_col in labels.columns:\n",
    "            vector_per_object_per_label = []\n",
    "            for neighbor,_ in neighbors.items():\n",
    "                vector_per_object_per_label.append(labels.loc[int(neighbor)][label_col])\n",
    "            vector_per_object.append(vector_per_object_per_label)\n",
    "        all_vectors[int(obj)] = vector_per_object\n",
    "        \n",
    "    knn_labels_df = pd.DataFrame(all_vectors.values(), columns=labels.columns)\n",
    "    knn_labels_df.index = list(all_vectors.keys())\n",
    "    return knn_labels_df\n",
    "\n",
    "%time knn_labels = get_knn_labels_from_ground_truth(ground_truths, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "013579e1-c5f0-4529-afd4-d12e43a4944c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>L1</th>\n",
       "      <th>L2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>118</th>\n",
       "      <td>[3, 8]</td>\n",
       "      <td>[7, 3]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>873</th>\n",
       "      <td>[8, 2]</td>\n",
       "      <td>[1, 8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>807</th>\n",
       "      <td>[2, 8]</td>\n",
       "      <td>[4, 8]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>244</th>\n",
       "      <td>[5, 6]</td>\n",
       "      <td>[4, 5]</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>890</th>\n",
       "      <td>[7, 1]</td>\n",
       "      <td>[9, 7]</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         L1      L2\n",
       "118  [3, 8]  [7, 3]\n",
       "873  [8, 2]  [1, 8]\n",
       "807  [2, 8]  [4, 8]\n",
       "244  [5, 6]  [4, 5]\n",
       "890  [7, 1]  [9, 7]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_labels.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "1260b1d8-9ea0-4d0b-8e1e-63ca804021a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# make sure that the config['data']['queries'] value corresponds to the path we used to store \n",
    "# the queries in the cell above\n",
    "queries = loader.load_queries()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a931c1f-b497-43a8-8cb9-cc4b891d936e",
   "metadata": {},
   "source": [
    "#### 4. Train and search in the LMI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "d2d676ce-8079-478c-b715-61a248b18128",
   "metadata": {},
   "outputs": [],
   "source": [
    "lmi = LMI(config['LMI'], dataset, labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "516b98eb-41b6-47bb-8f85-9a479a3e9402",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'level-0': {'max_iter': 10, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-1': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-2': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-3': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-4': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-5': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-6': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-7': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'},\n",
       " 'level-8': {'max_iter': 5, 'C': 10000, 'model': 'LogReg'}}"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_config = load_model_config(\n",
    "    config['LMI']['model-config'].replace(\n",
    "        'data-driven/models/model-kmeans.yml',\n",
    "        '100k/models/CoPhIR-100k-Mtree-200-LR-model.yml'\n",
    "    ), lmi.n_levels)\n",
    "model_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "e55c00c0-7652-496c-8c9a-71b6a9749af3",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:lmi.indexes.BaseIndex:Training model M.0 (root) on dataset(1000, 14) with {'max_iter': 10, 'C': 10000, 'model': 'LogReg'}.\n",
      "INFO:lmi.indexes.BaseIndex:Training level 1 with {'max_iter': 5, 'C': 10000, 'model': 'LogReg'}.\n",
      "INFO:lmi.indexes.BaseIndex:Finished training the LMI.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 147 ms, sys: 3.99 ms, total: 151 ms\n",
      "Wall time: 156 ms\n"
     ]
    }
   ],
   "source": [
    "%time lmi.train(model_config, rebuild=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "75fd9d6f-dc50-4b87-b7c7-b99fb75e721e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.87 ms, sys: 7 µs, total: 1.88 ms\n",
      "Wall time: 1.88 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([[(3, 7)]], [0.001852273941040039], [62])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time search_results, times, visited_objects_all = lmi.search(queries[0], [50])\n",
    "search_results, times, visited_objects_all"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb16a596-5572-46bb-98a4-8f1b1dd526d6",
   "metadata": {},
   "source": [
    "#### 5. Evaluate LMI's performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "11884b8d-fbb3-4804-9a21-e227c000db52",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:lmi.Experiment:Starting the search for 500 queries.\n",
      "INFO:lmi.Experiment:Evaluated 100/500 queries.\n",
      "INFO:lmi.Experiment:Evaluated 200/500 queries.\n",
      "INFO:lmi.Experiment:Evaluated 300/500 queries.\n",
      "INFO:lmi.Experiment:Evaluated 400/500 queries.\n",
      "INFO:lmi.Experiment:Search is finished, results are stored in: 'outputs/2022-06-23--14-38-09/search.csv'\n",
      "INFO:lmi.Experiment:Consumed memory by evaluating (MB): None\n"
     ]
    }
   ],
   "source": [
    "from lmi.Experiment import Evaluator\n",
    "import pandas as pd\n",
    "\n",
    "queries_df = pd.DataFrame(queries)\n",
    "queries_df = queries_df.set_index([0])\n",
    "\n",
    "e = Evaluator(lmi, ground_truths, queries_df, config)\n",
    "e.run_evaluate()\n",
    "e.generate_summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0c4cb563-b56c-4fbd-8179-84c8619d9b45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"model\": \"LMI\",\n",
      "    \"experiment\": \"outputs/2022-06-23--14-38-09\",\n",
      "    \"stop_conditions_perc\": [\n",
      "        0.0005,\n",
      "        0.001,\n",
      "        0.003,\n",
      "        0.005,\n",
      "        0.01,\n",
      "        0.05,\n",
      "        0.1,\n",
      "        0.2,\n",
      "        0.3,\n",
      "        0.5\n",
      "    ],\n",
      "    \"results\": {\n",
      "        \"0\": {\n",
      "            \"time\": 0.0009971680641174317,\n",
      "            \"score\": 0.928,\n",
      "            \"visited_objects\": 56\n",
      "        },\n",
      "        \"1\": {\n",
      "            \"time\": 0.001017770767211914,\n",
      "            \"score\": 0.928,\n",
      "            \"visited_objects\": 56\n",
      "        },\n",
      "        \"3\": {\n",
      "            \"time\": 0.0010340828895568847,\n",
      "            \"score\": 0.932,\n",
      "            \"visited_objects\": 56\n",
      "        },\n",
      "        \"5\": {\n",
      "            \"time\": 0.0010489230155944825,\n",
      "            \"score\": 0.932,\n",
      "            \"visited_objects\": 56\n",
      "        },\n",
      "        \"10\": {\n",
      "            \"time\": 0.001065969467163086,\n",
      "            \"score\": 0.934,\n",
      "            \"visited_objects\": 57\n",
      "        },\n",
      "        \"50\": {\n",
      "            \"time\": 0.0012597012519836425,\n",
      "            \"score\": 0.978,\n",
      "            \"visited_objects\": 81\n",
      "        },\n",
      "        \"100\": {\n",
      "            \"time\": 0.001591602325439453,\n",
      "            \"score\": 0.991,\n",
      "            \"visited_objects\": 121\n",
      "        },\n",
      "        \"200\": {\n",
      "            \"time\": 0.0028174004554748534,\n",
      "            \"score\": 0.998,\n",
      "            \"visited_objects\": 223\n",
      "        },\n",
      "        \"300\": {\n",
      "            \"time\": 0.004081925392150879,\n",
      "            \"score\": 1.0,\n",
      "            \"visited_objects\": 329\n",
      "        },\n",
      "        \"500\": {\n",
      "            \"time\": 0.005921130657196045,\n",
      "            \"score\": 1.0,\n",
      "            \"visited_objects\": 527\n",
      "        }\n",
      "    },\n",
      "    \"hw_info\": {\n",
      "        \"mem_data_load\": null,\n",
      "        \"mem_train\": null,\n",
      "        \"mem_exp\": null,\n",
      "        \"cpu_brand\": \"AMD EPYC 7532 32-Core Processor\",\n",
      "        \"cpu_hz\": \"3.0575 GHz\",\n",
      "        \"cpu_arch\": \"X86_64\"\n",
      "    }\n",
      "}"
     ]
    }
   ],
   "source": [
    "!cat outputs/2022-06-23--14-38-09/summary.json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc1b2548-34c3-4a7e-9920-3344a799785d",
   "metadata": {},
   "source": [
    "#### 6. Train and search in the LMI using a Multilabel NN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "d7f0966d-4372-4afa-a83e-6800e1f5dbe8",
   "metadata": {},
   "outputs": [],
   "source": [
    "lmi = LMI(config['LMI'], dataset, knn_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "c66856e0-0299-4f2e-8399-284b2431197b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'level-0': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.0001,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None}]}},\n",
       " 'level-1': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.0001,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None}]}},\n",
       " 'level-2': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.01,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}},\n",
       " 'level-3': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.001,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}},\n",
       " 'level-4': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.01,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}},\n",
       " 'level-5': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.01,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}},\n",
       " 'level-6': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.01,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}},\n",
       " 'level-7': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.01,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}},\n",
       " 'level-8': {'model': 'MultilabelNN',\n",
       "  'epochs': 10,\n",
       "  'learning_rate': 0.01,\n",
       "  'optimizer': 'adam',\n",
       "  'loss': 'categorical_crossentropy',\n",
       "  'hidden_layers': {'dense': [{'units': 282,\n",
       "     'activation': 'relu',\n",
       "     'dropout': None},\n",
       "    {'units': 1024, 'activation': 'relu', 'dropout': None},\n",
       "    {'units': 256, 'activation': 'relu', 'dropout': None}]}}}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_config = load_model_config(\n",
    "    config['LMI']['model-config'].replace(\n",
    "        'data-driven/models/model-kmeans.yml',\n",
    "        '100k/models/CoPhIR-100k-Mtree-200-multilabel-NN-model.yml'\n",
    "    ), lmi.n_levels)\n",
    "model_config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "e94734e3-7afd-463f-a1c5-0bde6b336073",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:lmi.indexes.BaseIndex:Training model M.0 (root) on dataset(1000, 14) with {'model': 'MultilabelNN', 'epochs': 10, 'learning_rate': 0.0001, 'optimizer': 'adam', 'loss': 'categorical_crossentropy', 'hidden_layers': {'dense': [{'units': 282, 'activation': 'relu', 'dropout': None}]}}.\n",
      "INFO:lmi.indexes.BaseIndex:Training level 1 with {'model': 'MultilabelNN', 'epochs': 10, 'learning_rate': 0.0001, 'optimizer': 'adam', 'loss': 'categorical_crossentropy', 'hidden_layers': {'dense': [{'units': 282, 'activation': 'relu', 'dropout': None}]}}.\n",
      "INFO:lmi.indexes.BaseIndex:Finished training the LMI.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.15 s, sys: 60.1 ms, total: 1.21 s\n",
      "Wall time: 1.3 s\n"
     ]
    }
   ],
   "source": [
    "%time lmi.train(model_config, rebuild=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "8377950c-69e7-4921-8385-9d851a76a401",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27.9 ms, sys: 4 ms, total: 31.9 ms\n",
      "Wall time: 33.7 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([[(1, 1)]], [0.033634185791015625], [1000])"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%time search_results, times, visited_objects_all = lmi.search(queries[0], [50])\n",
    "search_results, times, visited_objects_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b8981f3-2e40-4de9-9a48-2713d88a08a5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}