best-practices/repeatable-splitting/Reproducible ML - Maybe you shouldn't be using sklearn's train_test_split.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "16461af8",
   "metadata": {},
   "source": [
    "# ML Design Pattern: Repeatable Splitting\n",
    "\n",
    "\n",
    "## Reproducible ML: Maybe you shouldn't be using train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa138163",
   "metadata": {},
   "source": [
    "Reproducibility is critical for robust data science -- after all, it is a science.\n",
    "\n",
    "But reproducibility in ML can be surprisingly difficult. \n",
    "\n",
    "**The behaviour of your model doesn't only depend on your code, but also the underlying dataset that was used to train it**\n",
    "\n",
    "Therefore, you need to keep tight control on which data points were used to train and test your model to ensure reproducibility.\n",
    "\n",
    "A fundamental tenet of the ML workflow is splitting your data into training and testing sets. This involves deliberately withholding some data points from the model training in order to evaluate the performance of your model on 'unseen' data.\n",
    "\n",
    "It is vitally important to be able to reproducibly split your data across different training runs. For a few main reasons:\n",
    "- So you can use the same test data points to effectively compare the performance of different model candidates\n",
    "- To control as many variables as possible to help troubleshoot performance issues\n",
    "- To ensure that you, or your colleagues, can reproduce your model exactly\n",
    "\n",
    "**How you split your data can have a big effect on the perceived model performance**\n",
    "\n",
    "If you split your data 'randomly' there is a statistical chance that more outliers end up in the test set than the training set. As your model won't see many outliers during training, it will perform poorly on the test set when predicting 'outlier' values. \n",
    "\n",
    "Now imagine you randomly split the data again and the outliers now all reside in the training set and none in the test set. It is likely that your 'model performance' will increase. This performance increase has nothing to do with the model, just the statistical properties of the training/test sets.\n",
    "\n",
    "It is important to control and understand the training and test splits when comparing different model candidates and across multiple training runs.\n",
    "\n",
    "**Sklearn train_test_split**\n",
    " \n",
    "Probably, the most common way to split your dataset is to use Sklearn's [train_test_split](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) function.\n",
    "\n",
    "Out of the box, the `train_test_split` function will randomly split your data into a training set and a test set. Each time you run the function you will get a different split for your data. Not ideal for reproducibility.\n",
    "\n",
    "\"Ah!\" you say. \"I set the random seed so it is reproducible!\". \n",
    "\n",
    "Fair point. Setting random seeds is certainly an excellent idea and goes a long way to improve reproducibility. I would highly recommend setting random seeds for any functions which have non-deterministic outputs. \n",
    "\n",
    "**However, random seeds might not be enough to ensure reproducibility** \n",
    "\n",
    "In this post I will demonstrate that the `train_test_split` function is more sensitive than you might think, and explain why using a random seed does not always guarantee reproducibility, particularly if you need to retrain your model in the future."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72889ace",
   "metadata": {},
   "source": [
    "## What is the problem with train_test_split?\n",
    "\n",
    "**Setting the random reed only guarantees reproducible splits if the underlying data does not change in any way**\n",
    "\n",
    "The `train_test_split` is not *deterministic*. \n",
    "\n",
    "The splits from `train_test_split` are sensitive to any new data added to the dataset as well as the *ordering* of the underlying data.\n",
    "\n",
    "If your dataset is shuffled or amended in any way, the data will be split completely differently. It cannot be guaranteed that an individual data point will *always* be in the training set or *always* be in the test set. This means data points that were in the original training set might now end up in the test set and visa versa if the data was shuffled.\n",
    "\n",
    "**Therefore, for the same dataset, you can get different splits depending on how the rows in the dataset are ordered.** \n",
    "\n",
    "It is not a very robust solution. \n",
    "\n",
    "Even if one data point is removed, the order of two rows are switched, or a single data point is added you will get a *completely* different training and test split.\n",
    "\n",
    "This 'ultra sensitivity' to the data might be surprising and lead to unexpected model training results which can be hard to debug and reproduce if your dataset ordering changes.\n",
    "\n",
    "Let's demonstrate the issue with a simple demo.\n",
    "\n",
    "We will first download an example dataset from `sklearn.datasets` and create an 'index' column to uniquely identify each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a13c8945",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>mean radius</th>\n",
       "      <th>mean texture</th>\n",
       "      <th>mean perimeter</th>\n",
       "      <th>mean area</th>\n",
       "      <th>mean smoothness</th>\n",
       "      <th>mean compactness</th>\n",
       "      <th>mean concavity</th>\n",
       "      <th>mean concave points</th>\n",
       "      <th>mean symmetry</th>\n",
       "      <th>...</th>\n",
       "      <th>worst radius</th>\n",
       "      <th>worst texture</th>\n",
       "      <th>worst perimeter</th>\n",
       "      <th>worst area</th>\n",
       "      <th>worst smoothness</th>\n",
       "      <th>worst compactness</th>\n",
       "      <th>worst concavity</th>\n",
       "      <th>worst concave points</th>\n",
       "      <th>worst symmetry</th>\n",
       "      <th>worst fractal dimension</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>17.99</td>\n",
       "      <td>10.38</td>\n",
       "      <td>122.80</td>\n",
       "      <td>1001.0</td>\n",
       "      <td>0.11840</td>\n",
       "      <td>0.27760</td>\n",
       "      <td>0.3001</td>\n",
       "      <td>0.14710</td>\n",
       "      <td>0.2419</td>\n",
       "      <td>...</td>\n",
       "      <td>25.38</td>\n",
       "      <td>17.33</td>\n",
       "      <td>184.60</td>\n",
       "      <td>2019.0</td>\n",
       "      <td>0.1622</td>\n",
       "      <td>0.6656</td>\n",
       "      <td>0.7119</td>\n",
       "      <td>0.2654</td>\n",
       "      <td>0.4601</td>\n",
       "      <td>0.11890</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>20.57</td>\n",
       "      <td>17.77</td>\n",
       "      <td>132.90</td>\n",
       "      <td>1326.0</td>\n",
       "      <td>0.08474</td>\n",
       "      <td>0.07864</td>\n",
       "      <td>0.0869</td>\n",
       "      <td>0.07017</td>\n",
       "      <td>0.1812</td>\n",
       "      <td>...</td>\n",
       "      <td>24.99</td>\n",
       "      <td>23.41</td>\n",
       "      <td>158.80</td>\n",
       "      <td>1956.0</td>\n",
       "      <td>0.1238</td>\n",
       "      <td>0.1866</td>\n",
       "      <td>0.2416</td>\n",
       "      <td>0.1860</td>\n",
       "      <td>0.2750</td>\n",
       "      <td>0.08902</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>19.69</td>\n",
       "      <td>21.25</td>\n",
       "      <td>130.00</td>\n",
       "      <td>1203.0</td>\n",
       "      <td>0.10960</td>\n",
       "      <td>0.15990</td>\n",
       "      <td>0.1974</td>\n",
       "      <td>0.12790</td>\n",
       "      <td>0.2069</td>\n",
       "      <td>...</td>\n",
       "      <td>23.57</td>\n",
       "      <td>25.53</td>\n",
       "      <td>152.50</td>\n",
       "      <td>1709.0</td>\n",
       "      <td>0.1444</td>\n",
       "      <td>0.4245</td>\n",
       "      <td>0.4504</td>\n",
       "      <td>0.2430</td>\n",
       "      <td>0.3613</td>\n",
       "      <td>0.08758</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>11.42</td>\n",
       "      <td>20.38</td>\n",
       "      <td>77.58</td>\n",
       "      <td>386.1</td>\n",
       "      <td>0.14250</td>\n",
       "      <td>0.28390</td>\n",
       "      <td>0.2414</td>\n",
       "      <td>0.10520</td>\n",
       "      <td>0.2597</td>\n",
       "      <td>...</td>\n",
       "      <td>14.91</td>\n",
       "      <td>26.50</td>\n",
       "      <td>98.87</td>\n",
       "      <td>567.7</td>\n",
       "      <td>0.2098</td>\n",
       "      <td>0.8663</td>\n",
       "      <td>0.6869</td>\n",
       "      <td>0.2575</td>\n",
       "      <td>0.6638</td>\n",
       "      <td>0.17300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>20.29</td>\n",
       "      <td>14.34</td>\n",
       "      <td>135.10</td>\n",
       "      <td>1297.0</td>\n",
       "      <td>0.10030</td>\n",
       "      <td>0.13280</td>\n",
       "      <td>0.1980</td>\n",
       "      <td>0.10430</td>\n",
       "      <td>0.1809</td>\n",
       "      <td>...</td>\n",
       "      <td>22.54</td>\n",
       "      <td>16.67</td>\n",
       "      <td>152.20</td>\n",
       "      <td>1575.0</td>\n",
       "      <td>0.1374</td>\n",
       "      <td>0.2050</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.1625</td>\n",
       "      <td>0.2364</td>\n",
       "      <td>0.07678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  mean radius  mean texture  mean perimeter  mean area  \\\n",
       "0      0        17.99         10.38          122.80     1001.0   \n",
       "1      1        20.57         17.77          132.90     1326.0   \n",
       "2      2        19.69         21.25          130.00     1203.0   \n",
       "3      3        11.42         20.38           77.58      386.1   \n",
       "4      4        20.29         14.34          135.10     1297.0   \n",
       "\n",
       "   mean smoothness  mean compactness  mean concavity  mean concave points  \\\n",
       "0          0.11840           0.27760          0.3001              0.14710   \n",
       "1          0.08474           0.07864          0.0869              0.07017   \n",
       "2          0.10960           0.15990          0.1974              0.12790   \n",
       "3          0.14250           0.28390          0.2414              0.10520   \n",
       "4          0.10030           0.13280          0.1980              0.10430   \n",
       "\n",
       "   mean symmetry  ...  worst radius  worst texture  worst perimeter  \\\n",
       "0         0.2419  ...         25.38          17.33           184.60   \n",
       "1         0.1812  ...         24.99          23.41           158.80   \n",
       "2         0.2069  ...         23.57          25.53           152.50   \n",
       "3         0.2597  ...         14.91          26.50            98.87   \n",
       "4         0.1809  ...         22.54          16.67           152.20   \n",
       "\n",
       "   worst area  worst smoothness  worst compactness  worst concavity  \\\n",
       "0      2019.0            0.1622             0.6656           0.7119   \n",
       "1      1956.0            0.1238             0.1866           0.2416   \n",
       "2      1709.0            0.1444             0.4245           0.4504   \n",
       "3       567.7            0.2098             0.8663           0.6869   \n",
       "4      1575.0            0.1374             0.2050           0.4000   \n",
       "\n",
       "   worst concave points  worst symmetry  worst fractal dimension  \n",
       "0                0.2654          0.4601                  0.11890  \n",
       "1                0.1860          0.2750                  0.08902  \n",
       "2                0.2430          0.3613                  0.08758  \n",
       "3                0.2575          0.6638                  0.17300  \n",
       "4                0.1625          0.2364                  0.07678  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "# download an example dataset\n",
    "data = load_breast_cancer()\n",
    "df = pd.DataFrame(data[\"data\"], columns=data[\"feature_names\"])\n",
    "\n",
    "# create an 'index' column to use to uniquely identify each row\n",
    "df = df.reset_index(drop=False)\n",
    "\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9a2c124",
   "metadata": {},
   "source": [
    "Now let's split the data using Sklearn's `train_test_split`, setting the random state (seed)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "1224c72f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "TEST_RATIO = 0.1\n",
    "SEED = 42\n",
    "\n",
    "# split into training and test using a random seed\n",
    "x_train_skl, x_test_skl = train_test_split(df, test_size=TEST_RATIO, random_state=SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "adf9646c",
   "metadata": {},
   "source": [
    "Next, we shuffle the original dataframe and split the data again. We will still use the same random seed as before for consistency.\n",
    "\n",
    "Note that no new data has been added, we have just reordered the rows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ec9e29d1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# shuffle the orginal dataframe\n",
    "df_shuffled = df.sample(frac=1)\n",
    "\n",
    "# split the shuffled dataframe using the same random seed\n",
    "x_train_skl_shuffled, x_test_skl_shuffled = train_test_split(\n",
    "    df_shuffled, test_size=TEST_RATIO, random_state=SEED\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "329cab57",
   "metadata": {},
   "source": [
    "Ideally the rows contained in the `x_test_skl` and `x_test_skl_shuffled` test sets should be identical as we used the same random seed. \n",
    "\n",
    "However, when we compare the row ids contained in each test set, we notice they are different! Even though the random state (seed) was the same both times. Nothing in the data has changed, it was just shuffled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "c0f860ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compare the row ids included in the original test set vs shuffled test set\n",
    "# should return True if identical rows are included in each test set\n",
    "set(x_test_skl[\"index\"]) == set(x_test_skl_shuffled[\"index\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e4f4e6e",
   "metadata": {},
   "source": [
    "This highlights just how sensitive the `train_test_split` function is, even to ordering of the data.\n",
    "\n",
    "More importantly, if there was a change in the underlying data, such as a reordering, which resulted in different splits, it would be extremely difficult to reproduce the original data splits and debug model performance.\n",
    "\n",
    "I was certainly surprised by how sensitive this function was to something as trivial as the ordering of the underlying dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42c4c5c4",
   "metadata": {},
   "source": [
    "## What are the consequences of relying on a random seed when retraining the model with updated data?\n",
    "\n",
    "As mentioned previously, the results of your model can vary significantly depending on how the data was split.\n",
    "\n",
    "In the future, when you come to retrain your model with some updated data you want to be able to control as many variables as possible in order to effectively compare the accuracy and performance of each model.\n",
    "\n",
    "**You cannot guarantee reproducibility and transparency of splits with random seed**\n",
    "\n",
    "It is risky to rely on a random seed from a reproducibility point of view.\n",
    "\n",
    "The random seed only guarantees reproducibility when the dataset has not changed in any way.\n",
    "\n",
    "Can you be 100% sure the dataset has not changed between training runs? If a colleague has removed an outlier data point or if new rows have been added. Your data splits will be completely different to your original splits with no way to easily replicate the old data splits.\n",
    "\n",
    "You can use data versioning tools, such as [dvc](https://dvc.org/) to help keep track of changes, however, that doesn't prevent your data splits from changing. It would be better to protect against split changes in your code.\n",
    "\n",
    "Additionally, the process of splitting by the `train_test_split` function is not very transparent. It is non-deterministic. This means the same data point can be split differently and it is not clear why it appears in the training set one time and in the testing set the next.\n",
    "\n",
    "The lack of transparency makes it hard to understand or predict which split the data point will be placed. Now or in the future.\n",
    "\n",
    "**Difficult to effectively compare models**\n",
    "\n",
    "When comparing models, we want to be able to control as many variables as possible. That should include which data points were used for training and testing.\n",
    "\n",
    "If your data splits are significantly different between runs you might observe considerable differences in performance. For example if you have a couple of 'outlier' data points  that were in your training set for the original training run, but are now in your test set, your model performance might 'decline' as it could not predict outlier values in the test set as well as before.\n",
    "\n",
    "**Difficult to debug**\n",
    "\n",
    "If you can't effectively compare models, it can make it hard to debug performance issues.\n",
    "\n",
    "Imagine you add some new data points to your dataset and retrain your model, but the performance of the model drops.\n",
    "\n",
    "If you have used `train_test_split` with the random seed set, you will have completely different data splits as the underlying data has changed. It would be difficult to understand whether the model performance decline was due to the quality of the new data, or, as highlighted in the previous point, it was just because the data was split differently. So the model performance decline was due to statistical variation in the way the data was split."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "233f8bd6",
   "metadata": {},
   "source": [
    "## When might train_test_split not be appropriate?\n",
    "\n",
    "**If you need to retrain your model in the future on the original data + new data**\n",
    "\n",
    "As demonstrated, any underlying change to your existing data, be it reordering or even adding one additional data point will cause completely different data splits. Your original data splits will not be reproducible.\n",
    "\n",
    "If you are retraining the model with a *completely* new dataset it isn't a problem as obviously all the training and test data points will be different.\n",
    "\n",
    "But if you are training again with a dataset that includes your original data points, ideally you should be able to replicate their original data splits during the new training run. Even with the random seed set, `train_test_split` will not guarantee this. \n",
    "\n",
    "\n",
    "**If you are sampling or retrieving your source data from an evolving data source**\n",
    "\n",
    "In an ideal situation, you should have full control of your source dataset, however, sometimes this is not the case.\n",
    "\n",
    "For example, if you are using a table stored in BigQuery as the source that is used by many teams. You cannot guarantee the order of the rows returned by a query and new rows might be appended to the table in the meantime.\n",
    "\n",
    "Another example, would be if you are working with image data stored in a filesystem. If new images are added to your source folder you cannot guarantee the ordering of filepaths especially if new images are added.\n",
    "\n",
    "**If there is a possibility of data leakage**\n",
    "\n",
    "Imagine you are working with a dataset containing information about airline arrival times and you want to predict the chances of a plane arriving late. \n",
    "\n",
    "It is likely that rows of data at the same date will be correlated with each other. If we randomly split the data we risk data leakage. Therefore, we want a method to split the data such that data points at the same date only end up int the training set, or only end up in the test set.\n",
    "\n",
    "`train_test_split` does not allow for splitting the data based on a particular column value.\n",
    "  \n",
    "**If you have large datasets which do not fit in memory**\n",
    "\n",
    "If you need to distribute your data across many machines in order to parallelise data processing, using a non-deterministic method for splitting your training and test data can be problematic and difficult to ensure reproducibility.\n",
    "\n",
    "\n",
    "**If your experimentation or production code will be rewritten in another language**\n",
    "\n",
    "As the `train_test_split` is non-deterministic, the data splits will not be easily reproducible across languages. \n",
    "\n",
    "For example, you might want to compare the performance of your custom Python model to a model created using BigQuery's BQML which is defined using SQL. The splits from `train_test_split` Sklearn will not easily translate directly to SQL.\n",
    "\n",
    "It can also be common for teams to prototype models using Python, but then write their production systems in another language such as Java. To help the process of translating the prototype model into another language, ideally we should be able to split the data in the same way in both languages to ensure reproducibility and help debug any differences from the original model to the new model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64ce2b57",
   "metadata": {},
   "source": [
    "## The Solution: Hashing\n",
    "\n",
    "### What is hashing?\n",
    "\n",
    "> \"A hash function is any function that can be used to map data of arbitrary size to fixed-size values\" [Wikipedia](https://en.wikipedia.org/wiki/Hash_function)\n",
    "\n",
    "There are many different hashing algorithms, but essentially they allow you to reproducibly convert an input into an arbitrary value.\n",
    "\n",
    "The output of the hashing function is deterministic -- it will always be the same for the same input.\n",
    "\n",
    "\n",
    "### How does it work for splitting data reproducibly?\n",
    "\n",
    "In the context of data splitting, we can use hashing to reliably assign splits to individual data points. As this is a deterministic process, we can ensure that the data point is always assigned to the same split which aids reproducibility.\n",
    "\n",
    "The process works as follows:\n",
    "- Use a unique identifier for the data point (e.g. an ID or by concatenating multiple columns) and use a hashing algorithm to convert it to an arbitrary integer. Each unique data point will have a unique output from the hashing function.\n",
    "- Use a [modulo operation](https://en.wikipedia.org/wiki/Modulo_operation) to arbitrarily split the data into 'buckets'\n",
    "- Select all datapoints in a subset of buckets to be the training set and the rest to be in the test set\n",
    "\n",
    "```\n",
    "# example pseudo code\n",
    "\n",
    "id = \"0001\"\n",
    "\n",
    "# convert id into a hashed integer value\n",
    "hash_value = hash(id)\n",
    "\n",
    "# assign a bucket between 0 and 9\n",
    "bucket = hash_value % 10\n",
    "\n",
    "# add id to train set if less than 9 (i.e. approx 90% of the data)\n",
    "if bucket < 9:\n",
    "   train_set.append(id)\n",
    "else:\n",
    "   test_set.append(id)\n",
    "```\n",
    "\n",
    "\n",
    "### Reasons to use hashing\n",
    "\n",
    "**Deterministic**\n",
    "\n",
    "Hashing is robust to underlying changes in the data, unlike `train_test_split`.\n",
    "\n",
    "Using this method, an individual data point will *always* be assigned to the same bucket. If the data is reordered or new data is added, the assigned bucket will not change. This is preferable as a data point's train/test split assignment is now independent of the rest of the dataset. \n",
    "\n",
    "**Improves development and reduces chances of human error**\n",
    "\n",
    "When working on models in parallel with colleagues, it is very easy to accidentally forget to use random seeds or even use different random seeds. This leaves you open to the risk of human error.\n",
    "\n",
    "Using the same hashing algorithm removes the need to control reproducibility explicitly in your code with random seeds. As long as you agree with your team on which hashing algorithm to use you will always recreate the same splits. No risk of human error.  \n",
    "\n",
    "**Consistent splitting across raw and preprocessed data**\n",
    "\n",
    "During experimentation you might investigate different preprocessing steps and save intermediate and preprocessed data in a new file. You then might load this intermediate data at another stage to continue your analysis.\n",
    "\n",
    "As the preprocessed data is different to the raw data, using `train_test_split` and random seeds will give different splits for the raw and preprocessed data when loaded from a new file.\n",
    "\n",
    "Hashing will provide identical splits for raw and preprocessed data as long as the column used for calculating the hash value has not changed.\n",
    "\n",
    "**Storage and memory efficient**\n",
    "\n",
    "There are other strategies to combat reproducibility (discussed later in this article) such as explicitly saving data in a 'training' file and a 'test' file or adding a new column to your data to indicate which train/test split the data point belongs to.\n",
    "\n",
    "However, sometimes you are not able to save data to new files and add columns  -- for example, if you don't have permissions to copy or edit the original datasource or the data is too large.\n",
    "\n",
    "Hashing is deterministic and the data point split can be calculated 'on the fly' in memory when required without needing to explicitly change the underlying data or save into a new file."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1aad07d3",
   "metadata": {},
   "source": [
    "### Farmhash\n",
    "\n",
    "There are [many different hashing algorithms](https://en.wikipedia.org/wiki/List_of_hash_functions) which are used for multiple use cases such as checksums and cryptography.\n",
    "\n",
    "For the purpose of creating reproducible train/test splitting we need to use a 'fingerprint' hash function. [Fingerprint hash functions](https://devopedia.org/fingerprinting-algorithms) are lightweight, efficient and deterministic -- they will always return the same value for the same input.\n",
    "\n",
    "Cryptographic hash functions, such as MD5 and SHA1, are not suitable for this use case as they are not deterministic and they are also purposefully made to be computationally expensive.\n",
    "\n",
    "[Farmhash is developed by Google](https://github.com/google/farmhash) and [recommended for this use case](https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39). It has a simple [Python library](https://pypi.org/project/pyfarmhash/) implementation and is available across many other languages including [BigQuery SQL](https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39). \n",
    "\n",
    "> Another alternative to Farmhash is to use zlib and the crc32 checksum algorithm. An example of the implementation is shown in this notebook from [Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow](https://github.com/ageron/handson-ml2/blob/master/02_end_to_end_machine_learning_project.ipynb)\n",
    "\n",
    "Below is a demo of farmhash and how we can use it to assign buckets to our data points.\n",
    "\n",
    "\n",
    "[Python package on PyPI](https://pypi.org/project/pyfarmhash/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "6cafbb79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# install Python library\n",
    "# ! pip install pyfarmhash"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ee73e8c",
   "metadata": {},
   "source": [
    "We can hash a data point using its unique identifier (i.e. an ID or concatenation of column values).\n",
    "\n",
    "Let's start by converting an individual ID into a hashed integer value using farmhash's `fingerprint64` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "239e2402",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "6241004678967340495\n"
     ]
    }
   ],
   "source": [
    "import farmhash\n",
    "\n",
    "example_id = \"0001\"\n",
    "hashed_value = farmhash.fingerprint64(example_id)\n",
    "print(hashed_value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c472c4",
   "metadata": {},
   "source": [
    "We can now assign this datapoint to a 'bucket' using an arbitrary function. \n",
    "\n",
    "A useful method for this can be to use the modulo function. The integer outputs from the hashing algorithm are randomly distributed, therefore using the modulo function with a divisor of 10, for example, will split the data into 10 random buckets (from 0 to 9)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f1aba99b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "5\n"
     ]
    }
   ],
   "source": [
    "# assign a bucket using the modulo operation\n",
    "bucket = hashed_value % 10\n",
    "print(bucket)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3f7596",
   "metadata": {},
   "source": [
    "Therefore, our datapoint ID of \"0001\" would be assigned to bucket 5. When we use a divisor of 10 we will have 10 different buckets. Therefore, for example, we could assign all data points with a bucket of '1' to the test set to use 10% of the data for testing."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "819f6cba",
   "metadata": {},
   "source": [
    "### Splitting the dataset using Farmhash\n",
    "\n",
    "Now let's apply this splitting strategy to our dataset.\n",
    "\n",
    "The `hash_train_test_split` function below can be used to split a dataframe into training and test sets using a specified hash function. In this example, the function creates a new column to store the bucket assignments. This is just for demonstration purposes, there is no need to actually store the bucket values in your data as the bucket can be calculated reproducibly 'on the fly' from the row ID.\n",
    "\n",
    "I have broken each step of the process into individual functions. This is so we decouple the hashing function and test set checking functions from the main function. This allows us to swap different hash functions or test set checking functions as required in the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "61642fd1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from typing import Any, Callable\n",
    "\n",
    "TEST_RATIO = 0.1\n",
    "BUCKETS = 10\n",
    "\n",
    "# define some custom types (optional)\n",
    "HashFunc = Callable[[Any], int]\n",
    "TestSetCheckFunc = Callable[[int], bool]\n",
    "\n",
    "\n",
    "def generate_farmhash_fingerprint(value: Any) -> int:\n",
    "    \"\"\"Convert a value into a hashed value using farmhash\"\"\"\n",
    "    return farmhash.fingerprint64(str(value))\n",
    "\n",
    "\n",
    "def convert_hash_to_bucket(hashed_value: int, total_buckets: int) -> int:\n",
    "    \"\"\"Assign a bucket using modulo operator\"\"\"\n",
    "    return hashed_value % total_buckets\n",
    "\n",
    "\n",
    "def test_set_check(bucket: int) -> bool:\n",
    "    \"\"\"Check if the bucket should be included in the test set\n",
    "\n",
    "    This is an arbirtary function, you could change this for your own\n",
    "    requirements\n",
    "\n",
    "    In this case, the datapoint is assigned to the test set if the bucket\n",
    "    number is less than the test ratio x total buckets.\n",
    "    \"\"\"\n",
    "    return bucket < TEST_RATIO * BUCKETS\n",
    "\n",
    "\n",
    "def assign_hash_bucket(value: Any, hash_func: Callable) -> int:\n",
    "    \"\"\"Assign a bucket to an input value using hashing algorithm\"\"\"\n",
    "    hashed_value = hash_func(value)\n",
    "    bucket = convert_hash_to_bucket(hashed_value, total_buckets=BUCKETS)\n",
    "    return bucket\n",
    "\n",
    "\n",
    "def hash_train_test_split(\n",
    "    df: pd.DataFrame,\n",
    "    split_col: str,\n",
    "    approx_test_ratio: float,\n",
    "    hash_func: HashFunc,\n",
    "    test_set_check_func: TestSetCheckFunc,\n",
    ") -> tuple[pd.DataFrame, pd.DataFrame]:\n",
    "    \"\"\"Split the data into a training and test set based of a specific column\n",
    "\n",
    "    This function adds an additional column to the dataframe. This is for\n",
    "    demonstration purposes and is not required. The test set check could all\n",
    "    be completed in memory by adapting the test_set_check_func\n",
    "\n",
    "    Args:\n",
    "        df (pd.DataFrame): original dataset\n",
    "        split_col: name of the column to use for hashing which uniquely\n",
    "            identifies a datapoint\n",
    "        approx_test_ratio: float between 0-1. This is an approximate ratio as\n",
    "            the hashing algo will not necessarily provide a uniform bucket\n",
    "            distribution for small datasets\n",
    "        hash_func: hash function to use to encode the data\n",
    "        test_set_check_func: function used to check if the bucket should be\n",
    "            included in the test set\n",
    "    Returns:\n",
    "        tuple: Two dataframes, the first is the training set and the second\n",
    "            is the test set\n",
    "    \"\"\"\n",
    "\n",
    "    # assign bucket\n",
    "    df[\"bucket\"] = df[split_col].apply(assign_hash_bucket, hash_func=hash_func)\n",
    "\n",
    "    # generate 'mask' of boolean values which define the train/test split\n",
    "    in_test_set = df[\"bucket\"].apply(test_set_check_func)\n",
    "    return df[~in_test_set], df[in_test_set]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a1c4639",
   "metadata": {},
   "source": [
    "As before, we will create a train/test split using the original breast cancer dataset, but use the hashing method with Farmhash instead of sklearn's `train_test_split` with random seeds. \n",
    "\n",
    "Then we will shuffle the data, split the data again and compare the test set IDs to ensure the splits are the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e3b3e6dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a training and test set from original dataset using hashing method\n",
    "x_train_hash, x_test_hash = hash_train_test_split(\n",
    "    df,\n",
    "    split_col=\"index\",\n",
    "    approx_test_ratio=TEST_RATIO,\n",
    "    hash_func=generate_farmhash_fingerprint,\n",
    "    test_set_check_func=test_set_check,\n",
    ")\n",
    "\n",
    "\n",
    "# create a training and test set from shuffled dataset using hashing method\n",
    "x_train_hash_shuffled, x_test_hash_shuffled = hash_train_test_split(\n",
    "    df_shuffled,\n",
    "    split_col=\"index\",\n",
    "    approx_test_ratio=TEST_RATIO,\n",
    "    hash_func=generate_farmhash_fingerprint,\n",
    "    test_set_check_func=test_set_check,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "c9cdd52c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>bucket</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index  bucket\n",
       "0      0       5\n",
       "1      1       9\n",
       "2      2       9\n",
       "3      3       1\n",
       "4      4       3"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# show which bucket each row has been assigned for demo purposes\n",
    "x_train_hash[[\"index\", \"bucket\"]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "17bf78e6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# compare the row ids included in each test set\n",
    "set(x_test_hash[\"index\"]) == set(x_test_hash_shuffled[\"index\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60c35ebc",
   "metadata": {},
   "source": [
    "Problem solved! Even though the underlying dataframe is shuffled, the same row ids appear in the test dataset regardless."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4e1eb2c",
   "metadata": {},
   "source": [
    "## Considerations\n",
    "\n",
    "The hashing method is a little (although not much) more complicated than the common Sklearn `train_test_split`. As such, there are some additional important things to think about and be aware of when implementing this approach.\n",
    "\n",
    "**Hashing will not split your data exactly according to your specified train/test ratio**\n",
    "\n",
    "The output integers from the hashing algorithm are consistent but still random. By statistical chance, you may have slightly more outputs assigned to a particular bucket which means you might not get *exactly* 10% of your data being assigned to the test set. It might be slightly more or less. This is why I named the argument in the `hash_train_test_split` function 'approx_test_ratio' as the result will only be approximately that ratio.\n",
    "\n",
    "In our example above, we specified a ratio of 0.1 and would expect a test set size of 56. However, we actually ended up with only 46 records in the test set (8%)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "77b33e27",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(56, 46, 0.08)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_records = len(df)\n",
    "target_records = int(total_records * TEST_RATIO)\n",
    "hash_test_set_records = len(x_test_hash)\n",
    "actual_ratio = round(hash_test_set_records / total_records, 2)\n",
    "\n",
    "target_records, hash_test_set_records, actual_ratio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d954b77c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3UAAAFGCAYAAADNf7YKAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/MnkTPAAAACXBIWXMAAAsTAAALEwEAmpwYAABPPElEQVR4nO3debwcVZn/8c+XhF0kLBHZA8KIgIoaWQQxrAKiML9BBJFFEWYUGZcRBXUG3HFwXBBEETBBgbALIgJhCYisYd8lQoCEQAJhCTuB5/fHOZ1b6fR+b9++lft9v1731bdPbU8tXaeeqlNVigjMzMzMzMysnBbrdQBmZmZmZmbWOSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MSc1JnZmZmZmZWYqVM6iRNkxSFvzclzZX0mKTLJf1Y0nuajCMkDYn3OUg6KsdzVFX5Abl8fG8iW5ik8TmmA3ody0CTtIKkEyQ9Kun1PJ9/6nVc7ZI0Jsc+rdexdEO938tADzOYhuJvvb8kTc7zNK7XsZgNR/2pCyTtJunvkp4vHGttMvBRthzPkN6HD3VefsNDKZO6gkuBCcCpwF+BqcBY4HDgDkkXSnp7tyYuaVz+kUzu1jQG06I2Px34HfAfwDzgHNK2dWVPIzIbQE60zKwZSe8j1YGbAteT6sIJwJxexmVDR+Hiyphex9KuRbkeHNnrAPrp6IiYXCyQtBjwceBn+fNqSR+KiKerhn3X4ITYkuOAicBTvQ6kBUcARwMzex3IQJK0OLAb8AqwSUQ83+OQ+mMGaft+vdeBmJlZ6exOOj78UUR8u8exmFmLyp7ULSQi3gQukHQNcBPwL8D/AQdU9Xf/4EdXW0Q8RTkSOiJiJotYQpetSvo9zCh5QkdEvA4Mme3bzMxKZc38+WBPozCztpS9+WVdEfEM8JX89TPVzTDr3VMnaTVJx0maKukVSS/le6wukXRwob/JwFX560eq7vGbXOyvcplX0taS/iLpqXwf4O65n6ZtnSWtnO/3mp7j+qekH0hapka/DcdX6/6dNuan7j11SvbN8/xMIc7jJa1Z3X8eZv56kPQpSddLekHpHskrJG1Vb5k0ImltSb+W9JCkV3M8V0n6dK0YgEfy17Wr5n1Mi9M6Io//sTy9OfWmVxhux7w9zFK6h2+OpPslnSLp/VX9jpL0I0n35G3ylbwtTJZ0RFW/De+jkLSJpAvy9F6UdIukz1WWRZ3fRcfrSdKykr4h6Wal+zNezvNxlKS31BlmcUlfl3RvntcnJP1B0tr1ptMqSetI+qOkJ/O475H0X5IWOsnVaHvP3Zv91jaTdJqkR/J28ZSkKZK+K2mlFuPdSGkfFJK+XdXtXZJOlvRwnpdnlO4r/kRVf+Py+vtILrqqajsfV+i35e2yVZK2yXE9k7eba6tjrOpfkvaSdFleZq/mZfC7Rr9JSWtK+lnebl7M29t9SvuCjWv0v5GkU9X3u31K0sWSdq4z/vnbQx72XEmzC/O0TaHfXSVdLem5HMeFktZvEvsvJT2QfyPPK93TdIAk1ei/5X1CIyo0u5e0lKTvK9V/LyvtP78jaUQhxpMlzcjTu0vSZxqMe2VJP8nbT2WebpD0RdX+vc2vmyStJOnYvG2/psL9zZ1uH02WQ1vLvzB/x+Xl/mpeXj+WtIzqNPFSk2ZrDYbrqJ5pcxkcpbSf+Gwu+r369hHjq/r9UN7+n8jr5wlJ50javNl8qc6xUIcxL6503HFGXndz8+/h3rztrVhnuPnrQdLueTk+o6p7B3O8l+ft4Xml3/lual7PrqR0jHaX0v7hRUm3SvqqUsug6v6L+5b3SDo7L9M3JH2ljWXRVr0paTlJB0v6k9Lv/qUc722Svi1p6ar+D8jbSGWcD6vGMVM/1ktLx+BVw5SiHhwUEVG6P2AaEMC4Jv0JeDr3u3dVt0izv0DZqqSrUJGn8SfgTOBa4Fng/kK/hwOX5H6fAMYX/g4v9Dc593M88AZwF3AGMAn4WO7nqNzPUVXxHJDLLwD+mefl3Pz9+dztemCZquFqjq/GeMd3MD/jcz8H1FjWp+VurwGXkZqUPpTLngY+WCOWyH/fy8vn6rzM78vlrwJbtLl9bA48k4d/KMdxWY4rSPdgqmqezsndXqia95VbmN538rBT83qdmLeZebn82Abr4A3gurxN/Bm4HXizapkvA9yT+38SuDD3PxmYBbxSNe4xud9pNaa7LfBy7n5fHs9VOdZjqPG76M96AtYoxD6LdB/sBcDjuewOYIWqYRbLyyJyrBfnaT1OuqI9gQbbd511dFQeZkLeFmfkcf61sDzOBxarGm48Nbb3Vn5rpKbKb+bud+ft4i+ks98L7L+o8ZvM5duQ9j2vAftWddsrL/fK+M/J66VS9r1CvxvkeXkid7uEBbfzDdrdLltY5pPzuH6Zx3cncDrw98L29LUawy1O2s8F8BJwDXB2YVubA4ytMdyOwHO5nxnAeXmZ3JqnX71//QSpuXVl+Z2ep/VGLvt+jWlUtofjgBdJ+/OJwC307fs+DByax3MNcBZ9ddZMYKUa462s58jbx/mkfcncXHZqVf9t7ROarKdxeTzXAX/Ly/c80m/jxdztBOAdefup7FP/VliP+9QY73rAo4X5Pgu4qDDOScCSdfaLF+XpPE2qh88GftOf7aPJMmhr+edh3k6qlyv7trNJv5UXSPXyddQ4TilsC2Oa/G6qh+uknhlDnbqgzrR3J23jU/Nw19K3j/h8ob8v0Pc7uYn027mJvn3HQQ3mq+6xUIv78Orf8RqF9X4dffv12bn8n9Soxwvr4Vf588Y8H38D3pP7+UxhPm/J3a/P3yv15ULLFng3aR8UwGOk7fli+o5HLweWqLNv+R1pv/RP+uqMg1tYPh3Vm8BW9O1HrsnTnETfMeaNwFJV/Y8nbedB2seOp+qYqZP1QhvH4IVhhnQ9ONh/PQ+go6BbTOpyv5Nyvz+oKg8WTur+J5f/hsJBf+62JLB1Vdm43P/kBtOfXJlWvR8mzZO6ys51VKHbKqSDpAD+t5Xx1Rjv+A7mZzy1k7ov0pcQblQoHwEcW/iRVlfilfl7GvhAoXwx4MTcbVIb28ZS9B1I/BwYUei2MWnHFcC/Vw03phJjB9vjB4vzXChfvxDLZlXdKsnuh2oMtwawYeH7fvQd6Iys6ncEsG0r80I6EKwkU99lwcT2Q/QdwESNmNpeT6REv3Jg8ytg6UK3pYE/1NkOD83l04H1qtbtOYVYam7fTX5jlUqoWEmtn6cVwBdb2d5b+O3+ay6fC3y8zjazRqPfJLAPqWJ6Dtiuavj35G5zgZ2rum1U2O62qeo2mQb7zna2yxaWeWVaAXy9qtvHSfd8ziMfQBW6HZ2Hubq4jHK3L9F3YDuyUL4WfQch32Hh38laVdvt2+lLAL9W1e84+hKPj9bZHmoN95Nc/kAe94ertt1rcvf/rhpuVdKBzzxgfxb8Xa4J3Fa9DdLmPqHJehpXmKe/AcsXur2XlKi+AdwL/IIF96mHVNZHjfFWDvLPYsHf25p5GQXw46phDijEcimwXI3xtr19NJn/tpd/7lZJLCcV4wRWL8zfQr81Ok/qOqlnxtBBvUaD/V7eJl7P28Qnq7rtlctfAzZusD9omqTUmO5R1N7XLkfanyxeVb40cEoe5oQa46ush9epkVTm9VhJXD5X1e1f6Uump9WYbmU/ejgL7qdWpO+YtHo+Kss8gB9QdYKxheXTUb1J2q9vWz09YBQpCQvgmw2WX73tuO31QvvH4EO+Hhzsv54H0FHQ7SV1Z9TZeIKFk7rjc/nuLcYxjtaTussa9HNUnR/bAbn8TeDdNYbbJnd/ngUrzZrjqzHe8R3Mz3hqV3CVM5a1ztAtQWreGFSd0aVvR/OlGsOtkru9Ur1jaBDfvnmYh2sNQ3q6ZQAPVpWPocOkrkk8B+XxHlNV/iLwTIvjOCyP4yst9l9zXug7EHyAGhUGfQelUaNb2+sJ2DmXX19nesuSkuzXKVytK2xLB9QY5m2ks/N1t+86y6Tym3gReFuN7p+ts13U3N5rjLf6t3t7Lv+PFuNb4DdJ31W+6dT+7Z+Z+/9CnfHtkbufW1U+mcaVWcvbZQvzVJnWzXW6V84c/65QtmJev3Nrrafcz0V5uI8Xyn6Ryya2GNt/5/6vrdO9kjhUn6iobA/X1RhmhcLv5Ec1ulcS/Suryiu/u5/UiWVs7n5LoaytfUKTZTEuj+sN4F01uv+Jvn1q9dWFkfRdfVirUP5h+uqmFWuMc6dC92LdVfkdvEaNg8VOt48m89/J8l8r/z7nAe+oMczHC9vCuKpu0+ggqWsyD/XqmTEMfFJ3cu52Wp1hJ1L1u66ar7rHQk1iOor29/vLkOqXWTW6VdbDiXWGrSQYl9fpXtkHT6sq/0IuP7POcKvl7Xs2C7cYCtIV5xHN5q3GeLtRb65PnX14s+24k/VC+8fgQ74eHOy/RfaeuoLKPL7ZQr835c+f5DbWyw5gHOf1Y9g7I+Ku6sKIuIp0iX854AP9GH+/SFoDWJe0jP9Q3T0iXiM1zYR0AFHLRTWGe5LUjHJJoKX7j+hrK316pAeGVBtP+jGvJ2n1FsfZlNK9KLvlNvS/ze3jx5N2KpAe2FN0EzBK6X6e90m179nIbs6f35T0GUmjOgyzsmzOjPRAoWqntzCOdtbTLvnz3FrTi4gXgSmkA8MPwkLb0kLxRMQsUlPaTk3K46h2ep5mv7cLpft330uqtCa0OfgISb8BfkRqSrJ59W9f6Qm/lYPic+qM5+r8uUWb029nu2zVaXXKK/uKcYWybUhncq+us56g9rztlD9PajGmym+h3vo5JX9upXw/WZVLqgsi3cf9dL3u9D10YrWq8srv5Ow6sdxCumKwiaSlctlA7ROKHomI+2qUT82fV+V9+XwRMY+U7MGC81VZvn+OiIUegx8Rl5CaWdWru26NiGk1yjvdPhrpZPlvTWqJcENE/LN6gIj4M6m52IDqoJ7phsq6HV+ne+W3M65O9/4cC9WV91dfz/dj/T4vl1+TEqjRklZoM575xxJ1utcrb7g9RcTjpH3ByqSkqdoFEfFGnXHX1N96U8lWkr6ldP9xZfl9J/fS8XbV5npp+Ri8hPXgoFjknn5Zw8r5s5X3q/yBdF/Gp0lt6t+QdDe5nXFEXNePOB5p3ktdDzfoNo3UTGCNfoy/vyoHwTMj4pU6/TxU1W+1R+uUP086A75Une71Yqm5zCLiFUmP5/5WJyXF/SJpC1ITo0br4K1V379ISpD2zX/PSbqJ1Nb+1Ih4ohDzZEn/C3yd3GRR0v2kJrnnRsSlLYZaWTb1tsVWttF21tO6+fMYScc0Ge/o/FlZho9XH0AWTGsWZAP1totXJc2k77fUn+1i7fz5aES83Oawe5H2yzNJzfeeq9HPSvRtT7Oa1DejG3WsoeXtsg319l/T8mfxd1PZZj6mGg/sqVKct8oyb/Wprw33Ezm2N0nb80qke6aKptcZ7oXcf63uL+TP6n1ZZZ5vbuHYYSXSE3oHap9Q1GieWulenK9myxdSnbAqteuEevuiTrePRtpe/vRts43m7xFSE7YB0WE90w3N1m2zur4/x0ILUXrY1mmke2QbeSvp5GOr8XRaX1a2p7Nb2J5GA/9ocbyNdFxvSlqFlNh+qMH4296uOlwv7RyDl60eHBSLdFKXs+v35a8LXemqlq8m7CPpx8CuwJb571DgUEmnRMSBHYbT7sFdN3XrCm2zSrb+gLWvHPVHx7G0Q+npo+eTmiCeTHqgwFRgbkS8KWlH0r0hC+xxIuI+SRsAHyW1Z9+SdBZ6B+BISf+Wz2ZX+v9mvoKzG+lG5S1JTW4OknQZ6Z6AeS2GXW/ZNF0Hba6nyhWOq2meiA1oRT+Iav2W+rPt/Q1Yh9Rs6keSvhS5PUhBZbm+AfyxH9NaSLvbZRdU5u0B4IYm/d5Y+L/TZd7pcM1+B538Ts4kNWFu5NXKPwO8T4CBnaf5YXYwDNSvLzvdPhrpaPl30UL7lE7rmS4b6HXbqR+TEod7SfewTQGeqrTUySdxV6X+smkWT7v1ZWV7+gvNX1VV/f7kVuIZaCeRErq/k5q43gE8GxGvS1qCzrf5ttdLm8fgi3I92LFFOqkDPka6evA6qQ1tSyLiblLTp8ol3l1Il7Q/J+nMiOhP869OjGmhW/HKQuVMTc3HxdN3VnugVKa9mqQlI6LWTmDdqn67pTL+dWt1zM1nVqvqtz+2JlW0t0TE52t0X6/egHnndlH+IzdDOBL4MqniXr2q/4dJ9w79Ive/Feme0R2Bz5EeWNLI4/mz3vof02T4dj2WP8+OiONbHKa4LS1R56zjmH7EVHPYXHmtWhUDdPZbqlzNXFPS0m1erXuUdO/jFaSzhUtL+nxVMv0UqeJfmnSP4wsLj6Zz7W6XLRjTpLy4vCvbzF0RcUAb03gUeGf+q3dFqWgG6Ulo65KWda3YFiMd5LfSyqM/HiPtJ74fEfe0M+AA7BO6peF+uKpbO/vhTrePZuNsd/lXYh7ToJ96+9lO9ikd1zNdMIP0JNR1SfdxVRusur7ik/nzU/nYbb7cfO/tCw/SksdJ+5N268vH8nAnRMRfOpx2uzqqN/Py2YWUGO0aEc9W9dKf7arj9dLiMXjZ6sFBscjeU5dXwM/z11MbtL9vKCLejIiLSI9hh3SvTEXlh9Pt5Pi9kjaqLpT0Efqe0HRLoVPlB75BjWFE3/0n1Tqan4iYTmpysRjpEcDV01yc9CQ/aCO57lClDfXeqvEeJPLTzUhPaxuISqfyrpXH6nRv+f1B+Z6cw0hnAFeT1LDJQERUHjcNC26X9VyTP/fMO8pqe7cYaqv+mj8/2bCvgoh4jNSsZzFSU8QF5GWyQz9i2lHSyjXK987T/Gfenisa/ZaWpsZ9I7lpxp2kBwTt126Aefpbkyq1zwKnFbflfPXl8vx1j4XH0FDbv/F2t8sa9mlSPrlQdjnpJNz2bd4nVmluWOuAt5bKfqLe+qm8p+vaNq92daLt30k9HewTuqWyfD9e614mSR8lnUSprrua6XT7aKST5V95ncMWkhZKXCV9jPpNLxvtUzam78XfRQNWzwyAVn87k7sfCtB42Xyazq9eVurLevVivfIB+z23qh/15vJ5mLk1Ejqov++G5nXJgKyXesfgJawHB8Uil9RJWkzphYM3k84y3E9aEa0Mu59qvFRQ6SXBlRsti83EKjvn9eokEANFwAmSli/ENJr0/idIT28qXg24irTh7SRpy8IwI4AfApvWmU5/5udn+fP7+bJ1cZr/S3pa2CPUv6F1oJxN2omsA/y4mLxI2pD0KH+Anw7Q9Cr38GxbNd+LSfof0mX7BSi9mPZrdXYKHyP9Lp8n32gv6V+VXta6wO81JxXb56+tNF88m/S0yQ2Abxdv/pW0Genx5APpT6QDto9I+o1qvGxU0tslHVRVfGz+/EHxgEnSkqSnYy3Tj5iWAY7P46qM9x3A9/PXX1b1X7mKs6+kdxaGWZrUBGqtOtOpbGfHSNqluqOksUo3t9eUHz4zjrT89iLdn7FEoZfvkQ5uf6n0EuYFKkglm+ZmWUWV3/i7asTU1nbZhg9K+mrVtHah7x1Qx1XK83wfTzogvrD4myoMu6ykTyvdC1LxM1KCsJfSC5pHVA2zpqTiAzl+R3qC4laS/rOq361JzX0A/q+tOe3MMaTl+i1Jh9Ta9yq96Pz/Fb4P1D6hKyLib6Q6eDkW/r2tTr6yCBzX4D7sWuPtdPtopO3lnx/iciGpCdgJKjzQQdJqNK5fKvuUb0iaf6+SpDVJCXmtg92265kuOpb01M+9Jf1rsYOkTwJ7kvZNx9YYthsqy+aLVbGMJTUB7NTJpCtBO0jav2rcn6B+0nYi6Rhkf6WXuS9UX0laR9JCJ8D7qZN6s/KQs1GqeoG9pJ2ArzWYXt26JGt7vXRwDF6menBwNHo05lD9o+9RqsUXB04k7Syfoe9RwudT/7HHkWZ/gbI/5fLppEuufySddam8q+QaFn7nxq25272kmzxPAg4rdJ9Mk8cT0/rLx58iJUV/ou8dSzcBy9YY53H0PRr6irwsHiFtjL/M3cbXGK7Z/IynxmNzSRXR6bnbq6Qz52fQ95jdOTR4+XgL63pMG9tH8eXjU3Mcl1Ln5eN5mDF08OjnPGzlhZ+v5O1lYp7v1+l7XPbkQv+jctk80juQzsrD3JzL36TwKHz6HtdeeXn3H/M0K48Sv48F3y1Vd15IZ+sqL1y+N6+zK3MsP6tsMwO1nkg3cFfep/g86Qz36aQbs+/O8/pE1TAjSC9ODVKl+hfS/S4z8jz35+Xjp+ZxTM/jvJi+l49fSO1XL1TW74ukfc5FpPcxPk7f+3YWioW+R2JHXgZn5GHbefn48vS9rPtiFnz8+170PaZ6Wu7+x7yNVN7HeHTV+D5B37Z6Ien3fRKpudAo2tguW1jmk/NwlZeP35HX/bWF5XJYjeEWp+9R1fNI92KclctupG/73aBquJ3pe9fidNJ7xM4hJca1Xj6+W2Fcd+bYJtPay8cPqDPP06izv6Lx73Ib+n7Pj5PeZXUaaduvvGtpYqH/X9DGPqHJehpH1T6qzm+n5u+N+u9VW490cFuZpzPpezl3kM6y13v5+PgG8Xa0fTRZBm0t/zzMaqSrI5X1cDbpN/UC6X6/ei8fX4EFX8p+Hmkf/EL+/Hud4dqqZ5ptc02Wx3gab+dfJO0PIs/raXnZB+n3s9B76OptJ23EVHM7pO+x9UF6lcwZpKuJb+S4plG7bqpZXtXP/oX5nJLHV1mvlfryHzWGezd9r3F6mnSi/bS8fVT2/ze0s8xbWD4d1ZvAfxWW33Wk/WBlXf6w0q3G9P6Tvnr9HPrqkpU6XS90dgw+pOvBwf7reQAdbryVjSEKC3kuqQK5nPQ48I2bjGOhDZX0bp1fkBKlJ0jJyYy8IX2OqgooDzMm/3CeoO9llJML3SfT/6RuPOnpPb/L8bxKau74Q2okdHnYxYBvkM6WvEp6J8pZpEfTzh9vB/Mznjo7HlJit29eXs/m6T5Menztmq2uhzrrekyb28japCspD+c4niXtUPahKqErzHfblV8edgngm6Qk5eW8rC8ENqPGARPpkv9/kHYU95MS9JdIO/vTWPgFspuQ3pt1bWH9P0na8X6Fqhf0NpsX4P05vmfydG8DDiY1+wnSE7QGbD2Rnop3SF7+c0jJ9UxSJXkMtV/wuXhepvfl+Z1FqhTWobP3Fc0fhnTPx8Q8zlfzNA6jzrsQc/w/ztvSa6SDvpNJ9wQ0jIX0AIuz8jCVdxPdRGqbv2KhvwOo/5tclnRiJkgHfcsWur2DlDTdS0o6XyQd6F1KqnRXqzG+L5Aq2UpFGKTttK3tsoVlPrkw7u1z7M/lGP9Ok3cRkd719afCsnua9Bv7PbB7rfWVt4/jcsyv5Ondk8sWemEssDHp5NWMwjT+CuxSJ6bxdCGpy93fTtqn306qz17J45tMetDAOwr9bkIb+4Qmy3kcXUjqcreVSS01HsjzMzfHeEid9XcATZK6/mwfTcbX8vIvDDOaVL9V1sHDeX6XbbJc1iQdeM7Ow/2DtE9Yot5wtFnPtLLNNVgW42mSYJCuDp5H37tGnySdSNmi3e2kxZjqbod5/q/M28ALpDrtUNJx0DQ6TOpyf9uS9r9z8991wL+R9u1BjXdW5uGWJ71v9AbSfuhVUrJyPekK03vaXeYtLKOO6s08P9fnOJ/P87hP7laz7s/L9jt5WpUTKQssz3bXCx0cg+fhhmw9ONh/yjNnZsOcpH1JV7IuioiP9zoeM7OykjSZ9K6zbSJicm+jsYEm6b9JydlxEXFos/7NBsMid0+dmdUn6W2S1q5RvjnpqhnUf6msmZnZsCBprVr3ZuZ7go8gXd2ZMOiBmdWxqL/SwMwW9B5gktILPSvNCdel732Of4iIc3sVnJmZ2RCxI/BbSbeT7pET6b6rygM2fhARU3oUm9lCnNSZDS/3k+41/AjpnoDlSG3oryRdoRvQl3iamZmV1HWkK3FbAduRniA5h/QwjhMiPWrfbMjwPXVmZmZmZmYl5nvqzMzMzMzMSsxJnZmZmZmZWYk5qTMzMzMzMysxJ3VmZmZmZmYl5qTOzMzMzMysxJzUmZmZmZmZlZiTOjMzMzMzsxJzUmdmZmZmZlZiTurMzMzMzMxKzEmdmZmZmZlZiTmpMzMzMzMzKzEndWZmZmZmZiXmpM7MzMzMzKzEnNSZmZmZmZmVmJM6MzMzMzOzEnNSZ2ZmZmZmVmJO6szMzMzMzErMSZ2ZmZmZmVmJOamzRZ6koyT9sddxdELSGZJ2HwJxDMllKOlbkk7qwng/LunMgR6vmVl/SZomaftex9EuSaMl3S9p6UGY1vy6QdIYSSFpZP4+WdLn8//7SLqs2/HkaYWk9QZjWu2Q9IKkdbsw3pskbTTQ47X6nNTZQiR9WtKU/EOfKemvkrbqdVwAkg6QdG0Xxz9e0g+6Nf52piPpPcB7gQvy93GS3szrpfK3f9Uwe0m6T9KLkv4p6cOFbp+XNDUPd4mk1Xo1bwMlIn4UEZ9vpd92EtOI+DOwUV4HZjYMlDVZamSwEokWp3M4MD4iXs7D7CnpOkkvSZo8kPG0WjdExGkRseNATru/qpPQbouIt0TEQ6302+b29FPge51HZu1yUmcLkPQ14BfAj4BVgLWAXwO7dTCuhXZIg7WTWkT8O3BaRESh7PG8A678Tah0kLQD8BPgs8BywNbAQ7nbONI63Q1YEXgYOGMwZqLEzgAO7nUQZlYeSnxsVUXSksD+QPHE2hzS8cbRvYjJuu5CYBtJb+91IMOFdzw2n6TlSWdVDomI8yLixYh4PSL+HBGH5X4WuAqTrx5NL3yfJumbku4EXpS0Xj6zc6CkR4Erc3+fy1eUnpF0qaS1C+MISf8h6UFJz0o6PleU7wJ+A2yRrzY9W2c+1pF0taS5kiYBK1d1P1vSE5Kek3RNpXmApIOBfYBv5PH/OZcfnq96zZV0r6R/LYxrvTyt5yQ9pUKTPUkbSJokaY6kByTt2Wg6NewMXN1onVX5LvC9iLghIt6MiBkRMSN32xU4OyLuiYjXgO8DW0t6xxBYhgdI+ruk4/L47pe0XaH7apIuzMtxqqSDCt3mX30rnN3cX9KjeX18O3fbCfgW8Kkc1x2FaT+U43pY0j6F2ZwMfKyN5W9miyBJK0i6SNLsXGddJGmNQvfJkn4o6e/AS8C6knbM+/3nJP06708/Xximbh1YY/r7SnpE0tOVfVqh26aSrs915cy8H10id7sm93ZH3u99qoV5qbtPrBdzrenUmI3NgGcjYv7xQkRcHhFnAY+3sA7GSZou6RuSZuV53V3SLpL+keuHbxX6b6llhqpa/0j6kKSb83q7WdKHCt0mS/p+rq/mSrpM0sq1xwySDstxPi7pc1XdPibpNknPS3pM0lGFzpXl+WxenltIeoekK/M28JSk0ySNKoxvmqQjlOrXZyT9XtJShe4H5fpzjlJ9ulqh2/yrb0rHeMdL+kuexxuVjxPqbE8r523o2Tzuvymf1IiIV4BbgI82Ww82QCLCf/4jIgB2AuYBIxv0Mx74QeH7OGB64fs04HZgTWBpYAwQwKnAsrlsN2Aq8C5gJPAd4LrCOAK4CBhFulI4G9gpdzsAuLbJfFwP/AxYknS1ai7wx0L3z5GuZC1JOkt4e735y2WfBFYjnQT5FPAisGrudgbw7dxtKWCrXL4s8BjpqtlI4H3AU8CG9aZTNc1l83IYXbWsXwOeJF1p+zmwbO42Inc7PC/b6cBxwNK5+0+BXxfGtXoe/25DYBkeQNruvgosnrs/B6yYu19Dulq8FLBJ3h62zd2OqsRF37b2O9J29l7gVeBd1f0WlvHzwDvz91WBjQrdV8zje2uvf5v+85//uv9Hqr+2r1G+EvBvwDJ5v3c28KdC98nAo8BGeX8/Ou9b/l/+/mXgdeDzuf+GdWDVtDcEXsj74SXzfnleJU7gA8DmeTxjgPuArxSGD2C9Vual0T6xWczV06kxH4cAf6nT7fPA5CbrZlye7//J9cRBuS44Pc/HRsDLwDq5/1p1w8jC+qqsiwPIxxR5n/8MsG+ex73z95UKw/0T+BdSHTMZOLpOvDuR6uqN83I9vbiM8vy8m1Qnvif3u3uteHPZesAOeRsYTaoXf1G17d5NOvZaEfg7uR4GtiUdf7w/D/8r4Jpa645Ufz8NbJqXwWnAxAbb049JJ9sXz38fBlTofizws17/tofLn6/UWdFKwFMRMa+f4zk2Ih6L3G4+OyrSlb+Xgf8AfhwR9+Vp/QjYpOpM5dER8WxEPApcRTqYb0rSWsAHgf+OiFcj4hpggSthEXFKRMyNiFdJO/73Kl2lrCkizo6IxyNd/ToTeJC0w4NUUa8NrBYRr0RE5YzfrsC0iPh9RMyLiNuAc0nJTStG5c+5hbL7ScthVdJO+gOkCh5SU9nFgT1IO9VNSInkd3L3S4A9Jb1H6Sb1/yHtnJepnnAPliHALFIF9Xru/gDwMUlrAlsC38zL93bgJGC/etMCvhsRL0fEHcAdpOSunjeBjSUtHREzI+KeQrfKsh/VYHgzW8RFxNMRcW5EvBQRc4EfAh+p6m18pJYQ80itLO6J1OJlHunA9olCv63UgRV7ABdFxDV5f/vfpP1WJbZbIrXOmBcR04Df1oitnXmpt09sJ+ZaRrFgfdaJ14EfRsTrwERSC5Jf5rroHuBeGu/vm/kY8GBE/CEvzzNI9e7HC/38PiL+kY9lzqL+scmeud+7I+JFUj05X0RMjoi7cp14J+kEcaP1NjUiJuU6eTap7q/u/7h87DWHtF73zuX7AKdExK15GzqC1OJpTJ3JnR8RN+X1fFqDeYS0TlYF1s71998ionjLyFxchw4aJ3VW9DSwsvp/39tjTcrWBn6ZL9c/S2pXL9LVo4piBfgS8JYWp70a8EzeiVY8UvlH0ghJRys1BXyedHYLqpoXFknaT9LthXg3LvT/jRz7TZLuKTSxWBvYrDJMHm4foNW25c/mz+UqBRHxRETcmyuBh/O0/y13riTQv8oV8VOknf4uedjLgSNJieW0/DeXdEWv2mAvQ4AZVRXBIzmO1YA5+eCj2K24rVRradvJ8/cp0sHKzNzcZINCL5Vl/2yDaZnZIk7SMpJ+q9QE8nnSVZJRkkYUeivWcasVv+d9W3Ff20odWG9cL5Lq6kps/5Kbvz2RY/sRjffFdeelyT6xnZhreYZCfdahpyPijfx/pc57stD9ZVo/VqhlNQp1XVZd37R6bLLAeqser6TNJF2l1Az2OdIyb7TeVpE0UdKMvN7+WKP/6ulVmlguMF8R8QJpG6q37to5/jqGdAX3MqVmu4dXdV8O16GDxkmdFV1Paq62e4N+XmTBqzu1kpRoUvYY8O8RMarwt3REXNdCjLXGXTQTWEHSsoWytQr/f5rUjGR7YHlSMwdIldNC489nIX8HfInUBGMUqYmDYH6idVBErEZ6sMmvc9v0x4Crq+bxLRHxhVbmI1eulWYedXsj/4Yj4hnSQUNUdS+O8/iIWD8iViEldyPzvFQb1GWYrS6p+H0t0n0WjwMrSlquqtsM2rfQMo+ISyNiB9KZxvtznBXvIl1tfb6DaZnZouO/gHcCm0XEW0lNIWHBfVhx/zITKN6npuJ32qsDZ5Ka1FXGtQypVU3FCaR91/o5tm9VxdXWvDTYJ/an3ga4k8b12VDwOCl5Leq0vllgvbFgHQqpOeaFwJoRsTypCWPNOjT7US5/d15vn2Hh9Vw9vcq9igvMV67bV6Kz+VpAvkr6XxGxLvAJ4Gsq3BNPqkfv6O90rDVO6my+iHiO1CzveKUbkJeRtLiknSX9b+7tdmAXSSsqPdHoKx1M6jfAEep7uMbyklptlvgksIbyjeA15uERYArwXUlLKL2Kodh0YjlS4vo0KTn9UY3xF9/XUrm3bXaO9bOkq0zk759U303mz+R+3yTdE/gvSje4L57/Pqj0sJda06nlYgrNKyRtI2ltJWuSnhh2QaH/3wOHSnqbpBVI96hdlIddStLGedi1gBNJzVaeqZ7oYC/D7G3Af+bl9ElSRXBxRDwGXAf8OM/De4ADWfAJaq16EhijfBN3PvO5W67gXiXdt/Jmof+PAH/tYDpmVl6L531N5W8kaZ/3MunBFSuSWj008hfg3bkeHUm6n6x4ArSdOvAcYFdJW+V673sseOy2HOk+uBfyVbUvVA1fvT+uOy9N9onNYm5Wp91EuiI4/+qQUquPpUgnGBfLy3vxBuPototJ9fanJY1UeuDLhuR6tE1nAQdI2jAn4tXbzHKkViivSNqUdLK0YjZpuVevtxeA5/IyPKzGNA+RtEZer98GKg9uOwP4rKRNlJ5C+iPgxtxct10LrGdJuyo9ME6ke+HfyLGT1+0HgEkdTMc64KTOFhAR/wd8jXQv1mzS2bkvAX/KvfyBdNZlGnAZfTuNdqZxPunR+xNzM4K7SfcgtOJK4B7gCUlP1enn06Qnbc0h7UhPLXQ7ldQMYQap/f0NVcOeDGyYm5j8KSLuBf6PdBXzSdKNzX8v9P9B4EZJL5DOun05Ih7KzQV3BPYinSV7Is/zkrWmU2c+TgT2KVzBeh8pwXkxf94F/Geh/+8DNwP/IN0sfxupXT2kh4ycTqoUbsrz8991pguDuwwBbgTWJ93M/UNgj4ioNDHam3Q18HHgfODI3Jy0XWfnz6cl3Ura/30tj3cOKYkrHhDtTbo/xcyGj4tJSU/l7yjSw6CWJu2fbiDdo1xXbv7+SeB/SSe/NiSdKHs1d2+5Dsz3ih1C2n/PJJ08LDbl/Dppfz2XdFWtuk4+CpiQ98d7NpmXuvvEFmKunk71fLxGegjHZwrF+5KW8Qmke8FfZsHWEoMq1zm7kq5mPk26xWHXvD7bHddfScv6SlLzxCurevki8D1Jc0kn088qDPsSqR78e16em5Oebv1+UuL0F+C8GpM9nXRc9hCppc8P8vguJ9X355K2oXeQjk06cRQLruf1gctJxxbXkx7IdlXu9+OkB+A0fbqpDQwteBuLmQ0lkk4HzoqIP/U6lm6RdADpSWRD4gX3AJI+DuwbEQsdnJiZtSO3DpgO7FM44B12JI0G/ga8LxZ8kJr1k6RppHq0kxOeXSHpRuDAiKh1m4d1gV8EbTaERcSnm/dlAy0i/kzVEz/NzFol6aOkFggvk5rKiYVbNQwr+amNGzTt0RYJEbFZr2MYbtz80szMzGxgbUFqAvcUqRna7r46ZWbd5OaXZmZmZmZmJeYrdWZmZmZmZiXmpM7MzMzMzKzEnNSZmZmZmZmVmJM6MzMzMzOzEnNSZ2ZmZmZmVmJO6szMzMzMzErMSZ2ZmZmZmVmJOakzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMRG9jqAVqy88soxZsyYXodhZmZddssttzwVEaN7HUdZuH40Mxs+GtWRpUjqxowZw5QpU3odhpmZdZmkR3odQ5m4fjQzGz4a1ZFufmlmZmZmZlZiTurMzMzMzMxKzEmdmZmZmZlZiTmpMzMzMzMzKzEndWZmZmZmZiXmpM7MzKxA0imSZkm6u1C2oqRJkh7Mnyvkckk6VtJUSXdKen9hmP1z/w9K2r9Q/gFJd+VhjpWkRtMwMzNrxkmdmZnZgsYDO1WVHQ5cERHrA1fk7wA7A+vnv4OBEyAlaMCRwGbApsCRhSTtBOCgwnA7NZmGmZlZQ07qzMzMCiLiGmBOVfFuwIT8/wRg90L5qZHcAIyStCrwUWBSRMyJiGeAScBOudtbI+KGiAjg1Kpx1ZqGmZlZQ07qzMzMmlslImbm/58AVsn/rw48Vuhvei5rVD69RnmjaZiZmTU0stcBmBVNPubQQZ3euMN+NajTM7Pyi4iQFL2ahqSDSU09WWuttQZker+7/JIBGU+rDtq+unVrH8dS31CKx7HUNtixQPPtZqjweuruenJSZ2Zm1tyTklaNiJm5CeWsXD4DWLPQ3xq5bAYwrqp8ci5fo0b/jaaxgIg4ETgRYOzYsV1NLs2s3IZSImXd5eaXZmZmzV0IVJ5guT9wQaF8v/wUzM2B53ITykuBHSWtkB+QsiNwae72vKTN81Mv96saV61pmJmZNeQrdWZmZgWSziBdZVtZ0nTSUyyPBs6SdCDwCLBn7v1iYBdgKvAS8FmAiJgj6fvAzbm/70VE5eErXyQ9YXNp4K/5jwbTMDMza6hrSZ2kdwJnForWBf6H9KSvM4ExwDRgz/xkMDMzs56LiL3rdNquRr8BHFJnPKcAp9QonwJsXKP86VrTMDMza6ZrzS8j4oGI2CQiNgE+QDqDeT5+D4+ZmZmZmdmAGax76rYD/hkRj+D38JiZmZmZmQ2YwUrq9gLOyP/7PTxmZmZmZmYDpOtJnaQlgE8AZ1d3y/ci1H0Pj6QpkqbMnj27y1GamZmZmZmV02BcqdsZuDUinszfn8zv36HZe3giYmxEjB09evQghGlmZmZmZlY+g5HU7U1f00vwe3jMzMzMzMwGTFeTOknLAjsA5xWKjwZ2kPQgsH3+bmZmZmZmZh3o6svHI+JFYKWqMr+Hx8zMzMzMbIAM1tMvzczMzMzMrAuc1JmZmZmZmZWYkzozMzMzM7MSc1JnZmZmZmZWYk7qzMzMzMzMSsxJnZmZmZmZWYk5qTMzMzMzMysxJ3VmZmZmZmYl5qTOzMzMzMysxEb2OoDhaPIxhw76NMcd9qtBn6aZmZmZmXWfr9SZmZmZmZmVmJM6MzMzMzOzEnNSZ2ZmZmZmVmJO6szMzMzMzErMSZ2ZmZmZmVmJOakzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MS62pSJ2mUpHMk3S/pPklbSFpR0iRJD+bPFboZg5mZmZmZ2aJsZJfH/0vgkojYQ9ISwDLAt4ArIuJoSYcDhwPf7HIc1sDkYw4d1OmNO+xXgzo9MzMzM7NFWdeu1ElaHtgaOBkgIl6LiGeB3YAJubcJwO7disHMzMzMzGxR183ml+sAs4HfS7pN0kmSlgVWiYiZuZ8ngFW6GIOZmZmZmdkirZtJ3Ujg/cAJEfE+4EVSU8v5IiKAqDWwpIMlTZE0Zfbs2V0M08zMzMzMrLy6mdRNB6ZHxI35+zmkJO9JSasC5M9ZtQaOiBMjYmxEjB09enQXwzQzM2uNpK9KukfS3ZLOkLSUpHUk3ShpqqQz8z3kSFoyf5+au48pjOeIXP6ApI8WynfKZVPzfedmZmZNde1BKRHxhKTHJL0zIh4AtgPuzX/7A0fnzwu6FYPZosQPtBn6BnsdgdfTYJK0OvCfwIYR8bKks4C9gF2An0fEREm/AQ4ETsifz0TEepL2An4CfErShnm4jYDVgMsl/UuezPHADqQTozdLujAi7h3E2TQzsxLq9nvqDgVOk3QnsAnwI1Iyt4OkB4Ht83czM7MyGAksLWkk6YnOM4FtSa1RYMEHgBUfDHYOsJ0k5fKJEfFqRDwMTAU2zX9TI+KhiHgNmJj7NTMza6irrzSIiNuBsTU6bdfN6ZqZmQ20iJgh6afAo8DLwGXALcCzETEv9zYdWD3/vzrwWB52nqTngJVy+Q2FUReHeayqfLPqOCQdDBwMsNZaa/V/xszMrPS6faXOzMxskSBpBdKVs3VIzSaXBXYa7Dh8z7mZmVVzUmdmZtaa7YGHI2J2RLwOnAdsCYzKzTEB1gBm5P9nAGsC5O7LA08Xy6uGqVduZmbWkJM6MzOz1jwKbC5pmXxvXOUBYFcBe+R+ig8AuzB/J3e/Mr/K50Jgr/x0zHWA9YGbgJuB9fPTNJcgPUzlwkGYLzMzK7mu3lNnZosmP4nThqOIuFHSOcCtwDzgNuBE4C/AREk/yGUn50FOBv4gaSowh5SkERH35Cdn3pvHc0hEvAEg6UvApcAI4JSIuGew5s/MzMrLSZ2ZmVmLIuJI4Miq4odIT66s7vcV4JN1xvND4Ic1yi8GLu5/pGZmNpy4+aWZmZmZmVmJOakzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MSc1JnZmZmZmZWYk7qzMzMzMzMSsxJnZmZmZmZWYk5qTMzMzMzMysxJ3VmZmZmZmYl5qTOzMzMzMysxEb2OgCzoWryMYcO+jTHHfarQZ+mmZmZmZWbr9SZmZmZmZmVWFev1EmaBswF3gDmRcRYSSsCZwJjgGnAnhHxTDfjMDMzMzMzW1QNxpW6bSJik4gYm78fDlwREesDV+TvZmZmZmZm1oFeNL/cDZiQ/58A7N6DGMzMzMzMzBYJ3U7qArhM0i2SDs5lq0TEzPz/E8AqXY7BzMzMzMxskdXtp19uFREzJL0NmCTp/mLHiAhJUWvAnAQeDLDWWmt1OUwzMzMzM7Ny6uqVuoiYkT9nAecDmwJPSloVIH/OqjPsiRExNiLGjh49upthmpmZmZmZlVbXkjpJy0parvI/sCNwN3AhsH/ubX/ggm7FYGZmZmZmtqjrZvPLVYDzJVWmc3pEXCLpZuAsSQcCjwB7djEGM1vEDfZL4v2CeDMzMxtqupbURcRDwHtrlD8NbNet6ZqZmZmZmQ0nvXilgZmZmZmZmQ2Qbj/90szMesRNU83MzIYHX6kzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZXYsHlQih8YYGZmZmZmiyJfqTMzMzMzMysxJ3VmZmZmZmYl5qTOzMysRZJGSTpH0v2S7pO0haQVJU2S9GD+XCH3K0nHSpoq6U5J7y+MZ//c/4OS9i+Uf0DSXXmYYyWpF/NpZmbl4qTOzMysdb8ELomIDYD3AvcBhwNXRMT6wBX5O8DOwPr572DgBABJKwJHApsBmwJHVhLB3M9BheF2GoR5MjOzknNSZ2Zm1gJJywNbAycDRMRrEfEssBswIfc2Adg9/78bcGokNwCjJK0KfBSYFBFzIuIZYBKwU+721oi4ISICOLUwLjMzs7qc1JmZmbVmHWA28HtJt0k6SdKywCoRMTP38wSwSv5/deCxwvDTc1mj8uk1ys3MzBpyUmdmZtaakcD7gRMi4n3Ai/Q1tQQgX2GLbgYh6WBJUyRNmT17djcnZWZmJeGkzszMrDXTgekRcWP+fg4pyXsyN50kf87K3WcAaxaGXyOXNSpfo0b5AiLixIgYGxFjR48e3e+ZMjOz8mspqZO0ZStlZmZmQ8VA110R8QTwmKR35qLtgHuBC4HKEyz3By7I/18I7Jefgrk58FxupnkpsKOkFfIDUnYELs3dnpe0eX7q5X6FcZmZmdU1ssX+fkU6G9mszMzMbKjoRt11KHCapCWAh4DPkk6QniXpQOARYM/c78XALsBU4KXcLxExR9L3gZtzf9+LiDn5/y8C44Glgb/mPzMzs4YaJnWStgA+BIyW9LVCp7cCI7oZmJmZWSe6WXdFxO3A2BqdtqvRbwCH1BnPKcApNcqnABv3J0YzMxt+ml2pWwJ4S+5vuUL588Ae3QrKzMysH1x3mZnZsNIwqYuIq4GrJY2PiEcGKSYzM7OOue4yM7PhptV76paUdCIwpjhMRGzbbEBJI4ApwIyI2FXSOsBEYCXgFmDfiHit3cDNzMya6LjuMjMzK5NWk7qzgd8AJwFvtDmNLwP3ke5lAPgJ8POImCjpN8CBwAltjtPMzKyZ/tRdZmZmpdFqUjcvItpOvCStAXwM+CHwtfyI5m2BT+deJgBH4aTOzMwGXkd1l5mZWdm0+vLxP0v6oqRVJa1Y+WthuF8A3wDezN9XAp6NiHn5+3Rg9VoDSjpY0hRJU2bPnt1imGZmZvN1WneZmZmVSqtX6iovVT2sUBbAuvUGkLQrMCsibpE0rt3AIuJE4ESAsWPHRrvDm5nZsNd23WVmZlZGLSV1EbFOB+PeEviEpF2ApUj31P0SGCVpZL5atwYwo4Nxm5mZNdRh3WVmZlY6LSV1kvarVR4Rp9YbJiKOAI7Iw48Dvh4R+0g6m/SeoImks6gXtBeymZlZc53UXWZmZmXUavPLDxb+XwrYDrgV6KRi/CYwUdIPgNuAkzsYh5mZWTMDWXeZmZkNWa02vzy0+F3SKNKVtpZExGRgcv7/IWDTVoc1MzPrRH/rLjMzs7Jo9emX1V4EfK+CmZmViesuMzNbJLV6T92fSU8MAxgBvAs4q1tBmZmZ9ZfrLjMzGy5avafup4X/5wGPRMT0LsRjZmY2UFx3mZnZsNBS88uIuBq4H1gOWAF4rZtBmZmZ9ZfrLjMzGy5aSuok7QncBHwS2BO4UdIe3QzMzMysP1x3mZnZcNFq88tvAx+MiFkAkkYDlwPndCswMzOzfnLdZWZmw0KrT79crFIpZk+3MayZmVkvuO4yM7NhodUrdZdIuhQ4I3//FHBxd0IyMzMbEK67zMxsWGiY1ElaD1glIg6T9P+ArXKn64HTuh2cmZlZu1x3mZnZcNPsSt0vgCMAIuI84DwASe/O3T7exdjMzMw68Qtcd5mZ2TDS7N6CVSLirurCXDamKxGZmZn1j+suMzMbVpoldaMadFt6AOMwMzMbKKMadHPdZWZmi5xmSd0USQdVF0r6PHBLd0IyMzPrF9ddZmY2rDS7p+4rwPmS9qGvIhwLLAH8axfjMjMz69RXcN1lZmbDSMOkLiKeBD4kaRtg41z8l4i4suuRmZmZdcB1l5mZDTctvacuIq4CrupyLGZmZgPGdZeZmQ0Xze6pMzMzMzMzsyHMSZ2ZmZmZmVmJOakzMzMzMzMrsa4ldZKWknSTpDsk3SPpu7l8HUk3Spoq6UxJS3QrBjMzMzMzs0VdN6/UvQpsGxHvBTYBdpK0OfAT4OcRsR7wDHBgF2MwMzMzMzNbpHUtqYvkhfx18fwXwLbAObl8ArB7t2IwMzMbaJJGSLpN0kX5e80WKJKWzN+n5u5jCuM4Ipc/IOmjhfKdctlUSYcP+syZmVkpdfWeulzx3Q7MAiYB/wSejYh5uZfpwOrdjMHMzGyAfRm4r/C9XguUA4FncvnPc39I2hDYC9gI2An4da4vRwDHAzsDGwJ7537NzMwa6mpSFxFvRMQmwBrApsAGrQ4r6WBJUyRNmT17drdCNDMza5mkNYCPASfl76J+C5Td8ndy9+1y/7sBEyPi1Yh4GJhKqiM3BaZGxEMR8RowMfdrZmbW0KA8/TIiniW9AHYLYJSkykvP1wBm1BnmxIgYGxFjR48ePRhhmpmZNfML4BvAm/n7StRvgbI68BhA7v5c7n9+edUw9crNzMwa6ubTL0dLGpX/XxrYgdRc5Spgj9zb/sAF3YrBzMxsoEjaFZgVEbf0OA63ZDEzswWMbN5Lx1YFJuR7BBYDzoqIiyTdC0yU9APgNuDkLsZgZmY2ULYEPiFpF2Ap4K3AL8ktUPLVuGILlBnAmsD03EJleeDpQnlFcZh65fNFxInAiQBjx46NgZk1MzMrs64ldRFxJ/C+GuUPke4bMDMzK42IOAI4AkDSOODrEbGPpLNJLVAmsmALlAvz9+tz9ysjIiRdCJwu6WfAasD6wE2AgPUlrUNK5vYCPj04c2dmZmXWzSt1ZmZmw8E3qd0C5WTgD5KmAnNISRoRcY+ks4B7gXnAIRHxBoCkLwGXAiOAUyLinkGdEzMzKyUndWZmZm2KiMnA5Px/zRYoEfEK8Mk6w/8Q+GGN8ouBiwcwVDMzGwYG5emXZmZmZmZm1h1O6szMzMzMzErMSZ2ZmZmZmVmJOakzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MSc1JnZmZmZmZWYk7qzMzMzMzMSsxJnZmZmZmZWYk5qTMzMzMzMysxJ3VmZmZmZmYl5qTOzMzMzMysxJzUmZmZmZmZlZiTOjMzMzMzsxJzUmdmZmZmZlZiXUvqJK0p6SpJ90q6R9KXc/mKkiZJejB/rtCtGMzMzMzMzBZ13bxSNw/4r4jYENgcOETShsDhwBURsT5wRf5uZmZmZmZmHehaUhcRMyPi1vz/XOA+YHVgN2BC7m0CsHu3YjAzMzMzM1vUDco9dZLGAO8DbgRWiYiZudMTwCqDEYOZmZmZmdmiqOtJnaS3AOcCX4mI54vdIiKAqDPcwZKmSJoye/bsbodpZmZmZmZWSl1N6iQtTkroTouI83Lxk5JWzd1XBWbVGjYiToyIsRExdvTo0d0M08zMzMzMrLS6+fRLAScD90XEzwqdLgT2z//vD1zQrRjMzMzMzMwWdSO7OO4tgX2BuyTdnsu+BRwNnCXpQOARYM8uxmBmZmZmZrZI61pSFxHXAqrTebtuTdfMzMzMzGw4GZSnX5qZmZmZmVl3OKkzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MSc1JnZmZmZmZWYk7qzMzMWiBpTUlXSbpX0j2SvpzLV5Q0SdKD+XOFXC5Jx0qaKulOSe8vjGv/3P+DkvYvlH9A0l15mGMl1Xvfq5mZ2XxO6szMzFozD/iviNgQ2Bw4RNKGwOHAFRGxPnBF/g6wM7B+/jsYOAFSEggcCWwGbAocWUkEcz8HFYbbaRDmy8zMSs5JnZmZWQsiYmZE3Jr/nwvcB6wO7AZMyL1NAHbP/+8GnBrJDcAoSasCHwUmRcSciHgGmATslLu9NSJuiIgATi2My8zMrC4ndWZmZm2SNAZ4H3AjsEpEzMydngBWyf+vDjxWGGx6LmtUPr1GefW0D5Y0RdKU2bNn939mzMys9JzUmZmZtUHSW4Bzga9ExPPFbvkKW3Rz+hFxYkSMjYixo0eP7uakzMysJJzUmZmZtUjS4qSE7rSIOC8XP5mbTpI/Z+XyGcCahcHXyGWNyteoUW5mZtaQkzozM7MW5CdRngzcFxE/K3S6EKg8wXJ/4IJC+X75KZibA8/lZpqXAjtKWiE/IGVH4NLc7XlJm+dp7VcYl5mZWV0jex2AmZlZSWwJ7AvcJen2XPYt4GjgLEkHAo8Ae+ZuFwO7AFOBl4DPAkTEHEnfB27O/X0vIubk/78IjAeWBv6a/8zMzBpyUmdmZtaCiLgWqPfeuO1q9B/AIXXGdQpwSo3yKcDG/QjTzMyGITe/NDMzMzMzKzEndWZmZmZmZiXmpM7MzMzMzKzEnNSZmZmZmZmVWNeSOkmnSJol6e5C2YqSJkl6MH+u0K3pm5mZmZmZDQfdvFI3Htipquxw4IqIWB+4In83MzMzMzOzDnUtqYuIa4A5VcW7ARPy/xOA3bs1fTMzMzMzs+FgsO+pWyUiZub/nwBWqdejpIMlTZE0Zfbs2YMTnZmZmZmZWcn07EEp+aWs0aD7iRExNiLGjh49ehAjMzMzMzMzK4/BTuqelLQqQP6cNcjTNzMzMzMzW6QMdlJ3IbB//n9/4IJBnr6ZmZmZmdkipZuvNDgDuB54p6Tpkg4EjgZ2kPQgsH3+bmZmZmZmZh0a2a0RR8TedTpt161pmpmZmZmZDTc9e1CKmZmZmZmZ9Z+TOjMzMzMzsxJzUmdmZmZmZlZiTurMzMzMzMxKzEmdmZmZmZlZiTmpMzMzMzMzKzEndWZmZmZmZiXmpM7MzMzMzKzEnNSZmZmZmZmVmJM6MzMzMzOzEnNSZ2ZmZmZmVmJO6szMzMzMzErMSZ2ZmZmZmVmJOakzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MSc1JnZmZmZmZWYj1J6iTtJOkBSVMlHd6LGMzMzIYi15FmZtauQU/qJI0Ajgd2BjYE9pa04WDHYWZmNtS4jjQzs0704krdpsDUiHgoIl4DJgK79SAOMzOzocZ1pJmZta0XSd3qwGOF79NzmZmZ2XDnOtLMzNqmiBjcCUp7ADtFxOfz932BzSLiS1X9HQwcnL++E3hgUAPtszLwVI+mXctQisex1DaUYoGhFY9jqW+oxdMra0fE6F4H0Sut1JFDqH6EobXdOpb6hlI8jqW+oRSPYxma6taRIwc7EmAGsGbh+xq5bAERcSJw4mAFVY+kKRExttdxVAyleBxLbUMpFhha8TiW+oZaPNYzTevIoVI/wtDabh1LfUMpHsdS31CKx7GUTy+aX94MrC9pHUlLAHsBF/YgDjMzs6HGdaSZmbVt0K/URcQ8SV8CLgVGAKdExD2DHYeZmdlQ4zrSzMw60Yvml0TExcDFvZh2B4ZEE5eCoRSPY6ltKMUCQysex1LfUIvHesR1ZMccS31DKR7HUt9QisexlMygPyjFzMzMzMzMBk4v7qkzMzMzMzOzAeKkrgFJO0l6QNJUSYf3MI5TJM2SdHevYijEsqakqyTdK+keSV/ucTxLSbpJ0h05nu/2Mp4c0whJt0m6qMdxTJN0l6TbJU3pZSw5nlGSzpF0v6T7JG3RozjemZdJ5e95SV/pRSw5nq/mbfduSWdIWqpXsZi1aqjUjzkW15G1Y3H92DiWIVNHun5sGJPryBa5+WUdkkYA/wB2IL389WZg74i4twexbA28AJwaERsP9vSrYlkVWDUibpW0HHALsHsvlkuOR8CyEfGCpMWBa4EvR8QNvYgnx/Q1YCzw1ojYtYdxTAPGRsSQeLeLpAnA3yLipPxUv2Ui4tkexzSC9Lj4zSLikR5Mf3XSNrthRLws6Szg4ogYP9ixmLVqKNWPOR7XkbVjcf3YOJZpDJE60vVj3RhcR7bBV+rq2xSYGhEPRcRrwERgt14EEhHXAHN6Me1qETEzIm7N/88F7gNW72E8EREv5K+L57+enamQtAbwMeCkXsUwFElaHtgaOBkgIl7rdYWVbQf8s1cVVjYSWFrSSGAZ4PEexmLWiiFTP4LryAaxuH4sAdePTbmObJGTuvpWBx4rfJ9OD5OXoUjSGOB9wI09jmOEpNuBWcCkiOhlPL8AvgG82cMYKgK4TNItkg7ucSzrALOB3+emNydJWrbHMUF6B9gZvZp4RMwAfgo8CswEnouIy3oVj1mLXD+2YCjUka4fGxoqdaTrxzpcR7bHSZ11RNJbgHOBr0TE872MJSLeiIhNgDWATSX1pPmNpF2BWRFxSy+mX8NWEfF+YGfgkNxEqVdGAu8HToiI9wEvAr2+D2cJ4BPA2T2MYQXSFY51gNWAZSV9plfxmNnAGCp1pOvHhoZKHen6sX4criPb4KSuvhnAmoXva+SyYS+3zT8XOC0izut1PBW5ucJVwE49CmFL4BO5nf5EYFtJf+xRLJUzXETELOB8UpOpXpkOTC+cJT6HVIn10s7ArRHxZA9j2B54OCJmR8TrwHnAh3oYj1krXD82MBTrSNePCxtCdaTrx/pcR7bBSV19NwPrS1onn7HYC7iwxzH1XL7x+mTgvoj42RCIZ7SkUfn/pUk37t/fi1gi4oiIWCMixpC2lysjoidnlCQtm2/SJzfj2BHo2ZPhIuIJ4DFJ78xF2wE9eahCwd70uGkJqUnJ5pKWyb+t7Uj34JgNZa4f6xhKdaTrx/qGUh3p+rEh15FtGNnrAIaqiJgn6UvApcAI4JSIuKcXsUg6AxgHrCxpOnBkRJzci1hIZ9v2Be7K7fQBvhURF/conlWBCfkpTYsBZ0VEzx+VPASsApyf9oGMBE6PiEt6GxKHAqflg8CHgM/2KpBcie8A/HuvYgCIiBslnQPcCswDbgNO7GVMZs0MpfoRXEc24PqxvqFWR7p+rMF1ZHv8SgMzMzMzM7MSc/NLMzMzMzOzEnNSZ2ZmZmZmVmJO6szMzMzMzErMSZ2ZmZmZmVmJOakzMzMzMzMrMSd1NqxJ2l1SSNqgw+G/J2n7gY6rahoHSDquTvlsSbdLukfSOZKW6XAakyWNbbHfMZI+3cl0zMysHFw/zh+X60crBSd1NtztDVybP9sWEf8TEZcPbEhtOTMiNomIjYDXgE8NwjTHAK60zMwWba4f2zcG14/WI07qbNiS9BZgK+BAYK9C+aqSrsln+O6W9GFJIySNz9/vkvTV3O94SXvk/3eRdL+kWyQdK+miXH6UpFPy2b6HJP1nYVqfkXRTntZv80tikfRZSf+QdBPpZbbN5mUksCzwTHVc+fsLhf+/mefhDklHV41nsTzsD/I8HyPpZkl3Sqq8iPRo4MM55q+2s8zNzGzoc/3o+tHKZ2SvAzDrod2ASyLiH5KelvSBiLiFdJbt0oj4Ya5ElgE2AVaPiI0BJI0qjkjSUsBvga0j4mFJZ1RNawNgG2A54AFJJwDrkc4cbhkRr0v6NbCPpEnAd4EPAM8BVwG31ZmHT0naClgV+Afw50YzLGnnPN+bRcRLklYsdB4JnAbcnef9YOC5iPigpCWBv0u6DDgc+HpE7NpoWmZmVlquH10/Wsn4Sp0NZ3sDE/P/E+lrYnIz8FlJRwHvjoi5wEPAupJ+JWkn4PmqcW0APBQRD+fv1ZXWXyLi1Yh4CpgFrAJsR6qYbpZ0e/6+LrAZMDkiZkfEa8CZDebhzIjYBHg7cBdwWJN53h74fUS8BBARcwrdfkuusPL3HYH9cmw3AisB6zcZv5mZlZ/rR9ePVjJO6mxYymfgtgVOkjSNtLPfU5Ii4hpga2AGMF7SfhHxDPBeYDLwH8BJbU7y1cL/b5DO+gmYkNv8bxIR74yIozqZn4gI0lnIrXPRPPLvW9JiwBItjOY6YJt8VpUc36GF+NaJiMs6ic/MzMrB9WNNrh9tyHNSZ8PVHsAfImLtiBgTEWsCD5Pawq8NPBkRvyNVTu+XtDKwWEScC3wHeH/V+B4gnakck7+3ckP2FcAekt4GqSLN074R+IiklSQtDnyyxXnaCvhn/n8a6SwnwCeAxfP/k0hnWZepTLMw/MnAxcBZ+R6ES4Ev5BiQ9C+SlgXmkprJmJnZosf1I64frXx8T50NV3sDP6kqOzeX3wAcJul14AVgP2B14Pf5rB7AEcUBI+JlSV8ELpH0IqmJSkMRca+k7wCX5fG+DhwSETfkpi3XA88CtzcYTeWegcWA6cABufx3wAWS7gAuAV7M07xE0ibAFEmvkSqpbxVi+pmk5YE/APuQnuR1qyQBs4HdgTuBN/K4x0fEz5vNq5mZlYbrR9ePVkJKV6XNrL8kvSUiXsg7+OOBB71DNzOz4c71o1n3ufml2cA5KN80fQ+wPOnGajMzs+HO9aNZl/lKnZmZmZmZWYn5Sp2ZmZmZmVmJOakzMzMzMzMrMSd1ZmZmZmZmJeakzszMzMzMrMSc1JmZmZmZmZWYkzozMzMzM7MS+/8JZ6CvjfLKzAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 1080x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def plot_bucket_distributions(\n",
    "    ax, buckets: list[int], title: str, bar_color: str\n",
    ") -> None:\n",
    "    ax.bar(*np.unique(buckets, return_counts=True), color=bar_color)\n",
    "    ax.set_xticks(range(max(buckets)))\n",
    "    ax.set_xlabel(\"Assigned Bucket\")\n",
    "    ax.set_ylabel(\"Count\")\n",
    "    ax.set_title(title, pad=15)\n",
    "\n",
    "\n",
    "# create subplots\n",
    "fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2, figsize=(15, 4))\n",
    "\n",
    "# our example\n",
    "demo_example = x_train_hash[\"bucket\"].tolist() + x_test_hash[\"bucket\"].tolist()\n",
    "plot_bucket_distributions(\n",
    "    ax1,\n",
    "    demo_example,\n",
    "    title=\"Current dataset (569 datapoints)\",\n",
    "    bar_color=\"#BF8A6C\",\n",
    ")\n",
    "\n",
    "\n",
    "# large dataset example\n",
    "bigger_example = [farmhash.fingerprint64(str(i)) % 10 for i in range(1_000_000)]\n",
    "plot_bucket_distributions(\n",
    "    ax2,\n",
    "    bigger_example,\n",
    "    title=\"Large dataset (1 million datapoints)\",\n",
    "    bar_color=\"#9ABBB7\",\n",
    ")\n",
    "\n",
    "plt.suptitle(\n",
    "    \"Distribution of assigned buckets becomes more equal for larger datasets\",\n",
    "    fontsize=22,\n",
    "    y=1.1,\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26040d8d",
   "metadata": {},
   "source": [
    "The larger your dataset, the more uniform the bucket assignments will be and the closer to your desired ratio the splits will become (law of large numbers).\n",
    "\n",
    "Generally this should not be a problem though. Using a 90:10 train/test split is arbitrary. In reality, it doesn't matter if your split is a little over or under this target ratio.\n",
    "\n",
    "\n",
    "**How many buckets should you choose**\n",
    "\n",
    "I.e. which number should you use as the divisor for the modulus operation.\n",
    "\n",
    "This really depends on granularity of the desired test split. If you want to have a split of 90:10 you can split your data into 10 buckets using '10' as the divisor. Then choose one of the buckets to be your test set which will be approximately 10% of your data points. \n",
    "\n",
    "If you want 15% of your data for testing, you could use `100` as the divisor to split the data into 100 buckets. You could randomly choose 15 buckets from your data to get 15% of your data for testing. \n",
    "\n",
    "**Platform cross-compatibility quirks**\n",
    "\n",
    "Having said that hashing is consistent across platforms. That is true to an extent.\n",
    "\n",
    "While researching for this article, I tested to see if the dataset splits from my Python program were identical to a dataset stored in [BigQuery using SQL instead](https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39).\n",
    "\n",
    "To my surprise, I found some edge cases where the results differed.\n",
    "\n",
    "After a lot of head scratching, I came across two quirks of BigQuery which were causing the issue:\n",
    "1. [BigQuery does not support unsigned integers](https://stackoverflow.com/questions/51892989/how-does-bigquerys-farm-fingerprint-represent-a-64-bit-unsigned-int)\n",
    "2. BigQuery, [C and C++ have different implementations for the modulus operation](https://stackoverflow.com/questions/3883004/the-modulo-operation-on-negative-numbers-in-python) compared to Python when it comes to negative numbers\n",
    "\n",
    "The `farmhash.fingerprint64` method returns an *[unsigned](https://stackoverflow.com/questions/247873/signed-versus-unsigned-integers)* integer. Whereas the BigQuery `FARM_FINGERPRINT` function returns a `INT64` datatype. Therefore, BigQuery converts the farmhash output to a *signed* integer. The value of signed integer might be different to the original unsigned integer value and it might be negative.\n",
    "\n",
    "Ok, can't we just convert our Python farmhash value to a signed integer?\n",
    "\n",
    "Well, this leads to the second problem. In order to split the data into different buckets, we used the modulus operation. [Python and C (and BigQuery) have different implementations of the modulus operator](https://stackoverflow.com/questions/1907565/c-and-python-different-behaviour-of-the-modulo-operation#:~:text=Both%20variants%20are%20correct%2C%20however,same%20result%20as%20in%20Python.) which affects the results for negative numbers.\n",
    "\n",
    "Therefore, if our signed integer is negative, the bucket it is assigned will differ depending on if we calculated it using Python's modulus operator (`%`) or BigQuery's `MOD` function. \n",
    "\n",
    "> Note: this is nothing to do with the hash function or approach. This is to do with how different languages deal with unsigned integers and the modulus of negative numbers\n",
    "\n",
    "For example:\n",
    "\n",
    "```\n",
    "# in python\n",
    "-1%10 = 9\n",
    "\n",
    "# in BigQuery\n",
    "MOD(-1,10) = -1\n",
    "```\n",
    "\n",
    "Does this mean hashing is not platform independent? \n",
    "\n",
    "Well, not really, just that you need to check how unsigned integers are treated and the how modulus of negative numbers is implemented.\n",
    "\n",
    "We can fix our Python implementation to protect against these issues by converting the unsigned output of farmhash to a signed int and using a custom 'C-like' implementation of the modulus function.\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "\n",
    "def signed_hash_value(value:Any) -> int:\n",
    "    \"\"\"Calculate hash and convert unsigned int to a signed int\"\"\"\n",
    "    hashed_value = farmhash.fingerprint64(str(value))\n",
    "    return np.uint64(hashed_value).astype(\"int64\")\n",
    "    \n",
    "def c_mod(a:int, b:int)->int:\n",
    "    \"\"\"Modulus function implemented to behave like 'C' languages\"\"\"\n",
    "    res = a % b\n",
    "    if a < 0:\n",
    "        res -= b\n",
    "    return res\n",
    "\n",
    "# this will give the same results in Python and BigQuery (and C-languages)\n",
    "assigned_bucket = abs(c_mod(signed_hash_value,10))\n",
    "```\n",
    "\n",
    "> Note: [the order of the abs and mod functions is important](https://mentin.medium.com/be-careful-with-abs-function-8e91c78715d5)\n",
    ">\n",
    "> See this [StackOverflow thread](https://stackoverflow.com/questions/63341637/python-vs-bigquery-farmhash-sometimes-do-not-equal) for more information\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9615921",
   "metadata": {},
   "source": [
    "## Alternatives to hashing\n",
    "\n",
    "For completeness, here are two other common more explicit approaches for ensuring consistent train/test splitting. \n",
    "\n",
    "**Create an additional column in your data**\n",
    "\n",
    "You could use `train_test_split` (or another random splitting method) to initially define the splits. Then create an additional column in your dataset to explicitly record whether the data point should be included for training or test (or specify the fold for K-Fold validation).\n",
    "\n",
    "Here is an [example implementation by Abhishek Thakur](https://github.com/abhishekkrthakur/mlframework/blob/master/src/create_folds.py) who uses it to define the 'folds' for cross validation.\n",
    "\n",
    "This will ensure your splits are 'remembered' between training runs as they are explicitly defined. \n",
    "\n",
    "On the positive side, it is very transparent which data points belong to each split. However, a downside is that it increases the total size of your dataset which may not be sustainable for very large datasets. Additionally, if you do not have full control of the dataset (e.g. shared database table) you may not be able to add columns to the original schema.\n",
    "\n",
    "**Save your training and test data to different files**\n",
    "\n",
    "Another common approach is to store your training data and test data to individual files after splitting the data for the first time. For example, into files called `train.csv` and `test.csv`. If the data is too large for individual files you could also save multiple files into folders named `train` and `test` respectively.\n",
    "\n",
    "This can be a valid approach. However, sometimes it may not be feasible to make a copy of all your data and save into individual files. For example, if the dataset size is very large or you don't have the permissions to make a copy from the original source.\n",
    "\n",
    "The hashing approach can compute the deterministic data splits reproducibly on the fly and in-memory, preventing the need to copy data into individual files."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fba1533",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "Sklearn's `train_test_split` works well for introductory tutorials and small static datasets, however, in the real-world things get more complicated.\n",
    "\n",
    "Reproducibility is key for sustainably deploying ML models. The Sklearn `train_test_split` is surprisingly sensitive to changes in the underlying data and may not be appropriate, particularly if you add new data to your existing data.\n",
    "\n",
    "Hashing decouples an individual data point's train/test assignment from the rest of the dataset. This results in a more robust method for splitting your data.\n",
    "\n",
    "We demonstrated how to use [Farmhash](https://pypi.org/project/pyfarmhash/) to split our dataset and pointed out some key considerations to ensure cross-platform consistency.\n",
    "\n",
    "Hashing is a great solution, however, it can be overkill. If you are completing a one-off training on a small static dataset, Sklearn's `train_test_split` with a random seed will be sufficient. However, hashing is a great addition to a data scientist's toolbox to improve reproducibility and protect against unexpected changes in model performance.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b60f1b8",
   "metadata": {},
   "source": [
    "## References & Resources\n",
    "\n",
    "**Hashing**\n",
    "- [ML Design Pattern: Repeatable sampling](https://towardsdatascience.com/ml-design-pattern-5-repeatable-sampling-c0ccb2889f39) (inspiration for this post)\n",
    "- [Hash your data before you create the train-test split](https://www.bi-kring.nl/192-data-science/1340-reusing-data-for-ml-hash-your-data-before-you-create-the-train-test-split)\n",
    "- [Farmhash Algorithm Description](https://github.com/google/farmhash/blob/master/Understanding_Hash_Functions)\n",
    "- [Python Farmhash library](https://pypi.org/project/pyfarmhash/)\n",
    "- [Different hashing implementation from Hands-on Machine Learning with Scikit-Learn, Keras and TensorFlow](https://www.danli.org/2021/06/06/hands-on-machine-learning/) also [see notebook](https://github.com/ageron/handson-ml2/blob/master/02_end_to_end_machine_learning_project.ipynb)\n",
    "- [Google documentation: Considerations for Hashing](https://developers.google.com/machine-learning/data-prep/construct/sampling-splitting/randomization)\n",
    "\n",
    "**Differences between Python and BigQuery**\n",
    "- [Farmhash vs BigQuery implementation, GitHub issue](https://github.com/lovell/farmhash/issues/26)\n",
    "- [StackOverflow question: difference in results between BigQuery and Python](https://stackoverflow.com/questions/63341637/python-vs-bigquery-farmhash-sometimes-do-not-equal)\n",
    "- [BigQuery returns signed integers from FARM_FINGERPRINT function](https://stackoverflow.com/questions/51892989/how-does-bigquerys-farm-fingerprint-represent-a-64-bit-unsigned-int)\n",
    "- [Be careful about operation order in BigQuery](https://mentin.medium.com/be-careful-with-abs-function-8e91c78715d5)\n",
    "- [Reproducibly sampling in BigQuery](https://towardsdatascience.com/advanced-random-sampling-in-bigquery-sql-7d4483b580bb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8afae5a",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "#### Demo of BigQuery vs Python farmhash inconsistencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "958b81e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# example which returns negative int with bigquery\n",
    "example = \"1footrue\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "97c27e07",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python hash: 16905089972579912905\n",
      "Python Bucket: 5\n"
     ]
    }
   ],
   "source": [
    "# normal python method for assigning bucket\n",
    "python_hash = farmhash.fingerprint64(\"1footrue\")\n",
    "print(f\"Python hash: {python_hash}\")\n",
    "\n",
    "# assign to bucket\n",
    "python_bucket = python_hash % 10\n",
    "print(f\"Python Bucket: {python_bucket}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4aa4cfa3",
   "metadata": {},
   "source": [
    "The result of the farmhash algo is an *unsigned* integer.\n",
    "\n",
    "BigQuery converts unsigned integers into a signed integer. If we convert the Python unsigned integer to a signed integer, it becomes negative! This affects the bucket assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "d34ad1a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Signed int: -1541654101129638711\n",
      "Signed int bucket: 9\n"
     ]
    }
   ],
   "source": [
    "signed_int = np.uint64(python_hash).astype(\"int64\")\n",
    "print(f\"Signed int: {signed_int}\")\n",
    "\n",
    "signed_int_bucket = signed_int % 10\n",
    "print(f\"Signed int bucket: {signed_int_bucket}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7177aa62",
   "metadata": {},
   "source": [
    "We can't do anything about the fact that BigQuery returns signed integers, therefore it is easiest for us to just deal with it ourselves and convert Python's farmhash values into signed ints.\n",
    "\n",
    "However, there is a second problem. Python calculates the modulus of [negative numbers differently](http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html) to 'C' languages and by extension, BigQuery.\n",
    "\n",
    "If we take the modulus of -1541654101129638711 % 10 in BigQuery we actually get -1 instead of 9 (as shown above). We can fix this in our Python code by using the following implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "6e11a1a4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def c_mod(a, b):\n",
    "    res = a % b\n",
    "    if a < 0:\n",
    "        res -= b\n",
    "    return res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a5642a37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c_mod(signed_int, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "960979bd",
   "metadata": {},
   "source": [
    "Ok, now one final thing.\n",
    "\n",
    "Obviously we don't want negative numbers for our buckets -- we only what 10 buckets, not up to 20! Therefore we need to take the `abs` of the returned bucket. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "20592d14",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "abs(c_mod(signed_int, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94f93c85",
   "metadata": {},
   "source": [
    "So our datapoint will now be assigned to bucket 1 which will be consistent across Python and BigQuery (and C languages -- hopefully all languages, but I haven't checked...)\n",
    "\n",
    "[Note: the order of the abs and mod functions is important](https://mentin.medium.com/be-careful-with-abs-function-8e91c78715d5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aea9833e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}