example_circuits/circle_fc/training/circle_fc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "853cdb58",
   "metadata": {},
   "source": [
    "# Binary Classification with Neural Networks\n",
    "\n",
    "In this notebook, we will look at binary classification using a simple neural network. We will use a synthetic dataset from Scikit-learn, represented as circles, and leverage PyTorch and Skorch for defining and training our classification model. We will save the weights of the trained model in a json file which will be used in our zk program. We will also save some samples and their corresponding label predictions from our trained model in order to verify the correctness of our zk program."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4dd14f0",
   "metadata": {},
   "source": [
    "## 1. Importing Necessary Libraries\n",
    "\n",
    "Here, we import the libraries and packages necessary for data generation, manipulation, visualization, neural network modeling and training, and saving the model weights and test samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e430981e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.datasets import make_circles\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import torch\n",
    "from torch import nn\n",
    "import torch.nn.functional as F\n",
    "\n",
    "from skorch import NeuralNetClassifier\n",
    "\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b166dc6e",
   "metadata": {},
   "source": [
    "## 2. Setting Random Seeds\n",
    "\n",
    "To ensure reproducibility and get consistent results, we set random seeds for all potential sources of randomness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "648175d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "torch.manual_seed(0)\n",
    "torch.cuda.manual_seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e68ec738",
   "metadata": {},
   "source": [
    "## 3. Data Generation and Visualization\n",
    "\n",
    "For our binary classification task, we will create a synthetic dataset consisting of two classes. The `make_circles` function generates two concentric circles, each representing a class. Visualizing this dataset provides insights into its structure and the classification challenge at hand."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ad15789d",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAGFCAYAAAAfPZ8PAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABqQElEQVR4nO29e5Rb133f+8WBqNGD4gADyZM2kWObL5GixBE4I41ogBBxOQEF5SYrrR9XD7arseXVG4u0V5ZoXzfXScZSEkdkrijJsimOJC9FTJhx5PbGtcCZIWtzJNkWZ1xJZrrs4vDRlF65q64N0LXbOq41/N0/9tk4++yzzzmYB96/z1qzZgAcYA7O4/fbv3eMiAgMwzBMT2O1egcYhmGY1sPKgGEYhmFlwDAMw7AyYBiGYcDKgGEYhgErA4ZhGAasDBiGYRiwMmAYhmHAyoBhGIYBKwOGYRgGrAwYhmEYsDJgGIZhwMqAYRiGASsDhmEYBqwMGIZhGLAyYBiGYcDKgGEYhgErA4ZhGAasDBiGYRiwMmAYhmHAyoBhGIYBcEWrd4Bh2h3btnH+/HmsW7cO69evX7FtGaadYMuA6Wls28bx48dx9uxZ32vVahX37N6NjRs3olgsYsOGDbhn925cunRpWdsudj8YpikQw3QgU1NTND4+TjMzM57ny+UylUolsm079P2VSoWKhQIBqP0UCwWqVqu1bYqFAg3E43QUoIsAHQVoIB6nYqHg+7zFbLvY/WCYZsDKgOkozp07R4OplEd4DqZS9K//9b+mkXS6bqEaJbzL5TLBeZ6Unxedz1aVzWK2Xex+MEyzYGXAtIx6V/Eqg6kU9TtCUwrPfoDizu96hGo9wrtUKhGcz1K3uehsUyqVap81Pj5OAOgUQCWAbGfbU862ExMTgd9/qUqEYVYaVgZM01mqa+TZZ581Cs/HnM+oV6hKQf+CIrh1QS8F9X5tG/mZc3Nznu9gKd8FAA1qj03fr16Fo7MUJcowUbAyYJqOyTXSb1k0nE77BFy5XKbJyUnKZbM1waoLzxcCnjcJ1UqlQrlMxiuoAaqGCHoAlAfosGJtqN8hr1klQwFWyo5MxiPEF2sZhClRVhDMcmFlwDSEIOGkC8CKI4x1AXf+/Pma4LMc4XogwAJYjGVg9NE7AjxpWbRh7Vq6fXjYr6yc/SgWCjQ3N1f7f2Xtf+uP5XccCrAUxvJ5SsRi9KLzv14EKBGL0Vg+7zumpn1PWJYvhsIBaGYpsDJgVhTT6jWXzdaEk+4aKTrCWF9FD6ZSNBCP+xSA3F4VnkkIt4z+vB4ziFqJx5R9DtpmYmKCJiYmat+hBK9Voj8O+465TIY2bdxIcU1R9AE+ZRC070FWCAegmcXCyoBZUYwuIIiMn7m5OXrkkUcCV9W64D1oEK5V+C2JPoD+DMJdoz5/+/CwZ4Uc5aPfj2iXk/pTj2UQ9R3VmMNtAM3DbNGY9j3qs9llxCwGVgbMktFdQUGr1y/AH2DtA+ghRfCW4WbiSMH7QojAkxbDU0895bFEYtr/GUyl6MKFC6H7J4XnNECPRAjYv3Be6wNq7p08hHUirRK5Wn+xDuXygvN5SeczizBnIZn23WSFqJ8dFIBmGBOsDJhFExTIPHTokFE46QFW1QcP+P3p8vFBBLuGdFeIbduUXLPG+H8GU6nadtJyUT+rH6AbDMrqsPr/nP0tOft0GF4F58sm0vz4gat37bH+OaqLTd93GSsJynhiy4BZDKwMmEVjDGTGYrVVuSr4olwZ1wUoiqtXraoJvjMGhaEHSaempkL/j6xUrlarPkVmqlFQlZWqoKTQlavvQ4cO+YrdcpkMVavVWr1CLpv1KaABR8mpq3krYD8GUymqVqu+fdcVh8x46gfo+kSiVZcH06GwMmAiUd1BUa6WUXhdJg8j2gcf9Fl6Cmguk6HJyUnjilcWfgX9n/Hxcc/2UlBPT0+H7sOfQrikkprwru2jI+jDArgmBTQEEf+QnxeVEbVl8+ba97ZtmzasXRuowGqKiy0DZhGwMmBq6DEAkzto2FkFBwndSfgDvFEB1DCftxTaphRV9fl6LQOVSqUS+X0C3UZO3cBiArg1SyGT8aWTrjYciwr8QXE9tdX0f2eV47ec88/0FqwMmMAYwFg+71v1ronFwgWg83hW+Szp/tD99NLNsZhsmLDCK9mqQv8/asxApVgoUL9lhe7DxMQEzc/PG//n5OQkySBwUCWzCVlfoH6eycVWhLBIdKsjSoF9KOT4mQQ+N8tjiFgZMBTcLM2C3///sCO4dKEb5EYBQMPw+7fl4zz8weF+gEbS6UXta7FQoAsXLhib2MlsIhXV3VVPgJqIPFZKPZXMUSts27Zp3759tdV8TjmupyIUZdRrMs4gCbLypKKrt1keWw/dCyuDHifKvTILc5WwqVBKd6PkMhmylNXtLETmyxq4K+FnDJ9tATQ/P+/b13rbN8zMzBjbW6uoefum2oWRdDp0ZRxWybyYoi/1O1Xh72kUtPofSad9Qemko1yPApSAt3BN7u9h+F1PlnMewo4pWw/dDyuDLqSe1Zt+c4cVYo05wkUPVg5DpFrOG4RpPb5tNctmFsLq6LesQEG61MZuQcdI3zcbbsA77NhFKSWZTVQvasroGYA2Kccx6H8YXVdwg9KqMFf3N+hcZgOO6eTkpGcfudK5e2Fl0EUsZvUmb+6gfj9SmFwb8brqK5efJVfkUcJ7cnJyUavNlW75bKo5qEfAraRSIjJnG+UyGUpv3Rq5f7KiezZgXyYmJmr7G+V6MtUq5DIZbrXdI7Ay6CLCVm9h6aFBPvOxfJ7Wr1sXKvg+FCKo6hUiQRlDYd9xsQLchEkI1+P6aJRw1I9DPft35MiR0H2ZmJio7a8MLNd1LuGmqKq9mFZC+THtCSuDLiFKQOmBQ/XmNvnMZQAySthECdKVFN5ESxfgYSxGGUlW+nstdf/K5XItLqPHDyxFMe3aubMW56nnXO4A6Gnn78cff5wtgx6AlUGXUE8TtpqPOCCdUrp5PqHc5FHCZmZmJlSQNkJ4Ey1NgK8kjfpeS2Esn6c+TZjrnU/H8nm6zjlnQefykxB1Ijnts65PJGj76CglLaspyo9pDawMuoRI14X2nAX4b27HQtDN/3qETRStFt6Noh2+V5Ri0gPIpnM54PR1GoK/3Xa/c73o6cH6/2j1cWCWByuDLiKoCVs+wFq4ZfNmn2tgUrEQFuO7ZlpPkGKqJ432ueeeMxa+qQuKv4BoHrjasiiXyRARp5x2E6wMOhx1RTY3N+drmhaVQ57LZOjaWMyTzqjGDFTaYRXMLJ560mjlNmGuRjiK5LDyPk457R5YGXQo+opMN+FjAP1bBEwGU3L51TYOfEN3L/UEvKN6LR10riVZtBaVXMCLhs6ClUGHsmvnzprvN6j18VCAW0Dtkc855L1BPa6+oP5OarttNfOIU067C1YGHUa5XKYjR47UskKiisYOODfnAXh9vUT1de3kG7q7iHL1VatVymWzXqUBt7JZXhfD6XTdCwkOLncGrAzaEHnzTE9Pexqj6Su7o4gefaj+jOXznpVgPV07+QbuTeSch4MB14XsHSWvoYchqqBV9xMHlzsLVgZthOnmkbGAwVSqFqhT5+oGTRKTw1L+DKLGQO/5s5SunUxvERVnqFQqvlbclrLo4OByZ8HKoMEsxkQO6oS5SRP4ugJQhfkZGEZEwpsBQrT8rp1M9xMVZzBdr0nLopF0OnKCnGr1Mu0BK4MGsVgTuZ52EqorKAe3972qAEzBZDUDRMYAltO1k+ktTHEGef3sh7nBXT1pqvW6jjjm0BxYGTSIxZrIUe0kdME9CX866YhhO/0GVW+oZvbXYbqHSqXiq2eRQWZ5vV4VcS0ehJvYcE0s5pnxrP4fjjk0D1YGDWAp6Zrqe8oQgWFbec8meKeLPabcVHLbqGDysDY9jCuLmaUQ5M4satdrH/x9kPohrNgKRGuMoHhD4P/hxUrDYGXQAKJW+ePj40aFsGvnTlql3SB9AG0BKGFZxpGO6speKoiwgSgmuLKYqZeohc4auBZr0BS7I44i0N2ZSed6ly3XOcutubAyaAD1tpNWV+GVSoXeMTDgeX1IubnkttPT07WRjqaVva4g2PXDrCRRC50tyrUot5mDmIqnxwpM87Xla1zQ1nxYGTSIoKZxQwEm7/Y77jB2k5Q319zcXKBLR13Zs+uHaSRRC50DAI1qwl5mu0krYL+iLEzztS3wDIVWwMqgQZiE8hDcSk71wp6envYMjlfNZmlyywHo9fhPy+UyTUxM0MTEBN80zIoTlHigujHjzuLH5LpUU6N1RSFbqeSyWWNrjH4I65dZeVgZNBjbtml8fDzU5N27d29dbqWoVRJnXzDNIMz6lFbq/Py8p62Ffu3nAVodcV3LBZS+oGLLoDGwMmgCUab1o48+GqosfvXGG+vyn3L2BdNM6kk8CCo+OwzU5ieEpVNfhJspZxuueWblYGXQJMJy+k+fPh2qLF566aVIy4CzL5h2xXTtJy2r5gINswyCeiPx9bzysDJoEmGmdbFQoD7A6B+VoyWjCsSisjx4JcW0CtO1n3NmJ2yGuRZBLagcgqiy58y4xsLKoAGYyuflc/oAebmi3wv/IHILbm3AYubc8kqKaUd0t1KxUKCEZfniAn0QNQr6DGaOgTUWVgYriCmAO5bP+zo7ygvaVNafg2g1MRuwog/z03J7CaaTMC1wwhY0MzMzrd7lriZGRARmRbhn9268fvIknlxYwA4ArwB4KBbDz4jwHFB7bl88jtFduwDAt/0+AKMA7gWwB4Bt21i/fn1d///SpUt44N57UZqerj1XLBRw9NgxJJPJlfuiDLOCnD17FufOncPf//3f48EHH8RFADcqr88CuAvAI488gm3btmHdunV13xNM/bAyWCFs28bGjRtxFMD9yvNH4Qh1AOu15+Tfpu37LQvvHRvDy1NTi94XeXPxTcN0Evo9VIW4F0rKNhaAy4he5Ni2jfPnz/M9sBhaa5h0D5EBXMNzYdsP8zwBpgdRXZ15+Asx+wFKQPTqMrk/udZm6Vgt00Jdxtq1awEIV4/KrPN7neG5sO3/8q/+il07TM9x9NgxjO7ahT0AvgbgKQgr4Ubn9+cA/BjAP7p8GaXpaZw9e9bz/j333YfXT57EUQAXISzt10+exAP33tvEb9GZXNHqHegGpEmay2ax75vfBC0s4EYAXwXwDIA+AKcBXAUh7D8Wj6PoxAz2nTwJWlhATnuNTVumFyHNa71Dez3n/P6e8/vcuXO1e8W2bZSmpz2u1/sB0MIC9jiKg++rEFptmnQyJpP0hmTSN3RG70YqzVZuKscwXqSb6EBEZpH8UbPquNZmebAyWCLlcpmG02nqt6yaT/MAQFdA9FzRW0LkstnAlFCeJ8Aw/lqZHPyFmANw+xPlMpnQ93OtzeJgZbBITNbALvinNskxgHwxMkx96Cv7KkCD2n01BDHjYzCV8ljQsqgzl81yrc0S4QDyIvknv/mb+NaJE54A1asA5p2/a0ErAA8475F+znPnzjV5bxmmc9CTMJIQsYFNyjZvAbgtm8X3zp5FMplEtVrFPbt3Y+PGjSgWi5h99VWsSiSwB8A7IVJTR3ftwtFjx5r3RTqVVmujTqFSqdAOp59KUG92o2nKlgHD1E1QFX0ukzG6UsfyeUrEYh63bCIWo+2jo+x6XSRcdFYn9+zejVdOnMB/v3zZUyF5HEAR8FVNfh9iZfIwgOediuOlFJAxTC9RTxW9zN6Lx+MoFArBhZ6LqN5nOLU0ELWCkYhQmp7GAQD7IcxYefGtdX6rzwFuvcBBACNbt7KZyjB1kEwm8fLUlLGKvlqtYs9993kUBRCcfjo7O8vKYBGwMlCwbRtvvfUWPv+5z2H21Vdrz2/ZvBkA8EEAX4foH0QQF90cgCsBfFR5bhbAxyAshe8D+KPPfpYLyBhmEaxfv94nyNWCsh0AJuFfnAHuQuzAZz+LNWvW4LbbbmOlUA8tdlO1BWqGkOWksx2F6KE+pGQyHHUyHPQB3ttHR321BAk9u4jrBxhmyQSljQ4FpJ/y/bd4OJsI7orjAEQTrKchVhr/F9zsoDyAvQBeBnAYwAEAqy0LuUwG3/jWt/Afz51DLpMBIJppEcAl8QyzQpw/fx6A3yX05wB+Cniyh66AuI/V+++bMzP44PveV3ufbds4fvy4r51FT9NqbdRq1BVHCW7zOD1LyGQRmFYbQTNfOaOIYZZOVEEZAHoBoOmI7L5Dhw5RLptlq8FAz1sG6opDDQafd/6WK5EkhFUg/ZETExN4eWrKFwtYWFjwvE/CtQYMs3Q2bNiAYqGAffE4jkLE4o5C9PLa4VjkcQD/2dk+6P773Y9/HG+9+ipb7QZ6XhmohS4bINJE9wH4D87relfRi87vXC4HE5HdS9etA8Mwi0ftaKoWlP2/X/lKTVG87mwbdP+pbmDZCfWJhQVjB9Seo9WmSTuwI5Oh1ZZFB7SgcQygayB6Di2mtJ3HTzJM4zD18lKbPloQcxDU+y/p3M8AN7ILoqeVganPkOw4mlyzxvh8kH9R9kaxbZu7kTJMi7Btm2679Vbq0+J7fQBtj4gn9Ho8r6eVgVzBeyYpxWJ01RVX1NJL5fNJy6JcNuv7jLDJStyNlGGaj2kxNgTQHEAjTuopW+1+erYdRdDM4k0A/iOCZxPrJe737N7tH2rP7ScYpuU8++yzePDBBzEM4G8B/Nx5Xs5Rloyk0/jkv/pXuOaaa3p6ZnLPBpBNecs2hCLQnwfM2UBystKTCwsckGKYNmPHDnEX/xzA1RALujMAbnVejzm///0bb+B973sfisUiNmzYgHt278alS5eavr+tpmeUgV5kYsr6Oa/8XU82UFAhDKeRMkzr2bBhA3ZkMvhbiNnJaiHpEES6+BCABLhAFOgBZaD3O9+wYQNGtm3DT37yE1/e8ted92yGSC9V85kfApDLZj0mJKeRMkx789G9ewGIBZsNoATgUxBzEeTvJ8GppkAPKAPZauIwREsJAPj2G29gZGQEv/jFL7Atl8MeAO+CaDEBAN+F8Cmq+cxXpVL4N3/zN57PDiuEKRYKPet7ZJh2YWhoCIC3kPQd2m+27AVdrQxUn/5XIFYBqjn472dnsWrVKuSyWfRbluc1gutTzGUytclKOkGFMNyymmFaj7pgk4Wk/1X7zZa9oKuziY4fP45isYhTAO5CcIYQQl6bmZnB2NhY5P8y9V9nGKb1qANzLADXAXg3xMLvnc7vJ6C0n68jG1Cdd9It93tXzzOQPv2XncdB5qD+mg039eztt9+u63+Z+q8zDNN65MCcmZkZFAoF/DKEl8ACUHV+71G2L4ZY9qYBO/oktk6lq91E0kQ8YomvGWQOyteqAO4BsBHAP3ee/9M/+ZOeTDNjmG5DNpGcgljwfRXADIAvOq+Pj4/Dtm1jA0qJOmCn67KPWlvz1nhkNaIcWqNWHiZiMbLgDrQZcqoT1cpjrk5kmO4gqg12VKeA5b6/3elqywBwTcTT8/PYkE57Ar0/I8IXIEzGfwROM2OYbqbe7L/p6Wl85jOfwYkTJzzv7/q6olZro2Zj2zYdOXLEp+HVwTbc0ZBhupOwJpLnzp2jwVTK89pgKkUXLlwgIqKpqSkCQAe71DLo6gCyRI/8Sw0uC1HOQwzGAIKHa/damhnDdCPSU2DK/tu0fj3+oVLBUaDWZ+yjlQru2LYNI7ffXgsaPwxhUfw5gO/AsSx27er8BJJWa6NGYuoomstkaG5urtbJUH0tYYgrcMyAYbofueo3xQMsp2uxp7uxoa292sa+E+lqZVAsFChhWT6hP5hK0Q3JpK9N9RqArl61iucQMEyPMT4+bnQTn4qYgTAzMxPaxr6T6NoAsqw+ftfly7UUMPn7HyoVVC5d8o2/exrAz37xC8zMzKBUKkWmmTEM0x3ccccdAPzp5191fgcFjd9+++2uSTft2piBjPy/BW918f0QqnsPhBJQUU/w3Xff3fidZBimLSgUChhMpfDRSgUEtxr5sPP6JICbAawDsB5uLDEej6M0Pe2XMQsL2ONkIXZKLKFrLQNZfQwEa/Wvas9zsJhhepdvzc/jqlTKk35+dTKJG5JJ7AdQBLABwG0A9loWRtJpXLx4EUB3pJt2rTKQvcyB4MrjI05zOu42yjDMu9/9bvyXH/0IMzMzGB8fx8zMDEZuvx0LP/mJxwX0nwD89PJlzL/xBh588EEA3dHsrqsb1V26dKmWLvY5eBtRbcvlsGrVqq7sMcIwzPIJGo0rm1jOQiwkPwTg6lgMTxEtqtldu9EVMYOgDoLJZBLfO3sWv/Wbv4k9r75ae142okomk9xtlGEYI0EVxzLWaAP4MID/DuB3iOpudteudLRlsJgOglLox+NxLCwssPBnGCYU3TKoQlgEJWWbIoDPQsxVnpiYwC//8i93rGzp6JiBKaXrGydOYEcmg2effbbWT8i2bbz55pv40z/5ExQKhZ4ffM0wTDQbNmxALpPBRyFky/sBfAvavGQA/0x5T6cqAqCDLQOT1v4ggK/BnUUAAO8YGMB/rVZrQy2ehltqvq8D/XoMwzSPL33pS7j3gx+syZSwAVmSTo09dqxloPvz9gB4FUA/XM09BODn1SoOQCgIvciMO5IyDBPG0NAQLsONEwSlkO5HZxecAR2sDGQdwSsQgZwSgJ8DeApC0P8MouDsaYhiEaA7coGZ1mLbNo4fP84LiB5Bpqh/33kclEL6IDp/kdmxykD2Jn/IsqDWCkuBf155LMvPuiEXmGkN1WoVu3ffg40bN9ZiTrt338Mxpx7go3v3AgBGgVr8QNYmPQQgD1GVLOnURWbHKgMAeOrzn8fP43H8V+U5KfBVBbABIuq/D+AiM2ZJ3HffHpw8+TrU8OHJk6/j3nsfaPGeMY1maGgIADAC4KeAp0L5pxCxSpWOXWS2skvectmRydQ6Cu4A6Aqn86hsQb1JaUl9xtCyOpfNdlxnQab5yHGHwFHyNq98sSuGmjDRFAsF6res2nCbEkA2QEWAkl3S9r5jLQPbtvHKa68BAJ6HsADeBvATiGH27wTwPbia/FaIGEIOIo4AAJ/81Kc6LuLPNB+ZrBAUdeo0dwCzeI4eO4b3jo3BAvAIgAqAqwD8BoD/Ca+1MNqBBWdAB7uJ5A1qAXgT3tzfNc7zzz//PC4DOAgRYLYBnHJeBzrQjGMAND+I6zY9NEed+DrqfuSEtNHRUfwMrvD/l87r20dHjW3vOyrhoNWmyVJxTffgwRO2bVOxUKCBeLwrzLhep1KpUKFQ9Lj6CoViqKtvpaZPFQpFiscHHNfQRQJepHh8gAqF4rI+l+kcpMw56riIpKtIypuJiYnaddaJA286VhkQEQ2n05FD7MMGYDOdhSuQjzoC+WigQK5XcdSrLKrVqu/zMpkdNDk5yTGDHqFUKvnkTQWgvBaLLBYKNJbP00A87pmk2O6L0I5WBnKWcZhlILFtu6Pnk/Y69QZxpXDPZHKhisOkLNLpEZqfnw/dD9u2aXJykrLZ3KIsFKbzUS0DeQHKALIu9K065VI70dHKgIjYDdQjTE5OOoL3oqYMLhIAmpyc9Al3YIiAqlFxmKwMoJ8AK1KwL8ZCYboLVd6cinBTz4Z4LNqRjlEGQeZ8tVqlnJNiym6g7iWT2RFqGWQyqiVwioD9jnAv+hTHkSNHnM/aT4Dt+yzL6vcIdvXai7JQdMtCfy9bp52Nye0c5Kbez5bBymIKxAyn0zQ/P298LZfJsCLocKTQnJ6erv12V/reIC7QT2vWJJ3XxwkY0awDi4B5j8C++eZbtW22ONtcdB4/TABobm7O4EoaDrVQNm7cTKVSyfBey/M57FbqbGzbri0qgiyD1eis+oO2VwbSLDsMf6BmMJXquCANE4zJj+8KUYuA1wgY1F4fIuAa53XLsQZ010+CgMMUjw9QKjVIlpUg4BbD/9ni/D1LAOjWW4cI6NO2i4daBurnxWIJZ7s8AUlit1L3YXRTA7QdoF/R5FW7eyzaWhmoAZuic5Cl4D+A6LRSprMw+/EHHIHfT8Aa58Y6SECJXBfPJgJWRwjpmOJqGnIUhPp/ko7gH1TeY/mEuNgPqWC8FoqrYD6s7Eu4W2lmZqbVh51ZBtVqlUacrEb58w5NCVgAbR8dbWtFQNTGyqBcLtP4+DgB5kBNCeH+unYN0jBmonzxwAHlBlNdNPJ9+w2vEYn4gXjfjTf+qvIZYSv7VXVsp1sMfQTEyGvJnCEgPPANgDZv3kKHDh0KXcBwvKF9kdfufoByACXgzS5KAtTnWAbtTNspA1McYK1B8JfZMugq3KBukND8XU1Al0lYBxPOc4+RazUQARUSwWNVYF8RoFC8wlm4nYKUi9xOBp+lhSKVRD+5lswQAdsDlMoXSI8jABaNjt7pqV1YSqEd03zU3kVBMqnd5VLbKQPpg1M1a79jah11lEAJoGlHSfSjs4I0jJfTp09TOq0GfaN98cCV2nNx7fEQAVnyu3iSBFxfx/85SFHuHRFXMCmJh8lvyQyRN/D9mKOYVpMptVW+L5vNUT4/xmmsHUC1WqUtmzdTmLei3T0WbaUMTEUdqma90nPDg2KOkuikIA0jqFQqlMnkyBv0lYFW3Rc/pAnMPkU4DpE/aBwVP0g571H/j4wZWORaA0UyZS+JbcKUBAh4Qbkuz5BrpejWQJHcWgj1M+R3jRPwjPF/tfMqs5eoVCq0a+dOiiHcW6Ges3Z0+7WVMjCVe0vNajlWgCdzCKItdb9l0XA63VYHljHz7LPP0vvf/3669to1BqFdJb9rZ4hMhWPCNRO0eper8yAXzyQB79X+j0XC5x+1PzJWsNr5P7POPiWcbU0xDvl5OfJbKwPk1kLI/Stp33XU+D3aeZXZSxQLBeqDiA0MOXJJ9VbImEEuk6HJyUnKZbOea6pdFrBtpQyCLIPHIjSuzCxiZdC+fPvb36ZVq67WBOt7AoT2rLJNkECfcISmaZtTAUpCXXm/6Aj0OAHvI+BfOK/lyW8NXOcIe1VxmBTJOHmzn9aSm5EkYxpB+2Rrf+txDH9xHF/vrUdvmFmFyHxUrw8LoIE1IhMu5igG9fU+gMby+VZ/lfZTBhb8wyJWOwctyBf3AtrfH9frCEWgu3P6IwRk1GthVcnS/WRy8Yj3j45up61bb3MeS6vgGfJbAwlHyJtrBsRjNYtIvMeyEpRM3kDe7KQg5fawsn87SFgvqnXxsOd7ZLO52rFtR5dDryC9Gbp8shW5BICSlkUHHMWg9zJKOs+3+vy1lTKQB1YvLhtVNK96J7Fl0Bk8++yzixTaUrjKVbX+Wp5cF8tqw/sHSASQTSt4UUw2Oqq7ifpIFJ1Jq2DWEcAyAym6ZsD9EYpl585dtGPHTu21sPdmyS18UxURSP8ek5OTnGnUBtTTSl++diRiu4mJiZZ+l7ZSBlNTUwSIsXJ6v3AZM1Athn6InN6EZXEGURvzwAMPUPiqeK0m7NTsoLz2mingagrKHnb+/oSjMFbR1q3pkCZ1SQLGyG8V/LKy70FuKfE9xsfHaWZmprZK9/8f6T7SrZW1JNxAMmAdnGEkCuyCm+1xplHzUWMGunySSS8XFWUQ5OFgZUDm2oIhiLnFLwK0JhajTTfdRNcnEr5t1kC0peDVUPsSbhlIf7jM2VddIxYJt42sJQhK57TIH9BVhahIR83nx+oobpP7omYD1WcZqJap+f9UHYWgWytnIj9bzTBKpQZ5LnMbUa1WaSyf92U2WgDdOTpK0hpo99qotlAGQbUFptTRTQBNQlgM7XQgGTOuKyPIh58PEPD7SbhI+igq+Do6up28BWUgYDMBhxxF4gZnJyakYgmyUryZPDMzM7R58y3KvvvTX02rcdeXHFbcdlT5HbX9sKNMxH498sgjipLwb8/xs+YzNzdHtzi1BvJHHXTzIkBZ+D0ciViMA8hE4bUFFkTgRU8nLSrbcfuJ9sZ1ZTxDfpePXPlHZdfIbb3KRArharVK+fwYuUHgYKHqVjqHZxqpAr5arVIqNajtS7ifPtoCka9J11BUtlG/s+1XfP/f5DrjxVHzMS1qB+JxGsvnPZ6Pdq2NarkyCKotOIVwk2pGe8wXf/OoN3vFFYgHCDhCYpU+Q24dQI78aZymvHuQWOXnQoWwbdvKijnYfWKaZyw6jFqBn12tVp0iOff/p9PDoZPRguYmJxLXk7B4kiRiG1JJmqwneTxk+wq1SE+Nd+R9SoxpHlEFs7Zte6YttuPkxZYrg6CD+DDCgy0yZsDB4+axmOyVcrlMn/zkJ8nbvE3+yKKbSfIHbE0B4pxyCYgahLBgW9TwetM840KhSPPz85E36GJu4qD/c/LkyYDj8o8Mz8njMUrRldWcTdQqwgpmo67XdqHlyoDI3BM8qunTQQjfGwePm0dY9oq0FrxDXUTg1t8uWgZ3DzjPHSC3V48eU0hQ0OjKIIKEsH6dNGt1pv8fN54wS8JamiBgjvxutBwB58mtpwhvntcJAqdbibIMwtxB7VIn0hbKwDRKTg+8qIMjivCbYExjCfaBm7pv9hHwMfK3dzD5zR0rbyjt+P3d5xKJ651BNOHB2iDa0RQnCjqWRTIXs92gWASnQo9nO/e96QVMi9okRN2UafCWKYuylfGDtlAGEv3mNSoJiJJv1QTj4HHjCc6OyZPZh91HohV0WIbM+8iyVlMmk6v9H/UaqHeF34l4XVlBQl4GlQ8or/ub51lWkgqFIhehtRjjPHZFXumL16CAc6vc3m2lDHTkCuf5558n6RoymWC8Amo85tVskLVgEmJL93G36wp/OZgUnV9pvqA8L5WAGnAWP5lMjiYnJ2nr1jQBV5OYtCbqEmKxBOXzY63+ul2PaZU/qigCffFaT8C52bSlMjAd2KuuuILWwF/hN5hKtXp3ewZ/YNbUHbRM7iAaVYh5YwHp9Eht4H03CfnFEp4Bpaab+juoXn31tTQ8fDu58RlVsVgkKqpFJTaP12wsplV+Eq5LuwzQbzvnRlapAyJrUnZa0BVGs2lLZRBUhJbwXOwim4gtg+ZhXs1KYWWaLLafgHnf81u2bPXFB3rdnSHqGPS00n4SrTnU5w+QiCHEKJ0ecWIqCQp21eX5GDeYqFX+MPy1BXcMD9fmH6gupcNsGQjK5XKtKCisvkBqUo4ZtAbbtmvzqUWvnH4Sw+ATZHJjiMfjBFxDiUSKe+oYuHDhglbYBlqzJun8rfduGlL+vinAqtCD9Ad7/hg3iqg5LLJvkb647YO/e2krZyW3hTIwuYWC8nWHERyQYZqHG0P4DfJmDQU3W0ulBp0c++iMmF5lZmam1vCOSHXNHSARQzhA8fgAbd6sdzcNa3uxntSK7l4/xitN1ByW0NiA4bmwQsZG0hbKQHULnYo4eGsgUrV43nFzUNMU1b8rlYpvJVtP6mNUzx628ryYXHP+42465qaU3zwf4wZhSiu9Jua2Rwla3JZMz7Xo/LRcGZi0ahH+drCyvqCeIg5m+ZjSFFXhkkhcT7HYtSSGx/8+uTGCcEHP3TaXhlSk2WzO0H67j/zxBvmcfwgPH+OVR0+D12MEi7EMWnV+Wq4MTP62KkBbtYMp83U7qby7kwnu+Z8nf5/9YRKuIjnWMlzQR7WLYMwEK9LDBiuAFW4rsG2bRtJpTwLMEMyzWPr0BW+LPR1NUwZBVZFR/rb9baY9e4H6ev6rAcq4JoT6HAFlFvTdXEzWSKLbYn+YRCvs8DbdvJBqHCZ5VoWb+Sh/du7YQWN5b6JFqz0dDVcG9ZRcm/xtAwANwuAu4jhBw4kWOiXl77Vknm0c3AFU0o3FZI0kWknL2QbR7bPz+TFWvg0gqmHdvn37PPE3dTJeq2m4Mqin5LpardJIOu1VGABdgH8ecqu1Zy+weMvAvN3mzVva4iLvJkwuNlf5mnodmedHx2IJdss1gKiag7m5ubbqR6TSUGWwmJJruW2QW2hiYoIFSxMpFIpOj3+TMJF99ocjLAh256005sK/OImYjRpEPkzCXadux0NwmoHR0+EsgNutH5FKQ5VBlMmkp1CFHUTuxNh41GN84sQJMrc4kOlyORJtl8PdEZzG2Bimp6edY3zQEfLXBpyvXyavC4lTeRtNUBfmubm5uhbHrZJ1LbUM9N40poM4ls+3XaCl2zClkcbjfeRO1dpP6kB2V+jkSVQe62mNchurZQU0vYDrMjpMwNaA85UkvwvJVdjcs6hxBM2xCFocP/7448YZys2SdU2LGeir/cFUKvBLqwexnc2qbsGcRip76B8gETC2tVX/QUfwxMifTbSFgGfYL91gzC6jICttTYDC5kyuZlBPqx0LIuW0VbKu4crAtNofTKUooQ+6N3zpdmzz2m0EB4s/SuYRjGfIzShSg8iPEvAhcgPM7JduFvPz87R27XoKj9/EtHO52lH03LOokejZlFLgq4vjRCxGVyCiOK0J91DT6gzkal/6OoO+tBooXmzMgVk8wWmko+RPGR0gt0marQiarcpqM+cTRHyeGk90BhgIuJGATxDgdUXIc8pKe+WRno3DcDMj9epk9XErZZ2FJrF+/XrcfffdWFhYAADsUF6rAvii8/eDDz6IDRs24J7du3H99dcDAF7RPmvW+b1u3boG7nFvsHbtWucv9SjbAF4H8DSA+wHc6Px+AsBbAHIA1sM9E98BsAfATwH8jvI54nU+T43Ftm2cP38e2WwO8fg+AEcBfN/5vQ9A3vn7fwB4AcB/cR5fVH5bOHfuXCt2v2uxbRul6Wk8ubCAr0DcOUcB/B2AgwBWWxZu2bwZl5X3fEn7jKbKuoarG42wXkQmt1GxUKB+y6KHAZoFF56tNOJ8WOTNSTcNrVFdDk+Tm16ac1xGcqqZHHLPLSYajSnw729ip6aTqsNyOJjcaNQBNrrMU70huqUwBNAZuG0rctlsU/a36cqAyBtUjjpQ7x0d9ZlUY/k8B7xWCNdNtFU5zlao0HB/hkjEEITg1wURByYbiynwb1n99Eu/9I+dc6Cnk6pjNDnNtNFMTU0RAPpQiAvIFDTuVxTE1atWdU82kQlTUDnoQCXrCDQzS6NSqVAmk9ME/JUkfP9DZBpXCdxKwCSJoTZ+wc8tJpqDP0ZgmjQXNkbTr+S//OUv+/4Hn8vFY2rBY1rwPhbwvFwIvwvNTZRpiTKQ2LYdmG4VdaC4Inn5+FeWn1CEhX/mrrAYvkrAi2RZ/bR58xY+Dy3CH/jXhwoNkb/+Y4DcEZl6ZbmoKykUinT+/HluJLgMTOnwffBnEa22rMCFMAB6Ac1NlGmpMpCY4gLXhFgM6kXKBWhLw7uyLBNwhIDdmoAhEllDL5BfKbCgaCX+86ev+KvkHY8p4wcXAp4/TADIsvoplRrksaRLJCgd/jD8sYEdmUzogvdAL1kGRMKk0iuMLYDWR1gGs2C30XJwV5ZZn3AXwqKqHHbhRvj0pz9N6fQIC4o2wbXsogL+v0tu7Yf6/D9xfqtpwvKzDmqfxTUj9RCVDj8xMeGZHChnH+gdm4daINtargxMJlUSIsOo6BwY9UDJ13TlwBfp4nCziPpIuAn0FtRDpMYK4vG+mkuPh6a0B/4K5KCAv1mwiwAzyFtA+BfO7xeMioWDzOHUUyhrKkTTF8Ot8Hq0VBlEHbh5R/CrByoPMSxC17h8kdZPpVKh0dHtdQgR+aO3m+BslHbCtm3FYtMD/nEyxw6Kynk+4DxnEfDbznOfZIW/RMIabqqvezKILIuG0+mWzjdoqTKQK83ZAJNq0nl8QBFE3Jpi+YjV5LWRwh34LWWbTQR8hS2DNsXUpyibzdFzzz1H2ayeMZYnESPoJ9dFuIbM7kI3dZhdgfUR1LW0Wq22dYudligD4/QzZcUvD8wozL3AefLZ0nEDjweU4x9kGchtNimrSf/QFBYU7UNQau/8/DxdeeXV5BX2ORJpwgfI7XhqnljHSQKLx3Qu2rnFTkuUQVCcIA9vAEXXqmEal4mmXC7T+Pi4Yg0UyY0ZqG6EBAE3EXANASkSwWSpIOadlSVnE3UCaq3ARz7yEXLjAWpA+VToooArk1cOtgwUog6GtBLOwI2+63Bh0+IwtS1wawnGDO4BPUZg7lbKNQbti+mcb9s2opz7MrmtycOzkTgWtLJEeTi6criNicjUqzbSlN2Cv7hsiLxBxQOOFRCnvr5rDe4CtVvpAXYLdQCmVhXx+IAztEgfh7kq1DLgAUUrS5CH4/z58y31fLSdZcDN6FYWc2vjKvnbGKvWQVAMQfTEZ7dQexPezlqPDRwg4Gpy25D4K5NZ8a8ccsjNxMSEL3Oo1YO8WhozkGbSYwBdC1CMYwErTvi8gtXkHZF4Tai74F3veg9bah1A8DlXYwOmXkYJ7fH1JCuT+bwvj0qlQrt27vTVFMimm+0QS2jaPAOVo8eOYXTXLuwB8C4An4DotE4ARtJpzM/P4+WpKSSTyVbsXlfhn1dQhZhH8DqAwwAeg5gucT+AP9S2lYiu6n/3dxcauKfMSuE95zaA4wDOAviq8/wCgLsBfAveuQYWxLVRAnAAwI8A/DcA4FkHy2TPfffh1a9/Hf3wHvH5r38dD9x7L86fPw/AO+cFEGcDaNLxb7i6CeG9o6O+9q2JWIzG8vlW7lbX4fqPXySRCSTnG5ssAH22gSxSEhlEHEzsDHbu3EX+2MAVHndfsDtwjvSMsWw2x5b6EnHddsEr/6gJkF1rGQBiCtA3Xn/dN0vrKSKc+NrXcPbs2VbtWtdx7NhR7No1CjGN7GsA/sB5xWQBXAZwq7PtO53ftwD4AACeWtYpxGIxxGJXw12HDgG4VvkNBK9DPwrgTahr2G9+829x770PNHivuxO56geCj/jCwgKKhQL2xeOeOXUfi8dRLBSwfv36xu9ow9VNALL6OKyhkwr3Vl86/jRDWWOgty9IOs8TebuVPswZRB2EP4CsFxqqv2V6qd6GhKvMV4p6LAPbtlteR3VF49VNOK9AWASSWe31arWKPffdh9L0dO25YqGAo8eOcUyhTu67bw9Onnwdwg+8H+KoHwXwAMTKX7LZeR4QM45PO38fxK5dRRw7dhRM++OuROU6VD5+h/O7ADGFd7/yriEAFwBcAeBtBK1hz50715xVahexYcMGFAsF/LvpaeyFkPI5CFn3EIDrE4laTODlqSmcPXsW586dQzwex8LCAn70ox81R9Y1ReUYOH36tJhkBn9XUktZgbQ63arT8a8SdYtA1hhYTizBtRQsa4DS6RFeDXYY0ZbBEHkH4cjWE9cT8JDhve6Ma74Wlsbc3BzF4O9Qeov2uJX1Bi1TBsVCga6EmACkfuk+gO5yBkC3Q7pVp+NPMwyaYOb/u1Ao0tzcHLvnOhBv0oBaaHhTqBvIvQauI30ITio1yEHkJSLvw1mIwtoJgGx4p5rJhe5gKtWSBXBLlIEU8lsClMH20VEiau+mTp1CcAHSARJZJat9K8Q1a5I0NzfHow87GFMX01RqUHkc1Kl2P4nWIwnSK9E5brR0Ihe22uODLVgAt0QZuKtVcXBsgErOb/VLs2WwMvhXiWKGcdgKcXj4dp5o1gXofbxkCmOwZWCTeYwmB5GXi7EnEbzDulRLodkL4JZaBvWs+sfyeUrEYp4DyLUIi8O0Snz3u9dGrBDZb9ytFApFsiy9lqSfRG0BOec6+Npgi3xpmLKFNsE7rKuVlkFL6gw2bNiAHZkMAG+muw3gSefvG264ofb8z4g8We8/I2rOjnYJyWQSU1Mvw7ZtlEol2LaNgYEB51V5BmSl6qTyzlsB3ANgI4AiRPaJhTfffLNZu84sA9u2cfz4cV/NzrFjR7F9u15L8lMAH3S20KvWJSLX74orWp6E2JEkk0m8+Jd/iVw2W3vuewDyAP4Wbl1Bcs0a/HEr6g0apmYiqFarNJhKUT9AX4CYZQDlZzCVorm5uUhXErN4XMtsyPENewOF8fiVyuv+rJNsNtfqr8CEYGpfnU4P+7qPZjI5sqzVjsWXJ2/luYwZmMZocvxoqRhHXsLNMtJ/y5+uziYiIjpx4gRZzhfvB3wH6Nabb67LlcTUhyzcm5iYcC6yMyTSCb2BQstKUiKRYr9xh+LGiA5T2CAiv/vQNPbS9HgTWVaC40eLJCoGujoWM85Fbta91lJlMJxO1y60sGE3HEBeHubhNiBzmmGZ3GEn7DfuNLzZY7KmJDwJYG5ujtLpYcP1cZFEQFlWKct40uqaYuB7sH6isiP3t1jOtbQ30bffeKP2+Ffg9lYE3J4dw+l0a/t1dAFuBbLaLzEOQHZC3AHRzVTGBw46z1sQZ0VF+I25R1F74lYf/wpE99EnoXb/Wlh4AtPTJU8c4dOf/kN85zsXIK6LU8qnvQJRiX6381v2B/gDAG8B4G6mi0F2kzVHYoBfd37L6N07ncdNO8ZNUTkGpJZU/WXypwjQYefv+fl5nnu8DMx1BmXHJbAmdBUp3Ed9pPqNOb20vXHP9/66LDvz9THmnPcE+bvXFknNOGPLYHEEjby0HJlX1GShheZNmmuZMpAXoSlekIQoPss5lchEPPd4qZgHnRxRBMAYiWrTqKpULjzrFETqaHgdibyPzNdHlYAs+WMIRec18RmZTK6VX7MjMaWX5jIZGsvnqc+RfXpL/66edCbZslmMXgyKCRw6dKiVu9cVmFd+RxQBMEZun3vzKvLQoUOsiDsINzAsR1wGW3bhIzJBwEYSMYIDpGYVcWuKpVOpVCiXzXoUwntHR0NlYTPuvZYqg0OHDhFCAip6G2tmaXgrkM+QGzjWWxqbBUIikeIbvwOZn5+ndHok0LKT83g3b97iWBIml1CVeNDNymJKL01aFlkGWXiqibKwpcqA2000B28KoVwtbiLvzOOcbxUpBMIQuwQ6HNu2a0PYbdumSqVCo6PbNReQRe4ENNUl5C4K5PuZpRMl82TlcQX++EGjY6UtVQZE3G6imbz00kuKBTBJ3vGHj2vCQAqEM7XHLAg6D1Na8apVV2vneYjcZAFLee6M0bXELJ2o9NJrYjF6GKBRQ/yg0Z1LW64MWj3dp5fwBgulr3gj+WfljhAw71kVqhkoTOfgughlltgmcosMT5HIOuonb4HZQee5WM0q5PtxZYiyDGLKfTgEc9+iRi3KWq4MJJwt1DjK5TJNTk7Stm3Sf6wOuukjvQJZtCXI+1xFfG46i+AhN4cpfKbFJOnVx5xFtnLkMhlKWpave+mQYw3kpSUAc0fTRi3K2kYZMCuPufI4QW7++Fc0YWFKKR0iYA0lEtfzHOoOw582Kh/nyVxTIuMG232vs6to+VQqlZoXxFRbVYV3vkHQrIOutwwkLHBWDr+L4Khzkw+SV0GEtbGGx2XAK8XOIdgyCFsArHLONfekWmnULKIXnPPwgiLsPat/eGcbyOK0ro4ZSFStybGD5ROdP446BMMEuWmn3olovFLsDPyDjd4TsQBY7VgIs8bXOW60NPRYQdm598Imn72o3addn00kkVlFevUdZxUtDXNlqXrT7ydvQFFvVWyRN5toP7lDbmxeKXYIpsFG4QsAqfw/zJbBCmLKIipCxAXU2MEagLYCdMCxBHLZbNM8JW2hDLjeYOWJtgxs5/Ek+dsO5Al4htTgMTBq2IZXiu1OpVKhO+7YbjjHfSQCyUF9h67SFghJAiy+F5eIScZVIYLG8pzocYR3DAzQhQsXmraPbaEMjhw54tOapPjM9u3bxxfhEjDNPvaON5R+ZBkP2K8oCVVxWI4w0DOOWDi0M5VKhVKpQUfw6+dPWn9qTUlVOed6zQkr/+US1KQulUjQlWhtXyKiFikDPUgslUHYTAOOISwes4vAclb9qjKoJ5DMAcVOI5PZEXn+RBW6t++Q6FVVJNdlNMPnewUIalIn/w6Sf82q/G6qMggKEs/NzZEFUELzn/U7ZlSzKvC6FbWGw28tqINsjpIpLhCmKHil2J64bsIoRa8PtbFIuAQPk1tv8hhZ1mpuSbJCqPejG9sL9ox0ZQDZ1KApYVk0mBIjFnWf2RCaW4HXCwQHFLeQvxK5jzZt2syWQQeiCplwy2DWsQCuob6+aw2KgVOKG4mqtIMsg9kmLYabpgyCgsQ3AXQNRIOmi87vayDKsnn2ceNw5yDPUnAlcj9ls3c5fmdvxhGnlrY3rpAZIjdmYMoYE/fUmjVJqlarnlVrNpsjy/LGGvi8Lx/dTV4sFGqzDFTPSBLeCuSuKTrTU6sqAOV0MwjeKryDMGtKXo0uH2+20Vzo6jEWu4709gTcz779EUNuEo7Vp2cTbSTgDwj4EAGraevWtOe9UdlofA8uniA3+YULF2gsn/d5RvKaZ6Rr2lHolkERhq58zvPyS6/WNCXHDFYW0eu+n9y4QZBf+T0kMk2mCRgn4BMsEDqAYJfgZ8iNDU3Xzr96PqPqVNg6XzwmN7kq02zbpomJCXr00UdD3UYdbxkQuQfjMYT7yA44v1fplgNnE60oc3Nz2ooxyK98LQHXawLFosnJyVZ/BaYOpOtn8+YtzrnL186hek7T6RHP4Bu2DFaOqFoqPWMoKA21K2IGRP7UqsCe3hDB5DxAnwFotWXRyLZt3LOoAezcuYtEPxr/iERRiGQR8FGDm6GPRke3t3r3mUXgKv8kCbeft2+VZXnjAaY6FY4ZLI2oWip9wTs3N0fD6XRTF8MtqTOYnp4O1ZI3AjQPN7YwxBZCw8jnxxwl8Azp4w1FqwoQsJXMba656KyTcFf74WNO5Tk1uZk4m2hx6HGCqIwhNbtS/oyk0zQ/P9/wfW1ZBbJqBp0C6GGArnMsAlP/jmZO/OkVzK4Am7y1B/JniNwxiK7g4DnV7UVY1183DvACLSYewLNGlo4aJ8gjOmNoCKK+qhXyrmXKoFqt0q6dO30RdAugZ5wDE9nZjy/OZVF/MzvZ+rro24aVQXtgml1RKBRpbm6uNv/4ueeeq8symJ6eZuG/Auhxgir8c43VjKFWy7uWKQOigCK0WIz6nAMge35zvUFjqL+ZHRHwmPPcjGeb0dHt7DZoAzKZHWRZq0mMrJQKPEFAXLPw4uSOufT3rUokvIkCmUyOJicnWTEsgaB5x7PK8d0Pd55BqcXyrmXKICq6rv6wZdA4opvZVcg/InHIETRDBPRToVDkoUQtolKpUCaT086P3nROVRDXKgpCTwqIO3UJ5oZ2HC9YHEEy7gvwd1sYBej+XrUMgrSm1IITExNUKpUol802PcWql4huZlck84jEJAHzBHzBJ1RYaDQPUVimdyQdIGAHubGBknKL/baiIEokLL0SCZdgmJV4kDOJloApRbTP8YAcBegM/AkyfQAdboG8a1vLQM1o4AlojaNcLtPk5CTdfPMtzvH9sCPcB8h1DYVNS7uB9EwjFhrNwezmq5BeLQ7kyA3+/ymZ40ThQWWhMLjGYLGY5Jcq90wJMv2a5TCWzzdF3rVUGVgwR9ct5YKT7oeZmRl2Q6wgpoCj+DlFrisIEQLiQ6HKgs9VYzEnAJgsOTX4H6TgoxS/TVx9vHRkRpbsCXYR0QHjDwPUb1lN84K03E2U14SRfDw5OckWQQNxYwVSaKgZJlKgfCJCQMhmd9yyoBUED7wPG2kpYwAJ8saJRLBZuJz0wsOi53NYyS+d06dP1xRAZMAYzY2PtoWbyHa+uDoEWsYKuL5g5QnOIhpyhMVh8rYs6A8QENyyoNV4EwCiXD1q0Fh3JYnH2awejB4iMQebq49XArVDaVRbHhvNzZxsi9RSPTi8w5n+o6ZdeQ4SC5llEVxfcIb8GSamrBOZraIqEG5Z0ArMCQBBlsEzzu8REgkABxwFcsBzzmzbpsnJSV+WEicGLA+5CHsGbr2BBREj8MhAuIVoPWEZEJmDK7t27qT01q2e53IQhRlcX7AyRNcXfIIsazXddNMm5TzMkpiKpVsJawi4koVGizl58iRdd10iwJJLkphpIK2BQySmmUWfM64+Xjn0DMo5iEwiPc30FogsIzntMZfNNmX/WqoMJOoFVywUKGFZvnSrQYh0K7YMVobg+gI3p3x0dLu22qwS8A6DG2EN9fcPcHFSC3EHEJl6TBVJpJLqFp5Fhw4d4nPWJExt/GUm0azjCZEteeQ5soCmdQduC2UgmZqaIgC0CeZ0q1UQA6SZ5WNyL2Szotp0enq61mXROylLZpzIHHVZocwxglYi7xvh9pHnRe0xNeMoiH7FNXSQYrEEu/MaRFARZr1t/CfgtvJv1n3VFsqgUqlQzokTyJ8hmOcfw1EI7IZYGVSrzJxumiXgLm1VaQ5Q7tu3jxVCkymXy/T+97/fsOovkogBgYD/XVHs/qAxn7OVI2iaWbVardX05LLZ2mtBmUQPo/kJMy1XBpVKhQZTKX+nPni7+el9v1OJBLslVhh/uqlsVV10VpofUFxGQUVoHDNoBl7FLeMEem2BKvxlIaG+jcUxuBUkaJqZry31tm2hloGqRJpFy5WBzBwKS69SH/9b8HyDRhAdVP63BNxE7nAUPUBpkQgycwVyM3AVd3gHUuA65W/zNjMzM63+Ol1BVFeFgwYFYcqmHEmnW7LIbakycAWQ11wqw+1Y+gK8fb95vkFjiG5nLX/WkGnqGTDmEzJstTUGr+KOOm/PR27DlsHKENVvrWRQEKrLqNULWwst5Pz587W/XwFQBXAPgI0A/rnz/L8AsAfAbQD+EEAJwJMA7gdwo/P7iYUFlKancfbs2Wbtetexdu1a569XtFdmlb8tAFcA+HPn+f0AVgNYA2BS2S4HADh37lwjdrXnce+bHQCizlsmcpt169at8B72JvIeCjoT6lHOOb8/+alPwbZtlEol2LaNl6emkEwmG7ynAbREBTnIFc6Qs9qXv01Nm+oq3+YVzrJIp0fIXG2spirWM/sg2jLgltdLx2sZTBHwbvK3l+gnETOQ56Toc++xO2/l0QtpH4OY6b4pwHXUTtd/y2MGsq7gJoSnWq1GHeXbbXRgOxF3YLqelXJYeRzkjniY9HoFOWlLFfpBE7k45lM/5XKZNm/eQsAVynE0ue5kvOCicw77+Lg3GLWQVi8mG4JbTNaOru2WKwO9Cjlo1b/l5ptrB7gf8AVd2u3AdiqiP36/I9xnFeEuK1bDM4nET56A/8cofLLZuygWS5Ca1RKLJSifH2v1V297ojOI1hCwlUQdyIBjLXiP//z8PFtkTSCXzVLSsgJbU7dj0kvLlYFkeno6ctUve6boQZdcJsNppitE8LCbz1B4JhFIDEiR7iLplnCFlVAyJoUiitk4qyWc8AyiMrlFZrZPSfOs6uYRlVXUrtd52ygDouDGdaZVf00xaMVq7ahxOxG1GE1YC6uVVb9uBegupKA01Ye17fwjNdl1IdBjKm6c4AAB48pxNI0lHSGRCuw+xwul5hGZVVQq0dTUFO3du5ceffTRtjk3baUMFjvVLKjAg11GK0u1WqVMZoci4G1y2x6oK9CDznX/uPN4VlMGpzQl4R/E0utBzaCYynPPPUf+uMAQiZRe01hSd9tUapAVbBOJsgwG+vs959EC6Pbh4ZZ7N9pKGUjq6ZRY79hMZuXIZnOG4Sf9mpDy+qm97a5fJNfXHT5Za35+vtVftyWYqsDj8QG69tp+8scI5LEPiuNwEWCrCPJyXL1qla/bQj+8weZWeTfaUhnUQz2mGLOymOMJfSQ6ZV4kd7aB3s5iKwn3xgABW6iePkfp9HCrv27TCa4CjxpJqVtg6txiLgJsBSYvR1QLitkWezdaWnRmwrZtHD9+PLKALLLAgwtpVpxkMompqZdh2zYeeeQR59nnAHwEwM8AvAXgaXhLAp8E8B2IAjULopzwsvKp5jP4xhvfXvEiwnqvrVbx1ltvOX/t0F4ZDHg+5/z+c+15vcxJbMdFgM0jmUzi5akpT0HZr//GbwAAFgCoV6A8i/8DLS6ibbr6CSCs25+JcrlMw+k09VsWp5m2AH/7iqi2CPs1l9IoAbdQ8CAW0UBtJYrTml3bELXPQa974zKLsQxk/OAMua67nG87tgxaR6VSodsdy6Am3yA6M+t92Frl3WgbZVBvMPj06dM0kk57Dmo7+Nt6jcUPY1eDzTIPfoyATZ5zqRa5rV+/wSjAF6sggvzwK+1Hr1QqikD3K50wpeQezyHn2OhV4IOG55Mksrn0oLFFwEPElcbtg1G+QRSiyb5r8qZpVdyzLZRBPcFgOfNAFp2pBzVpWS3r9NfL+KelyZiBLsiK5Pdlm4rVUgScJ3/66lYCPkOWlXAmetW/wo/qxrpS10ylUqGBAfMUOFlQ51dKBygWW03DwyOKpXWG/KmiRQIuBDxf9Xwf4BO+48rpuq2jXC7XBkUFybdVEFMcW+3daAtlEBUMnpycpMFUilY7j4MOaq9moLQKU0A5kbi+DoFlk9d9pGbHXO+seA8blEKCRJVt/Sv8qG6sK2WKZzI5CpspIIsqxfP+2gCRLaQqLZvEzOk15FWu1zjbBQWNr6Gbb76VZmZmuNK4hZjc3kHyLaYp757OJoqyDGQUfn/EQR1Op1v9VXoSPRU4k8mR6JujCzLVSjA1uFN94/4aBH/ztegVfiMsA91N5Y6dDPbpf+QjH1GUUtB3u4JisWtJZF6Z+wlFtQVJJFJsBbQBqlvoVMQidhZi1sFqy2rpWN+2UAZEwXm5aoVx1EHlIFl7UK1WKZ8fI3ORlBrkzGsC7QVnu1MRwnVCUSJiRRzWbsHvznqMLGs1ZTK5RX0vk8/f67YKngXxznf+qvO3qZVEhfwjKeWxGyJgnrxFfrJWw1vvkUhcz4qgDTAtbosQsQFVvrVLrEDSNsogqPp4cnKy9vgo3OE26kHtByiL1kTgmWBs26aJiQn6vd/7PUqldH+6RaI+wZQ1sz9SuLouKLejaiazw1jF6XVneRXUYvzpbhO//SQLuoRQvilAeT2m7W+CXDeP+t1MlkKSgKsNQn+AxFxq7/fIZnOsCNoEk9u7ClDecy2Ix+qc91bXSLWNMpCoLgc1+DLkKIHDhoPaB9AWtgzamm3bbidgtSJIhwyCLuEIOSkwgyyDUUVg9jnC0buyTqeHfTEkt4J68VlFp0+f9glgrzK6Xfk+Z3z7A8RJuM10xRGVhRUL/J8cF2g+9WSyRbm9P/7xj4e+3vOWgYop+HIlQBs0JaAGXiyAto+O1j6Dh6e0D2a/fdUgMGMEXEnuCt60KpbvUVNVtzuv+YPO/tTN8NhB0HXjDv7RA8Tq/1NTO/Vt1XhHjtzur9I1FmQFrSGhRN2W4pwq2nwWWwdlcnsnLIsGU6mavOpHe7Xib0tlYMrJTcRitXqCrfCnl/YDlN66ddEnjWk80fOV/yl5rYWjZF5dF53nQcKHrrqNzEFnKTjdfXjBUSRl5zNmCQA9/vjjlE4PL0mRiJ/POP/zQefxwYBtbRKKUE8TDfrs1SSsCq+LLZ8f42u6iSy2KWa1WqWxvHdhEgco4bz3DIS3o53kVNspgygTS40fmF7PZbPcybTNiBamUjibtpEB1xnDe1RhfCr0f1x99XUeYep34aiP8wQ8Y1AkujI7pbx/l0HAq2m1er8g9bsFWUHF2v7HYqqLjZvPNZN6m2LqVmWxUKB+y6L9zntNn3HAeb4dZhy0nTKIqjlYv3Zt6Ovt6ItjTBk9ekZRlPXwAnmrbuX73+O8/msh71fdNnnSh+64LhzV/eMKYm+NgMz+0QV/nPxBYFMqrUwbld8lR8Ck89ukSNR6jGD3FtM46qmD0r0RO5wsSCmLOmF+e9spgygt/NJLL4W+3u4HvFcJnqAmM4rqccXoq/qrQj5P/qi1C/VYKOpj4UI6cuQIrVmTJHf1risUU7qorgD6SWQHqfvbRyLOIZXDGhL1GTnDZ5iLzPiabjxRMimXyfi8EastyyOLymj/hWrbKQOi6IlnQa/r2rgdD3ivo09Qi2pnEY8P0PDw7TQ+Pk6PPvoouVZCjkT2kb7C7yOv9aFW7EZZHyXt8QOa8lH/VgV/1OfC+W4JAjaRZa2mrVuHKDw7yQ0Wh80s4Gu6OUTVQeky5zFN+JchgsamWgOrTc5jWyqDqIlnYa+bTlo/QIMprsxsN0zWQljvIe/ox8VaESDXFVOvZQACVhGwl9y6AhlfUAV/lMXhjXdkMjmlfkYqKW8Rnfr98/kxn4uNYwbNJaoOSvVGnAZoI0S2Y78jg15w3jPquR7dNPl2sPDaUhlIgiaeyUCNKc+6Wq3W0rfkzxBEWhcHkdsT/TyHTbrzzmMOW4m/h8yFXDeQm9apxi6GyOvL99cuiC6rawMEv9+q8cYMvK6dqKD6xMRE7fublCY3n2sN+rUpW5EcBKgC0Jgm7G9Q/ra013IQdVPtYuG1tTLQqSdtVN5kByGCNrJHOLuKugP/POYwy6DeQi6Tu8Y0WzhBwg01RP520gkSbabVzxkiN5vI79oxBdXDVvz1jINlmoNJFiXgT3kfcIT+WsNrSYiC2XZZpHaUMshlMpS0rNC0UR6H2d24/YFM6ZjuYJxwy0H2NnLrDIB/Qa7bJsrtM0/+bKKcI/jl523xKQxd0POKv3PR6w5kimhYYku7d1vuCGVQqVTqDg7XmxPMdCbuavoZEq4c7wp/aChNf/RHfxQhzG3Dc6o1UW+g2Sa3grikfd486W6mIEHPK/7OwiRjolJHO2GB2hHKoFgo+FK1wg5oVDYS05mY/ew2AR8iwFu4I7qm6paDKdtIdPt88MEHSVgUAxQ9ZtKkTGZJr4OIxwcok8mxoO8yTN6HqNTRTligtr0ykAIgygzTg8jckqL7WMygGpMLJp8fo2TyBs0SGCLLStDmzbfUHrtxhPqUSVD6Kbt8upMg78MQ/P2G+iECx1evWtX2C9S2VwaqFja1r04qWUIyy2h6ejow24jpXJYyqEbvgivef5C86ZxqYLmf3BbamzXFcQvpwed8fozm5+dr/4NdPt2NDBybGs0lLIveMTDguT5kA80LFy60/QK17ZWBqoWrjkLwpGdls3T+/Hnfgbba9IAzy2OxGTgqUZbFxo03aav8i+QGmt1Rnfv27fOkfjK9g3RBPwN/K30pa+QcD9M10s6LhbZXBkT+GMABiHLvkW3bqFQqGcvBBxyzrd1MMWZ5LCcDJ8qymJ+fd4rewucptOONzDQek3vIBuhhg6u6E+kIZWCKAZgKy9SpQTKWcKBLThTjZakrrCjLolqtUjabM8YMuOq3t+n2tPWOUAYSKQCMbarhnScqT9ALXXKimJWhXstifn7eGWjDNQCMoNvT1mNEROggbNvGxo0bcRTA/crzRwHsAWADWK88/gSAx5z3rV+/vtm7y7QpZ8+exblz57Bu3brQ66Le7Zje4J7du/H6yZN4YmEBOQCzAD4Wj2N01y68PDXV6t1bFh2nDI4fP45isYiLAG5Unv8+gHcCeAGABeBjAH4M4BrLwo6xsY4/UQzDtJ5Lly7hgXvvRWl6uvZcsVDA0WPHkEwmW7hny+eKVu/AYlm7di0A4BV4LYNZ5/c/d35vAlAFsOm223D02LGm7R/DMN1LMpnEy1NTOHv2LE6dOoUf/OAH+KVf+iX86Ec/6nhl0HGWAWA21R4C8I+d358D8D1l+27R3AzDtJ5qtYoPvu99+NrXv47LyvNj+TwmX3qpY+VMRyoDk6k2mErhB5UKLADXAfi/AbwDwA8B/LHi07NtG+fPn2cfMMMwdaHLjFw2i2++9hquAfB5ADsgPBUPxWLY/mu/1rEu6Y5UBhI9uDczM4NCoYAhAG8p28nHt2zejL/97ndrz7PFwDBMENVqFXvuu8+z6Lw+kcCPfvzj2uMiRLJKEkoSS4cmq1it3oHlsH79etx99921A7+wsAALwEWIE6P+tgCc/+53Pc+/fvIkHrj33pbsO8Mw7c2e++7D6ydP1mTGEIBf/PjHHhnyLQC/5Wyfc36fO3eu2bu6InRcADkMy7JwGcCTAEYA/AcAtwN4AkJj74UbdL4fAC0sYM/0NM6ePduRmpxhmMZg2zZK09O1FHYbwrugprTfD1FgsAfAXQDksnLdunVN3deVoquUweXLIpzzPIAHlOfzzu/NECf1PIB18GpyVgYMw0jOnz8PQMQDACEz1McSKUP+PYDXIVzPnSpLOtpNpLN27VpYAN6E1030JsQX/SyAjRB+vg0ACs77rriiq3QiwzDLRE1hB1xB+Yq2nUxp/wMAPwcw/uijDd+3RtFVygAALgN4CsKEu9H5/aTz/PfhVRL/H8QB+LVf+zXcs3s3Ll261JJ9ZhimvdiwYQOKhQIesizcBmA3hKz4KITskLLkYxCLyw867/vhD3/Yit1dEbpKGeimnUSacv8nvEricxBK4iA4mMwwjIgVHD9+HGfPnsXRY8fQl0ziP0EI/rcgapn2QHQ72ANg1HlNWgidGi8AuixmEFWd/Ova9lJJbAbwBAeTGaZnMaWR7shk8INKxRM0/i6EvPjPAMYhLIKXIYpdc9lsR8uOrrIMpGm3Lx73mnLxOCznsUpNm8OfFqauEBiG6W5+6zd/E6+cOIGDcN3Ib3zzmwD8noZJAP8AYD9cC+GqVAr/5m/+pnk73AC6ShkAwNFjxzC6a5fXlNu1C/9bPu9XEhD+vvVwFcP111+Pe3bvxsaNG1EsFrFhwwaOJzBMl1KtVnFXNotXXnsN//3yZTwM4F9CyIXfd7IT1aCxDeDPAU8bilwmg++dPdvxxasdXYEchl6d/MYbbyAzOoqf/eIXtW0SAL4K4D/BbUMLAN84cQIfuXwZvw6hOPY5rz3+5JPcyoJhuoi7sll85xvfwOeIam0l9kHEAg4DeBeAfsvCH1++jL8G8DXlvSPpND7/zDMYHh5u+n43hNaMUWg+g6kU9QN00Bl4c9AZaC1nJQ+n0/TFL36x9lj+5AD6sLKd/OHZygzTOZTLZc9kvEqlQjsymdBhNR9S5qxbjrzwDNTqspG6PaEMpqamQk+6/DGd8H6A4j1wITBMN1KpVHwjc4uFAo3l87TasnxjLCvwD7qPUhqdPuFM0nUxAxOnT58GEJxy+k8hYgiXATwNb/rp7wFYMDz/xMICSk72EcMw7YneX0j2JPt3X/sa/sCJCTwFQN7Fe+AvWg0KJHd6LyKdnlAGd9xxB4Dg6sEvw21foZ/wwYDnu+1CYJhuQ/YXenJhobaQGwHw2wsLuAxx3wPAAYiOBHcCKMFftGoKJAPdUVug0hPKoFAoYDCV8lUPPgTRifAixAUB+E/4DwKe77YLgWG6DbUItQrgHoh2NAchBN/34LUAzjjv0xd+/4ez/V7L8qWsd3IvIh+t9lM1iwsXLtBgKuXxBQ4BVFV8gENObOBFx4/4IkADAA06vz3Pc8yAYdqacrlc8/UXnXv4KECnnPtfjwE8FvC8jA3kstmuTiLpGWUgmZmZoQceeMAXOCKAzhiyhoYAes353c0XAsN0I8VCgfqdQLEU8iXnHtbv/4vO/Z+0rMCFn23bnqykbqJr6wzCsG0bGzdu9JSZA+6koi9/+ct4++238fmnnsLsa6/VXs9lMvidvXtx2223dY9pyDBdzKVLl3DHyAjOnj+PUwD+J4A4RMfioPs/l81i9tVXa8/3ykTEnlQGAHDP7t14/eRJPLGwgBxEDOBjyqxkiV68xjBMZ/HSSy/hg+9/v6dqeBDA/4LoaGy6/3vxvu9ZZXDp0iU8cO+9nsZUvbICYJhuRh9gf8/u3fjG9DSehju8/qMA/geAt5X39fr937PKQGJaAegXE8Mw7Y+p82guk8Hsa68FuoRu3bIFn33sMb7XwcrAg+li6vXVAsN0CtL1++TCQs0C2GtZ+G+XL+PvIOoGJN+HaGQJiMVfrysCoEfqDOolqFrxgXvv5ZbWDNPGmArM7gfw5OXLuAzgS9r2s8rfXDgqYMvAISrDSIWtBYZpL44fP45isYiLMFsA1wB4Bkqw2Hn+LbBlIGHLwOGtt94CENx2Yj8WZy2wJcEwzUMfYC+RFsDCFVd4Zpy8E8DfWVZ3VRAvl9aUN7Qfd95xR3hnwpBup1CK0IK6JHKBGsM0llwmY+wgMOTchyPbtvF9GQK7iSBW8Zs2bsQqCHNSzT1+CMA2AP9O2V6anvsB7IUzEEMZjqMHsfYZ6hcYhllZvvSlL+HeD37QU09QBPBZALcCKJVKWLduXc/VD9TLFa3egXZgdnYWlyGUwN/AGyOwANylb+/8fhBuoIoWFrDHyUJS4w7qa2fPnuULkGEaxNDQEC5DNKLbDDHbfD3E/QigpgD4HjTDMQOFuwG8DDHntAQh9C8DeEzrVvgQgDzEhSbJKX9zu2uGaT4bNmxAsVDAH8fjqAC4Cm530R2ZDM6dO8cxvBBYGQDI5YS4lsGn9RCK4aLzeNv27Z7g008BfFD7DDVVbRLAcbgDM7jdNcM0h6PHjmF01y7P/boqkcArr72GYrGIDRs24K4dO3Dp0qUW72n7wTEDh8FUCj+vVvE5eOMFfQMD+EGl4qlU/vjevca+RukdO/C3Z87gB5VK7XOHILIWto+NccyAYZZJvd0B5P36B5/+NL735pv4w8uX8QG4rSiuSqXwvbNnOT1cpbXx6/ZA9j0fMrSvhmHGabVaDZyr2m9ZtB+gWbgzlAdTKc5aYJhlUG+Wnhx8Pzc3RzlndnFte2d+SW0+QSbTom/TnrAyIKJSqVTrb247/c5t5zEAKpVKxvepvc1Pnz7tm4VQBOhwgEJhGKY+yuUyDafT1G9ZdNS5L49qcwZ0ZWE5CzHP9s49eVG5R/m+dOFsIngLVu6HGxhWsxBUpqencfr0adx55524++67AQC3b9uG6wBPZ8R9AP7Bec+5c+c4i4FhFoGpV9hfQqSLqll6J06cwKE/+7NaK5lfgcgAfBpaVh9EDGFS+R98Xyq0Whu1C8VCgQbi8dDRlufOnfONzhxMpejkyZOhBWuIWIFI05ZXKUyvYroH5D1pWt2TtsJX77+wSWYAaHWIC7iXYWXgEBQHUH2Sg6mUz/TsByi5Zk3oxTecThOR/4LnamWm1wm6B+bm5iI7Asi/P6QJ/7KmHPT3bgIoYVk8w1yDlYFG0IzTqampyNV/0GsnT570XfC5TIbG8nn/ykezRhimmzGu/uNxGk6nQxdYDytWgkn4FwFKwtuaoh/ujHNedPlhZVAn4+PjoRfn+rVrA91MxUKBEpbly1YKXfmw+cp0OTKLbykLLAC0A6BJx0rIO8Je3n+HAerT7rVcNkuTk5N8bwXAyqBOoiyDL3/5y6Hm7pCzkpEroP2aaasrl6AMJobpBsrlcuQCS2YEqQuspGVRDKDNmqC/BaCY9lyxUKD5+XmOx9UJK4NFIGMGuuk5mErVttHdTDJtVVckUX5NvniZbsQUIxiCyP/X74Gvwl/7UywU6B0DA8bY3WAqFejmZaJhZbAILly4YMwmunDhQuB7pCkMgE7BrWEgzbQ9BeEHXROLUS6bbeK3YpjmYYoR9DtCv7b6h5sxRIoV/cUvfjHStTQxMcGKYIlwO4olcOLECXzrW9/CnXfeibGxscjtt4+O4vTp057WurdDtNV9DkAM8LwWAzC0dSuOPPsshoeHV3LXGaZl1DtN8AYApwH0O8+XlNeG02l8+403AieaSXga4eLhRnVLYGxsDL//+79flyIAgNXXXovrAM9s5TKAL0IIfvnaGYheRgTgze98ByMjI7hn925uqsV0BbOzomVjUFff34VoP/02RF+wPQBeh/e+OetMJAyaaDYL7zRCZhG02jTpdhaTMVHUgsycasp0A3qcIGqaYFQ2keXcF57AsuZa4tjb4mHLoMGcP38eQPBqSL4mZyg8CWFCy6E5TywsoOQMxmGYdsc0+3vPfffVWkXkIaYDqvNBPgbRYkI2hdDvDZUchEt17datnjbVt8FtH6N+Bs8QqR9WBg0malC3fO288zcPxmE6kWq1int278bGjRtrcwPu2b0b8/PzKE1P48mFBdwP4CUAdwIeQX4ZwOeUz9LvDRhe+4u/+ivYto0jR44AAH4bQNKwHc8QWQStNk16gbBB3RZACYAe41RTpoMJqiQeCagknnWu6w8bson6nfvCAigRi4X2C1P/d9R2TDisDJpAtVr1paQOAbTG+ZEl8qYiG76omXZnOZXEtradvDe+AtBe+KuITW0k6ukrxkTDqaVN4tKlS/it3/gNzL72Wu25OIDVEK12b4QwoZ8B8L+U93GKHNPuHD9+HMViMTDdc8vmzfj+d7/rmSL4MQCjEDPH1bTQz0BkEKnppDEAjx86hGKxWNd0s6gpaEwArdZGvYZt23TkyJHIFRMXzzCdQlSrlvn5eUolEt6VO9yqY9UyyMOfUdcP0IjT+ZdpHGwZtIiRbdtCi2dKpVJtcA4QPfu13tmwDLNSqMNnLIh6Gc/q35kLfuWVV3oG1FwJkTVXVLa75c47a1ZzUFGabdt8bTeSVmujXiWyX3vIzIORdJrm5+cDX2d/KdMM1KDxGZj7CJnatMsAsX69RrWt5uaNjYWVQQupJwsiqJeLFXKzqZ9RLpfpyJEj7HZiVhQZNN4Pt1iMADrgCO6ZmZlF9xGqd4HENAZWBi0kKgtC3kwH4G1wJ2+O1bFY7eYpK9vI1987Our5bAugsXyerQZmWVQqlVrKaO26dWIAchU/MTFR69gbtNIfHx/3CXhOE20drAzagKC2u5OTkx5zWt50Z5y//4kSdFO3ycNNU1UthiREqh7fWMxy2LVzpy/lsw+gMXiDwTsymciUU30BxGmirYMDyG3MXdks3nrtNTwNUZn8CoB9EAHmtyACa/8MorvjU8o2vwPgJwgOxAEcjOt1pqen8fLLL2NwcBAf+MAH6r4WbNvGpo0bfdfcXgD/DSJVehiiInhfPI5ViQR+8eMf44mFhVpg+SEA7wbwFee9++JxjO7ahZenpmr/h9NEW0CrtRFjJsrfmoO5arkC0HsQPkUN4GBcr3Lu3Dm6YWDA5z68K5uta/Ut06KDrsst8KeM5rJZz/8bgnmYDccEWgv3JmpTohrc3QLgM5bl22YPgIrzd1g/pL//+7/HzMyMr6kY092894478L+qVU9b6H4A33r1VV/LZ1PTOUnQdfkxuD2C5HOf/NSnYNs2xsfHAQiLIGl4L/ffajGt1kaMmXpK/HWfrDpKswjR80hv89sHb1qf/Jv9st1PVHEYnNV5WDpzPRarbhnIFX/Ue9kyaC2sDNqYoMyKkXSabNumcrlMliPkXwToBcU9VIUI6Kk3tAXQlQA9AyUN1THbB+JxymUyPD+2i4kaQA/HfRiVzrxzxw5fA7kExFD6JEQCAzeV6zxYGbQxUZkVMnVPzyZSV142QB8KeE1dlR3QPoMthe6jHsvg+eefD93mOoj4Qs6xSmvXi7MAeTHiGuJsofaFlUEHEJR6qprdNkSdQU6xFNSVV+K660JXhS8ov+uZsFYul9mK6EAGUylPZ9wvwN8ZNOw6eQhuHYG8XmzDdhMTE6H7EXRNM62DlUGHo5vdhw0398i2bXVbBtOOUtkPt5JUhdtfdC7lcpmef/55Gujv97gO++HWoxyIuE5Gnd+PPPII+/+7DFYGHU6Q2T0/P0+Tk5Mec152hFSthn6A1kL4fAc1JWIKLhcLBeq3LNoPMaCE5zS3D0HWmkmB3z48TB/4wAeMAn0TDHM14O09NDExwf7/LoOVQZdgMrvlzSpXe89A+HZNAj8Bf+tgNbhcLBTo9OnTxorow4bVILuRmkeUtbbYKWSfB0Ir32POuWb/f3fByqBL0dP4iopVMAvQwwD1WxaNpNM1N1KUC+mWzZt9LS4G4AawS6VSqGBSFQQri5UjSNgXC4UlTSGTLsJPwtwTa/voqOf/s/+/O2Bl0KXoTcKqBqtACunJycm6gstRQsW27UDBpI/9VFeeuUyGJicne0KYrLQSjMoQkoHeoHM7nE7XXD2mNtRDzvPSpTiYSvHKv0thZdClBK0I1RbDUdua0k7DhErU5xxUFITMR9f723erm2GlA+/65wWdl6j2EfPz87XPMTU3VM9Nrs6WFUxnwsqgi6k3wFepVHwph2rQcCAep9sjXEnz8/ORLYtLhvepLbcPBuxfp2Fa/Ye5coLeE2ZF6PGgoPMyNzdnPrfacZ6eng79HD2rjOk+WBl0MfUG+IqFAiUsy+cikCvC6535tXLlGCRUItsNBCgI9e9OTk0MWv1HDW3RC7jG8nkay+cDz1tYPEg/L0HnVnf3RCpybmzY9bAy6AHCAny6YJHFa59whEDSKVaTPwlNqEhhNzk5Sbls1qgw+iEsjCAFof69EsJnuX75pb4/aPUfNc5xtWV53pOIxagPCLQi6o0H6UpInltpTehWSKgi70DlzCwOVgY9TtiK0ORDHoDIQwdAL730ks/f/Az87TGuXrWK1mgKwtPDxhFmyxU+y/XLL+b9usJYStaO6h6LsqTU4xL0v2QW0Be/+MXIc2tSuFw30NuwMuhxggSLaVaCLtxkJorJb21DpK+qWSkmF5SesbIc4RPll1+J9wcpjKiMrP5rr/UJ2qRlkRUmrEMEuCq4TVlAuUyGHn/88UWt9rluoLdhZcAYV4SrLStUuKmr3RLCM1oA0FcgBp9A+9l8001LWonrz9Xr5jCt6EulUmQAVf6PEUcB6gojFzHi8VoIC0kX2PXGWKRyloPm1erysCwgC/B1GI1SkFw30JuwMmCMK8Ko+bW3bN5cUwDqHAXTtno1ax7CndQPka4YJnxMK/E777iD0lu3ep4LqqaVCmlyctL3OXrtQ9j7h5UB8GGBYJ+bBaAdAP22PG433+z5riZFLGMGplW/fiyjsrw+A3+vKl7tMyZYGTA1dKFsElT9jhDVg5NBGS0yrVGPOxRRX3xAdd1IwWiMZcTjZCn7U4Y3EH778LCxR/8Q6mvQJvsxRSmcEUVpAKAbNEFsQVTwhg2AV7OJ1O+ah4i1qN+hHleTWoDGq30mCFYGTCAmQaUWHkX5rVXrQgpnNXtoVgosJZBZLpfpyJEjNDEx4XPdFB3BGCa0r4vFavuhr6Kj3DFDMKfOSiUTZQGp7qr9EO3EE7rwhlipq26acrlMExMTPmGtfv/TEf87LAjN6aFMPbAyYCIJcuMYlYXSWiJo+I58LAPMctTirp07fQLcchSNFMRRq/PEddfVVvwDjvB8IeI9MlB7xqBAdPdTWE6/JJfJ0OoI4Q2APvKRj9DI8LDRhVMulz2TyYYjvsNqy/K5p1YiQ4vpHVgZMMsmbPiOBYNrwxG6/ZZVE6LFQoH6DNtKwS6D1KfqELK6yydyRW/4DPkz7Pjk5cr7PPytvq9PJOjChQu1712tViOHCak/Q44iCurjpLYDqbdwTX4mp4cy9cLKgGkYkcPTM5naKjhK2O1XXg9anctVvMkSML1HKhr1cZ8jfJ+GW0+hCtiss53+ml7RG1V1LN1kagxFfV32cRoC6Bq4FpXpO4yk00QklLI+w0K1NhgmDFYGTMOot+hJbhe1kpbB1MPwu57UiltTMLgKf0xDX+HHIIa6B63a+519UF1QqgWTy2Y9338sn/fFIGoxA4OC8Pj3lf2+yXkuaB7F/Py879hzeiizWFgZMA1jMbn/UZbBzMxMreWFx42TTnuEoQxqS4FtmgV9VSxGn4CbRbTasmjL5s1kGYS8adVez3ciEspAT+u0IKyLqkk5wu+6kq+r9QLqPAp2/zArBSsDpqHU2+JAjRnoOff6ttPT0zQ+Pm7spKkGtfVgcNBMhWKhQH/9138dLuThtVKirB1VEdoATQD08QhFcgDhfZx0RcjuH2YlYWXANJR6WxxUq1Uay+d9Anwsn6e5uTkqlUo0NzdXd7sE6SaZmZmhUqlEuWzWV2eQtKyaa6ee9tuLsQyCPu8GmBv5ye89mEpRQs8MUpQnu3+YRsHKgGkK9Qox27ZrOfcm4d8H4TtXi830nH1T24ooAV7PgJ+BeJzG8vm65gMEfd6fARTXFN71iQQ999xzPFeYaSmsDJi2xdQ4Lglz8DXMaqg3kB1UcW1pn1etVuty2YS5yGZmZgJdXURsATDNh5UB05bUOyhH7R201KHwUuAGVVwHzWeOEti8ymc6iRgRERimzTh+/DiKxSIuArhRef77AN4JoATgbgBHAexxXjsK4H5lW/mabdv4+N69eP3kSTyxsIAcgFkAH4vHMbprF16emvL877Nnz+LcuXNYt24d1q9fv+zvstKfxzCNgJUB05bYto2NGzcGCvhZABchBPrarVsx/8YbwYqjVMLo6CgeuPdelKana68XCwUcPXYMyWSy4d+HYdodVgZM23LP7t2+1fzeWAw/IcJlZ5tioYA/fOQR3H777aGWgVyR8yqdYcywMmDalkuXLhlX8+OPPoof/vCHHoFuUhxBbiCGYfywMmDannpW80GKg91ADFMfrAyYroLdQAyzNFgZMAzDMLBavQMMwzBM62FlwDAMw7AyYBiGYVgZMAzDMGBlwDAMw4CVAcMwDANWBgzDMAxYGTAMwzBgZcAwDMOAlQHDMAwDVgYMwzAMWBkwDMMwYGXAMAzDgJUBwzAMA1YGDMMwDFgZMAzDMGBlwDAMw4CVAcMwDANWBgzDMAxYGTAMwzBgZcAwDMMA+P8Bhzfm6ulGLeIAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_name = \"circle_fc\"\n",
    "\n",
    "# Generate the dataset\n",
    "data, label = make_circles(n_samples=1000, factor = 0.4, noise = 0.025)\n",
    "\n",
    "# Visualize the dataset\n",
    "def plot_data(data, label):\n",
    "    reds = label == 0\n",
    "    blues = label == 1\n",
    "    plt.scatter(data[reds, 0], data[reds, 1], c=\"red\", s=20, edgecolor='k')\n",
    "    plt.scatter(data[blues, 0], data[blues, 1], c=\"blue\", s=20, edgecolor='k')\n",
    "    plt.gca().set_aspect('equal', adjustable='box')\n",
    "    plt.axis('off')\n",
    "    plt.savefig(f\"{model_name}_visualization.png\", format='png', dpi=300, bbox_inches='tight')\n",
    "    plt.show()\n",
    "\n",
    "plot_data(data, label)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d4514bf",
   "metadata": {},
   "source": [
    "## 4. Data Preparation\n",
    "\n",
    "Before training our neural network, we need to convert the dataset into a data type suitable for neural network trainnig, and split it into training and testing subsets to ensure we have separate data for model evaluation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "215cd02e",
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = data.astype(np.float32), label.astype(np.int64)\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b316648e",
   "metadata": {},
   "source": [
    "## 5. Neural Network Architecture\n",
    "\n",
    "Our neural network is a fully-connected network which comprises an input layer, one hidden layer with relu activation function, and an output layer. Additionally, dropout is included for regularization. We also define the number of epochs and learning rate for our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "555b0959",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ClassifierModule(nn.Module):\n",
    "    def __init__(\n",
    "            self,\n",
    "            input_dim=2,\n",
    "            hidden_dims=[10],\n",
    "            output_dim=2,\n",
    "            nonlin=F.relu,\n",
    "            dropout=0.5,\n",
    "    ):\n",
    "        super(ClassifierModule, self).__init__()\n",
    "        self.nonlin = nonlin\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        \n",
    "        dims = [input_dim] + hidden_dims + [output_dim]\n",
    "        self.layers = nn.ModuleList()\n",
    "        for i in range(len(dims) - 1):\n",
    "            self.layers.append(nn.Linear(dims[i], dims[i+1]))\n",
    "\n",
    "    def forward(self, X, **kwargs):\n",
    "        for i, layer in enumerate(self.layers):\n",
    "            X = layer(X)\n",
    "            if i != len(self.layers) - 1:  # If not the last layer\n",
    "                X = self.nonlin(X)\n",
    "                X = self.dropout(X)\n",
    "            else:  # If last layer\n",
    "                X = F.softmax(X, dim=-1)\n",
    "        return X\n",
    "    \n",
    "net = NeuralNetClassifier(\n",
    "    ClassifierModule,\n",
    "    max_epochs=100,\n",
    "    lr=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0cd75f62",
   "metadata": {},
   "source": [
    "## 6. Model Training\n",
    "\n",
    "We now proceed to train our neural network model using the training data. Training involves feeding our network the training data multiple times (epochs) and adjusting the model weights to minimize the classification error."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f8754a09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  epoch    train_loss    valid_acc    valid_loss     dur\n",
      "-------  ------------  -----------  ------------  ------\n",
      "      1        \u001b[36m0.7103\u001b[0m       \u001b[32m0.6687\u001b[0m        \u001b[35m0.6878\u001b[0m  0.0271\n",
      "      2        \u001b[36m0.7015\u001b[0m       0.5375        \u001b[35m0.6831\u001b[0m  0.0096\n",
      "      3        \u001b[36m0.6938\u001b[0m       0.6562        \u001b[35m0.6808\u001b[0m  0.0091\n",
      "      4        \u001b[36m0.6886\u001b[0m       \u001b[32m0.6875\u001b[0m        \u001b[35m0.6778\u001b[0m  0.0054\n",
      "      5        \u001b[36m0.6854\u001b[0m       0.6375        \u001b[35m0.6734\u001b[0m  0.0054\n",
      "      6        \u001b[36m0.6788\u001b[0m       0.6062        \u001b[35m0.6694\u001b[0m  0.0088\n",
      "      7        \u001b[36m0.6786\u001b[0m       0.6125        \u001b[35m0.6662\u001b[0m  0.0076\n",
      "      8        \u001b[36m0.6670\u001b[0m       0.6562        \u001b[35m0.6609\u001b[0m  0.0065\n",
      "      9        \u001b[36m0.6668\u001b[0m       0.6687        \u001b[35m0.6554\u001b[0m  0.0051\n",
      "     10        \u001b[36m0.6566\u001b[0m       0.6875        \u001b[35m0.6495\u001b[0m  0.0075\n",
      "     11        0.6649       \u001b[32m0.7000\u001b[0m        \u001b[35m0.6446\u001b[0m  0.0068\n",
      "     12        \u001b[36m0.6526\u001b[0m       \u001b[32m0.7438\u001b[0m        \u001b[35m0.6379\u001b[0m  0.0091\n",
      "     13        \u001b[36m0.6525\u001b[0m       \u001b[32m0.8000\u001b[0m        \u001b[35m0.6325\u001b[0m  0.0058\n",
      "     14        \u001b[36m0.6362\u001b[0m       \u001b[32m0.8250\u001b[0m        \u001b[35m0.6254\u001b[0m  0.0094\n",
      "     15        0.6444       \u001b[32m0.8375\u001b[0m        \u001b[35m0.6205\u001b[0m  0.0256\n",
      "     16        0.6442       0.8375        \u001b[35m0.6155\u001b[0m  0.0135\n",
      "     17        \u001b[36m0.6213\u001b[0m       \u001b[32m0.8625\u001b[0m        \u001b[35m0.6081\u001b[0m  0.0086\n",
      "     18        0.6218       0.8375        \u001b[35m0.6012\u001b[0m  0.0071\n",
      "     19        0.6263       0.8500        \u001b[35m0.5957\u001b[0m  0.0064\n",
      "     20        0.6228       \u001b[32m0.8688\u001b[0m        \u001b[35m0.5909\u001b[0m  0.0067\n",
      "     21        \u001b[36m0.6148\u001b[0m       0.8625        \u001b[35m0.5854\u001b[0m  0.0087\n",
      "     22        \u001b[36m0.6125\u001b[0m       \u001b[32m0.8750\u001b[0m        \u001b[35m0.5793\u001b[0m  0.0082\n",
      "     23        0.6145       0.8688        \u001b[35m0.5745\u001b[0m  0.0069\n",
      "     24        \u001b[36m0.5999\u001b[0m       \u001b[32m0.8812\u001b[0m        \u001b[35m0.5686\u001b[0m  0.0057\n",
      "     25        0.6083       0.8750        \u001b[35m0.5637\u001b[0m  0.0062\n",
      "     26        0.6058       0.8750        \u001b[35m0.5593\u001b[0m  0.0051\n",
      "     27        \u001b[36m0.5966\u001b[0m       0.8750        \u001b[35m0.5540\u001b[0m  0.0069\n",
      "     28        \u001b[36m0.5963\u001b[0m       \u001b[32m0.8875\u001b[0m        \u001b[35m0.5486\u001b[0m  0.0064\n",
      "     29        \u001b[36m0.5880\u001b[0m       0.8750        \u001b[35m0.5447\u001b[0m  0.0093\n",
      "     30        \u001b[36m0.5839\u001b[0m       0.8812        \u001b[35m0.5383\u001b[0m  0.0048\n",
      "     31        0.5893       0.8812        \u001b[35m0.5351\u001b[0m  0.0090\n",
      "     32        0.5851       0.8812        \u001b[35m0.5313\u001b[0m  0.0049\n",
      "     33        \u001b[36m0.5808\u001b[0m       0.8875        \u001b[35m0.5271\u001b[0m  0.0091\n",
      "     34        \u001b[36m0.5765\u001b[0m       0.8875        \u001b[35m0.5216\u001b[0m  0.0067\n",
      "     35        \u001b[36m0.5726\u001b[0m       \u001b[32m0.8938\u001b[0m        \u001b[35m0.5161\u001b[0m  0.0087\n",
      "     36        0.5773       0.8938        \u001b[35m0.5123\u001b[0m  0.0094\n",
      "     37        \u001b[36m0.5602\u001b[0m       0.8812        \u001b[35m0.5071\u001b[0m  0.0115\n",
      "     38        0.5789       0.8812        \u001b[35m0.5045\u001b[0m  0.0106\n",
      "     39        \u001b[36m0.5557\u001b[0m       0.8812        \u001b[35m0.4982\u001b[0m  0.0055\n",
      "     40        0.5613       0.8812        \u001b[35m0.4941\u001b[0m  0.0061\n",
      "     41        \u001b[36m0.5552\u001b[0m       \u001b[32m0.9000\u001b[0m        \u001b[35m0.4901\u001b[0m  0.0052\n",
      "     42        0.5586       \u001b[32m0.9437\u001b[0m        \u001b[35m0.4860\u001b[0m  0.0061\n",
      "     43        \u001b[36m0.5471\u001b[0m       0.9000        \u001b[35m0.4801\u001b[0m  0.0054\n",
      "     44        \u001b[36m0.5456\u001b[0m       0.9125        \u001b[35m0.4757\u001b[0m  0.0070\n",
      "     45        \u001b[36m0.5380\u001b[0m       0.8938        \u001b[35m0.4713\u001b[0m  0.0052\n",
      "     46        0.5409       0.8875        \u001b[35m0.4671\u001b[0m  0.0074\n",
      "     47        \u001b[36m0.5307\u001b[0m       \u001b[32m0.9500\u001b[0m        \u001b[35m0.4615\u001b[0m  0.0069\n",
      "     48        0.5459       \u001b[32m0.9563\u001b[0m        \u001b[35m0.4591\u001b[0m  0.0083\n",
      "     49        0.5349       0.9563        \u001b[35m0.4551\u001b[0m  0.0080\n",
      "     50        \u001b[36m0.5245\u001b[0m       0.9563        \u001b[35m0.4509\u001b[0m  0.0102\n",
      "     51        \u001b[36m0.5151\u001b[0m       0.9500        \u001b[35m0.4447\u001b[0m  0.0085\n",
      "     52        0.5314       0.9563        \u001b[35m0.4419\u001b[0m  0.0069\n",
      "     53        \u001b[36m0.4992\u001b[0m       0.9500        \u001b[35m0.4367\u001b[0m  0.0062\n",
      "     54        0.5255       0.9563        \u001b[35m0.4345\u001b[0m  0.0097\n",
      "     55        0.5127       \u001b[32m0.9625\u001b[0m        \u001b[35m0.4283\u001b[0m  0.0077\n",
      "     56        0.5009       0.9625        \u001b[35m0.4245\u001b[0m  0.0054\n",
      "     57        0.5017       0.9625        \u001b[35m0.4235\u001b[0m  0.0054\n",
      "     58        0.5015       0.9625        \u001b[35m0.4179\u001b[0m  0.0090\n",
      "     59        0.5001       0.9625        \u001b[35m0.4110\u001b[0m  0.0072\n",
      "     60        0.5133       0.9625        \u001b[35m0.4077\u001b[0m  0.0096\n",
      "     61        0.5077       0.9625        \u001b[35m0.4044\u001b[0m  0.0078\n",
      "     62        0.5087       0.9625        \u001b[35m0.4011\u001b[0m  0.0056\n",
      "     63        \u001b[36m0.4677\u001b[0m       0.9625        \u001b[35m0.3919\u001b[0m  0.0062\n",
      "     64        0.4972       0.9625        \u001b[35m0.3903\u001b[0m  0.0053\n",
      "     65        0.4861       0.9625        \u001b[35m0.3846\u001b[0m  0.0087\n",
      "     66        0.4933       \u001b[32m0.9688\u001b[0m        \u001b[35m0.3822\u001b[0m  0.0055\n",
      "     67        \u001b[36m0.4636\u001b[0m       0.9688        \u001b[35m0.3770\u001b[0m  0.0051\n",
      "     68        0.4637       0.9688        \u001b[35m0.3716\u001b[0m  0.0051\n",
      "     69        0.4861       0.9688        \u001b[35m0.3666\u001b[0m  0.0075\n",
      "     70        0.4726       0.9688        \u001b[35m0.3639\u001b[0m  0.0090\n",
      "     71        0.4954       \u001b[32m0.9750\u001b[0m        0.3643  0.0157\n",
      "     72        0.4667       \u001b[32m0.9812\u001b[0m        \u001b[35m0.3601\u001b[0m  0.0074\n",
      "     73        \u001b[36m0.4574\u001b[0m       0.9812        \u001b[35m0.3549\u001b[0m  0.0081\n",
      "     74        0.4590       \u001b[32m0.9875\u001b[0m        \u001b[35m0.3499\u001b[0m  0.0076\n",
      "     75        0.4614       0.9875        \u001b[35m0.3446\u001b[0m  0.0078\n",
      "     76        \u001b[36m0.4512\u001b[0m       0.9875        \u001b[35m0.3403\u001b[0m  0.0130\n",
      "     77        0.4920       0.9875        0.3408  0.0081\n",
      "     78        0.4835       0.9875        \u001b[35m0.3380\u001b[0m  0.0062\n",
      "     79        \u001b[36m0.4318\u001b[0m       0.9875        \u001b[35m0.3299\u001b[0m  0.0060\n",
      "     80        0.4372       0.9875        \u001b[35m0.3250\u001b[0m  0.0049\n",
      "     81        0.4546       \u001b[32m1.0000\u001b[0m        \u001b[35m0.3225\u001b[0m  0.0054\n",
      "     82        0.4714       1.0000        \u001b[35m0.3205\u001b[0m  0.0051\n",
      "     83        0.4707       1.0000        0.3208  0.0051\n",
      "     84        0.4441       1.0000        \u001b[35m0.3180\u001b[0m  0.0050\n",
      "     85        0.4444       1.0000        \u001b[35m0.3120\u001b[0m  0.0060\n",
      "     86        \u001b[36m0.4288\u001b[0m       1.0000        \u001b[35m0.3059\u001b[0m  0.0099\n",
      "     87        0.4473       1.0000        0.3062  0.0093\n",
      "     88        0.4504       1.0000        \u001b[35m0.3019\u001b[0m  0.0054\n",
      "     89        0.4437       1.0000        \u001b[35m0.3001\u001b[0m  0.0053\n",
      "     90        \u001b[36m0.4086\u001b[0m       1.0000        \u001b[35m0.2941\u001b[0m  0.0080\n",
      "     91        0.4173       1.0000        \u001b[35m0.2901\u001b[0m  0.0125\n",
      "     92        0.4410       1.0000        0.2904  0.0081\n",
      "     93        0.4285       1.0000        \u001b[35m0.2880\u001b[0m  0.0066\n",
      "     94        0.4314       1.0000        \u001b[35m0.2844\u001b[0m  0.0047\n",
      "     95        0.4332       1.0000        \u001b[35m0.2828\u001b[0m  0.0059\n",
      "     96        0.4308       1.0000        \u001b[35m0.2807\u001b[0m  0.0079\n",
      "     97        0.4269       1.0000        \u001b[35m0.2778\u001b[0m  0.0094\n",
      "     98        \u001b[36m0.4075\u001b[0m       1.0000        \u001b[35m0.2716\u001b[0m  0.0055\n",
      "     99        0.4393       1.0000        0.2718  0.0098\n",
      "    100        0.4194       1.0000        \u001b[35m0.2683\u001b[0m  0.0057\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<class 'skorch.classifier.NeuralNetClassifier'>[initialized](\n",
       "  module_=ClassifierModule(\n",
       "    (dropout): Dropout(p=0.5, inplace=False)\n",
       "    (layers): ModuleList(\n",
       "      (0): Linear(in_features=2, out_features=10, bias=True)\n",
       "      (1): Linear(in_features=10, out_features=2, bias=True)\n",
       "    )\n",
       "  ),\n",
       ")"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c88893e4",
   "metadata": {},
   "source": [
    "## 7. Model Testing\n",
    "\n",
    "We then evaluate our model's performance on unseen data (test data). This gives us an indication of whether our model has truly learned the underlying patterns in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1e96fffc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 100.0%\n"
     ]
    }
   ],
   "source": [
    "y_pred = net.predict(X_test)\n",
    "\n",
    "accuracy = (sum(y_test == y_pred) / len(y_test)) * 100\n",
    "\n",
    "print(f\"Accuracy: {accuracy}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4065a4fd",
   "metadata": {},
   "source": [
    "## 8. Save Model Parameters\n",
    "\n",
    "A zk program usually does not support `floats`. Therefore, we define a base_scaling_factor, which is used to scale the inputs and the weights and convert them to integers. Since bias is added to `(input * weight)`, it has a scaling factor at the first layer which is twice of the base_scaling_factor. The scaling factor for biases in subsequent layers is adjusted accordingly.\n",
    "\n",
    "We will save the trained model parameters (weights and biases) in a JSON file, making them ready for integration into a zk program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "906f78be",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved model parameters at circle_fc_parameters.json\n"
     ]
    }
   ],
   "source": [
    "base_scaling_factor = 10**6\n",
    "\n",
    "def get_scaling_factor(layer_idx):\n",
    "    scaling_factors = {\n",
    "        \"weight\": base_scaling_factor,\n",
    "        \"bias\": base_scaling_factor ** (layer_idx + 1)\n",
    "    }\n",
    "    return scaling_factors\n",
    "\n",
    "model_data = {}\n",
    "\n",
    "for param in net.get_all_learnable_params():\n",
    "    model_data[param[0]] = param[1].detach().numpy().flatten()\n",
    "\n",
    "model_json = {}\n",
    "num_layers = len(model_data) // 2  # considering weight and bias for each layer\n",
    "\n",
    "for i in range(num_layers):\n",
    "    weight_key = f\"layers.{i}.weight\"\n",
    "    bias_key = f\"layers.{i}.bias\"\n",
    "    \n",
    "    scaling = get_scaling_factor(i + 1)\n",
    "        \n",
    "    model_json[f\"w{i+1}\"] = (model_data[weight_key] * scaling[\"weight\"]).round().astype(int).tolist()\n",
    "    model_json[f\"b{i+1}\"] = (model_data[bias_key] * scaling[\"bias\"]).round().astype(int).tolist()\n",
    "\n",
    "# Save to JSON\n",
    "with open(f\"{model_name}_parameters.json\", \"w\") as f:\n",
    "    json.dump(model_json, f, indent=4)\n",
    "\n",
    "print(f\"Saved model parameters at {model_name}_parameters.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "767c0147",
   "metadata": {},
   "source": [
    "## 9. Save Test Samples\n",
    "\n",
    "We also save the test sample inputs and corresponding model predictions in a json file. They will serve as a benchmark, enabling us to test the correctness of our zk program. The samples are again scaled appropriately for use in zk programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "2f12f031",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved test samples at circle_fc_samples.json\n"
     ]
    }
   ],
   "source": [
    "samples_json = {}\n",
    "\n",
    "for (i, sample) in enumerate(zip(X_test, y_pred)):\n",
    "    samples_json[f\"in{i+1}\"] = (sample[0]*(base_scaling_factor)).round().astype(int).tolist()\n",
    "    samples_json[f\"out{i+1}\"] = sample[1].tolist()\n",
    "    \n",
    "# Save to JSON\n",
    "with open(f\"{model_name}_samples.json\", \"w\") as f:\n",
    "    json.dump(samples_json, f, indent=4)\n",
    "\n",
    "print(f\"Saved test samples at {model_name}_samples.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7f9f179",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}