architectures/headers/add_headers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ac3cb663",
   "metadata": {},
   "source": [
    "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Fvertex-ai-mlops%2Farchitectures%2Fheaders&file=add_headers.ipynb)\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://colab.research.google.com/github/statmike/vertex-ai-mlops/blob/main/architectures/headers/add_headers.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Fvertex-ai-mlops%2Fmain%2Farchitectures%2Fheaders%2Fadd_headers.ipynb\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://github.com/statmike/vertex-ai-mlops/blob/main/architectures/headers/add_headers.ipynb\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/vertex-ai-mlops/main/architectures/headers/add_headers.ipynb\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "507455bd-a45b-4289-9f6b-590977801da0",
   "metadata": {},
   "source": [
    "# Add Headers To All Notebooks and `.md` Files\n",
    "\n",
    "Include tracking links, and links to view/load content in Colab, Colab Enterprise, GitHub, Vertex AI Workbench Instances\n",
    "\n",
    "For more details on working with files to update with `nbformat` package see the companion notebook: [add_helpers_example](./add_helpers_example.ipynb)\n",
    "\n",
    "Reference:\n",
    "- https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/ed16b60537639c949e7ea436edb37996860e57ab/notebooks/notebook_template.ipynb#L38"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea259dee-31fe-4cb3-885a-b0b692b6b089",
   "metadata": {},
   "source": [
    "---\n",
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c0187b6-1f74-4c5c-b6fb-245960bdc881",
   "metadata": {},
   "source": [
    "Imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "05f17c9c-bd1d-46d8-83cf-8453c0cc1a65",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "nbformat.NO_CONVERT"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os, json, urllib.parse, IPython\n",
    "\n",
    "import nbformat as nbf\n",
    "nbf.NO_CONVERT"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d90585-8cec-49ab-838f-525e59274ef3",
   "metadata": {},
   "source": [
    "## Test For Parsing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "36f20361-8f1b-4dcd-9511-67db4941848b",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'user%2Frepo%2Fpath%2Fto+the%2Ffile%2Ffile+name.ipynb'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urllib.parse.quote_plus('user/repo/path/to the/file/file name.ipynb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "2d563ecc-e923-49cc-8b9e-b5857644e90a",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'user/repo/path/to%20the/file/file%20name.ipynb'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urllib.parse.quote('user/repo/path/to the/file/file name.ipynb')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "2a58ec6f-8183-4b54-b6f8-51e423c2e207",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https%3A%2F%2Fgithub.com'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urllib.parse.quote_plus('https://github.com')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "210ed2f2-5fc0-4947-9703-8879c3ddab6c",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'https%3A//github.com'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "urllib.parse.quote('https://github.com')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28f05157-6ee7-4711-a6db-e8b5fda4d5c5",
   "metadata": {},
   "source": [
    "---\n",
    "## Parse And Update\n",
    "\n",
    "**Notes:** URL Encoding\n",
    "- `urllib.parse.qoute_plus()` replaces spaces with `+` which works for the query part of a URL (after a `?`)\n",
    "- `urllib.parse.qoute()` replaces spaces with `%20` which works for the main URL"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ad75c283-12f4-4e3a-a0f2-2503eff32f61",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def make_urls(root, file):\n",
    "    prefixes = dict(\n",
    "        colab = 'https://colab.research.google.com/github/',\n",
    "        ecolab = 'https://console.cloud.google.com/vertex-ai/colab/import/',\n",
    "        github = 'https://github.com/',\n",
    "        github_raw = 'https://raw.githubusercontent.com/',\n",
    "        wbi = 'https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=',\n",
    "        track = 'https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?',\n",
    "    )\n",
    "    \n",
    "    path = root.replace('/home/jupyter/', 'statmike/').split('/')\n",
    "    \n",
    "    urls = dict(\n",
    "        track = prefixes['track'] + f\"path={urllib.parse.quote_plus('/'.join(path))}&file={urllib.parse.quote_plus(file)}\",\n",
    "        colab = prefixes['colab'] + '/'.join(path[0:2]) + '/blob/main/' + urllib.parse.quote('/'.join(path[2:])) + '/' + urllib.parse.quote(file),\n",
    "        ecolab = prefixes['ecolab'] + urllib.parse.quote_plus(prefixes['github_raw'] + '/'.join(path[0:2]) + '/main/' + urllib.parse.quote('/'.join(path[2:])) + '/' + urllib.parse.quote(file)),\n",
    "        github = prefixes['github'] + '/'.join(path[0:2]) + '/blob/main/' + urllib.parse.quote('/'.join(path[2:])) + '/' + urllib.parse.quote(file),\n",
    "        wbi = prefixes['wbi'] + prefixes['github_raw'] + '/'.join(path[0:2]) + '/main/' + urllib.parse.quote('/'.join(path[2:])) + '/' + urllib.parse.quote(file)\n",
    "    )\n",
    "    return urls"
   ]
  },
  {
   "cell_type": "raw",
   "id": "15117d1a-9a62-4bfd-a726-285b3d8e79b6",
   "metadata": {
    "tags": []
   },
   "source": [
    "def make_urls(root, file):\n",
    "    # url prefixes\n",
    "    colab_prefix = 'https://colab.research.google.com/github/'\n",
    "    ecolab_prefix = 'https://console.cloud.google.com/vertex-ai/colab/import/'\n",
    "    github_prefix = 'https://github.com/'\n",
    "    github_raw_prefix = 'https://raw.githubusercontent.com/'\n",
    "    wbi_prefix = 'https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url='\n",
    "    track_prefix = 'https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?'\n",
    "    \n",
    "    # custom tracking\n",
    "    path = urllib.parse.quote_plus(root.replace('/home/jupyter/', 'statmike/'))\n",
    "    file = urllib.parse.quote_plus(file)\n",
    "    track =  track_prefix + f\"path={path}&file={file}\"\n",
    "    \n",
    "    # GH URL Parts\n",
    "    gh_base = os.path.join(root.replace('/home/jupyter/', 'statmike/'), file).split('/')\n",
    "    colab = colab_prefix + '/'.join(gh_base[0:2]) + '/blob/main/' + urllib.parse.quote('/'.join(gh_base[2:]))\n",
    "    ecolab = ecolab_prefix + urllib.parse.quote(github_raw_prefix + '/'.join(gh_base[0:2]) + '/main/' + '/'.join(gh_base[2:]))\n",
    "    github = github_prefix + '/'.join(gh_base[0:2]) + '/blob/main/' + urllib.parse.quote('/'.join(gh_base[2:]))\n",
    "    wbi = wbi_prefix + urllib.parse.quote(github_raw_prefix + '/'.join(gh_base[0:2]) + '/main/' + '/'.join(gh_base[2:]))\n",
    "    download = github_raw_prefix + '/'.join(gh_base[0:2]) + '/main/' + urllib.parse.quote('/'.join(gh_base[2:]))\n",
    "    filename = gh_base[-1]\n",
    "    \n",
    "    return dict(track=track, colab=colab, ecolab=ecolab, github=github, wbi=wbi, download=download, filename=filename)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f183a858-3878-4143-b92c-1bb83d13d47b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def make_header(root, file):\n",
    "    urls = make_urls(root, file)\n",
    "    \n",
    "    if file.endswith('.ipynb'):\n",
    "        header = f'''![tracker]({urls['track']})\n",
    "<!--- header table --->\n",
    "<table align=\"left\">\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"{urls['colab']}\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
    "      <br>Run in<br>Colab\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"{urls['ecolab']}\">\n",
    "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
    "      <br>Run in<br>Colab Enterprise\n",
    "    </a>\n",
    "  </td>      \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"{urls['github']}\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"{urls['wbi']}\">\n",
    "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
    "      <br>Open in<br>Vertex AI Workbench\n",
    "    </a>\n",
    "  </td>\n",
    "</table>'''    \n",
    "\n",
    "    if file.endswith('.md'):\n",
    "        header = f'''![tracker]({urls['track']})\n",
    "<!--- header table --->\n",
    "<table align=\"left\">     \n",
    "  <td style=\"text-align: center\">\n",
    "    <a href=\"{urls['github']}\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
    "      <br>View on<br>GitHub\n",
    "    </a>\n",
    "  </td>\n",
    "</table><br/><br/><br/><br/>\n",
    "\n",
    "---\n",
    "''' \n",
    "        \n",
    "    return header"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "a0c0f31e-8d0e-411b-81d8-71fc62c391fb",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "![tracker](https://us-central1-vertex-ai-mlops-369716.cloudfunctions.net/pixel-tracking?path=statmike%2Frepo%2Fpath%2Fto+the%2Ffile&file=name+of+file.ipynb)\n",
      "<!--- header table --->\n",
      "<table align=\"left\">\n",
      "  <td style=\"text-align: center\">\n",
      "    <a href=\"https://colab.research.google.com/github/statmike/repo/blob/main/path/to%20the/file/name%20of%20file.ipynb\">\n",
      "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Google Colaboratory logo\">\n",
      "      <br>Run in<br>Colab\n",
      "    </a>\n",
      "  </td>\n",
      "  <td style=\"text-align: center\">\n",
      "    <a href=\"https://console.cloud.google.com/vertex-ai/colab/import/https%3A%2F%2Fraw.githubusercontent.com%2Fstatmike%2Frepo%2Fmain%2Fpath%2Fto%2520the%2Ffile%2Fname%2520of%2520file.ipynb\">\n",
      "      <img width=\"32px\" src=\"https://lh3.googleusercontent.com/JmcxdQi-qOpctIvWKgPtrzZdJJK-J3sWE1RsfjZNwshCFgE_9fULcNpuXYTilIR2hjwN\" alt=\"Google Cloud Colab Enterprise logo\">\n",
      "      <br>Run in<br>Colab Enterprise\n",
      "    </a>\n",
      "  </td>      \n",
      "  <td style=\"text-align: center\">\n",
      "    <a href=\"https://github.com/statmike/repo/blob/main/path/to%20the/file/name%20of%20file.ipynb\">\n",
      "      <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">\n",
      "      <br>View on<br>GitHub\n",
      "    </a>\n",
      "  </td>\n",
      "  <td style=\"text-align: center\">\n",
      "    <a href=\"https://console.cloud.google.com/vertex-ai/workbench/deploy-notebook?download_url=https://raw.githubusercontent.com/statmike/repo/main/path/to%20the/file/name%20of%20file.ipynb\">\n",
      "      <img src=\"https://lh3.googleusercontent.com/UiNooY4LUgW_oTvpsNhPpQzsstV5W8F7rYgxgGBD85cWJoLmrOzhVs_ksK_vgx40SHs7jCqkTkCk=e14-rj-sc0xffffff-h130-w32\" alt=\"Vertex AI logo\">\n",
      "      <br>Open in<br>Vertex AI Workbench\n",
      "    </a>\n",
      "  </td>\n",
      "</table>\n"
     ]
    }
   ],
   "source": [
    "print(make_header('/home/jupyter/repo/path/to the/file', 'name of file.ipynb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "29e11f5a-2a76-4fb0-8623-267ebf195761",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving File:\n",
      "\t/home/jupyter/vertex-ai-mlops/MLOps/Experiment Tracking\n",
      "\tVertex AI Experiments.ipynb\n",
      "Saving File:\n",
      "\t/home/jupyter/vertex-ai-mlops/MLOps/Pipelines\n",
      "\tVertex AI Pipelines - Testing.ipynb\n"
     ]
    }
   ],
   "source": [
    "for root, dirs, files in os.walk(r'/home/jupyter/vertex-ai-mlops'): #os.walk('../.'):\n",
    "#for root, dirs, files in os.walk(r'/home/jupyter/vertex-ai-mlops/architectures'):    \n",
    "    #if root.startswith('/home/jupyter/vertex-ai-mlops/architectures/headers') and not root.endswith(('.ipynb_checkpoints')) and '/temp' not in root: # testing\n",
    "    if not root.endswith(('.ipynb_checkpoints')) and '/temp' not in root: # full repository\n",
    "        for file in files:\n",
    "            # process for markdown files (most readme.md files)\n",
    "            if file.endswith(('.md')):\n",
    "                # generate current header\n",
    "                header = make_header(root, file)\n",
    "                \n",
    "                # read file\n",
    "                with open(os.path.join(root, file), 'r') as reader:\n",
    "                    content = reader.read()\n",
    "                    \n",
    "                # detect existing header in file\n",
    "                if content.startswith('<!--- header table --->') or content.startswith('![tracker](https://'):\n",
    "                    end_index = content.index('#')\n",
    "                    prior = content[0:end_index]\n",
    "                else:\n",
    "                    end_index = 0\n",
    "                    prior = ''\n",
    "                            \n",
    "                # is header changed?\n",
    "                if header != prior:\n",
    "                    update = True\n",
    "                else:\n",
    "                    update = False\n",
    "                    \n",
    "                # save updated file\n",
    "                if update:\n",
    "                    print(f'Saving File:\\n\\t{root}\\n\\t{file}')\n",
    "                    with open(os.path.join(root, file), 'w') as writer:\n",
    "                        writer.write(header + content[end_index:])\n",
    "             \n",
    "            # process for notebooks\n",
    "            if file.endswith(('.ipynb')):\n",
    "                # generate current header\n",
    "                header = make_header(root, file)\n",
    "                \n",
    "                # read file\n",
    "                nb = nbf.read(os.path.join(root, file), nbf.NO_CONVERT)\n",
    "                \n",
    "                # detect existing header in file\n",
    "                if nb['cells'][0]['cell_type'] == 'markdown':\n",
    "                    content = nb['cells'][0]['source']\n",
    "                    if content.startswith('<!--- header table --->') or content.startswith('![tracker](https://'):\n",
    "                        end_index = len(content) # content.index('</table>') + 10\n",
    "                        prior = content # [0:end_index]\n",
    "                    else:\n",
    "                        end_index = 0\n",
    "                        prior = ''\n",
    "                \n",
    "                # is header changed?\n",
    "                if header != prior:\n",
    "                    update = True\n",
    "                else:\n",
    "                    update = False\n",
    "                    \n",
    "                # save updated file\n",
    "                if update:\n",
    "                    print(f'Saving File:\\n\\t{root}\\n\\t{file}')\n",
    "                    header_cell = nbf.v4.new_markdown_cell(source = header)\n",
    "                    if end_index == 0:\n",
    "                        nb['cells'].insert(0, header_cell)\n",
    "                    else:\n",
    "                        nb['cells'][0] = header_cell\n",
    "                    nbf.write(nb, os.path.join(root, file), version = nbf.NO_CONVERT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52f51e4b-06cb-4bf8-a5b1-9a92011494d5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "conda-base-py",
   "name": "workbench-notebooks.m124",
   "type": "gcloud",
   "uri": "us-docker.pkg.dev/deeplearning-platform-release/gcr.io/workbench-notebooks:m124"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "conda-base-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}