Google_Colab_tutorial.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Google_Colab_tutorial.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1hjKE6QCD9ez",
        "colab_type": "text"
      },
      "source": [
        "## Intro.\n",
        "\n",
        "It is example codes and script for explaining about Google Colab. \n",
        "\n",
        "My ways are not the optimal way to use Colab, but I think these are helpful enough to increase experiment speed.\n",
        "\n",
        "\n",
        "## What is Google Colab.\n",
        "https://www.youtube.com/watch?v=inN8seMm7UI\n",
        "\n",
        "Google Colab(GC) is **free**  machine learning tool for studying and researching.\n",
        "\n",
        "It means that you can run your codes in GC, which contains a powerful GPU machine.\n",
        "\n",
        "and its only limitation is the maximum running time, which is 12 hours.\n",
        "\n",
        "and it is connected to your Google drive so you can access GC GPU anytime.\n",
        "\n",
        "So there is no reason not to use GC."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h3udZ7cCG2a6",
        "colab_type": "text"
      },
      "source": [
        "# How to Use GC\n",
        "\n",
        "## Connect with your Google drive\n",
        "\n",
        "If you want to use GC, Google drive is needed for saving codes, datasets, training log, and so on.\n",
        "\n",
        "so the first step is connecting to Google Driver.\n",
        "\n",
        "\n",
        "Look below code"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "odw4J82VExjd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cleyIr7JITja",
        "colab_type": "text"
      },
      "source": [
        "this code connect your google drive to GC\n",
        "\n",
        "it is run by ctrl + Enter.\n",
        "\n",
        "if you run this code, return some URL and you can fine authorization code in there.\n",
        "\n",
        "After printed \n",
        "\n",
        "\"Mounted at /content/gdrive\"\n",
        "\n",
        "you can access you google driver in GC like below image and your home path is set to ''/content'' \n",
        "\n",
        "![](https://drive.google.com/uc?id=1bmWrRZYzyb1b4WkaXt9a9bgM9cTEoLyk)\n",
        "\n",
        "\n",
        "It is Korean, but the button's location is the same I think\n",
        "\n",
        "Next, I'll make some folder in my Google drive to upload new codes by the below code.\n",
        "\n",
        "As you can see if you add '!' in front of code line, it runs in the terminal\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fa57dSchNeJj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!mkdir gdrive/My\\ Drive/testing_folder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EWpl_eUDPPVe",
        "colab_type": "text"
      },
      "source": [
        "![](https://drive.google.com/uc?id=1PvNolorXaIgs9ls9ldvCypa-hdtfj5aX)\n",
        "\n",
        "\n",
        "As you can see, a new folder is made in Google drive.\n",
        "\n",
        "it is possible to make a new folder in Google driver surely, but I want to show that it is not so different from our computer.\n",
        "\n",
        "\n",
        "## Upload codes in Google Colab\n",
        "\n",
        "The next step is uploading a bunch of codes. I'll explain how to code in GC, but it is not so comfortable.\n",
        "\n",
        "So if you have to modify many things, work on your computer is better.\n",
        "\n",
        "if you upload codes in the folder, you can see in GC also.\n",
        "\n",
        "\n",
        "![](https://drive.google.com/uc?id=1w1_xa1apDJhe2xcoXFuZBHJEA-16bBMe)\n",
        "\n",
        "\n",
        "## GPU setting\n",
        "You surely want to run your codes in GPU. So you have to change the runtime type.\n",
        "\n",
        "![](https://drive.google.com/uc?id=17X0FqZzLCjEeJhn_13WPZFnb6McPcMzH)\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8SfVIbgHVks4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mi2xNhzFVwRp",
        "colab_type": "text"
      },
      "source": [
        "it's done, and Tesla T4 is yours!\n",
        "\n",
        "## Start Tensorboard\n",
        "I think analyzing the learning plot is very important. so I'll explain about how to run Tensorboard in GC\n",
        "\n",
        "I set training path to\" test\"\n",
        "\n",
        "So Tensorboard is run by the below codes."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yUb-PUKwXIMv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-amd64.zip\n",
        "!unzip ngrok-stable-linux-amd64.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sjnnUfeDauZK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LOG_DIR = 'test'\n",
        "\n",
        "get_ipython().system_raw(\n",
        "    'tensorboard --logdir={} --host=0.0.0.0 --port=6006 &'\n",
        "    .format(LOG_DIR))\n",
        "\n",
        "get_ipython().system_raw('./ngrok http 6006 &')\n",
        "\n",
        "!curl -s http://localhost:4040/api/tunnels | python3 -c \\\n",
        "    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dTcypKPuLgfJ",
        "colab_type": "text"
      },
      "source": [
        "when all process is done, you can see some link in last line and the page show Tensorboard\n",
        "\n",
        "you can check running of Tensorboard by the process manager"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x4iKnACILxSx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ps -A"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UETRQM251RKB"
      },
      "source": [
        "As you can see, Tensorboard is running backend.\n",
        "\n",
        "## Training the network\n",
        "\n",
        "Surely, you need some codes for training, but if you don't have, you can clone the example codes by below cell.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KDGfyE_v1sxo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!git clone https://github.com/sseung0703/Google_Colab_tutorial.git"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iNnFmnv01twj",
        "colab_type": "text"
      },
      "source": [
        ""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L9Fq_uduYMCi",
        "colab_type": "code",
        "cellView": "code",
        "colab": {}
      },
      "source": [
        "!python /content/Google_Colab_tutorial/train.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JS82dKZffMeK",
        "colab_type": "text"
      },
      "source": [
        "![](https://drive.google.com/uc?id=1WCkdnD_aRTHrNU1UxrR26EEWPDUI5w8p)\n",
        "\n",
        "training result is not so good so you have to modify hyper-parameters.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1e6W7G9Zc_8r",
        "colab_type": "text"
      },
      "source": [
        "## Modifying Code\n",
        "If you want to modify just some of the hyper-parameters, re-uploading codes are so uncomfortable.\n",
        "\n",
        "so we need some text editor that can modify the codes in Google drive.\n",
        "\n",
        "Note that the codes have to be placed in your drive.\n",
        "\n",
        "![](https://drive.google.com/uc?id=1T3BvM1kPUBiOFePmebcS65oMCkW8cdMC)\n",
        "\n",
        "Text Editor for Drive is suitable for this, I think.\n",
        "\n",
        "it can access your Google Drive and edit the codes.\n",
        "\n",
        "So I modify some hyper-parameters and run again.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vIu9zBxPenmH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!python /content/Google_Colab_tutorial/train.py"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rQGkNcjoeyom",
        "colab_type": "text"
      },
      "source": [
        "later one which called test0 is better than the former one\n",
        "\n",
        "![](https://drive.google.com/uc?id=1wfpIlwhfDrVMSLXG30PzXB-l38tF0RZB)\n",
        "\n",
        "and if you want to change logdir of Tensorboard, you have to kill running Tensorboard and rerun the Tensorboard\n",
        "\n",
        "all progress is explained in below codes"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3woZGfsvMkZZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ps -A"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L__XJIRLMoP_",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!kill -9 322"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8eDene_bMsDH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "LOG_DIR = 'test/test1'\n",
        "\n",
        "get_ipython().system_raw(\n",
        "    'tensorboard --logdir={} --host=0.0.0.0 --port=6006 &'\n",
        "    .format(LOG_DIR))\n",
        "\n",
        "get_ipython().system_raw('./ngrok http 6006 &')\n",
        "\n",
        "!curl -s http://localhost:4040/api/tunnels | python3 -c \\\n",
        "    \"import sys, json; print(json.load(sys.stdin)['tunnels'][0]['public_url'])\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "toec57lJNCMH",
        "colab_type": "text"
      },
      "source": [
        "That's all and enjoy you free GPU :)"
      ]
    }
  ]
}