Deep_GCN_Spam.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Deep GCN Spam.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/zaidalyafeai/Notebooks/blob/master/Deep_GCN_Spam.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "metadata": {
        "id": "UjoTbUQVnCz8",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "!pip install --upgrade torch-scatter\n",
        "!pip install --upgrade torch-sparse\n",
        "!pip install --upgrade torch-cluster\n",
        "!pip install --upgrade torch-spline-conv \n",
        "!pip install torch-geometric"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "xjY9vtO9MgoL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "![alt text](https://raw.githubusercontent.com/rusty1s/pytorch_geometric/master/docs/source/_static/img/pyg_logo_text.svg?sanitize=true)"
      ]
    },
    {
      "metadata": {
        "id": "P3UffAf8M2Gw",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Intorduction"
      ]
    },
    {
      "metadata": {
        "id": "_4_eVOI2M4Uo",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "PyTorch Geometric [PyG](https://github.com/rusty1s/pytorch_geometric) is a geometric deep learning (GDN) extension library for PyTorch. In general GDN is used to generalize deep learning for non-Ecludian data. For the most part, CNN doesn't work very good for 3D shapes, point clouds and graph structures. Moreover, many real life datasets are inherently non-ecludian like social communicatin datasets, molecular structures, network traffic . etc ... \n",
        "\n",
        "Graph convolutional networks (GCN) come to the rescue to generalize CNNs to work for non-ecludian datasets. The basic architecture is illustrated below \n",
        "\n",
        "![alt text](https://tkipf.github.io/graph-convolutional-networks/images/gcn_web.png)\n",
        "\n",
        "where the input is a graph $G = (V,E)$ represented as \n",
        "\n",
        "*   Feature repsentation for each node $N \\times D$ where N is the number of nodes in the graph and $D$ is the number of features per node. \n",
        "*   A matrix repsentation of the graph in the form $2\\times L$ where $L$ is the number of edges in the graph. Each column in the matrix represents an edge between two nodes. \n",
        "*  Edge attributes of the form $L \\times R$ where R is the number of features per each edge. \n",
        "\n",
        "The output is of form $N \\times F$ where $F$ is the number of features per each node in the graph. \n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "YeA0slcJnQik",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import os.path as osp\n",
        "import torch\n",
        "import torch.nn.functional as F\n",
        "from torch_geometric.nn import SplineConv\n",
        "from torch_geometric.data import Data\n",
        "from random import shuffle, randint\n",
        "import networkx as nx\n",
        "import matplotlib.pyplot as plt\n",
        "import random "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "6pQ-c3ftL_gp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Dataset\n",
        "\n",
        "We will simulate a spammer vs non-spammer graph network. Given a node which represents a client that can send emails to different node (another client). \n",
        "\n",
        "Spammers have some similarities \n",
        "\n",
        "*   More likely to send lots of emails (more edges)\n",
        "*   More likely to send lots of data through email (we will represent an edge feature is the number of bytes where the value [0, 1] where 1 represents more bytes sent)\n",
        "*   Each spammer has an associated trust value which is given by the server. If the node is more likely to be a spammer then the value will be closer to 1. \n",
        "\n",
        "Non-spammers have the opposite features. In the next code snippet will try to simulate all of these features through randomization\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "MhlVjcdM7l6H",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "labels = []\n",
        "N = 1000 \n",
        "nodes = range(0, N)\n",
        "node_features = []\n",
        "edge_features = []\n",
        "\n",
        "for node in nodes:\n",
        "  \n",
        "  #spammer \n",
        "  if random.random() > 0.5:\n",
        "    #more likely to have many connections with a maximum of 1/5 of the nodes in the graph \n",
        "    nb_nbrs = int(random.random() * (N/5))\n",
        "    #more likely to have sent many bytes\n",
        "    node_features.append((random.random()+1) / 2.)\n",
        "    #more likely to have a high trust value \n",
        "    edge_features += [(random.random()+2)/3.] * nb_nbrs\n",
        "    #associate a label \n",
        "    labels.append(1)\n",
        "    \n",
        "  #non-spammer \n",
        "  else:\n",
        "    #at most connected to 10 nbrs \n",
        "    nb_nbrs = int(random.random() * 10 + 1)\n",
        "    #associate more bytes and random bytes \n",
        "    node_features.append(random.random())\n",
        "    edge_features += [random.random()] * nb_nbrs\n",
        "    labels.append(0)\n",
        "  \n",
        "  #connect to some random nodes \n",
        "  nbrs = np.random.choice(nodes, size = nb_nbrs)\n",
        "  nbrs = nbrs.reshape((1, nb_nbrs))\n",
        "  \n",
        "  #add the edges of nbrs \n",
        "  node_edges = np.concatenate([np.ones((1, nb_nbrs), dtype = np.int32) * node, nbrs], axis = 0)\n",
        "  \n",
        "  #add the overall edges \n",
        "  if node == 0:\n",
        "    edges = node_edges\n",
        "  else:\n",
        "    edges = np.concatenate([edges, node_edges], axis = 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qvfuQZv5lcM8",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Create a data structure "
      ]
    },
    {
      "metadata": {
        "id": "W1tyghgVFinu",
        "colab_type": "code",
        "outputId": "2c970876-76f9-4ef7-c8a5-d9a222b0a768",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "x = torch.tensor(np.expand_dims(node_features, 1), dtype=torch.float)\n",
        "y = torch.tensor(labels, dtype=torch.long)\n",
        "\n",
        "edge_index = torch.tensor(edges, dtype=torch.long)\n",
        "edge_attr = torch.tensor(np.expand_dims(edge_features, 1), dtype=torch.float)\n",
        "\n",
        "data = Data(x = x, edge_index=edge_index, y =y, edge_attr=edge_attr )\n",
        "print(data)"
      ],
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Data(edge_attr=[49077, 1], edge_index=[2, 49077], x=[1000, 1], y=[1000])\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bGcoGWzKlkHy",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "We will create a trian/test mask where we split the data into training and test. This is necessary because during optimizing the loss when training we don't want to include the nodes part of the testing process "
      ]
    },
    {
      "metadata": {
        "id": "WRwBaYmyoLDX",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "data.train_mask = torch.zeros(data.num_nodes, dtype=torch.uint8)\n",
        "data.train_mask[:int(0.8 * data.num_nodes)] = 1 #train only on the 80% nodes\n",
        "data.test_mask = torch.zeros(data.num_nodes, dtype=torch.uint8) #test on 20 % nodes \n",
        "data.test_mask[- int(0.2 * data.num_nodes):] = 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "H2YFmL6kl5Dh",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Deep GCN\n",
        "\n",
        "We will use [SplineConv](https://arxiv.org/abs/1711.08920) layer for the convolution. We will illsue exponential ReLU as an activation function and dropout for regulaization"
      ]
    },
    {
      "metadata": {
        "id": "MTlX4IBkoOnm",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Net(torch.nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        self.conv1 = SplineConv(1, 16, dim=1, kernel_size=5)\n",
        "        self.conv2 = SplineConv(16, 32, dim=1, kernel_size=5)\n",
        "        self.conv3 = SplineConv(32, 64, dim=1, kernel_size=7)\n",
        "        self.conv4 = SplineConv(64, 128, dim=1, kernel_size=7)\n",
        "        self.conv5 = SplineConv(128, 128, dim=1, kernel_size=11)\n",
        "        self.conv6 = SplineConv(128, 2, dim=1, kernel_size=11)\n",
        "\n",
        "    def forward(self):\n",
        "        x, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr\n",
        "        x = F.elu(self.conv1(x, edge_index, edge_attr))\n",
        "        x = self.conv2(x, edge_index, edge_attr)\n",
        "        x = F.elu(self.conv3(x, edge_index, edge_attr))\n",
        "        x = self.conv4(x, edge_index, edge_attr)\n",
        "        x = F.elu(self.conv5(x, edge_index, edge_attr))\n",
        "        x = self.conv6(x, edge_index, edge_attr)\n",
        "        x = F.dropout(x, training = self.training)\n",
        "        return F.log_softmax(x, dim=1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "pULYL97tmYel",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Optimization \n",
        "\n",
        "We will use nll_loss which can be used for classification of arbitrary classes"
      ]
    },
    {
      "metadata": {
        "id": "Hhabp4QvoP6V",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate_loss(mode = 'train'):\n",
        "  \n",
        "  #use masking for loss evaluation \n",
        "  if mode == 'train':\n",
        "    loss = F.nll_loss(model()[data.train_mask], data.y[data.train_mask])\n",
        "  else:\n",
        "    loss = F.nll_loss(model()[data.test_mask], data.y[data.test_mask])\n",
        "  return loss\n",
        "\n",
        "def train():\n",
        "  #training \n",
        "  model.train()\n",
        "  optimizer.zero_grad()\n",
        "  loss = evaluate_loss()\n",
        "  loss.backward()\n",
        "  optimizer.step()\n",
        "  return loss.detach().cpu().numpy() \n",
        "\n",
        "def test():\n",
        "  #testing \n",
        "  model.eval()\n",
        "  logits, accs = model(), []\n",
        "  loss = evaluate_loss(mode = 'test').detach().cpu().numpy() \n",
        "\n",
        "  for _, mask in data('train_mask', 'test_mask'):\n",
        "      pred = logits[mask].max(1)[1]\n",
        "      acc = pred.eq(data.y[mask]).sum().item() / mask.sum().item()\n",
        "      accs.append(acc)\n",
        "  return [loss] + accs"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "y0XicLqpmqwR",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Setup the model \n",
        "We will create the model and setup training using adam optimizer"
      ]
    },
    {
      "metadata": {
        "id": "sDvcl5eLoRb3",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "model, data = Net().to(device), data.to(device)\n",
        "optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "FyyfCGZimtX2",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Training and Testing"
      ]
    },
    {
      "metadata": {
        "id": "qsslw_68oS52",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 3400
        },
        "outputId": "aa47e6c7-8985-4409-bfe5-805c0e3df3ae"
      },
      "cell_type": "code",
      "source": [
        "losses = []\n",
        "for epoch in range(1, 200):\n",
        "  train_loss = train()\n",
        "  log = 'Epoch: {:03d}, train_loss: {:.3f}, test_loss:{:.3f}, train_acc: {:.2f}, test_acc: {:.2f}'\n",
        "  test_loss = test()[0]\n",
        "  losses.append([train_loss,test_loss])\n",
        "  print(log.format(epoch, train_loss, *test()))"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch: 001, train_loss: 0.692, test_loss:0.687, train_acc: 0.54, test_acc: 0.51\n",
            "Epoch: 002, train_loss: 0.686, test_loss:0.680, train_acc: 0.80, test_acc: 0.81\n",
            "Epoch: 003, train_loss: 0.680, test_loss:0.670, train_acc: 0.82, test_acc: 0.83\n",
            "Epoch: 004, train_loss: 0.671, test_loss:0.656, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 005, train_loss: 0.657, test_loss:0.635, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 006, train_loss: 0.639, test_loss:0.606, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 007, train_loss: 0.613, test_loss:0.570, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 008, train_loss: 0.585, test_loss:0.525, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 009, train_loss: 0.554, test_loss:0.477, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 010, train_loss: 0.513, test_loss:0.433, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 011, train_loss: 0.503, test_loss:0.403, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 012, train_loss: 0.524, test_loss:0.388, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 013, train_loss: 0.501, test_loss:0.385, train_acc: 0.82, test_acc: 0.83\n",
            "Epoch: 014, train_loss: 0.478, test_loss:0.393, train_acc: 0.82, test_acc: 0.83\n",
            "Epoch: 015, train_loss: 0.466, test_loss:0.414, train_acc: 0.81, test_acc: 0.82\n",
            "Epoch: 016, train_loss: 0.476, test_loss:0.427, train_acc: 0.81, test_acc: 0.77\n",
            "Epoch: 017, train_loss: 0.472, test_loss:0.416, train_acc: 0.81, test_acc: 0.80\n",
            "Epoch: 018, train_loss: 0.480, test_loss:0.398, train_acc: 0.82, test_acc: 0.83\n",
            "Epoch: 019, train_loss: 0.483, test_loss:0.389, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 020, train_loss: 0.463, test_loss:0.385, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 021, train_loss: 0.460, test_loss:0.386, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 022, train_loss: 0.467, test_loss:0.392, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 023, train_loss: 0.462, test_loss:0.400, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 024, train_loss: 0.462, test_loss:0.406, train_acc: 0.81, test_acc: 0.83\n",
            "Epoch: 025, train_loss: 0.459, test_loss:0.405, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 026, train_loss: 0.468, test_loss:0.400, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 027, train_loss: 0.472, test_loss:0.393, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 028, train_loss: 0.461, test_loss:0.389, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 029, train_loss: 0.476, test_loss:0.387, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 030, train_loss: 0.476, test_loss:0.387, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 031, train_loss: 0.459, test_loss:0.387, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 032, train_loss: 0.466, test_loss:0.389, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 033, train_loss: 0.457, test_loss:0.388, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 034, train_loss: 0.463, test_loss:0.386, train_acc: 0.82, test_acc: 0.84\n",
            "Epoch: 035, train_loss: 0.454, test_loss:0.385, train_acc: 0.82, test_acc: 0.85\n",
            "Epoch: 036, train_loss: 0.456, test_loss:0.385, train_acc: 0.82, test_acc: 0.85\n",
            "Epoch: 037, train_loss: 0.466, test_loss:0.386, train_acc: 0.82, test_acc: 0.85\n",
            "Epoch: 038, train_loss: 0.452, test_loss:0.387, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 039, train_loss: 0.462, test_loss:0.389, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 040, train_loss: 0.456, test_loss:0.389, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 041, train_loss: 0.455, test_loss:0.388, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 042, train_loss: 0.446, test_loss:0.387, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 043, train_loss: 0.466, test_loss:0.386, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 044, train_loss: 0.456, test_loss:0.386, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 045, train_loss: 0.450, test_loss:0.384, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 046, train_loss: 0.446, test_loss:0.383, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 047, train_loss: 0.440, test_loss:0.383, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 048, train_loss: 0.469, test_loss:0.383, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 049, train_loss: 0.456, test_loss:0.383, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 050, train_loss: 0.445, test_loss:0.381, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 051, train_loss: 0.456, test_loss:0.378, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 052, train_loss: 0.453, test_loss:0.376, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 053, train_loss: 0.444, test_loss:0.374, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 054, train_loss: 0.451, test_loss:0.372, train_acc: 0.83, test_acc: 0.84\n",
            "Epoch: 055, train_loss: 0.447, test_loss:0.371, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 056, train_loss: 0.446, test_loss:0.370, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 057, train_loss: 0.433, test_loss:0.369, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 058, train_loss: 0.446, test_loss:0.365, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 059, train_loss: 0.438, test_loss:0.363, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 060, train_loss: 0.443, test_loss:0.360, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 061, train_loss: 0.446, test_loss:0.359, train_acc: 0.83, test_acc: 0.85\n",
            "Epoch: 062, train_loss: 0.431, test_loss:0.360, train_acc: 0.84, test_acc: 0.86\n",
            "Epoch: 063, train_loss: 0.437, test_loss:0.360, train_acc: 0.84, test_acc: 0.86\n",
            "Epoch: 064, train_loss: 0.419, test_loss:0.357, train_acc: 0.85, test_acc: 0.86\n",
            "Epoch: 065, train_loss: 0.416, test_loss:0.353, train_acc: 0.85, test_acc: 0.87\n",
            "Epoch: 066, train_loss: 0.429, test_loss:0.349, train_acc: 0.85, test_acc: 0.87\n",
            "Epoch: 067, train_loss: 0.415, test_loss:0.343, train_acc: 0.86, test_acc: 0.88\n",
            "Epoch: 068, train_loss: 0.401, test_loss:0.337, train_acc: 0.87, test_acc: 0.88\n",
            "Epoch: 069, train_loss: 0.403, test_loss:0.330, train_acc: 0.87, test_acc: 0.88\n",
            "Epoch: 070, train_loss: 0.408, test_loss:0.325, train_acc: 0.88, test_acc: 0.89\n",
            "Epoch: 071, train_loss: 0.414, test_loss:0.324, train_acc: 0.89, test_acc: 0.89\n",
            "Epoch: 072, train_loss: 0.389, test_loss:0.323, train_acc: 0.89, test_acc: 0.89\n",
            "Epoch: 073, train_loss: 0.406, test_loss:0.318, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 074, train_loss: 0.388, test_loss:0.310, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 075, train_loss: 0.394, test_loss:0.304, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 076, train_loss: 0.372, test_loss:0.300, train_acc: 0.90, test_acc: 0.91\n",
            "Epoch: 077, train_loss: 0.353, test_loss:0.298, train_acc: 0.90, test_acc: 0.91\n",
            "Epoch: 078, train_loss: 0.378, test_loss:0.298, train_acc: 0.90, test_acc: 0.91\n",
            "Epoch: 079, train_loss: 0.370, test_loss:0.299, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 080, train_loss: 0.375, test_loss:0.303, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 081, train_loss: 0.374, test_loss:0.309, train_acc: 0.89, test_acc: 0.90\n",
            "Epoch: 082, train_loss: 0.381, test_loss:0.312, train_acc: 0.89, test_acc: 0.90\n",
            "Epoch: 083, train_loss: 0.382, test_loss:0.304, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 084, train_loss: 0.397, test_loss:0.304, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 085, train_loss: 0.363, test_loss:0.306, train_acc: 0.90, test_acc: 0.90\n",
            "Epoch: 086, train_loss: 0.369, test_loss:0.309, train_acc: 0.90, test_acc: 0.90\n",
            "Epoch: 087, train_loss: 0.389, test_loss:0.316, train_acc: 0.89, test_acc: 0.89\n",
            "Epoch: 088, train_loss: 0.396, test_loss:0.314, train_acc: 0.90, test_acc: 0.90\n",
            "Epoch: 089, train_loss: 0.398, test_loss:0.312, train_acc: 0.90, test_acc: 0.90\n",
            "Epoch: 090, train_loss: 0.359, test_loss:0.312, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 091, train_loss: 0.378, test_loss:0.316, train_acc: 0.90, test_acc: 0.90\n",
            "Epoch: 092, train_loss: 0.370, test_loss:0.319, train_acc: 0.90, test_acc: 0.91\n",
            "Epoch: 093, train_loss: 0.379, test_loss:0.321, train_acc: 0.90, test_acc: 0.90\n",
            "Epoch: 094, train_loss: 0.341, test_loss:0.313, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 095, train_loss: 0.377, test_loss:0.312, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 096, train_loss: 0.371, test_loss:0.315, train_acc: 0.90, test_acc: 0.91\n",
            "Epoch: 097, train_loss: 0.374, test_loss:0.322, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 098, train_loss: 0.367, test_loss:0.319, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 099, train_loss: 0.359, test_loss:0.315, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 100, train_loss: 0.365, test_loss:0.312, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 101, train_loss: 0.377, test_loss:0.315, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 102, train_loss: 0.352, test_loss:0.323, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 103, train_loss: 0.362, test_loss:0.327, train_acc: 0.90, test_acc: 0.88\n",
            "Epoch: 104, train_loss: 0.362, test_loss:0.312, train_acc: 0.91, test_acc: 0.91\n",
            "Epoch: 105, train_loss: 0.356, test_loss:0.307, train_acc: 0.91, test_acc: 0.92\n",
            "Epoch: 106, train_loss: 0.346, test_loss:0.309, train_acc: 0.92, test_acc: 0.92\n",
            "Epoch: 107, train_loss: 0.366, test_loss:0.318, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 108, train_loss: 0.360, test_loss:0.322, train_acc: 0.91, test_acc: 0.89\n",
            "Epoch: 109, train_loss: 0.377, test_loss:0.311, train_acc: 0.92, test_acc: 0.92\n",
            "Epoch: 110, train_loss: 0.362, test_loss:0.304, train_acc: 0.91, test_acc: 0.92\n",
            "Epoch: 111, train_loss: 0.396, test_loss:0.304, train_acc: 0.91, test_acc: 0.92\n",
            "Epoch: 112, train_loss: 0.360, test_loss:0.314, train_acc: 0.91, test_acc: 0.90\n",
            "Epoch: 113, train_loss: 0.368, test_loss:0.332, train_acc: 0.89, test_acc: 0.85\n",
            "Epoch: 114, train_loss: 0.352, test_loss:0.309, train_acc: 0.92, test_acc: 0.91\n",
            "Epoch: 115, train_loss: 0.355, test_loss:0.302, train_acc: 0.91, test_acc: 0.92\n",
            "Epoch: 116, train_loss: 0.352, test_loss:0.301, train_acc: 0.91, test_acc: 0.92\n",
            "Epoch: 117, train_loss: 0.371, test_loss:0.307, train_acc: 0.92, test_acc: 0.91\n",
            "Epoch: 118, train_loss: 0.364, test_loss:0.336, train_acc: 0.90, test_acc: 0.84\n",
            "Epoch: 119, train_loss: 0.361, test_loss:0.325, train_acc: 0.90, test_acc: 0.86\n",
            "Epoch: 120, train_loss: 0.345, test_loss:0.304, train_acc: 0.92, test_acc: 0.92\n",
            "Epoch: 121, train_loss: 0.359, test_loss:0.306, train_acc: 0.91, test_acc: 0.92\n",
            "Epoch: 122, train_loss: 0.357, test_loss:0.304, train_acc: 0.92, test_acc: 0.92\n",
            "Epoch: 123, train_loss: 0.367, test_loss:0.312, train_acc: 0.92, test_acc: 0.91\n",
            "Epoch: 124, train_loss: 0.346, test_loss:0.325, train_acc: 0.92, test_acc: 0.87\n",
            "Epoch: 125, train_loss: 0.350, test_loss:0.317, train_acc: 0.92, test_acc: 0.89\n",
            "Epoch: 126, train_loss: 0.354, test_loss:0.307, train_acc: 0.92, test_acc: 0.92\n",
            "Epoch: 127, train_loss: 0.325, test_loss:0.304, train_acc: 0.93, test_acc: 0.92\n",
            "Epoch: 128, train_loss: 0.363, test_loss:0.306, train_acc: 0.93, test_acc: 0.92\n",
            "Epoch: 129, train_loss: 0.348, test_loss:0.312, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 130, train_loss: 0.330, test_loss:0.320, train_acc: 0.91, test_acc: 0.89\n",
            "Epoch: 131, train_loss: 0.346, test_loss:0.314, train_acc: 0.92, test_acc: 0.90\n",
            "Epoch: 132, train_loss: 0.336, test_loss:0.309, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 133, train_loss: 0.336, test_loss:0.306, train_acc: 0.92, test_acc: 0.92\n",
            "Epoch: 134, train_loss: 0.346, test_loss:0.307, train_acc: 0.93, test_acc: 0.92\n",
            "Epoch: 135, train_loss: 0.359, test_loss:0.312, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 136, train_loss: 0.336, test_loss:0.317, train_acc: 0.92, test_acc: 0.89\n",
            "Epoch: 137, train_loss: 0.332, test_loss:0.314, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 138, train_loss: 0.353, test_loss:0.311, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 139, train_loss: 0.333, test_loss:0.311, train_acc: 0.93, test_acc: 0.92\n",
            "Epoch: 140, train_loss: 0.359, test_loss:0.313, train_acc: 0.93, test_acc: 0.92\n",
            "Epoch: 141, train_loss: 0.362, test_loss:0.323, train_acc: 0.91, test_acc: 0.88\n",
            "Epoch: 142, train_loss: 0.359, test_loss:0.327, train_acc: 0.91, test_acc: 0.88\n",
            "Epoch: 143, train_loss: 0.336, test_loss:0.312, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 144, train_loss: 0.337, test_loss:0.309, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 145, train_loss: 0.356, test_loss:0.310, train_acc: 0.93, test_acc: 0.92\n",
            "Epoch: 146, train_loss: 0.336, test_loss:0.315, train_acc: 0.93, test_acc: 0.90\n",
            "Epoch: 147, train_loss: 0.357, test_loss:0.327, train_acc: 0.92, test_acc: 0.87\n",
            "Epoch: 148, train_loss: 0.340, test_loss:0.320, train_acc: 0.93, test_acc: 0.90\n",
            "Epoch: 149, train_loss: 0.349, test_loss:0.314, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 150, train_loss: 0.352, test_loss:0.314, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 151, train_loss: 0.330, test_loss:0.315, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 152, train_loss: 0.331, test_loss:0.318, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 153, train_loss: 0.341, test_loss:0.321, train_acc: 0.93, test_acc: 0.90\n",
            "Epoch: 154, train_loss: 0.340, test_loss:0.323, train_acc: 0.93, test_acc: 0.89\n",
            "Epoch: 155, train_loss: 0.351, test_loss:0.319, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 156, train_loss: 0.340, test_loss:0.318, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 157, train_loss: 0.360, test_loss:0.320, train_acc: 0.93, test_acc: 0.91\n",
            "Epoch: 158, train_loss: 0.336, test_loss:0.324, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 159, train_loss: 0.336, test_loss:0.324, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 160, train_loss: 0.331, test_loss:0.323, train_acc: 0.94, test_acc: 0.90\n",
            "Epoch: 161, train_loss: 0.335, test_loss:0.324, train_acc: 0.94, test_acc: 0.92\n",
            "Epoch: 162, train_loss: 0.359, test_loss:0.324, train_acc: 0.94, test_acc: 0.90\n",
            "Epoch: 163, train_loss: 0.338, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 164, train_loss: 0.321, test_loss:0.328, train_acc: 0.93, test_acc: 0.90\n",
            "Epoch: 165, train_loss: 0.328, test_loss:0.326, train_acc: 0.93, test_acc: 0.90\n",
            "Epoch: 166, train_loss: 0.322, test_loss:0.324, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 167, train_loss: 0.342, test_loss:0.325, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 168, train_loss: 0.312, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 169, train_loss: 0.331, test_loss:0.328, train_acc: 0.94, test_acc: 0.90\n",
            "Epoch: 170, train_loss: 0.312, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 171, train_loss: 0.341, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 172, train_loss: 0.318, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 173, train_loss: 0.333, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 174, train_loss: 0.333, test_loss:0.326, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 175, train_loss: 0.354, test_loss:0.331, train_acc: 0.93, test_acc: 0.89\n",
            "Epoch: 176, train_loss: 0.321, test_loss:0.326, train_acc: 0.94, test_acc: 0.89\n",
            "Epoch: 177, train_loss: 0.321, test_loss:0.318, train_acc: 0.95, test_acc: 0.91\n",
            "Epoch: 178, train_loss: 0.341, test_loss:0.316, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 179, train_loss: 0.320, test_loss:0.317, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 180, train_loss: 0.301, test_loss:0.322, train_acc: 0.94, test_acc: 0.89\n",
            "Epoch: 181, train_loss: 0.303, test_loss:0.318, train_acc: 0.94, test_acc: 0.90\n",
            "Epoch: 182, train_loss: 0.310, test_loss:0.317, train_acc: 0.95, test_acc: 0.92\n",
            "Epoch: 183, train_loss: 0.320, test_loss:0.321, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 184, train_loss: 0.313, test_loss:0.325, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 185, train_loss: 0.335, test_loss:0.327, train_acc: 0.93, test_acc: 0.89\n",
            "Epoch: 186, train_loss: 0.316, test_loss:0.321, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 187, train_loss: 0.302, test_loss:0.319, train_acc: 0.94, test_acc: 0.92\n",
            "Epoch: 188, train_loss: 0.306, test_loss:0.320, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 189, train_loss: 0.343, test_loss:0.318, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 190, train_loss: 0.314, test_loss:0.311, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 191, train_loss: 0.310, test_loss:0.315, train_acc: 0.94, test_acc: 0.90\n",
            "Epoch: 192, train_loss: 0.334, test_loss:0.315, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 193, train_loss: 0.316, test_loss:0.312, train_acc: 0.94, test_acc: 0.92\n",
            "Epoch: 194, train_loss: 0.328, test_loss:0.315, train_acc: 0.94, test_acc: 0.90\n",
            "Epoch: 195, train_loss: 0.303, test_loss:0.324, train_acc: 0.94, test_acc: 0.89\n",
            "Epoch: 196, train_loss: 0.304, test_loss:0.318, train_acc: 0.94, test_acc: 0.91\n",
            "Epoch: 197, train_loss: 0.328, test_loss:0.309, train_acc: 0.95, test_acc: 0.91\n",
            "Epoch: 198, train_loss: 0.323, test_loss:0.309, train_acc: 0.95, test_acc: 0.91\n",
            "Epoch: 199, train_loss: 0.296, test_loss:0.312, train_acc: 0.95, test_acc: 0.91\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "adWu02_enxNp",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# References\n",
        "[1] https://github.com/rusty1s/pytorch_geometric\n",
        "\n",
        "[2] https://rusty1s.github.io/pytorch_geometric/build/html/notes/introduction.html\n",
        "\n",
        "[3] https://tkipf.github.io/graph-convolutional-networks/"
      ]
    }
  ]
}