machine_learning_artistic_style_transfer_en.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "name": "machine-learning-artistic-style-transfer-en.ipynb",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/shellerbrand/machine-learning-for-artistic-style/blob/update-lmwl-2020/machine_learning_artistic_style_transfer_en.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxYzWnjJr86x"
      },
      "source": [
        "# New Works by old Masters - Image Style Transfer Using Neural Networks\n",
        "\n",
        "Visual recognition of objects is a task that humans excel at. In recent days, by using neural networks computers have also become very reliable at detecting objects in specific contexts.\n",
        "\n",
        "Particularly good performance has been reported for the class of [Convolutional Neural Networks](https://en.wikipedia.org/wiki/Convolutional_Neural_Network) (ConvNets). These networks consists of a sequence of layers. The outputs of the first layers correspond to pixel-level patterns and the outputs further to the end of the sequences describe larger-scale patterns and objects.\n",
        "\n",
        "\n",
        "A research team from Tübingen [has described](https://arxiv.org/abs/1508.06576) how the style of paintings can be described via the activations of the earlier layers of a ConvNet. The have also shown a procedure, by which an image can be created such that it corresponds to one image in terms of content (the __content image__) and in style to another image (the  __style image__).\n",
        "\n",
        "<img src=\"https://github.com/shellerbrand/machine-learning-for-artistic-style/raw/master/resources/vgg-convnet-content-and-style.jpg\" alt=\"Convolutional Neural Network\" width=\"700\"/>\n",
        "\n",
        "__With this notebook you can experiment with Neural Style Transfer.__\n",
        "\n",
        "The example was created for use in the course [Lernen, wie Maschinen lernen](https://www.mzl.uni-muenchen.de/lehramtpro/programm/Lernen_-wie-Maschinen-lernen/index.html) at LMU Munich. We are using some modified code from the [Google Magenta Repo](https://github.com/tensorflow/magenta) that implements the style transfer scheme described by [Ghiasi et al.](https://arxiv.org/abs/1705.06830).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "52Um4cugyInh"
      },
      "source": [
        "---\n",
        "\n",
        "## Preparation\n",
        "\n",
        "These steps only have to be run once."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "5tbnXXX2sNYx"
      },
      "source": [
        "#@title Load required models and copy style-transfer code to this runtime\n",
        "!curl https://raw.githubusercontent.com/shellerbrand/machine-learning-for-artistic-style/master/dist/stylization-lib.tar.gz -o image-stylization-lib.tar.gz\n",
        "\n",
        "import sys\n",
        "IN_COLAB = 'google.colab' in sys.modules\n",
        "# Ensure the required version of scipy is installed in colab\n",
        "if IN_COLAB:\n",
        "  !pip install numpy==1.16.3\n",
        "  !pip install tensorflow==1.13.2\n",
        "  !pip install keras-applications==1.0.7\n",
        "  !pip install keras-preprocessing==1.0.9\n",
        "  !pip install scipy==1.2.1\n",
        "\n",
        "# The Model can be downloaded as per Google Magenta project\n",
        "!curl https://storage.googleapis.com/download.magenta.tensorflow.org/models/arbitrary_style_transfer.tar.gz -o image-stylization-checkpoint.tar.gz\n",
        "\n",
        "# Unpack\n",
        "!tar -zxvf image-stylization-lib.tar.gz\n",
        "!tar -zxvf image-stylization-checkpoint.tar.gz\n",
        "print('\\nDone.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "8HPnqEClr86y"
      },
      "source": [
        "#@title Load helper functions (required for loading and display of images)\n",
        "\n",
        "# \n",
        "# Funktionen um Bilder zu laden vom Image-Upload Tool\n",
        "#\n",
        "\n",
        "print('Loading functions for downloading style and content images.')\n",
        "\n",
        "import requests\n",
        "import os\n",
        "import shutil\n",
        "\n",
        "HOST = 'https://imageupload.hellerbit.com'\n",
        "API_KEY = 'bce8a814f0b3882ac6f2810464128b42dbfee105e1ba84311a77b3f4'\n",
        "DIR = './tmp/'\n",
        "EXP_CONTENT = '2_inhalt'\n",
        "EXP_STYLE = '2_stil'\n",
        "OUTDIR = './output/'\n",
        "\n",
        "\n",
        "if not os.path.isdir(DIR):\n",
        "    print('Making the directory: {}'.format(DIR))\n",
        "    os.makedirs(DIR)\n",
        "\n",
        "def clear_output():\n",
        "    if os.path.isdir(OUTDIR):\n",
        "        shutil.rmtree(OUTDIR)\n",
        "        os.mkdir(OUTDIR)\n",
        "\n",
        "def clear_experiment(experiment_id):\n",
        "    expdir = os.path.join(DIR, experiment_id)\n",
        "    if os.path.isdir(expdir):\n",
        "        shutil.rmtree(expdir)\n",
        "\n",
        "def download_experiment_images(experiment_id):\n",
        "    url = ''.join([HOST, '/api/{}/all.txt'.format(experiment_id)])\n",
        "    headers = {'APIKEY': API_KEY}\n",
        "    resp = requests.get(url, headers=headers)\n",
        "\n",
        "    print(resp.text)\n",
        "    files = resp.text.split('\\n')\n",
        "    # Make the directory for the experiment\n",
        "    expdir = os.path.join(DIR, experiment_id)\n",
        "    if not os.path.isdir(expdir):\n",
        "            os.makedirs(expdir)\n",
        "\n",
        "    for line in files:\n",
        "        print(\"Loading this file: {}\".format(line))\n",
        "        url = ''.join([HOST, '/api/{}/{}'.format(experiment_id, line)])\n",
        "        path = os.path.join(expdir, line)\n",
        "        print(\"\\t from URL {}\".format(url))\n",
        "        r = requests.get(url, stream=True, headers=headers)\n",
        "        if r.status_code == 200:\n",
        "            with open(path, 'wb') as f:\n",
        "                for chunk in r:\n",
        "                    f.write(chunk)\n",
        "                    \n",
        "#\n",
        "# Funktionen zur Anzeige von Bildern laden\n",
        "#\n",
        "\n",
        "print('Loading functions for showing images.')\n",
        "# Configuration of matplotlib\n",
        "import matplotlib.pyplot as plt\n",
        "import matplotlib as mpl\n",
        "mpl.rcParams['figure.figsize'] = (10,10)\n",
        "mpl.rcParams['axes.grid'] = False\n",
        "\n",
        "# python image library, numpy etc.\n",
        "import numpy as np\n",
        "from PIL import Image\n",
        "import time\n",
        "import functools\n",
        "\n",
        "from tensorflow.python.keras.preprocessing import image as kp_image\n",
        "\n",
        "# Helper function for loading images from a path\n",
        "def load_img(path_to_img):\n",
        "  max_dim = 1024\n",
        "  img = Image.open(path_to_img)\n",
        "  long = max(img.size)\n",
        "  scale = max_dim/long\n",
        "  img = img.resize((round(img.size[0]*scale), round(img.size[1]*scale)), \n",
        "                   Image.ANTIALIAS)\n",
        "  \n",
        "  img = kp_image.img_to_array(img)\n",
        "  \n",
        "  # We need to broadcast the image array such that it has a batch dimension \n",
        "  img = np.expand_dims(img, axis=0)\n",
        "  return img\n",
        "\n",
        "# Helper function for showing an image object (as per PIL)\n",
        "def imshow(img, title=None):\n",
        "  # Remove the batch dimension\n",
        "  out = np.squeeze(img, axis=0)\n",
        "  # Normalize for display \n",
        "  out = out.astype('uint8')\n",
        "  plt.imshow(out)\n",
        "  if title is not None:\n",
        "    plt.title(title)\n",
        "  plt.axis('off')\n",
        "  plt.imshow(out)\n",
        "  \n",
        "\n",
        "print('\\nDone.')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9K7v2hAAr862"
      },
      "source": [
        "---\n",
        "\n",
        "\n",
        "## Upload Images\n",
        "\n",
        "In the next step the images are imported, for which the style-transfer will be applied. You can adjust the images that server as model for the style. These __style images__ can be uploaded via the following link:\n",
        "\n",
        "https://imageupload.hellerbit.com/images/2_stil/\n",
        "\n",
        "The __content images__, to which the style will be applied are uploaded on this link:\n",
        "\n",
        "https://imageupload.hellerbit.com/images/2_inhalt/\n",
        "\n",
        "__Please note:__ You have to login to the image upload portal and before that you need to create an account. You can go ahead right now if you just want to see what the following steps look like for a demo-account. All accounts are deleted on every day at midnight Berlin time. Your images can only be accessed via your login or by using the API-key.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w8toep0Kr862"
      },
      "source": [
        "<img src=\"https://github.com/shellerbrand/machine-learning-for-artistic-style/raw/master/resources/iphone_upload_tool_list_en.jpg\" width=\"300\" style=\"float:right;\">\n",
        "\n",
        "Now please enter the API-key for loading the images. When you run the next cell, then this key will be used to download the images to this notebook."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "_kyOe2cXr864"
      },
      "source": [
        "#@title Please enter the key for image download:\n",
        "API_KEY = \"3c7f5f9a4d227972f0989cb7c1be7ad0835990d03f84ed8efbae839a\" #@param {type:\"string\"}\n",
        "clear_experiment(EXP_CONTENT)\n",
        "clear_experiment(EXP_STYLE)\n",
        "clear_output()\n",
        "download_experiment_images(EXP_CONTENT)\n",
        "download_experiment_images(EXP_STYLE)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5wptVGUt7Zze"
      },
      "source": [
        "## Transfer Image-Styles\n",
        "\n",
        "Now the image style can be transferred.\n",
        "\n",
        "In order to do that, for every __style image__ a vector S is computed by the network for style-analysis. This vector described the style of the image and can be passed to the style-transfer network, which transfers this style onto the __content image__.\n",
        "\n",
        "<img src=\"https://github.com/shellerbrand/machine-learning-for-artistic-style/raw/master/resources/networks-for-analysis-and-transfer.jpg\" alt=\"Convolutional Neural Network\" width=\"700\"/>\n",
        "\n",
        "The networks used in this example have already been trained by Google in their Project Magenta. In simple terms, training means that the network for style-analysis has already seen tens of thousands of images. Because of the experience it has gained from looking at all these pictures it can also describe the style for images it has not seen before. In a similar fashion the style-transfer network has been trained with many images on how to transfer a specific style onto a content-image.\n",
        "\n",
        "Both of these pre-trained networks are now applied for all combinations of __style images__ and __content images__ that have been uploaded in the previous step.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "iXlSva7Hr867",
        "scrolled": true
      },
      "source": [
        "#@title Start style-transfer\n",
        "from lib.arbitrary_image_stylization_with_weights_func import arbitrary_stylization_with_weights, StyleTransferConfig\n",
        "# from lib.arbitrary_image_stylization_with_weights import main as dostyle\n",
        "import tensorflow as tf\n",
        "\n",
        "c = StyleTransferConfig()\n",
        "c.content_images_paths = os.path.join(DIR,'2_inhalt/*')\n",
        "c.output_dir = OUTDIR\n",
        "c.style_images_paths = os.path.join(DIR,'2_stil/*')\n",
        "c.checkpoint = './arbitrary_style_transfer/model.ckpt'\n",
        "c.image_size = 512\n",
        "c.style_image_size = 256\n",
        "print(c)\n",
        "\n",
        "with tf.device('/gpu:0'):\n",
        "  arbitrary_stylization_with_weights(c)\n",
        "\n",
        "print(\"\\nDone.\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PpHk--79r86_"
      },
      "source": [
        "## View images\n",
        "\n",
        "If the style-transfer has finished without errors, then you can run the following steps to view the images."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "dr12ukjYr87D"
      },
      "source": [
        "#@title View content images\n",
        "# Get all content images\n",
        "cols = 3\n",
        "basewidth=20\n",
        "cfiles = os.listdir(path=os.path.join(DIR, EXP_CONTENT))\n",
        "#print(\"{},{}\".format(basewidth,len(files)/cols*basewidth))\n",
        "plt.figure(num=1, figsize=(basewidth,len(cfiles)/(cols*cols)*basewidth))\n",
        "pind = 1\n",
        "for fi in cfiles:\n",
        "    path = os.path.join(DIR, EXP_CONTENT, fi)\n",
        "    # print(path)\n",
        "    im = load_img(path_to_img=path).astype('uint8')    \n",
        "    plt.subplot(np.ceil(len(cfiles)/cols),cols,pind)\n",
        "    imshow(im,title=fi)\n",
        "    pind = pind + 1\n",
        "    \n",
        "print(\"The images that styles will be applied to:\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "B327ydjkr87G"
      },
      "source": [
        "#@title View style images\n",
        "# Get all style images\n",
        "basewidth=20\n",
        "sfiles = os.listdir(path=os.path.join(DIR, EXP_STYLE))\n",
        "cols = len(sfiles)+1\n",
        "#print(\"{},{}\".format(basewidth,len(files)/cols*basewidth))\n",
        "plt.figure(num=1, figsize=(basewidth,len(sfiles)/(cols*cols)*basewidth))\n",
        "plt.axis('off')\n",
        "\n",
        "pind = 1\n",
        "sfiles.sort()\n",
        "for fi in sfiles:\n",
        "    path = os.path.join(DIR, EXP_STYLE, fi)\n",
        "    # print(path)\n",
        "    im = load_img(path_to_img=path).astype('uint8')    \n",
        "    plt.subplot(np.ceil(len(sfiles)/cols),cols,pind)\n",
        "    imshow(im,title=fi)\n",
        "    pind = pind + 1\n",
        "    \n",
        "print(\"The style images:\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "iHyKeDh2r87Q"
      },
      "source": [
        "#@title View images with style-transfer applied\n",
        "# Stylized Images\n",
        "from re import match\n",
        "\n",
        "cols = 3\n",
        "basewidth=20\n",
        "files = os.listdir(path=os.path.join(OUTDIR))\n",
        "files = [x for x in files if match('^zzResult.*',x)]\n",
        "#print(\"{},{}\".format(basewidth,len(files)/cols*basewidth))\n",
        "plt.figure(num=1, figsize=(basewidth,len(cfiles)/(len(sfiles)+1)*basewidth))\n",
        "pind = 1\n",
        "files.sort()\n",
        "for fi in files:\n",
        "    path = os.path.join(OUTDIR, fi)\n",
        "    # print(path)\n",
        "    im = load_img(path_to_img=path).astype('uint8')    \n",
        "    plt.subplot(len(cfiles)+1,len(sfiles)+1,pind)\n",
        "    imshow(im,title=fi)\n",
        "    pind = pind + 1\n",
        "    \n",
        "print(\"The images that result from style-transfer:\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "fyOsDJCxC2Op"
      },
      "source": [
        "#@title Download all images in a .zip file\n",
        "\n",
        "if IN_COLAB:\n",
        "    !zip output.zip output/*\n",
        "    from google.colab import files\n",
        "    files.download(\"output.zip\")\n",
        "else:\n",
        "    print(\"Not in colab - skipping.\")\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vB8asjdgwOxK"
      },
      "source": [
        "---\n",
        "\n",
        "## References\n",
        "\n",
        "* Gatys, L. et al; A Neural Algorithm of Artistic Style, Sep 2015, [arxiv](https://arxiv.org/abs/1508.06576)\n",
        "* Google Magenta: Fast Style Transfer for Arbitrary Styles, [Github](https://github.com/tensorflow/magenta/blob/2c3ae9b0dd64b06295e48e2ee5654e3d207035fc/magenta/models/arbitrary_image_stylization/README.md)  \n",
        "* Ghiasi, G. et al.; \n",
        "Exploring the structure of a real-time, arbitrary neural artistic stylization network\n",
        "Golnaz Ghiasi, Aug 2017, [arxiv](https://arxiv.org/abs/1705.06830)\n",
        "\n",
        "The Source-Code for this notebook and the tool for uploading images is or will be published here:\n",
        "https://github.com/shellerbrand/machine-learning-for-artistic-style\n"
      ]
    }
  ]
}