site/en/guide/migrate.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "wJcYs_ERTnnI"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "HMUDt0CiUJk9"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "77z2OchJTk0l"
      },
      "source": [
        "# Migrate your TensorFlow 1 code to TensorFlow 2\n",
        "\n",
        "\u003ctable class=\"tfo-notebook-buttons\" align=\"left\"\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://www.tensorflow.org/guide/migrate\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" /\u003e\n",
        "    View on TensorFlow.org\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/migrate.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /\u003e\n",
        "    Run in Google Colab\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/migrate.ipynb\"\u003e\n",
        "    \u003cimg src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /\u003e\n",
        "    View source on GitHub\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "  \u003ctd\u003e\n",
        "    \u003ca href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/migrate.ipynb\"\u003e\u003cimg src=\"https://www.tensorflow.org/images/download_logo_32px.png\" /\u003eDownload notebook\u003c/a\u003e\n",
        "  \u003c/td\u003e\n",
        "\u003c/table\u003e"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "meUTrR4I6m1C"
      },
      "source": [
        "Important: This doc for users of low level TensorFlow APIs. If you are using the high level APIs (`tf.keras`) there may be little or no action you need to take to make your code fully TensorFlow 2.0 compatible. Check your [optimizer's default learning rate](#keras_optimizer_lr). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "C0V10enS1_WU"
      },
      "source": [
        "It is still possible to run 1.X code, unmodified ([except for contrib](https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md)), in TensorFlow 2.0:\n",
        "\n",
        "```\n",
        "import tensorflow.compat.v1 as tf\n",
        "tf.disable_v2_behavior()\n",
        "```\n",
        "\n",
        "However, this does not let you take advantage of many of the improvements made in TensorFlow 2.0. This guide will help you upgrade your code, making it simpler, more performant, and easier to maintain."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "GUp_x0bOgaac"
      },
      "source": [
        "## Automatic conversion script\n",
        "\n",
        "The first step, before attempting to implement the changes described in this doc,  is to try running the [upgrade script](./upgrade.md).\n",
        "\n",
        "This will do an initial pass at upgrading your code to TensorFlow 2.0. But it can't make your code idiomatic to  2.0. Your code may still make use of `tf.compat.v1` endpoints to access placeholders, sessions, collections, and other 1.x-style functionality."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0w5LiSYsy1mh"
      },
      "source": [
        "## Top-level behavioral changes\n",
        "\n",
        "If your code works in TensorFlow 2.0 using `tf.compat.v1.disable_v2_behavior()`, there are still global behavioral changes you may need to address. The major changes are:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1y-W0Mz_zB6Y"
      },
      "source": [
        "* *Eager execution, `v1.enable_eager_execution()`* : Any code that implicitly uses a `tf.Graph` will fail. Be sure to wrap this code in a `with tf.Graph().as_default()` context. \n",
        "    \n",
        "* *Resource variables, `v1.enable_resource_variables()`*: Some code may depends on non-deterministic behaviors enabled by TF reference variables. \n",
        "Resource variables are locked while being written to, and so provide more intuitive consistency guarantees.\n",
        "\n",
        "  * This may change behavior in edge cases.\n",
        "  * This may create extra copies and can have higher memory usage.\n",
        "  * This can be disabled by passing `use_resource=False` to the `tf.Variable` constructor.\n",
        "\n",
        "* *Tensor shapes, `v1.enable_v2_tensorshape()`*: TF 2.0 simplifies the behavior of tensor shapes. Instead of `t.shape[0].value` you can say `t.shape[0]`. These changes should be small, and it makes sense to fix them right away. See [TensorShape](#tensorshape) for examples.\n",
        "\n",
        "* *Control flow, `v1.enable_control_flow_v2()`*: The TF 2.0 control flow implementation has been simplified, and so produces different graph representations. Please [file bugs](https://github.com/tensorflow/tensorflow/issues) for any issues."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_Ni9zLLvwcOR"
      },
      "source": [
        "## Make the code 2.0-native\n",
        "\n",
        "\n",
        "This guide will walk through several examples of converting TensorFlow 1.x code to TensorFlow 2.0. These changes will let your code take advantage of performance optimizations and simplified API calls.\n",
        "\n",
        "In each case, the pattern is:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uP0O8Pc45LNs"
      },
      "source": [
        "### 1. Replace `v1.Session.run` calls\n",
        "\n",
        "Every `v1.Session.run` call should be replaced by a Python function.\n",
        "\n",
        "* The `feed_dict` and `v1.placeholder`s become function arguments.\n",
        "* The `fetches` become the function's return value. \n",
        "* During conversion eager execution allows easy debugging with standard Python tools like `pdb`.\n",
        "\n",
        "After that add a `tf.function` decorator to make it run efficiently in graph. See the [Autograph Guide](function.ipynb) for more on how this works.\n",
        "\n",
        "Note that:\n",
        "\n",
        "* Unlike `v1.Session.run` a `tf.function` has a fixed return signature, and always returns all outputs. If this causes performance problems, create two separate functions.\n",
        "\n",
        "* There is no need for a `tf.control_dependencies` or similar operations: A `tf.function` behaves as if it were run in the order written. `tf.Variable` assignments and `tf.assert`s, for example, are executed automatically.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "jlBOqROL5NmN"
      },
      "source": [
        "### 2. Use Python objects to track variables and losses\n",
        "\n",
        "All name-based variable tracking is strongly discouraged in TF 2.0. Use Python objects to to track variables.\n",
        "\n",
        "Use `tf.Variable` instead of `v1.get_variable`.\n",
        "\n",
        "Every `v1.variable_scope` should be converted to a Python object. Typically this will be one of:\n",
        "\n",
        "* `tf.keras.layers.Layer`\n",
        "* `tf.keras.Model`\n",
        "* `tf.Module`\n",
        "\n",
        "If you need to aggregate lists of variables (like `tf.Graph.get_collection(tf.GraphKeys.VARIABLES)`), use the `.variables` and `.trainable_variables` attributes of the `Layer` and `Model` objects.\n",
        "\n",
        "These `Layer` and `Model` classes implement several other properties that remove the need for global collections. Their `.losses` property can be a replacement for using the `tf.GraphKeys.LOSSES` collection.\n",
        "\n",
        "See the [keras guides](keras.ipynb) for details.\n",
        "\n",
        "Warning: Many `tf.compat.v1` symbols  use the global collections implicitly.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "rGFhBzoF5FIq"
      },
      "source": [
        "### 3. Upgrade your training loops\n",
        "\n",
        "Use the highest level API that works for your use case.  Prefer `tf.keras.Model.fit` over building your own training loops.\n",
        "\n",
        "These high level functions manage a lot of the low-level details that might be easy to miss if you write your own training loop. For example, they automatically collect the regularization losses, and set the `training=True` argument when calling the model.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oaY37_6L4la5"
      },
      "source": [
        "\n",
        "### 4. Upgrade your data input pipelines\n",
        "\n",
        "Use `tf.data` datasets for data input. These objects are efficient, expressive, and integrate well with tensorflow.\n",
        "\n",
        "They can be passed directly to the `tf.keras.Model.fit` method.\n",
        "\n",
        "```\n",
        "model.fit(dataset, epochs=5)\n",
        "```\n",
        "\n",
        "They can be iterated over directly standard Python:\n",
        "\n",
        "```\n",
        "for example_batch, label_batch in dataset:\n",
        "    break\n",
        "```\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_Mwsd0SK4oIs"
      },
      "source": [
        "#### 5. Migrate off `compat.v1` symbols \n",
        "\n",
        "The `tf.compat.v1` module contains the complete TensorFlow 1.x API, with its original semantics.\n",
        "\n",
        "The [TF2 upgrade script](upgrade.ipynb)  will convert symbols to their 2.0 equivalents if such a conversion is safe, i.e., if it can determine that the behavior of the 2.0 version is exactly equivalent (for instance, it will rename `v1.arg_max` to `tf.argmax`, since those are the same function). \n",
        "\n",
        "After the upgrade script is done with a piece of code, it is likely there are many mentions of `compat.v1`. It is worth going through the code and converting these manually to the 2.0 equivalent (it should be mentioned in the log if there is one)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "X_ilfTGJ4Yml"
      },
      "source": [
        "## Converting models\n",
        "\n",
        "### Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "bad2N-Z115W1"
      },
      "outputs": [],
      "source": [
        "from __future__ import absolute_import, division, print_function, unicode_literals\n",
        "try:\n",
        "  # %tensorflow_version only exists in Colab.\n",
        "  %tensorflow_version 2.x\n",
        "except Exception:\n",
        "  pass\n",
        "import tensorflow as tf\n",
        "\n",
        "\n",
        "import tensorflow_datasets as tfds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FB99sqHX2Q5m"
      },
      "source": [
        "### Low-level variables \u0026 operator execution\n",
        "\n",
        "Examples of low-level API use include:\n",
        "\n",
        "* using variable scopes to control reuse\n",
        "* creating variables with `v1.get_variable`.\n",
        "* accessing collections explicitly\n",
        "* accessing collections implicitly with methods like :\n",
        "\n",
        "  * `v1.global_variables`\n",
        "  * `v1.losses.get_regularization_loss`\n",
        "\n",
        "* using `v1.placeholder` to set up graph inputs\n",
        "* executing graphs with `Session.run`\n",
        "* initializing variables manually\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "e582IjyF2eje"
      },
      "source": [
        "#### Before converting\n",
        "\n",
        "Here is what these patterns may look like in code using TensorFlow 1.x.\n",
        "\n",
        "```python\n",
        "in_a = tf.placeholder(dtype=tf.float32, shape=(2))\n",
        "in_b = tf.placeholder(dtype=tf.float32, shape=(2))\n",
        "\n",
        "def forward(x):\n",
        "  with tf.variable_scope(\"matmul\", reuse=tf.AUTO_REUSE):\n",
        "    W = tf.get_variable(\"W\", initializer=tf.ones(shape=(2,2)),\n",
        "                        regularizer=tf.contrib.layers.l2_regularizer(0.04))\n",
        "    b = tf.get_variable(\"b\", initializer=tf.zeros(shape=(2)))\n",
        "    return W * x + b\n",
        "\n",
        "out_a = forward(in_a)\n",
        "out_b = forward(in_b)\n",
        "\n",
        "reg_loss = tf.losses.get_regularization_loss(scope=\"matmul\")\n",
        "\n",
        "with tf.Session() as sess:\n",
        "  sess.run(tf.global_variables_initializer())\n",
        "  outs = sess.run([out_a, out_b, reg_loss],\n",
        "      \t        feed_dict={in_a: [1, 0], in_b: [0, 1]})\n",
        "\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "QARwz4Xd2lc2"
      },
      "source": [
        "#### After converting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x0AVzBFRBPcU"
      },
      "source": [
        "In the converted code:\n",
        "\n",
        "* The variables are local Python objects.\n",
        "* The `forward` function still defines the calculation.\n",
        "* The `Session.run` call is replaced with a call to `forward`\n",
        "* The optional `tf.function` decorator can be added for performance.\n",
        "* The regularizations are calculated manually, without referring to any global collection.\n",
        "* **No sessions or placeholders.**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "lXEZoLMP2cWJ"
      },
      "outputs": [],
      "source": [
        "W = tf.Variable(tf.ones(shape=(2,2)), name=\"W\")\n",
        "b = tf.Variable(tf.zeros(shape=(2)), name=\"b\")\n",
        "\n",
        "@tf.function\n",
        "def forward(x):\n",
        "  return W * x + b\n",
        "\n",
        "out_a = forward([1,0])\n",
        "print(out_a)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YmE96A_1jZTg"
      },
      "outputs": [],
      "source": [
        "out_b = forward([0,1])\n",
        "\n",
        "regularizer = tf.keras.regularizers.l2(0.04)\n",
        "reg_loss = regularizer(W)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ycDxY9nL268-"
      },
      "source": [
        "### Models based on `tf.layers`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "K-bIk7wL48U7"
      },
      "source": [
        "The `v1.layers` module is used to contain layer-functions that relied on `v1.variable_scope` to define and reuse variables."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8I_qKpT73KyM"
      },
      "source": [
        "#### Before converting\n",
        "```python\n",
        "def model(x, training, scope='model'):\n",
        "  with tf.variable_scope(scope, reuse=tf.AUTO_REUSE):\n",
        "    x = tf.layers.conv2d(x, 32, 3, activation=tf.nn.relu,\n",
        "          kernel_regularizer=tf.contrib.layers.l2_regularizer(0.04))\n",
        "    x = tf.layers.max_pooling2d(x, (2, 2), 1)\n",
        "    x = tf.layers.flatten(x)\n",
        "    x = tf.layers.dropout(x, 0.1, training=training)\n",
        "    x = tf.layers.dense(x, 64, activation=tf.nn.relu)\n",
        "    x = tf.layers.batch_normalization(x, training=training)\n",
        "    x = tf.layers.dense(x, 10, activation=tf.nn.softmax)\n",
        "    return x\n",
        "\n",
        "train_out = model(train_data, training=True)\n",
        "test_out = model(test_data, training=False)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "b8_Ii7CQ3fK-"
      },
      "source": [
        "#### After converting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BsAseSMfB9XN"
      },
      "source": [
        "* The simple stack of layers fits neatly into `tf.keras.Sequential`. (For more complex models see [custom layers and models](keras/custom_layers_and_models.ipynb), and [the functional API](keras/functional.ipynb).)\n",
        "* The model tracks the variables, and regularization losses.\n",
        "* The conversion was one-to-one because there is a direct mapping from `v1.layers` to `tf.keras.layers`.\n",
        "\n",
        "Most arguments stayed the same. But notice the differences:\n",
        "\n",
        "* The `training` argument is passed to each layer by the model when it runs.\n",
        "* The first argument to the original `model` function (the input `x`) is gone. This is because object layers separate building the model from calling the model.\n",
        "\n",
        "\n",
        "Also note that:\n",
        "\n",
        "* If you were using regularizers of initializers from  `tf.contrib`, these have more argument changes than others.\n",
        "* The code no longer writes to collections, so functions like `v1.losses.get_regularization_loss` will no longer return these values, potentially breaking your training loops."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "DLAPORrN3lct"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Conv2D(32, 3, activation='relu',\n",
        "                           kernel_regularizer=tf.keras.regularizers.l2(0.04),\n",
        "                           input_shape=(28, 28, 1)),\n",
        "    tf.keras.layers.MaxPooling2D(),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dropout(0.1),\n",
        "    tf.keras.layers.Dense(64, activation='relu'),\n",
        "    tf.keras.layers.BatchNormalization(),\n",
        "    tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "train_data = tf.ones(shape=(1, 28, 28, 1))\n",
        "test_data = tf.ones(shape=(1, 28, 28, 1))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "6nWh6IXvkMKv"
      },
      "outputs": [],
      "source": [
        "train_out = model(train_data, training=True)\n",
        "print(train_out)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YnAdIDLlj3go"
      },
      "outputs": [],
      "source": [
        "test_out = model(test_data, training=False)\n",
        "print(test_out)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "sAgqwCJBMx_x"
      },
      "outputs": [],
      "source": [
        "# Here are all the trainable variables.\n",
        "len(model.trainable_variables)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "uX6knaYMNM8p"
      },
      "outputs": [],
      "source": [
        "# Here is the regularization loss.\n",
        "model.losses"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "9moqw5E_4Cwl"
      },
      "source": [
        "### Mixed variables \u0026 `v1.layers`\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "80DEsImmq6VX"
      },
      "source": [
        "Existing code often mixes lower-level TF 1.x variables and operations with higher-level `v1.layers`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oZe9L6RR4OcP"
      },
      "source": [
        "#### Before converting\n",
        "```python\n",
        "def model(x, training, scope='model'):\n",
        "  with tf.variable_scope(scope, reuse=tf.AUTO_REUSE):\n",
        "    W = tf.get_variable(\n",
        "      \"W\", dtype=tf.float32,\n",
        "      initializer=tf.ones(shape=x.shape),\n",
        "      regularizer=tf.contrib.layers.l2_regularizer(0.04),\n",
        "      trainable=True)\n",
        "    if training:\n",
        "      x = x + W\n",
        "    else:\n",
        "      x = x + W * 0.5\n",
        "    x = tf.layers.conv2d(x, 32, 3, activation=tf.nn.relu)\n",
        "    x = tf.layers.max_pooling2d(x, (2, 2), 1)\n",
        "    x = tf.layers.flatten(x)\n",
        "    return x\n",
        "\n",
        "train_out = model(train_data, training=True)\n",
        "test_out = model(test_data, training=False)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "y6ORX7cD4TkD"
      },
      "source": [
        "#### After converting"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "2BaRwog5CBpz"
      },
      "source": [
        "To convert this code, follow the pattern of mapping layers to layers as in the previous example.\n",
        "\n",
        "A `v1.variable_scope` is effectively a layer of its own. So rewrite it as a `tf.keras.layers.Layer`. See [the guide](keras/custom_layers_and_models.ipynb) for details.\n",
        "\n",
        "The general pattern is:\n",
        "\n",
        "* Collect layer parameters in `__init__`.\n",
        "* Build the variables in `build`.\n",
        "* Execute the calculations in `call`, and return the result.\n",
        "\n",
        "The `v1.variable_scope` is essentially a layer of its own. So rewrite it as a `tf.keras.layers.Layer`. See [the guide](keras/custom_layers_and_models.ipynb) for details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "YcCAjNuP4NVh"
      },
      "outputs": [],
      "source": [
        "# Create a custom layer for part of the model\n",
        "class CustomLayer(tf.keras.layers.Layer):\n",
        "  def __init__(self, *args, **kwargs):\n",
        "    super(CustomLayer, self).__init__(*args, **kwargs)\n",
        "\n",
        "  def build(self, input_shape):\n",
        "    self.w = self.add_weight(\n",
        "        shape=input_shape[1:],\n",
        "        dtype=tf.float32,\n",
        "        initializer=tf.keras.initializers.ones(),\n",
        "        regularizer=tf.keras.regularizers.l2(0.02),\n",
        "        trainable=True)\n",
        "\n",
        "  # Call method will sometimes get used in graph mode,\n",
        "  # training will get turned into a tensor\n",
        "  @tf.function\n",
        "  def call(self, inputs, training=None):\n",
        "    if training:\n",
        "      return inputs + self.w\n",
        "    else:\n",
        "      return inputs + self.w * 0.5"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "dR_QO6_wBgMm"
      },
      "outputs": [],
      "source": [
        "custom_layer = CustomLayer()\n",
        "print(custom_layer([1]).numpy())\n",
        "print(custom_layer([1], training=True).numpy())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "VzqaIf4E42oY"
      },
      "outputs": [],
      "source": [
        "train_data = tf.ones(shape=(1, 28, 28, 1))\n",
        "test_data = tf.ones(shape=(1, 28, 28, 1))\n",
        "\n",
        "# Build the model including the custom layer\n",
        "model = tf.keras.Sequential([\n",
        "    CustomLayer(input_shape=(28, 28, 1)),\n",
        "    tf.keras.layers.Conv2D(32, 3, activation='relu'),\n",
        "    tf.keras.layers.MaxPooling2D(),\n",
        "    tf.keras.layers.Flatten(),\n",
        "])\n",
        "\n",
        "train_out = model(train_data, training=True)\n",
        "test_out = model(test_data, training=False)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dS5ed_jjOkvh"
      },
      "source": [
        "Some things to note:\n",
        "\n",
        "* Subclassed Keras models \u0026 layers need to run in both v1 graphs (no automatic control dependencies) and in eager mode\n",
        "  * Wrap the `call()` in a `tf.function()` to get autograph and automatic control dependencies\n",
        "\n",
        "* Don't forget to accept a `training` argument to `call`.\n",
        "    * Sometimes it is a `tf.Tensor`\n",
        "    * Sometimes it is a Python boolean.\n",
        "\n",
        "* Create model variables in constructor or `Model.build` using `self.add_weight()`.\n",
        "  * In `Model.build` you have access to the input shape, so can create weights with matching shape.\n",
        "  * Using `tf.keras.layers.Layer.add_weight` allows Keras to track variables and regularization losses.\n",
        "\n",
        "* Don't keep `tf.Tensors` in your objects.\n",
        "  * They might get created either in a `tf.function` or in the eager context, and these tensors behave differently.\n",
        "  * Use `tf.Variable`s for state, they are always usable from both contexts\n",
        "  * `tf.Tensors` are only for intermediate values."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ulaB1ymO4pw5"
      },
      "source": [
        "### A note on Slim \u0026 contrib.layers\n",
        "\n",
        "A large amount of older TensorFlow 1.x code uses the [Slim](https://ai.googleblog.com/2016/08/tf-slim-high-level-library-to-define.html) library, which was packaged with TensorFlow 1.x as `tf.contrib.layers`. As a `contrib` module, this is no longer available in TensorFlow 2.0, even in `tf.compat.v1`. Converting code using Slim to TF 2.0 is more involved than converting repositories that use `v1.layers`. In fact, it may make sense to convert your Slim code to `v1.layers` first, then convert to Keras.\n",
        "\n",
        "* Remove `arg_scopes`, all args need to be explicit\n",
        "* If you use them, split `normalizer_fn` and `activation_fn` into their own layers\n",
        "* Separable conv layers map to one or more different Keras layers (depthwise, pointwise, and separable Keras layers)\n",
        "* Slim and `v1.layers` have different arg names \u0026 default values\n",
        "* Some args have different scales\n",
        "* If you use Slim pre-trained models, try out Keras's pre-traimed models from `tf.keras.applications` or [TF Hub](https://tfhub.dev/s?q=slim%20tf2)'s TF2 SavedModels exported from the original Slim code.\n",
        "\n",
        "Some `tf.contrib` layers might not have been moved to core TensorFlow but have instead been moved to the [TF add-ons package](https://github.com/tensorflow/addons).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1w72KrXm4yZR"
      },
      "source": [
        "## Training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "56PQxTgy2bpI"
      },
      "source": [
        "There are many ways to feed data to a `tf.keras` model. They will accept Python generators and Numpy arrays as input.\n",
        "\n",
        "The recommended way to feed data to a model is to use the `tf.data` package, which contains a collection of high performance classes for manipulating data.\n",
        "\n",
        "If you are still using `tf.queue`, these are now only supported as data-structures, not as input pipelines."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "m6htasZ7iBB4"
      },
      "source": [
        "### Using Datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "loTPH2Pz4_Oj"
      },
      "source": [
        "The [TensorFlow Datasets](https://tensorflow.org/datasets) package (`tfds`) contains utilities for loading predefined datasets as `tf.data.Dataset` objects.\n",
        "\n",
        "For this example, load the MNISTdataset, using `tfds`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "BMgxaLH74_s-"
      },
      "outputs": [],
      "source": [
        "datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True)\n",
        "mnist_train, mnist_test = datasets['train'], datasets['test']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "hPJhEuvj5VfR"
      },
      "source": [
        "Then prepare the data for training:\n",
        "\n",
        "  * Re-scale each image.\n",
        "  * Shuffle the order of the examples.\n",
        "  * Collect batches of images and labels.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "StBRHtJM2S7o"
      },
      "outputs": [],
      "source": [
        "BUFFER_SIZE = 10 # Use a much larger value for real code.\n",
        "BATCH_SIZE = 64\n",
        "NUM_EPOCHS = 5\n",
        "\n",
        "\n",
        "def scale(image, label):\n",
        "  image = tf.cast(image, tf.float32)\n",
        "  image /= 255\n",
        "\n",
        "  return image, label"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "SKq14zKKFAdv"
      },
      "source": [
        " To keep the example short, trim the dataset to only return 5 batches:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "_J-o4YjG2mkM"
      },
      "outputs": [],
      "source": [
        "train_data = mnist_train.map(scale).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n",
        "test_data = mnist_test.map(scale).batch(BATCH_SIZE)\n",
        "\n",
        "STEPS_PER_EPOCH = 5\n",
        "\n",
        "train_data = train_data.take(STEPS_PER_EPOCH)\n",
        "test_data = test_data.take(STEPS_PER_EPOCH)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XEqdkH54VM6c"
      },
      "outputs": [],
      "source": [
        "image_batch, label_batch = next(iter(train_data))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mSev7vZC5GJB"
      },
      "source": [
        "### Use Keras training loops\n",
        "\n",
        "If you don't need low level control of your training process, using Keras's built-in `fit`, `evaluate`, and `predict` methods is recommended. These methods provide a uniform interface to train the model regardless of the implementation (sequential,  functional, or sub-classed).\n",
        "\n",
        "The advantages of these methods include:\n",
        "\n",
        "* They accept Numpy arrays, Python generators and, `tf.data.Datasets`\n",
        "* They apply regularization, and activation losses automatically.\n",
        "* They support `tf.distribute` [for multi-device training](distributed_training.ipynb).\n",
        "* They support arbitrary callables as losses and metrics.\n",
        "* They support callbacks like `tf.keras.callbacks.TensorBoard`, and custom callbacks.\n",
        "* They are performant, automatically using TensorFlow graphs.\n",
        "\n",
        "Here is an example of training a model using a `Dataset`. (For details on how this works see [tutorials](../tutorials).)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "uzHFCzd45Rae"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Conv2D(32, 3, activation='relu',\n",
        "                           kernel_regularizer=tf.keras.regularizers.l2(0.02),\n",
        "                           input_shape=(28, 28, 1)),\n",
        "    tf.keras.layers.MaxPooling2D(),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dropout(0.1),\n",
        "    tf.keras.layers.Dense(64, activation='relu'),\n",
        "    tf.keras.layers.BatchNormalization(),\n",
        "    tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "# Model is the full model w/o custom layers\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "model.fit(train_data, epochs=NUM_EPOCHS)\n",
        "loss, acc = model.evaluate(test_data)\n",
        "\n",
        "print(\"Loss {}, Accuracy {}\".format(loss, acc))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "akpeOb09YBhq"
      },
      "source": [
        "### Write your own loop\n",
        "\n",
        "If the Keras model's training step works for you, but you need more control outside that step, consider using the `tf.keras.Model.train_on_batch` method,  in your own data-iteration loop.\n",
        "\n",
        "Remember: Many things can be implemented as a `tf.keras.callbacks.Callback`.\n",
        "\n",
        "This method has many of the advantages of the methods mentioned in the previous section, but gives the user control of the outer loop.\n",
        "\n",
        "You can also use `tf.keras.Model.test_on_batch` or `tf.keras.Model.evaluate` to check performance during training.\n",
        "\n",
        "Note: `train_on_batch` and `test_on_batch`, by default return the loss and metrics for the single batch. If you pass `reset_metrics=False` they return accumulated metrics and you must remember to appropriately reset the metric accumulators. Also remember that some metrics like `AUC` require `reset_metrics=False` to be calculated correctly.\n",
        "\n",
        "To continue training the above model:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "eXr4CyJMtJJ6"
      },
      "outputs": [],
      "source": [
        "# Model is the full model w/o custom layers\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "metrics_names = model.metrics_names\n",
        "\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "  #Reset the metric accumulators\n",
        "  model.reset_metrics()\n",
        "\n",
        "  for image_batch, label_batch in train_data:\n",
        "    result = model.train_on_batch(image_batch, label_batch)\n",
        "    print(\"train: \",\n",
        "          \"{}: {:.3f}\".format(metrics_names[0], result[0]),\n",
        "          \"{}: {:.3f}\".format(metrics_names[1], result[1]))\n",
        "  for image_batch, label_batch in test_data:\n",
        "    result = model.test_on_batch(image_batch, label_batch,\n",
        "                                 # return accumulated metrics\n",
        "                                 reset_metrics=False)\n",
        "  print(\"\\neval: \",\n",
        "        \"{}: {:.3f}\".format(metrics_names[0], result[0]),\n",
        "        \"{}: {:.3f}\".format(metrics_names[1], result[1]))\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "LQTaHTuK5S5A"
      },
      "source": [
        "\u003ca name=\"custom_loop\"\u003e\u003c/a\u003e\n",
        "\n",
        "### Customize the training step\n",
        "\n",
        "If you need more flexibility and control, you can have it by implementing your own training loop. There are three steps:\n",
        "\n",
        "1. Iterate over a Python generator or `tf.data.Dataset` to get batches of examples.\n",
        "2. Use `tf.GradientTape` to collect gradients.\n",
        "3. Use one of the `tf.keras.optimizers` to apply weight updates to the model's variables.\n",
        "\n",
        "Remember:\n",
        "\n",
        "* Always include a `training` argument on the `call` method of subclassed layers and models.\n",
        "* Make sure to call the model with the `training` argument set correctly.\n",
        "* Depending on usage, model variables may not exist until the model is run on a batch of data.\n",
        "* You need to manually handle things like regularization losses for the model.\n",
        "\n",
        "Note the simplifications relative to v1:\n",
        "\n",
        "* There is no need to run variable initializers. Variables are initialized on creation.\n",
        "* There is no need to add manual control dependencies. Even in `tf.function` operations act as in eager mode."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gQooejfYlQeF"
      },
      "outputs": [],
      "source": [
        "model = tf.keras.Sequential([\n",
        "    tf.keras.layers.Conv2D(32, 3, activation='relu',\n",
        "                           kernel_regularizer=tf.keras.regularizers.l2(0.02),\n",
        "                           input_shape=(28, 28, 1)),\n",
        "    tf.keras.layers.MaxPooling2D(),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dropout(0.1),\n",
        "    tf.keras.layers.Dense(64, activation='relu'),\n",
        "    tf.keras.layers.BatchNormalization(),\n",
        "    tf.keras.layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "optimizer = tf.keras.optimizers.Adam(0.001)\n",
        "loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()\n",
        "\n",
        "@tf.function\n",
        "def train_step(inputs, labels):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = model(inputs, training=True)\n",
        "    regularization_loss = tf.math.add_n(model.losses)\n",
        "    pred_loss = loss_fn(labels, predictions)\n",
        "    total_loss = pred_loss + regularization_loss\n",
        "\n",
        "  gradients = tape.gradient(total_loss, model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
        "\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "  for inputs, labels in train_data:\n",
        "    train_step(inputs, labels)\n",
        "  print(\"Finished epoch\", epoch)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kS7WW5Z75ve3"
      },
      "source": [
        "### New-style metrics and losses\n",
        "\n",
        "In TensorFlow 2.0, metrics and losses are objects. These work both eagerly and in `tf.function`s. \n",
        "\n",
        "A loss object is callable, and expects the (y_true, y_pred) as arguments:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "C5_TVrBlbBcy"
      },
      "outputs": [],
      "source": [
        "cce = tf.losses.CategoricalCrossentropy(from_logits=True)\n",
        "cce([[1, 0]], [[-1.0,3.0]]).numpy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JFDc1v0XbAyB"
      },
      "source": [
        "A metric object has the following methods:\n",
        "\n",
        "* `Metric.update_state()` — add new observations\n",
        "* `Metric.result()` —get the current result of the metric, given the observed values\n",
        "* `Metric.reset_states()` — clear all observations.\n",
        "\n",
        "The object itself is callable. Calling updates the state with new observations, as with `update_state`, and returns the new result of the metric.\n",
        "\n",
        "You don't have to manually initialize a metric's variables, and because TensorFlow 2.0 has automatic control dependencies, you don't need to worry about those either.\n",
        "\n",
        "The code below uses a metric to keep track of the mean loss observed within a custom training loop."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HAbA0fKW58CH"
      },
      "outputs": [],
      "source": [
        "# Create the metrics\n",
        "loss_metric = tf.keras.metrics.Mean(name='train_loss')\n",
        "accuracy_metric = tf.keras.metrics.SparseCategoricalAccuracy(name='train_accuracy')\n",
        "\n",
        "@tf.function\n",
        "def train_step(inputs, labels):\n",
        "  with tf.GradientTape() as tape:\n",
        "    predictions = model(inputs, training=True)\n",
        "    regularization_loss = tf.math.add_n(model.losses)\n",
        "    pred_loss = loss_fn(labels, predictions)\n",
        "    total_loss = pred_loss + regularization_loss\n",
        "\n",
        "  gradients = tape.gradient(total_loss, model.trainable_variables)\n",
        "  optimizer.apply_gradients(zip(gradients, model.trainable_variables))\n",
        "  # Update the metrics\n",
        "  loss_metric.update_state(total_loss)\n",
        "  accuracy_metric.update_state(labels, predictions)\n",
        "\n",
        "\n",
        "for epoch in range(NUM_EPOCHS):\n",
        "  # Reset the metrics\n",
        "  loss_metric.reset_states()\n",
        "  accuracy_metric.reset_states()\n",
        "\n",
        "  for inputs, labels in train_data:\n",
        "    train_step(inputs, labels)\n",
        "  # Get the metric results\n",
        "  mean_loss = loss_metric.result()\n",
        "  mean_accuracy = accuracy_metric.result()\n",
        "\n",
        "  print('Epoch: ', epoch)\n",
        "  print('  loss:     {:.3f}'.format(mean_loss))\n",
        "  print('  accuracy: {:.3f}'.format(mean_accuracy))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Hf718XCgDAGJ"
      },
      "source": [
        "### Keras optimizers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "A6El-NxAQ8aF"
      },
      "source": [
        "The optimizers in `v1.train`, like `v1.train.AdamOptimizer` and `v1.train.GradientDescentOptimizer`, have equivalents in `tf.keras.optimizers`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "qgP89WdSDQx-"
      },
      "source": [
        "#### Convert `v1.train` to `keras.optimizers`\n",
        "\n",
        "Here are things to keep in mind when converting your optimizers:\n",
        "\n",
        "* Upgrading your optimizers [may make old checkpoints incompatible](#checkpoints).\n",
        "* All epsilons now default to `1e-7` instead of `1e-8` (which is negligible in most use cases).\n",
        "* `v1.train.GradientDescentOptimizer` can be directly replaced by `tf.keras.optimizers.SGD`. \n",
        "* `v1.train.MomentumOptimizer` can be directly replaced by the `SGD` optimizer using the momentum argument: `tf.keras.optimizers.SGD(..., momentum=...)`.\n",
        "* `v1.train.AdamOptimizer` can be converted to use `tf.keras.optimizers.Adam`. The `beta1` and `beta2` arguments have been renamed to `beta_1` and `beta_2`.\n",
        "* `v1.train.RMSPropOptimizer` can be converted to `tf.keras.optimizers.RMSprop`. The `decay` argument has been renamed to `rho`.\n",
        "* `v1.train.AdadeltaOptimizer` can be converted directly to `tf.keras.optimizers.Adadelta`.\n",
        "* `tf.train.AdagradOptimizer` can be converted directly to `tf.keras.optimizers.Adagrad`.\n",
        "* `tf.train.FtrlOptimizer` can be converted directly to `tf.keras.optimizers.Ftrl`. The `accum_name` and `linear_name` arguments have been removed.\n",
        "* The `tf.contrib.AdamaxOptimizer` and `tf.contrib.NadamOptimizer`, can be converted directly to `tf.keras.optimizers.Adamax` and `tf.keras.optimizers.Nadam`. The `beta1`, and `beta2` arguments have been renamed to `beta_1` and `beta_2`.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Ef60-wJ2bR3l"
      },
      "source": [
        "\n",
        "#### New defaults for some `tf.keras.optimizers`\n",
        "\u003ca id=\"keras_optimizer_lr\"\u003e\u003c/a\u003e\n",
        "\n",
        "Warning: If you see a change in convergence behavior for your models, check the default learning rates.\n",
        "\n",
        "There are no changes for `optimizers.SGD`, `optimizers.Adam`, or `optimizers.RMSprop`.\n",
        "\n",
        "The following default learning rates have changed:\n",
        "\n",
        "* `optimizers.Adagrad` from 0.01 to 0.001\n",
        "* `optimizers.Adadelta` from 1.0 to 0.001\n",
        "* `optimizers.Adamax` from 0.002 to 0.001\n",
        "* `optimizers.Nadam` from 0.002 to 0.001"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5Cf1ks48Q3uc"
      },
      "source": [
        "### TensorBoard"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "0tx7FyM_RHwJ"
      },
      "source": [
        "TensorFlow 2.0 includes significant changes to the `tf.summary` API used to write summary data for visualization in TensorBoard.  For a general introduction to the new tf.summary, there are [several tutorials available](https://www.tensorflow.org/tensorboard/r2/get_started) that use the TF 2.0 API. This includes a [TensorBoard TF2.0 Migration Guide](https://www.tensorflow.org/tensorboard/r2/migrate_tf2)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "JmMLBKs66DeA"
      },
      "source": [
        "## Saving \u0026 Loading\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "5_QKn3Kl6TUu"
      },
      "source": [
        "\u003ca id=\"checkpoints\"\u003e\u003c/a\u003e\n",
        "### Checkpoint compatibility\n",
        "\n",
        "TensorFlow 2.0 uses [object-based checkpoints](checkpoint.ipynb).\n",
        "\n",
        "Old-style name-based checkpoints can still be loaded, if you're careful.\n",
        "The code conversion process may result in variable name changes, but there are workarounds.\n",
        "\n",
        "The simplest approach it to line up the names of the new model with the names in the checkpoint:\n",
        "\n",
        "* Variables still all have a `name` argument you can set.\n",
        "* Keras models also take a `name` argument as which they set as the prefix for their variables.\n",
        "* The `v1.name_scope` function can be used to set variable name prefixes. This is very different from `tf.variable_scope`. It only affects names, and  doesn't track variables \u0026 reuse.\n",
        "\n",
        "If that does not work for your use-case, try the `v1.train.init_from_checkpoint` function. It takes an `assignment_map` argument, which specifies the mapping from old names to new names.\n",
        "\n",
        "Note: Unlike object based checkpoints, which can [defer loading](checkpoint.ipynb#loading_mechanics), name-based checkpoints require that all variables be built when the function is called. Some models defer building variables until you call `build` or run the model on a batch of data.\n",
        "\n",
        "The [TensorFlow Estimator repository](https://github.com/tensorflow/estimator/blob/master/tensorflow_estimator/python/estimator/tools/checkpoint_converter.py) includes a [conversion tool](#checkpoint_converter) to upgrade the checkpoints for premade estimators from TensorFlow 1.X to 2.0. It may serve as an example of how to build a tool fr a similar use-case."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_ONjobDD6Uur"
      },
      "source": [
        "### Saved models compatibility\n",
        "\n",
        "There are no significant compatibility concerns for saved models.\n",
        "\n",
        "* TensorFlow 1.x saved_models work in TensorFlow 2.0.\n",
        "* TensorFlow 2.0 saved_models even load work in TensorFlow 1.x if all the ops are supported."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-KpXprsf1tyb"
      },
      "source": [
        "### A Graph.pb or Graph.pbtxt "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tz4eAAGY19MM"
      },
      "source": [
        "There is no straightforward way to upgrade a raw `Graph.pb` file to TensorFlow 2.0. Your best bet is to upgrade the code that generated the file.\n",
        "\n",
        "But, if you have a \"Frozen graph\" (a `tf.Graph` where the variables have been turned into constants), then it is possible to convert this to a [`concrete_function`](https://tensorflow.org/guide/concrete_function) using `v1.wrap_function`:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "1xl_fFhs1ur6"
      },
      "outputs": [],
      "source": [
        "def wrap_frozen_graph(graph_def, inputs, outputs):\n",
        "  def _imports_graph_def():\n",
        "    tf.compat.v1.import_graph_def(graph_def, name=\"\")\n",
        "  wrapped_import = tf.compat.v1.wrap_function(_imports_graph_def, [])\n",
        "  import_graph = wrapped_import.graph\n",
        "  return wrapped_import.prune(\n",
        "      tf.nest.map_structure(import_graph.as_graph_element, inputs),\n",
        "      tf.nest.map_structure(import_graph.as_graph_element, outputs))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bQHGXyYC4GT-"
      },
      "source": [
        "For example, here is a frozed graph for Inception v1, from 2016:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "obfto7Bp3hHj"
      },
      "outputs": [],
      "source": [
        "path = tf.keras.utils.get_file(\n",
        "    'inception_v1_2016_08_28_frozen.pb',\n",
        "    'http://storage.googleapis.com/download.tensorflow.org/models/inception_v1_2016_08_28_frozen.pb.tar.gz',\n",
        "    untar=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FHE3Ot4gSCJg"
      },
      "source": [
        "Load the `tf.GraphDef`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HqL_OTwr4n20"
      },
      "outputs": [],
      "source": [
        "graph_def = tf.compat.v1.GraphDef()\n",
        "loaded = graph_def.ParseFromString(open(path,'rb').read())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "T9ai6Kh6SsQ4"
      },
      "source": [
        "Wrap it into a `concrete_function`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "pajiVteo6H0U"
      },
      "outputs": [],
      "source": [
        "inception_func = wrap_frozen_graph(\n",
        "    graph_def, inputs='input:0',\n",
        "    outputs='InceptionV1/InceptionV1/Mixed_3b/Branch_1/Conv2d_0a_1x1/Relu:0')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FvknOO-MSw4n"
      },
      "source": [
        "Pass it a tensor as input:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gsPSIjAE7nry"
      },
      "outputs": [],
      "source": [
        "input_img = tf.ones([1,224,224,3], dtype=tf.float32)\n",
        "inception_func(input_img).shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ewl9P3oZ6ZtR"
      },
      "source": [
        "## Estimators"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YprVP9g3l6eG"
      },
      "source": [
        "### Training with Estimators\n",
        "\n",
        "Estimators are supported in TensorFlow 2.0.\n",
        "\n",
        "When you use estimators, you can use `input_fn()`, `tf.estimator.TrainSpec`, and `tf.estimator.EvalSpec` from TensorFlow 1.x.\n",
        "\n",
        "Here is an example using `input_fn` with train and evaluate specs."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "N5kZeJsF8lS2"
      },
      "source": [
        "#### Creating the input_fn and train/eval specs"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "AOlXGO4J6jDh"
      },
      "outputs": [],
      "source": [
        "# Define the estimator's input_fn\n",
        "def input_fn():\n",
        "  datasets, info = tfds.load(name='mnist', with_info=True, as_supervised=True)\n",
        "  mnist_train, mnist_test = datasets['train'], datasets['test']\n",
        "\n",
        "  BUFFER_SIZE = 10000\n",
        "  BATCH_SIZE = 64\n",
        "\n",
        "  def scale(image, label):\n",
        "    image = tf.cast(image, tf.float32)\n",
        "    image /= 255\n",
        "\n",
        "    return image, label[..., tf.newaxis]\n",
        "\n",
        "  train_data = mnist_train.map(scale).shuffle(BUFFER_SIZE).batch(BATCH_SIZE)\n",
        "  return train_data.repeat()\n",
        "\n",
        "# Define train \u0026 eval specs\n",
        "train_spec = tf.estimator.TrainSpec(input_fn=input_fn,\n",
        "                                    max_steps=STEPS_PER_EPOCH * NUM_EPOCHS)\n",
        "eval_spec = tf.estimator.EvalSpec(input_fn=input_fn,\n",
        "                                  steps=STEPS_PER_EPOCH)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_o6J48Nj9H5c"
      },
      "source": [
        "### Using a Keras model definition"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "IXCQdhGq9SbB"
      },
      "source": [
        "There are some differences in how to construct your estimators in TensorFlow 2.0.\n",
        "\n",
        "We recommend that you define your model using Keras, then use the `tf.keras.estimator.model_to_estimator` utility to turn your model into an estimator. The code below shows how to use this utility when creating and training an estimator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "aelsClm3Cq4I"
      },
      "outputs": [],
      "source": [
        "def make_model():\n",
        "  return tf.keras.Sequential([\n",
        "    tf.keras.layers.Conv2D(32, 3, activation='relu',\n",
        "                           kernel_regularizer=tf.keras.regularizers.l2(0.02),\n",
        "                           input_shape=(28, 28, 1)),\n",
        "    tf.keras.layers.MaxPooling2D(),\n",
        "    tf.keras.layers.Flatten(),\n",
        "    tf.keras.layers.Dropout(0.1),\n",
        "    tf.keras.layers.Dense(64, activation='relu'),\n",
        "    tf.keras.layers.BatchNormalization(),\n",
        "    tf.keras.layers.Dense(10, activation='softmax')\n",
        "  ])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "HJb6f8dtl6rr"
      },
      "outputs": [],
      "source": [
        "model = make_model()\n",
        "\n",
        "model.compile(optimizer='adam',\n",
        "              loss='sparse_categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "estimator = tf.keras.estimator.model_to_estimator(\n",
        "  keras_model = model\n",
        ")\n",
        "\n",
        "tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "-ptTxL1q6flL"
      },
      "source": [
        "### Using a custom `model_fn`\n",
        "\n",
        "If you have an existing custom estimator `model_fn` that you need to maintain, you can convert your `model_fn` to use a Keras model.\n",
        "\n",
        "However, for compatibility reasons, a custom `model_fn` will still run in 1.x-style graph mode. This means there is no eager execution and no automatic control dependencies."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Avgqf8IKfd51"
      },
      "source": [
        "\u003ca name=\"minimal_changes\"\u003e\u003c/a\u003e\n",
        "\n",
        "#### Custom model_fn with minimal changes\n",
        "To make your custom `model_fn` work in TF 2.0, if you prefer minimal changes to the existing code, `tf.compat.v1` symbols such as `optimizers` and `metrics` can be used.\n",
        "\n",
        "Using a Keras models in a custom `model_fn` is similar to using it in a custom training loop:\n",
        "\n",
        "* Set the `training` phase appropriately, based on the `mode` argument.\n",
        "* Explicitly pass the model's `trainable_variables` to the optimizer.\n",
        "\n",
        "But there are important differences, relative to a [custom loop](#custom_loop):\n",
        "\n",
        "* Instead of using `Model.losses`, extract the losses using `Model.get_losses_for`.\n",
        "* Extract the model's updates using `Model.get_updates_for`.\n",
        "\n",
        "Note: \"Updates\" are changes that need to be applied to a model after each batch. For example, the moving averages of the mean and variance in a `layers.BatchNormalization` layer.\n",
        "\n",
        "The following code creates an estimator from a custom `model_fn`, illustrating all of these concerns."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "iY16eZKW606-"
      },
      "outputs": [],
      "source": [
        "def my_model_fn(features, labels, mode):\n",
        "  model = make_model()\n",
        "\n",
        "  optimizer = tf.compat.v1.train.AdamOptimizer()\n",
        "  loss_fn = tf.keras.losses.SparseCategoricalCrossentropy()\n",
        "\n",
        "  training = (mode == tf.estimator.ModeKeys.TRAIN)\n",
        "  predictions = model(features, training=training)\n",
        "\n",
        "  if mode == tf.estimator.ModeKeys.PREDICT:\n",
        "    return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions)\n",
        "\n",
        "  reg_losses = model.get_losses_for(None) + model.get_losses_for(features)\n",
        "  total_loss = loss_fn(labels, predictions) + tf.math.add_n(reg_losses)\n",
        "\n",
        "  accuracy = tf.compat.v1.metrics.accuracy(labels=labels,\n",
        "                                           predictions=tf.math.argmax(predictions, axis=1),\n",
        "                                           name='acc_op')\n",
        "\n",
        "  update_ops = model.get_updates_for(None) + model.get_updates_for(features)\n",
        "  minimize_op = optimizer.minimize(\n",
        "      total_loss,\n",
        "      var_list=model.trainable_variables,\n",
        "      global_step=tf.compat.v1.train.get_or_create_global_step())\n",
        "  train_op = tf.group(minimize_op, update_ops)\n",
        "\n",
        "  return tf.estimator.EstimatorSpec(\n",
        "    mode=mode,\n",
        "    predictions=predictions,\n",
        "    loss=total_loss,\n",
        "    train_op=train_op, eval_metric_ops={'accuracy': accuracy})\n",
        "\n",
        "# Create the Estimator \u0026 Train\n",
        "estimator = tf.estimator.Estimator(model_fn=my_model_fn)\n",
        "tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XVxHmU2ccfAG"
      },
      "source": [
        "#### Custom `model_fn` with TF 2.0 symbols\n",
        "If you want to get rid of all TF 1.x symbols and upgrade your custom `model_fn` to native TF 2.0, you need to update the optimizer and metrics to `tf.keras.optimizers` and `tf.keras.metrics`.\n",
        "\n",
        "In the custom `model_fn`, besides the above [changes](#minimal_changes), more upgrades need to be made:\n",
        "\n",
        "* Use [`tf.keras.optimizers`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers) instead of `v1.train.Optimizer`.\n",
        "* Explicitly pass the model's `trainable_variables` to the `tf.keras.optimizers`.\n",
        "* To compute the `train_op/minimize_op`,\n",
        "  * Use `Optimizer.get_updates()` if the loss is scalar loss `Tensor`(not a callable). The first element in the returned list is the desired `train_op/minimize_op`. \n",
        "  * If the loss is a callable (such as a function), use `Optimizer.minimize()` to get the `train_op/minimize_op`.\n",
        "* Use [`tf.keras.metrics`](https://www.tensorflow.org/api_docs/python/tf/keras/metrics) instead of `tf.compat.v1.metrics` for evaluation.\n",
        "\n",
        "For the above example of `my_model_fn`, the migrated code with 2.0 symbols is shown as:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "uf8J3nloeze2"
      },
      "outputs": [],
      "source": [
        "def my_model_fn(features, labels, mode):\n",
        "  model = make_model()\n",
        "\n",
        "  training = (mode == tf.estimator.ModeKeys.TRAIN)\n",
        "  loss_obj = tf.keras.losses.SparseCategoricalCrossentropy()\n",
        "  predictions = model(features, training=training)\n",
        "\n",
        "  # Get both the unconditional losses (the None part)\n",
        "  # and the input-conditional losses (the features part).\n",
        "  reg_losses = model.get_losses_for(None) + model.get_losses_for(features)\n",
        "  total_loss = loss_obj(labels, predictions) + tf.math.add_n(reg_losses)\n",
        "\n",
        "  # Upgrade to tf.keras.metrics.\n",
        "  accuracy_obj = tf.keras.metrics.Accuracy(name='acc_obj')\n",
        "  accuracy = accuracy_obj.update_state(\n",
        "      y_true=labels, y_pred=tf.math.argmax(predictions, axis=1))\n",
        "\n",
        "  train_op = None\n",
        "  if training:\n",
        "    # Upgrade to tf.keras.optimizers.\n",
        "    optimizer = tf.keras.optimizers.Adam()\n",
        "    # Manually assign tf.compat.v1.global_step variable to optimizer.iterations\n",
        "    # to make tf.compat.v1.train.global_step increased correctly.\n",
        "    # This assignment is a must for any `tf.train.SessionRunHook` specified in\n",
        "    # estimator, as SessionRunHooks rely on global step.\n",
        "    optimizer.iterations = tf.compat.v1.train.get_or_create_global_step()\n",
        "    # Get both the unconditional updates (the None part)\n",
        "    # and the input-conditional updates (the features part).\n",
        "    update_ops = model.get_updates_for(None) + model.get_updates_for(features)\n",
        "    # Compute the minimize_op.\n",
        "    minimize_op = optimizer.get_updates(\n",
        "        total_loss,\n",
        "        model.trainable_variables)[0]\n",
        "    train_op = tf.group(minimize_op, *update_ops)\n",
        "\n",
        "  return tf.estimator.EstimatorSpec(\n",
        "    mode=mode,\n",
        "    predictions=predictions,\n",
        "    loss=total_loss,\n",
        "    train_op=train_op,\n",
        "    eval_metric_ops={'Accuracy': accuracy_obj})\n",
        "\n",
        "# Create the Estimator \u0026 Train.\n",
        "estimator = tf.estimator.Estimator(model_fn=my_model_fn)\n",
        "tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "g1l6VnOTodfA"
      },
      "source": [
        "### Premade Estimators\n",
        "\n",
        "[Premade Estimators](https://www.tensorflow.org/guide/premade_estimators) in the family of `tf.estimator.DNN*`, `tf.estimator.Linear*` and `tf.estimator.DNNLinearCombined*` are still supported in the TensorFlow 2.0 API, however, some arguments have changed:\n",
        "\n",
        "1. `input_layer_partitioner`: Removed in 2.0.\n",
        "2. `loss_reduction`: Updated to `tf.keras.losses.Reduction` instead of `tf.compat.v1.losses.Reduction`. Its default value is also changed to `tf.keras.losses.Reduction.SUM_OVER_BATCH_SIZE` from `tf.compat.v1.losses.Reduction.SUM`.\n",
        "3. `optimizer`, `dnn_optimizer` and `linear_optimizer`: this arg has been updated to `tf.keras.optimizers` instead of the `tf.compat.v1.train.Optimizer`. \n",
        "\n",
        "To migrate the above changes:\n",
        "1. No migration is needed for `input_layer_partitioner` since [`Distribution Strategy`](https://www.tensorflow.org/guide/distributed_training) will handle it automatically in TF 2.0.\n",
        "2. For `loss_reduction`, check [`tf.keras.losses.Reduction`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/losses/Reduction) for the supported options.\n",
        "3. For `optimizer` args, if you do not pass in an `optimizer`, `dnn_optimizer` or `linear_optimizer` arg, or if you specify the `optimizer` arg as a `string` in your code, you don't need to change anything. `tf.keras.optimizers` is used by default. Otherwise, you need to update it from `tf.compat.v1.train.Optimizer` to its corresponding [`tf.keras.optimizers`](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers)\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "v0Kljg-AHyqv"
      },
      "source": [
        "#### Checkpoint Converter\n",
        "\u003ca id=\"checkpoint_converter\"\u003e\u003c/a\u003e\n",
        "\n",
        "The migration to `keras.optimizers` will break checkpoints saved using TF 1.x, as `tf.keras.optimizers` generates a different set of variables to be saved in checkpoints. To make old checkpoint reusable after your migration to TF 2.0, try the [checkpoint converter tool](https://github.com/tensorflow/estimator/blob/master/tensorflow_estimator/python/estimator/tools/checkpoint_converter.py)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "h9FiYN9mIPli"
      },
      "outputs": [],
      "source": [
        "! curl -O https://raw.githubusercontent.com/tensorflow/estimator/master/tensorflow_estimator/python/estimator/tools/checkpoint_converter.py"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "DMc6zDJaJwNw"
      },
      "source": [
        "The tool has builtin help:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9JNZFX3rJLXv"
      },
      "outputs": [],
      "source": [
        "! python checkpoint_converter.py -h"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "dt8ct9XCFqls"
      },
      "source": [
        "\u003ca id=\"tensorshape\"\u003e\u003c/a\u003e\n",
        "\n",
        "## TensorShape\n",
        "\n",
        "This class was simplified to hold `int`s, instead of `tf.compat.v1.Dimension` objects. So there is no need to call `.value()` to get an `int`.\n",
        "\n",
        "Individual `tf.compat.v1.Dimension` objects are still accessible from `tf.TensorShape.dims`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "x36cWcmM8Eu1"
      },
      "source": [
        "\n",
        "\n",
        "The following demonstrate the differences between TensorFlow 1.x and TensorFlow 2.0."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "PbpD-kHOZR4A"
      },
      "outputs": [],
      "source": [
        "# Create a shape and choose an index\n",
        "i = 0\n",
        "shape = tf.TensorShape([16, None, 256])\n",
        "shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "kDFck03neNy0"
      },
      "source": [
        "If you had this in TF 1.x:\n",
        "\n",
        "```python\n",
        "value = shape[i].value\n",
        "```\n",
        "\n",
        "Then do this in TF 2.0:\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "KuR73QGEeNdH"
      },
      "outputs": [],
      "source": [
        "value = shape[i]\n",
        "value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bPWPNKRiZmkd"
      },
      "source": [
        "If you had this in TF 1.x:\n",
        "\n",
        "```python\n",
        "for dim in shape:\n",
        "    value = dim.value\n",
        "    print(value)\n",
        "```\n",
        "\n",
        "Then do this in TF 2.0:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y6s0vuuprJfc"
      },
      "outputs": [],
      "source": [
        "for value in shape:\n",
        "  print(value)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YpRgngu3Zw-A"
      },
      "source": [
        "If you had this in TF 1.x (Or used any other dimension method):\n",
        "\n",
        "```python\n",
        "dim = shape[i]\n",
        "dim.assert_is_compatible_with(other_dim)\n",
        "```\n",
        "\n",
        "Then do this in TF 2.0:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "LpViGEcUZDGX"
      },
      "outputs": [],
      "source": [
        "other_dim = 16\n",
        "Dimension = tf.compat.v1.Dimension\n",
        "\n",
        "if shape.rank is None:\n",
        "  dim = Dimension(None)\n",
        "else:\n",
        "  dim = shape.dims[i]\n",
        "dim.is_compatible_with(other_dim) # or any other dimension method"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "GaiGe36dOdZ_"
      },
      "outputs": [],
      "source": [
        "shape = tf.TensorShape(None)\n",
        "\n",
        "if shape:\n",
        "  dim = shape.dims[i]\n",
        "  dim.is_compatible_with(other_dim) # or any other dimension method"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "3kLLY0I3PI-l"
      },
      "source": [
        "The boolean value of a `tf.TensorShape` is `True` if the rank is known, `False` otherwise."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "-Ow1ndKpOnJd"
      },
      "outputs": [],
      "source": [
        "print(bool(tf.TensorShape([])))      # Scalar\n",
        "print(bool(tf.TensorShape([0])))     # 0-length vector\n",
        "print(bool(tf.TensorShape([1])))     # 1-length vector\n",
        "print(bool(tf.TensorShape([None])))  # Unknown-length vector\n",
        "print(bool(tf.TensorShape([1, 10, 100])))       # 3D tensor\n",
        "print(bool(tf.TensorShape([None, None, None]))) # 3D tensor with no known dimensions\n",
        "print()\n",
        "print(bool(tf.TensorShape(None)))  # A tensor with unknown rank."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8u63n5S7Y9IX"
      },
      "source": [
        "## Other Changes\n",
        "\n",
        "* Remove `tf.colocate_with`: TensorFlow's device placement algorithms have improved significantly. This should no longer be necessary. If removing it causes a performance degredation [please file a bug](https://github.com/tensorflow/tensorflow/issues).\n",
        "\n",
        "* Replace `v1.ConfigProto` usage with the equivalent functions from `tf.config`.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "vKX6AdTAQhB-"
      },
      "source": [
        "## Conclusions\n",
        "\n",
        "The overall process is:\n",
        "\n",
        "1. Run the upgrade script.\n",
        "2. Remove contrib symbols.\n",
        "3. Switch your models to an object oriented style (Keras).\n",
        "4. Use `tf.keras` or `tf.estimator` training and evaluation loops where you can.\n",
        "5. Otherwise, use custom loops, but be sure to avoid sessions \u0026 collections.\n",
        "\n",
        "\n",
        "It takes a little work to convert code to idiomatic TensorFlow 2.0, but every change results in:\n",
        "\n",
        "* Fewer lines of code.\n",
        "* Increased clarity and simplicity.\n",
        "* Easier debugging.\n",
        "\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "migrate.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}