06 - Work with Data.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Work with Data\n",
        "\n",
        "Data is the foundation on which machine learning models are built. Managing data centrally in the cloud, and making it accessible to teams of data scientists who are running experiments and training models on multiple workstations and compute targets is an important part of any professional data science solution.\n",
        "\n",
        "In this notebook, you'll explore two Azure Machine Learning objects for working with data: *datastores*, and *datasets*."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Connect to your workspace\n",
        "\n",
        "To get started, connect to your workspace.\n",
        "\n",
        "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366182180
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import azureml.core\n",
        "from azureml.core import Workspace\n",
        "\n",
        "# Load the workspace from the saved config file\n",
        "ws = Workspace.from_config()\n",
        "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Create a compute cluster\n",
        "\n",
        "To run a script as an experiment using the ScriptRunConfig, you'll need a compute target.\n",
        "\n",
        "> **Important**: Change *your-compute-cluster* to a suitable name for your compute cluster in the code below before running it - you can specify the name of an existing cluster if you have one. Cluster names must be globally unique names between 2 to 16 characters in length. Valid characters are letters, digits, and the - character."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core.compute import ComputeTarget, AmlCompute\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "cluster_name = \"your-compute-cluster\"\n",
        "\n",
        "try:\n",
        "    # Check for existing compute target\n",
        "    training_cluster = ComputeTarget(workspace=ws, name=cluster_name)\n",
        "    print('Found existing cluster, use it.')\n",
        "except ComputeTargetException:\n",
        "    # If it doesn't already exist, create it\n",
        "    try:\n",
        "        compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_DS11_V2', max_nodes=2)\n",
        "        training_cluster = ComputeTarget.create(ws, cluster_name, compute_config)\n",
        "        training_cluster.wait_for_completion(show_output=True)\n",
        "    except Exception as ex:\n",
        "        print(ex)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Work with datastores\n",
        "\n",
        "In Azure ML, *datastores* are references to storage locations, such as Azure Storage blob containers. Every workspace has a default datastore - usually the Azure storage blob container that was created with the workspace. If you need to work with data that is stored in different locations, you can add custom datastores to your workspace and set any of them to be the default.\n",
        "\n",
        "### View datastores\n",
        "\n",
        "Run the following code to determine the datastores in your workspace:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366185140
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Get the default datastore\n",
        "default_ds = ws.get_default_datastore()\n",
        "\n",
        "# Enumerate all datastores, indicating which is the default\n",
        "for ds_name in ws.datastores:\n",
        "    print(ds_name, \"- Default =\", ds_name == default_ds.name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can also view and manage datastores in your workspace on the **Datastores** page for your workspace in [Azure Machine Learning studio](https://ml.azure.com).\n",
        "\n",
        "### Upload data to a datastore\n",
        "\n",
        "Now that you have determined the available datastores, you can upload files from your local file system to a datastore so that it will be accessible to experiments running in the workspace, regardless of where the experiment script is actually being run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366190608
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Dataset\n",
        "from azureml.data.datapath import DataPath\n",
        "\n",
        "Dataset.File.upload_directory(src_dir='data',\n",
        "                              target=DataPath(default_ds, 'diabetes-data/')\n",
        "                              )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Work with datasets\n",
        "\n",
        "Azure Machine Learning provides an abstraction for data in the form of *datasets*. A dataset is a versioned reference to a specific set of data that you may want to use in an experiment. Datasets can be *tabular* or *file*-based.\n",
        "\n",
        "### Create a tabular dataset\n",
        "\n",
        "Let's create a dataset from the diabetes data you uploaded to the datastore, and view the first 20 records. In this case, the data is in a structured format in a CSV file, so we'll use a *tabular* dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366193530
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Dataset\n",
        "\n",
        "# Get the default datastore\n",
        "default_ds = ws.get_default_datastore()\n",
        "\n",
        "#Create a tabular dataset from the path on the datastore (this may take a short while)\n",
        "tab_data_set = Dataset.Tabular.from_delimited_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
        "\n",
        "# Display the first 20 rows as a Pandas dataframe\n",
        "tab_data_set.take(20).to_pandas_dataframe()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As you can see in the code above, it's easy to convert a tabular dataset to a Pandas dataframe, enabling you to work with the data using common python techniques.\n",
        "\n",
        "### Create a file Dataset\n",
        "\n",
        "The dataset you created is a *tabular* dataset that can be read as a dataframe containing all of the data in the structured files that are included in the dataset definition. This works well for tabular data, but in some machine learning scenarios you might need to work with data that is unstructured; or you may simply want to handle reading the data from files in your own code. To accomplish this, you can use a *file* dataset, which creates a list of file paths in a virtual mount point, which you can use to read the data in the files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366196795
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#Create a file dataset from the path on the datastore (this may take a short while)\n",
        "file_data_set = Dataset.File.from_files(path=(default_ds, 'diabetes-data/*.csv'))\n",
        "\n",
        "# Get the files in the dataset\n",
        "for file_path in file_data_set.to_path():\n",
        "    print(file_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Register datasets\n",
        "\n",
        "Now that you have created datasets that reference the diabetes data, you can register them to make them easily accessible to any experiment being run in the workspace.\n",
        "\n",
        "We'll register the tabular dataset as **diabetes dataset**, and the file dataset as **diabetes files**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366200005
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# Register the tabular dataset\n",
        "try:\n",
        "    tab_data_set = tab_data_set.register(workspace=ws, \n",
        "                                        name='diabetes dataset',\n",
        "                                        description='diabetes data',\n",
        "                                        tags = {'format':'CSV'},\n",
        "                                        create_new_version=True)\n",
        "except Exception as ex:\n",
        "    print(ex)\n",
        "\n",
        "# Register the file dataset\n",
        "try:\n",
        "    file_data_set = file_data_set.register(workspace=ws,\n",
        "                                            name='diabetes file dataset',\n",
        "                                            description='diabetes files',\n",
        "                                            tags = {'format':'CSV'},\n",
        "                                            create_new_version=True)\n",
        "except Exception as ex:\n",
        "    print(ex)\n",
        "\n",
        "print('Datasets registered')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can view and manage datasets on the **Datasets** page for your workspace in [Azure Machine Learning studio](https://ml.azure.com). You can also get a list of datasets from the workspace object:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366204696
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "print(\"Datasets:\")\n",
        "for dataset_name in list(ws.datasets.keys()):\n",
        "    dataset = Dataset.get_by_name(ws, dataset_name)\n",
        "    print(\"\\t\", dataset.name, 'version', dataset.version)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "The ability to version datasets enables you to redefine datasets without breaking existing experiments or pipelines that rely on previous definitions. By default, the latest version of a named dataset is returned, but you can retrieve a specific version of a dataset by specifying the version number, like this:\n",
        "\n",
        "```python\n",
        "dataset_v1 = Dataset.get_by_name(ws, 'diabetes dataset', version = 1)\n",
        "```\n",
        "\n",
        "\n",
        "### Train a model from a tabular dataset\n",
        "\n",
        "Now that you have datasets, you're ready to start training models from them. You can pass datasets to scripts as *inputs* in the estimator being used to run the script.\n",
        "\n",
        "Run the following two code cells to create:\n",
        "\n",
        "1. A folder named **diabetes_training_from_tab_dataset**\n",
        "2. A script that trains a classification model by using a tabular dataset that is passed to it as an argument."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366207914
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Create a folder for the experiment files\n",
        "experiment_folder = 'diabetes_training_from_tab_dataset'\n",
        "os.makedirs(experiment_folder, exist_ok=True)\n",
        "print(experiment_folder, 'folder created')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "%%writefile $experiment_folder/diabetes_training.py\n",
        "# Import libraries\n",
        "import os\n",
        "import argparse\n",
        "from azureml.core import Run, Dataset\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import joblib\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import roc_auc_score\n",
        "from sklearn.metrics import roc_curve\n",
        "\n",
        "# Get the script arguments (regularization rate and training dataset ID)\n",
        "parser = argparse.ArgumentParser()\n",
        "parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')\n",
        "parser.add_argument(\"--input-data\", type=str, dest='training_dataset_id', help='training dataset')\n",
        "args = parser.parse_args()\n",
        "\n",
        "# Set regularization hyperparameter (passed as an argument to the script)\n",
        "reg = args.reg_rate\n",
        "\n",
        "# Get the experiment run context\n",
        "run = Run.get_context()\n",
        "\n",
        "# Get the training dataset\n",
        "print(\"Loading Data...\")\n",
        "diabetes = run.input_datasets['training_data'].to_pandas_dataframe()\n",
        "\n",
        "# Separate features and labels\n",
        "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
        "\n",
        "# Split data into training set and test set\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
        "\n",
        "# Train a logistic regression model\n",
        "print('Training a logistic regression model with regularization rate of', reg)\n",
        "run.log('Regularization Rate',  np.float(reg))\n",
        "model = LogisticRegression(C=1/reg, solver=\"liblinear\").fit(X_train, y_train)\n",
        "\n",
        "# calculate accuracy\n",
        "y_hat = model.predict(X_test)\n",
        "acc = np.average(y_hat == y_test)\n",
        "print('Accuracy:', acc)\n",
        "run.log('Accuracy', np.float(acc))\n",
        "\n",
        "# calculate AUC\n",
        "y_scores = model.predict_proba(X_test)\n",
        "auc = roc_auc_score(y_test,y_scores[:,1])\n",
        "print('AUC: ' + str(auc))\n",
        "run.log('AUC', np.float(auc))\n",
        "\n",
        "os.makedirs('outputs', exist_ok=True)\n",
        "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
        "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
        "\n",
        "run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "> **Note**: In the script, the dataset is passed as a parameter (or argument). In the case of a tabular dataset, this argument will contain the ID of the registered dataset; so you could write code in the script to get the experiment's workspace from the run context, and then get the dataset using its ID; like this:\n",
        ">\n",
        "> ```\n",
        "> run = Run.get_context()\n",
        "> ws = run.experiment.workspace\n",
        "> dataset = Dataset.get_by_id(ws, id=args.training_dataset_id)\n",
        "> diabetes = dataset.to_pandas_dataframe()\n",
        "> ```\n",
        ">\n",
        "> However, Azure Machine Learning runs automatically identify arguments that reference named datasets and add them to the run's **input_datasets** collection, so you can also retrieve the dataset from this collection by specifying its \"friendly name\" (which as you'll see shortly, is specified in the argument definition in the script run configuration for the experiment). This is the approach taken in the script above.\n",
        "\n",
        "Now you can run a script as an experiment, defining an argument for the training dataset, which is read by the script.\n",
        "\n",
        "> **Note**: The **Dataset** class depends on some components in the **azureml-dataprep** package, so you need to include this package in the environment where the training experiment will be run. The **azureml-dataprep** package is included in the **azure-defaults** package."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366222016
        },
        "scrolled": true,
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment, ScriptRunConfig, Environment\n",
        "from azureml.core.runconfig import DockerConfiguration\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "\n",
        "# Create a Python environment for the experiment (from a .yml file)\n",
        "env = Environment.from_conda_specification(\"experiment_env\", \"environment.yml\")\n",
        "\n",
        "# Get the training dataset\n",
        "diabetes_ds = ws.datasets.get(\"diabetes dataset\")\n",
        "\n",
        "# Create a script config\n",
        "script_config = ScriptRunConfig(source_directory=experiment_folder,\n",
        "                              script='diabetes_training.py',\n",
        "                              arguments = ['--regularization', 0.1, # Regularizaton rate parameter\n",
        "                                           '--input-data', diabetes_ds.as_named_input('training_data')], # Reference to dataset\n",
        "                              environment=env,\n",
        "                              docker_runtime_config=DockerConfiguration(use_docker=True),\n",
        "                              compute_target=cluster_name) \n",
        "\n",
        "# submit the experiment\n",
        "experiment_name = 'mslearn-train-diabetes'\n",
        "experiment = Experiment(workspace=ws, name=experiment_name)\n",
        "run = experiment.submit(config=script_config)\n",
        "RunDetails(run).show()\n",
        "run.wait_for_completion()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "> **Note:** The **--input-data** argument passes the dataset as a *named input* that includes a *friendly name* for the dataset, which is used by the script to read it from the **input_datasets** collection in the experiment run. The string value in the **--input-data** argument is actually the registered dataset's ID.  As an alternative approach, you could simply pass `diabetes_ds.id`, in which case the script can access the dataset ID from the script arguments and use it to get the dataset from the workspace, but not from the **input_datasets** collection.\n",
        "\n",
        "The first time the experiment is run, it may take some time to set up the Python environment - subsequent runs will be quicker.\n",
        "\n",
        "When the experiment has completed, in the widget, view the **azureml-logs/70_driver_log.txt** output log and the metrics generated by the run.\n",
        "\n",
        "### Register the trained model\n",
        "\n",
        "As with any training experiment, you can retrieve the trained model and register it in your Azure Machine Learning workspace."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366229781
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Model\n",
        "\n",
        "run.register_model(model_path='outputs/diabetes_model.pkl', model_name='diabetes_model',\n",
        "                   tags={'Training context':'Tabular dataset'}, properties={'AUC': run.get_metrics()['AUC'], 'Accuracy': run.get_metrics()['Accuracy']})\n",
        "\n",
        "for model in Model.list(ws):\n",
        "    print(model.name, 'version:', model.version)\n",
        "    for tag_name in model.tags:\n",
        "        tag = model.tags[tag_name]\n",
        "        print ('\\t',tag_name, ':', tag)\n",
        "    for prop_name in model.properties:\n",
        "        prop = model.properties[prop_name]\n",
        "        print ('\\t',prop_name, ':', prop)\n",
        "    print('\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Train a model from a file dataset\n",
        "\n",
        "You've seen how to train a model using training data in a *tabular* dataset; but what about a *file* dataset?\n",
        "\n",
        "When you're using a file dataset, the dataset argument passed to the script represents a mount point containing file paths. How you read the data from these files depends on the kind of data in the files and what you want to do with it. In the case of the diabetes CSV files, you can use the Python **glob** module to create a list of files in the virtual mount point defined by the dataset, and read them all into Pandas dataframes that are concatenated into a single dataframe.\n",
        "\n",
        "Run the following two code cells to create:\n",
        "\n",
        "1. A folder named **diabetes_training_from_file_dataset**\n",
        "2. A script that trains a classification model by using a file dataset that is passed to is as an *input*."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366232676
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Create a folder for the experiment files\n",
        "experiment_folder = 'diabetes_training_from_file_dataset'\n",
        "os.makedirs(experiment_folder, exist_ok=True)\n",
        "print(experiment_folder, 'folder created')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "%%writefile $experiment_folder/diabetes_training.py\n",
        "# Import libraries\n",
        "import os\n",
        "import argparse\n",
        "from azureml.core import Dataset, Run\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import joblib\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LogisticRegression\n",
        "from sklearn.metrics import roc_auc_score\n",
        "from sklearn.metrics import roc_curve\n",
        "import glob\n",
        "\n",
        "# Get script arguments (rgularization rate and file dataset mount point)\n",
        "parser = argparse.ArgumentParser()\n",
        "parser.add_argument('--regularization', type=float, dest='reg_rate', default=0.01, help='regularization rate')\n",
        "parser.add_argument('--input-data', type=str, dest='dataset_folder', help='data mount point')\n",
        "args = parser.parse_args()\n",
        "\n",
        "# Set regularization hyperparameter (passed as an argument to the script)\n",
        "reg = args.reg_rate\n",
        "\n",
        "# Get the experiment run context\n",
        "run = Run.get_context()\n",
        "\n",
        "# load the diabetes dataset\n",
        "print(\"Loading Data...\")\n",
        "data_path = run.input_datasets['training_files'] # Get the training data path from the input\n",
        "# (You could also just use args.dataset_folder if you don't want to rely on a hard-coded friendly name)\n",
        "\n",
        "# Read the files\n",
        "all_files = glob.glob(data_path + \"/*.csv\")\n",
        "diabetes = pd.concat((pd.read_csv(f) for f in all_files), sort=False)\n",
        "\n",
        "# Separate features and labels\n",
        "X, y = diabetes[['Pregnancies','PlasmaGlucose','DiastolicBloodPressure','TricepsThickness','SerumInsulin','BMI','DiabetesPedigree','Age']].values, diabetes['Diabetic'].values\n",
        "\n",
        "# Split data into training set and test set\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=0)\n",
        "\n",
        "# Train a logistic regression model\n",
        "print('Training a logistic regression model with regularization rate of', reg)\n",
        "run.log('Regularization Rate',  np.float(reg))\n",
        "model = LogisticRegression(C=1/reg, solver=\"liblinear\").fit(X_train, y_train)\n",
        "\n",
        "# calculate accuracy\n",
        "y_hat = model.predict(X_test)\n",
        "acc = np.average(y_hat == y_test)\n",
        "print('Accuracy:', acc)\n",
        "run.log('Accuracy', np.float(acc))\n",
        "\n",
        "# calculate AUC\n",
        "y_scores = model.predict_proba(X_test)\n",
        "auc = roc_auc_score(y_test,y_scores[:,1])\n",
        "print('AUC: ' + str(auc))\n",
        "run.log('AUC', np.float(auc))\n",
        "\n",
        "os.makedirs('outputs', exist_ok=True)\n",
        "# note file saved in the outputs folder is automatically uploaded into experiment record\n",
        "joblib.dump(value=model, filename='outputs/diabetes_model.pkl')\n",
        "\n",
        "run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Just as with tabular datasets, you can retrieve a file dataset from the **input_datasets** collection by using its friendly name. You can also retrieve it from the script argument, which in the case of a file dataset contains a mount path to the files (rather than the dataset ID passed for a tabular dataset).\n",
        "\n",
        "Next we need to change the way we pass the dataset to the script - it needs to define a path from which the script can read the files. You can use either the **as_download** or **as_mount** method to do this. Using **as_download** causes the files in the file dataset to be downloaded to a temporary location on the compute where the script is being run, while **as_mount** creates a mount point from which the files can be streamed directly from the datastore.\n",
        "\n",
        "You can combine the access method with the **as_named_input** method to include the dataset in the **input_datasets** collection in the experiment run (if you omit this, for example by setting the argument to `diabetes_ds.as_mount()`, the script will be able to access the dataset mount point from the script arguments, but not from the **input_datasets** collection)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366252096
        },
        "scrolled": true,
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "from azureml.core.runconfig import DockerConfiguration\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "\n",
        "# Get the training dataset\n",
        "diabetes_ds = ws.datasets.get(\"diabetes file dataset\")\n",
        "\n",
        "# Create a script config\n",
        "script_config = ScriptRunConfig(source_directory=experiment_folder,\n",
        "                                script='diabetes_training.py',\n",
        "                                arguments = ['--regularization', 0.1, # Regularizaton rate parameter\n",
        "                                             '--input-data', diabetes_ds.as_named_input('training_files').as_download()], # Reference to dataset location\n",
        "                                environment=env, # Use the environment created previously\n",
        "                                docker_runtime_config=DockerConfiguration(use_docker=True),\n",
        "                                compute_target=cluster_name)\n",
        "\n",
        "# submit the experiment\n",
        "experiment_name = 'mslearn-train-diabetes'\n",
        "experiment = Experiment(workspace=ws, name=experiment_name)\n",
        "run = experiment.submit(config=script_config)\n",
        "RunDetails(run).show()\n",
        "run.wait_for_completion()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When the experiment has completed, in the widget, view the **azureml-logs/70_driver_log.txt** output log to verify that the files in the file dataset were downloaded to a temporary folder to enable the script to read the files.\n",
        "\n",
        "### Register the trained model\n",
        "\n",
        "Once again, you can register the model that was trained by the experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649366260993
        },
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Model\n",
        "\n",
        "run.register_model(model_path='outputs/diabetes_model.pkl', model_name='diabetes_model',\n",
        "                   tags={'Training context':'File dataset'}, properties={'AUC': run.get_metrics()['AUC'], 'Accuracy': run.get_metrics()['Accuracy']})\n",
        "\n",
        "for model in Model.list(ws):\n",
        "    print(model.name, 'version:', model.version)\n",
        "    for tag_name in model.tags:\n",
        "        tag = model.tags[tag_name]\n",
        "        print ('\\t',tag_name, ':', tag)\n",
        "    for prop_name in model.properties:\n",
        "        prop = model.properties[prop_name]\n",
        "        print ('\\t',prop_name, ':', prop)\n",
        "    print('\\n')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "> **More Information**: For more information about training with datasets, see [Training with Datasets](https://docs.microsoft.com/azure/machine-learning/how-to-train-with-datasets) in the Azure ML documentation."
      ]
    }
  ],
  "metadata": {
    "kernel_info": {
      "name": "python38-azureml"
    },
    "kernelspec": {
      "display_name": "Python 3.8 - AzureML",
      "language": "python",
      "name": "python38-azureml"
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    },
    "vscode": {
      "interpreter": {
        "hash": "f2b2cd046deda8eabef1e765a11d0ec9aa9bd1d31d56ce79c815a38c323e14ec"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}