03_predictive_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Predictive Models\n",
    "\n",
    "In the last two notebooks we had a look at two of the components of the Basis Mixer. In this notebook we add the third part of the puzzle: the **Predictive Models**.\n",
    "\n",
    "A predictive model is defined as a mathematical which maps score information (encoded by the basis functions) $\\mathbf{\\Phi}$ to expressive parameters $\\mathbf{Y}$\n",
    "\n",
    "$$F(\\boldsymbol{\\Phi}) = \\mathbf{Y}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "## Defining and Building Predictive Models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib notebook\n",
    "import numpy as np\n",
    "import os\n",
    "from basismixer import make_datasets\n",
    "from helper import init_dataset, data\n",
    "from basismixer.utils import load_pyc_bz, save_pyc_bz\n",
    "from helper.predictions import (construct_model, setup_output_directory, train_model, \n",
    "                                split_datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_dir = setup_output_directory()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The configuration of the full predictive model can be defined as a list of models defined for individual (or groups) of expressive parameters.\n",
    "\n",
    "The structure of this configuration can be summarized as follows:\n",
    "\n",
    "* `onsetwise`: specifies if the model predicts *notewise* or *onsetwise* parameters.\n",
    "* `basis_functions`: A list with the basis functions (or the familiy of basis functions) used to specify the model. Alternatively, we can use a dataset to specify the basis functions.\n",
    "* `parameter_names`: Name of the expressive parameters\n",
    "* `model`: A dictionary specifying the architecture of the model\n",
    "    * `constructor`: A list specifying the module and method used to construct the model\n",
    "    * `args`: Arguments for the constructor.\n",
    "* `train_args`: A dictionary containing parameters for training the model.\n",
    "\n",
    "Let us define an example of such an architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "model_config = [\n",
    "    dict(onsetwise=True,\n",
    "         basis_functions=['polynomial_pitch_basis',\n",
    "                          'loudness_direction_basis',\n",
    "                          'tempo_direction_basis',\n",
    "                          'articulation_basis',\n",
    "                          'duration_basis',\n",
    "                          'slur_basis',\n",
    "                          'fermata_basis',\n",
    "                          'grace_basis',\n",
    "                          'metrical_basis'],\n",
    "         parameter_names=['velocity_trend', 'beat_period_standardized'],\n",
    "         model=dict(constructor=['basismixer.predictive_models', 'RecurrentModel'],\n",
    "                    args=dict(recurrent_size=128,\n",
    "                              n_layers=1,\n",
    "                              hidden_size=64)),\n",
    "         seq_len=50,\n",
    "         train_args=dict(\n",
    "             optimizer=['Adam', dict(lr=1e-3)],\n",
    "             epochs=1,\n",
    "             save_freq=1,\n",
    "             early_stopping=100,\n",
    "             batch_size=100\n",
    "         )\n",
    "    )\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Making the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "init_dataset() # download the corpus if necessary; set some variables\n",
    "\n",
    "# path to the MusicXML and Match files\n",
    "xmlfolder = os.path.join(data.DATASET_DIR, 'musicxml')\n",
    "matchfolder = os.path.join(data.DATASET_DIR, 'match')\n",
    "\n",
    "dataset_fn = os.path.join(data.DATASET_DIR, 'vienna_4x22.pyc.bz')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if dataset_fn is not None and os.path.exists(dataset_fn):\n",
    "    datasets = load_pyc_bz(dataset_fn)\n",
    "else:\n",
    "    datasets = make_datasets(model_config,\n",
    "                             xmlfolder, \n",
    "                             matchfolder)\n",
    "    if dataset_fn is not None:\n",
    "        save_pyc_bz(datasets, dataset_fn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "\n",
    "## Training the models\n",
    "\n",
    "Given a training set of expressive performances aligned to their scores, we can train the models in a supervised way by minimizing the *mean squared error* between predictions and the observed expressive parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "models = []\n",
    "test_sets = []\n",
    "for (dataset, in_names, out_names), config in zip(datasets, model_config):\n",
    "    \n",
    "    # Build model\n",
    "    model, model_out_dir = construct_model(config, in_names, out_names, out_dir)\n",
    "    # Split datasets\n",
    "    train_set, valid_set, test_set = split_datasets(dataset)\n",
    "    # Train Model\n",
    "    train_model(model, train_set, valid_set, config, model_out_dir)\n",
    "    \n",
    "    models.append(model)\n",
    "    test_sets.append(test_set)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "autoscroll": false,
    "ein.hycell": false,
    "ein.tags": "worksheet-0",
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from helper.plotting import plot_predictions_and_targets\n",
    "\n",
    "for model, test_set in zip(models, test_sets):\n",
    "    \n",
    "    basis = test_set.datasets[0].basis\n",
    "    idx = test_set.datasets[0].idx\n",
    "    n_basis = test_set.datasets[0].n_basis\n",
    "    inputs = np.zeros((len(basis), n_basis))\n",
    "    inputs[:, idx] = basis\n",
    "    targets = test_set.datasets[0].targets\n",
    "    preds = model.predict(inputs)\n",
    "\n",
    "    plot_predictions_and_targets(preds, targets)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "name": "03_predictive_models.ipynb"
 },
 "nbformat": 4,
 "nbformat_minor": 2
}