notebooks/00-brain-decoding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# studying the brain with naturalistic stimuli\n",
    "[Functional magnetic resonance imaging (fMRI)](https://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging) offers a way to non-invasively record activity throughout the brain in human subjects. It can take tens of thousands of measurements from small (~1-2 mm$^3$) regions of the brain (**voxels**) evey few seconds. Our lab uses this technology to study the brain while people do things that resemble normal human activities. Data from experiments using these \"naturalistic\" stimuli can tell us things about the brain that are hard to learn from highly synthetic stimuli and behaviors, but can be challenging to analyze.\n",
    "\n",
    "You're about to work with data collected for a recent experiment performed in our lab. We had subjects do something that comes very naturally to people: listen to stories. One great source for stories is the [The Moth Radio Hour](https://themoth.org/). Since 2009, the show has featured engaging speakers telling personal stories about a wide variety of topics.\n",
    "\n",
    "# understanding the brain through encoding models\n",
    "Subjects lay down in the scanner and listened to hours of stories over the course of several sessions while we recorded their brain activity. We analyzed the data using an **encoding model**, that is, a model that takes the stimulus features as input and produces a prediction of brain activity. This result was published in [Huth et al. 2016](http://www.nature.com/nature/journal/v532/n7600/full/nature17637.html).\n",
    "\n",
    "# decoding the brain\n",
    "## example: direct decoding\n",
    "It is also useful to be able to predict the stimulus from the brain activity, often referred to as **brain decoding**. We have tried several methods of brain decoding in our lab with varying degrees of success. Here we show an example in which we simply swap the inputs to the linear model used to perform encoding. That is, whereas in the encoding model we learn linear weights to predict responses from stimuli, in this decoding model (which we refer to as **direct decoding**) we learn linear weights to predict stimuli from the responses. The results are so-so, but should give you an idea for baseline performance against which you can gauge your decoder's success.\n",
    "\n",
    "We'll split the data into a training and test set, fit the direct decoding on the training set and see how it performs on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T17:36:28.064976",
     "start_time": "2016-10-05T17:36:28.057466"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# add some useful code to your python path\n",
    "import sys, os\n",
    "import os.path as op\n",
    "sys.path.append(op.join(os.getcwd(), op.pardir, 'src'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T17:36:31.999718",
     "start_time": "2016-10-05T17:36:31.655269"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import h5py\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline\n",
    "try: plt.style.use('ggplot')\n",
    "except: pass"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# story names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T17:36:42.668828",
     "start_time": "2016-10-05T17:36:42.663070"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "train_stories = ['alternateithicatom', 'avatar', 'howtodraw', 'legacy',\n",
    "                 'life', 'myfirstdaywiththeyankees', 'naked',\n",
    "                 'odetostepfather', 'souls']\n",
    "\n",
    "test_stories = ['undertheinfluence']\n",
    "all_stories = train_stories + test_stories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# computing stimulus features\n",
    "The stimuli are stored as [`TextGrid`](http://www.fon.hum.uva.nl/praat/manual/TextGrid.html) files, which is basically a list of words and time stamps for when they occur. We have some helper functions to parse these into a useable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T17:36:44.635354",
     "start_time": "2016-10-05T17:36:43.678672"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from stimulus_utils import load_grids_for_stories, load_generic_trfiles\n",
    "grids = load_grids_for_stories(all_stories, grid_dir=op.join(op.pardir, 'data/grids'))\n",
    "trfiles = load_generic_trfiles(all_stories, root=op.join(op.pardir, 'data/trfiles'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T17:36:44.751163",
     "start_time": "2016-10-05T17:36:44.636928"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from dsutils import make_word_ds, make_word2vec_model\n",
    "wordseqs = make_word_ds(grids, trfiles) # dictionary of {storyname : word DataSequence}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## semantic vectors\n",
    "The 2016 paper used a custom word embedding space to generate features from the stimuli. Around the same time the paper came out, [`word2vec`](https://www.tensorflow.org/versions/r0.9/tutorials/word2vec/index.html) hit the streets, and it seems to work just about as well. For generality, we'll use `word2vec` as a feature space for our stimuli. You can download [a pre-trained model from Google](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing).\n",
    "\n",
    "You'll also need to install the python module [`gensim`](https://radimrehurek.com/gensim/) to load and use the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T17:36:47.331325",
     "start_time": "2016-10-05T17:36:47.326852"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## uncomment and run to install\n",
    "# !pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:15.859144",
     "start_time": "2016-10-05T16:29:10.143900"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# download pre-trained model, unzip, and add the path here\n",
    "semantic_model_path = ''\n",
    "\n",
    "from gensim.models.word2vec import Word2Vec\n",
    "semantic_model = Word2Vec.load_word2vec_format(semantic_model_path,\n",
    "                                               binary=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:15.942922",
     "start_time": "2016-10-05T16:31:15.860722"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "semanticseqs = dict() # dictionary to hold projected stimuli {story name : projected DataSequence}\n",
    "for story in all_stories:\n",
    "    semanticseqs[story] = make_word2vec_model(wordseqs[story], semantic_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:16.959033",
     "start_time": "2016-10-05T16:31:15.944504"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "interptype = 'lanczos'\n",
    "downsampled_semanticseqs = dict() # dictionary to hold downsampled stimuli\n",
    "for story in all_stories:\n",
    "    downsampled_semanticseqs[story] = semanticseqs[story].chunksums(interptype, window=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we trim ~20 seconds from the beginning and the end to remove artifacts from our detrending method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:16.963600",
     "start_time": "2016-10-05T16:31:16.960527"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "trim = [10,10]\n",
    "stimuli = {st:downsampled_semanticseqs[st][trim[0]:-trim[1]]\n",
    "           for st in all_stories}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## save stimulus features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:30.308438",
     "start_time": "2016-10-05T16:31:30.291673"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with h5py.File(op.join(op.pardir, 'data/features.h5'), 'w') as f:\n",
    "    for story in stimuli:\n",
    "        f[story] = stimuli[story]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## load pre-computed stimulus features\n",
    "If you want to concentrate your efforts on decoding instead of designing features, you can download our pre-computed `word2vec` features by running the `download_features.sh` script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:40.681788",
     "start_time": "2016-10-05T16:31:38.405489"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## uncomment and run to download\n",
    "# !bash ../download_features.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:55.568139",
     "start_time": "2016-10-05T16:31:55.554933"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with h5py.File(op.join(op.pardir, 'data/features.h5'), 'r') as f:\n",
    "    stimuli = {st:f[st].value for st in f}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:56.617790",
     "start_time": "2016-10-05T16:31:56.601475"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ridge.utils import zscore"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:57.388497",
     "start_time": "2016-10-05T16:31:57.371137"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Xtrain = np.vstack([zscore(stimuli[story], 0)\n",
    "                    for story in train_stories])\n",
    "Xtest = np.vstack([zscore(stimuli[story], 0)\n",
    "                    for story in test_stories])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Xtrain` and `Xtest` contain the training and test stimulus information. The dimensions are time points x word embedding space dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:31:59.127628",
     "start_time": "2016-10-05T16:31:59.123066"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3433, 300) (304, 300)\n"
     ]
    }
   ],
   "source": [
    "print Xtrain.shape, Xtest.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# responses\n",
    "We've preprocessed the neural data to save you time. If you haven't run the shell script `download_responses.sh` already, do so now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:33:38.380079",
     "start_time": "2016-10-05T16:32:40.730614"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "## uncomment and run to download\n",
    "# !bash ../download_responses.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:33:43.979713",
     "start_time": "2016-10-05T16:33:40.075419"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Ytrain = []\n",
    "Ytest = []\n",
    "with h5py.File(op.join(op.pardir, 'data/responses.h5'), 'r') as f:\n",
    "    for story in train_stories:\n",
    "        Ytrain.append(zscore(f[story].value))\n",
    "    for story in test_stories:\n",
    "        Ytest.append(zscore(f[story].value))\n",
    "Ytrain = np.nan_to_num(np.vstack(Ytrain))\n",
    "Ytest = np.nan_to_num(np.vstack(Ytest))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Ytrain` and `Ytest` contain the training and test response data from one subject. The dimensions are time points x voxels. The sampling period in our fMRI experiments is **2.0045 seconds**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:33:47.206945",
     "start_time": "2016-10-05T16:33:47.202364"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3433, 92836) (304, 92836)\n"
     ]
    }
   ],
   "source": [
    "print Ytrain.shape, Ytest.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The biological signal measured my fMRI is slow to respond to the underlying neural activity that causes it. For this reason, information about the stimulus is available in the responses observed several seconds after the stimulus actually occurs. We delay the responses so that we are predicting the stimulus at each time point from the voxel responses 6 seconds after the stimulus occurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:33:51.542823",
     "start_time": "2016-10-05T16:33:50.962641"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from ridge.utils import make_delayed\n",
    "delays = [-3]\n",
    "delYtrain = make_delayed(Ytrain, delays)\n",
    "delYtest = make_delayed(Ytest, delays)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sample decoding: direct decoding\n",
    "To fit this model we use L2-regularized (or ridge) regression. Some optimized code is available from [Alex Huth's github page](https://github.com/alexhuth/ridge.git)\n",
    "\n",
    "Even fully optimized, this can take a long time to run (and time is of the essence!) so you can download prefit weights by running the script `download_model.sh`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:33:55.110660",
     "start_time": "2016-10-05T16:33:55.098678"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "saving model to ../models/bsridge-directdecoding-alphas2to4-nboots10.h5\n"
     ]
    }
   ],
   "source": [
    "alpha_range = [2,4]\n",
    "alphs = np.logspace(*alpha_range, num=15)\n",
    "nboots = 10 # Number of cross-validation runs.\n",
    "chunklen = 40 # \n",
    "nchunks = 20\n",
    "\n",
    "model_dir = op.join(op.pardir, 'models')\n",
    "model_path_fmt = os.path.join(model_dir, 'bsridge-directdecoding'\n",
    "                              '-alphas{alpha_lo}to{alpha_hi}'\n",
    "                              '-nboots{nboots}.h5').format\n",
    "\n",
    "model_path = model_path_fmt(alpha_lo=alpha_range[0], alpha_hi=alpha_range[1],\n",
    "                            nboots=nboots)\n",
    "print 'saving model to', model_path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### run ridge regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from ridge import bootstrap_ridge\n",
    "\n",
    "wt, corr, alphas, bscorrs, valinds = bootstrap_ridge(delYtrain, Xtrain,\n",
    "                                                     delYtest, Xtest,\n",
    "                                                     alphs, nboots,\n",
    "                                                     chunklen, nchunks,\n",
    "                                                     singcutoff=1e-10,\n",
    "                                                     single_alpha=False,\n",
    "                                                     use_corr=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T15:18:15.265582",
     "start_time": "2016-10-05T14:50:51.352611"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with h5py.File(model_path, 'w') as f:\n",
    "    f['wt'] = wt\n",
    "    f['alphas'] = alphas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### load previously run ridge regression parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## uncomment and run to download\n",
    "# !bash ../download_model.sh\n",
    "# model_path = op.join(op.pardir, 'models/bsridge-directdecoding-alphas2to4-nboots10.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:36:42.071896",
     "start_time": "2016-10-05T16:36:42.046761"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with h5py.File(model_path, 'r') as f:\n",
    "    wt = f['wt'].value\n",
    "    alphas = f['alphas'].value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### predicting embedding vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:36:44.034007",
     "start_time": "2016-10-05T16:36:43.768349"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Xhat = delYtest.dot(wt)\n",
    "ntrials_test = delYtest.shape[0]\n",
    "\n",
    "decoding_perf = np.empty(ntrials_test)\n",
    "for i in xrange(ntrials_test):\n",
    "    decoding_perf[i] = np.corrcoef(Xhat[i,:], Xtest[i,:])[0,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### De-embedding words\n",
    "The direct decoding provides us with decoded word embedding vectors, but we actually want to decode the words that the subject heard. We consider the decoded embedding vectors to be the sum of the embedding vectors from the words that we presented and use L1-regularized regression to estimate the sparse matrix of word activations that led to the word embedding vectors. We'll use L1-regularized (LASSO) regression from the [`sklearn`](http://scikit-learn.org/stable/) python package to do this. You can use `pip` to install it.\n",
    "\n",
    "The vocabulary we'll decode into is provided by wikipedia's list of [1,000 most common english words](https://en.wiktionary.org/wiki/Category:1000_English_basic_words) (at the time we scraped them, there were actually 985 words). To save you time, we've included the list in a text file named `basis_vocabulary.txt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T15:30:58.239731",
     "start_time": "2016-10-05T15:30:58.235615"
    },
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## uncomment and run to install sklearn\n",
    "# !pip install sklearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:36:56.886173",
     "start_time": "2016-10-05T16:36:56.863818"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "basis_vocab = np.loadtxt('basis_vocabulary.txt', str)\n",
    "basis_vocab_vectors = np.empty((basis_vocab.size, semantic_model.vector_size))\n",
    "for wi, word in enumerate(basis_vocab):\n",
    "    try:\n",
    "        vec = semantic_model[word]\n",
    "    except KeyError:\n",
    "        vec = np.zeros(semantic_model.vector_size)\n",
    "    basis_vocab_vectors[wi,:] = vec"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:36:57.636001",
     "start_time": "2016-10-05T16:36:57.631820"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(985, 300) (304, 300)\n"
     ]
    }
   ],
   "source": [
    "print basis_vocab_vectors.shape, Xhat.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:37:03.386157",
     "start_time": "2016-10-05T16:36:58.504131"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Lasso\n",
    "\n",
    "lasso = Lasso(alpha=0.001, positive=True)\n",
    "lasso.fit(basis_vocab_vectors.T, Xhat.T)\n",
    "\n",
    "decoded_words = lasso.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Results\n",
    "The output for this simple example will just be a list of words that were deemed most likely to have been present in the original story. That is, we made no attempt to impose correct syntax on the output.\n",
    "\n",
    "Let's look at the original and decoded text side-by-side. You'll notice a few times where the decoded output comes close to the thematic content of the original, but overall it's not very convincing. We hope you can do better!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:40:19.353454",
     "start_time": "2016-10-05T16:40:19.295073"
    },
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 i don't ::: is want kind this be anything something god how seem\n",
      "1 do it to protect the innocent because i don't know any ::: want may anything not nature is kind hate god fun\n",
      "2 stories where the people in them are innocent ::: be nature people is kind sad may special city say\n",
      "3 and i don't do it to be shocking ::: be nature say sad such sound may yes music school\n",
      "4 either though i admit to ::: i too how lesson sad enjoy what always am possible\n",
      "5 to once telling a story ::: you i say thing she kind how should may always\n",
      "6 where i exaggerated the length of my penis ::: her special everybody bird raise music boy hobby report depend\n",
      "7  ::: mother student evening report music study boy chocolate flower fever\n",
      "8 in my own defense i exaggerated down and not ::: city will woman nature fear parent be fever school science\n",
      "9 up and i did it for the laughs ::: if is sweet seem parent story miss say must are\n",
      "10 you ::: be seem should she parent if sing thing miss difficult\n",
      "11 the story i'm telling you tonight does not ::: be wish why person learn she write excited i parent\n",
      "12 have a lie in it ::: i will me glad write happen take tomorrow agree she\n",
      "13 or many laughs for that matter ::: i probably say she me something am letter glad really\n",
      "14 when i'm done telling you this story you may think it's nothing ::: i really probably should what she may you say thing\n",
      "15 but a lie but i'd ask you to trust ::: i we something very should really be why you what\n",
      "16 me on this ::: clothes shoe\n",
      "17 at the age of nineteen i fell prey ::: shoe coat chair clothes brown sand heavy orange body east\n",
      "18 to a powerful and deeply ::: structure heavy elephant substance report shirt body market furniture police\n",
      "19 corrupting influence ::: bright machine shirt potato structure head weak sound noise strong\n",
      "20 it dogged me for six years ::: produce at net by bright extremely heavy base are station\n",
      "21 costing me many a friend and in process ::: event telephone fresh office week an man report middle increase\n",
      "22 bringing my family to ruin ::: pair hammer east join rubber copper fold shoulder top office\n",
      "23 it crippled me ::: down pair hammer arrive finish ten fold morning front clothes\n",
      "24 to such an extent that i have spent the intervening ::: hundred several position ten shirt finish clothes down pair morning\n",
      "25 fifteen years recovering from it ::: nine position ago shirt several pair half ball lately team\n",
      "26 the influence ::: \n",
      "27 i speak of is hope ::: \n",
      "28  ::: \n",
      "29 now you should know at the get ::: \n",
      "30 go there's nothing in my childhood to ::: never you yet have either is know enemy who correct\n",
      "31 suggest i might find myself on such a wayward ::: yet either who enemy substance already active by power sex\n",
      "32 path as that ::: thin substance object\n",
      "33 my parents loved me terribly they ::: active position nature therefore who question thin such sentence number\n",
      "34 taught me right from wrong ::: active past nature sentence poison examination skill substance oil\n",
      "35 they taught me to be courageous in the face ::: examination\n",
      "36 of bullies they taught me patience ::: \n",
      "37 and forgiveness they taught ::: i comb\n",
      "38 me that love would see you through ::: how have position never thin nor size question these wide\n",
      "39 any misfortune ::: \n",
      "40 my trouble began on ::: examination\n",
      "41 independence day not the ::: have i the we how since nor never should is\n",
      "42 independence day but my independence day ::: the have who since yet examination position nor are fast\n",
      "43 my independence day occurred on memorial ::: five nine across from north century past search class above\n",
      "44 day nineteen eighty two that was the day i ::: nine pair chair past female across century over male length\n",
      "45 told my family i was gay ::: chair prince pink coat bedroom pair male neck ruler photograph\n",
      "46 the act itself ::: shout spoon hair lip prince soft cupboard coat tall star\n",
      "47 mom dad i'm gay ::: know shout you i listen easy worry fail clever please\n",
      "48 was relatively unexceptional ::: i know listen easy write your wonder not nor understand\n",
      "49 in fact it should have been ::: you not about talk any real is type how nature\n",
      "50 more exceptional and i've always sort of wished ::: not kind know if how thin polite behave easy dirty\n",
      "51 that it had been however ::: thin\n",
      "52 subsequent events overshadowed ::: \n",
      "53 it and it pales by comparison ::: \n",
      "54 the subsequent events occurred ::: your i position our order question understand how best method\n",
      "55 in my absence after the fact as i was ::: my i their is since out grandfather show stay home\n",
      "56 in my car driving back to college ::: over month seven east from join five home nine evening\n",
      "57 to take my final freshman exams ::: five home join its month hour evening over children collect\n",
      "58 i remember being on the highway and thinking ::: down sure we put i watch door me let everybody\n",
      "59 how you know i kind of expected my parents ::: know i you something think do easy anything really want\n",
      "60 to freak out a little and you ::: know do you i think something what really easy anything\n",
      "61 know to my surprise they had not freaked out ::: you know if really anything how kind little want something\n",
      "62 they'd been calm and cool and collected ::: you too behave know fun laugh want always dirty often\n",
      "63 oddly calm ::: i know you too understand hello seem funny afraid behave\n",
      "64 cool and collected but still i was really ::: i\n",
      "65 happy as i drove back to school ::: i library\n",
      "66 meanwhile subsequent events were busy unfolding ::: entrance bright green chair photograph model library student visit structure\n",
      "67 back home ::: green on train explain direction chair entrance library friendly search\n",
      "68 my mother was going through the house where i grew ::: your she telephone bridge business police bus smell everybody hall\n",
      "69 up and was gathering together things ::: its police office in shake her an people call road\n",
      "70 i'd made for her um a jewelry box ::: hospital visit threat travel\n",
      "71 when i was in four h and a painting ::: be\n",
      "72 when i was sixteen um ::: \n",
      "73 a box of letters con or a box containing ::: me i thing am something anything music husband hobby goodbye\n",
      "74 the letters that i'd written them from school which ::: i friend sure go me glad sorry you get am\n",
      "75 i used to do every week ::: you i ask mean know worry sorry glad hello bite\n",
      "76 she was removing photographs ::: east hit pull near fork pocket slip back hundred pour\n",
      "77 from the walls and placing them in little piles around the house ::: near nine east float chair desk taxi coin bedroom hour\n",
      "78 and she was directing my father ::: me friend she my i sorry fool hurt knife sure\n",
      "79 who never dared not ::: know i me is he should probably really always everybody\n",
      "80 follow her direction to take ::: will here me one nothing happen put every really people\n",
      "81 the bed and the desk and the chair ::: nine morning at its school east home bedroom tie evening\n",
      "82 and the lamp and the smith corona ::: nine east bedroom pour near rent across branch taxi rob\n",
      "83 my smith corona even and to put them all ::: i bus entrance hit hundred near clothes pour bedroom hers\n",
      "84 in the front yard next ::: float pocket floor tie bag front shoe hit hundred taxi\n",
      "85 to the rock garden not too close to the maple ::: \n",
      "86 tree ::: \n",
      "87 my clothes my books ::: i bedroom village hello taxi morning\n",
      "88 my bookcases ::: bedroom taxi hall visit entrance\n",
      "89 my report cards my ::: pupil i\n",
      "90 farah fawcett posters my shoes ::: village report group substance nor god member newspaper education sex\n",
      "91 three years worth of interview magazines ::: nature report study substance extremely general yet parent science business\n",
      "92 the good ones with the andy warhol covers ::: study nature report substance general modern\n",
      "93 everything ::: \n",
      "94 then with my ::: tell student why town door learn here we everybody visit\n",
      "95 brother and my sister and my grandparents ::: my grandfather hello everybody across sister study member door born\n",
      "96 watching my mother ::: his nine pair arm head pen chair police photograph desk\n",
      "97 removed a cigarette from this tiny crocheted case ::: pair hammer shoulder rubber nearly nine pink iron leaf his\n",
      "98 she always kept them in and she ::: leaf piano thick hammer pour dance float chair pin rubber\n",
      "99 lit the cigarette and then she took the match ::: hit double his shelf pocket dance nose air float sword\n",
      "100 and put it to the pile of things there in the front ::: double shoulder pull sand boil dance wild ice foot sock\n",
      "101 yard that contained the soul ::: how can real you no trust expect fat depend need\n",
      "102 and complete record of my existence ::: that can real else not how me may your true\n",
      "103 in my family ::: know anything easy understand happen think careful not useful afraid\n",
      "104  ::: try happen you win slow next begin tomorrow kill\n",
      "105 it burned for seven and a half hours ::: ten per length its shelf below road pour plastic sail\n",
      "106 thanks in part to the addition of some lighter ::: silly your nearly even hers pull shelf must enough film\n",
      "107 fluid to help get the larger pieces of furniture ::: \n",
      "108 going ::: i\n",
      "109 all of it all that was ::: i yes my cry too glad hers none little really\n",
      "110 me prior to that memorable ::: i ago yes ten my this will hers their here\n",
      "111 memorial day up ::: i nine ten ago bedroom century total daughter prize seat\n",
      "112 in flames according to my sister ::: i my home excited ago hall since cry nine family\n",
      "113 who years later recounted these details ::: i know never we is probably me family ago are\n",
      "114 to me it was a ::: game seven ago spring never one boy last sport terrible\n",
      "115 mighty impressive blaze ::: \n",
      "116 {cg} in ::: pleased\n",
      "117 their eagerness to feed it and ::: really kind not you can here understand into how i\n",
      "118 due to an unexpected wind off the fields ::: on behind into move narrow spring east deep free every\n",
      "119 around the house the sugar maple ::: its in pair middle east over old third across rubber\n",
      "120 that was older than my great grandfather ::: pair nearly its past corner on green across rubber above\n",
      "121 caught a spark in its branches ::: float pair across nearly rubber foot third shoulder ice corner\n",
      "122 and was sacrificed ::: float wire net rubber\n",
      "123  ::: \n",
      "124 they cut off all communication ::: i chair\n",
      "125 with me they emptied and ::: his back home join after near gate board front morning\n",
      "126 closed our joint bank account ::: five home past children his month middle after at office\n",
      "127 there goes college ::: seven wife nine evening join ago office village school home\n",
      "128 they barred the door they ::: nine wife home school evening son branch join pupil ago\n",
      "129 stopped talking stopped answering ::: him join ago he stay son am hour speak member\n",
      "130 my letters stopped taking my calls they stopped ::: \n",
      "131 anything with me they just ::: \n",
      "132 stopped ::: \n",
      "133 i was ::: \n",
      "134 completely disbelieving i mean this didn't ::: i\n",
      "135 make any sense uh all of my friends ::: i my husband wife student pupil education science skin office\n",
      "136 had you know stories about telling their families they were ::: she may my in that about children any say always\n",
      "137 gay and they all ended the same way sooner or later ::: never think should we i may be nothing people important\n",
      "138 everything worked out fine so ::: i pleased\n",
      "139 i even had a friend named neil whose ::: library\n",
      "140 parents had done the same thing you know at first they just ::: my what i why we she very say entrance may\n",
      "141 stopped talking to him but ::: my i you she about here look afraid in everybody\n",
      "142 one year later they were inviting his ::: my your next seven into telephone on daughter i silver\n",
      "143 new boyfriend to come home with him for the holidays ::: her at home grandfather his off hit children my invite\n",
      "144 everyone counseled ::: you who she him glad let ask children tell month\n",
      "145 me to have a little patience ::: me think i yet be we he pretty everybody try\n",
      "146 and have a little hope ::: \n",
      "147 and this is how it ::: i understand what mean therefore may if think you correct\n",
      "148 starts slowly just a ::: how i mean understand nature correct wonder may useful sorry\n",
      "149 little hope just enough to get you through ::: useful\n",
      "150 but hope ::: are nature result report must period amount key measure may\n",
      "151 is cumulative a little bit ::: yet nature produce more that expect measure period certain key\n",
      "152 here and a little bit there it builds ::: nature substance key produce useful prevent power thin mix measure\n",
      "153 up in the system until it becomes something toxic ::: substance nature thin structure modern skin key plastic leaf oil\n",
      "154 denial ::: nature substance thin object measure depend leaf letter useful method\n",
      "155 i mean ::: nature useful method behave thin sweet clever substance soft depend\n",
      "156 their reaction had been yes ::: how you think real should is why may amount anyone\n",
      "157 extreme but not the worst that could happen ::: know we anything why do may should is extremely true\n",
      "158 the thing to do was to be a good son ::: always really if not know do anything never useful should\n",
      "159 to make them proud to ::: we always should think really know useful him do but\n",
      "160 earn back their love ::: my position off friend ladder her flat your student shoot\n",
      "161 so i got a job and ::: his ten home collect join pair school top office her\n",
      "162 then another and then a third three shifts ::: his join east its nine collect home on library pair\n",
      "163 three restaurants ::: pair home near join morning off collect below climb plate\n",
      "164 six days a week that ::: hour visit enter join ladder collect basket slip\n",
      "165 would show them ::: join receive collect visit\n",
      "166 but they weren't watching ::: hammer chase slip rob ball brother game fork shirt dinner\n",
      "167 i wrote them ::: rice shirt\n",
      "168 letters lots of letters ::: village\n",
      "169 about nothing you know it's uh tuesday ::: football furniture pupil\n",
      "170 and it's hot or my new roommate ::: his question star male who pair spring have cupboard past\n",
      "171 is named kathy ::: his five free middle male brush month old coat famous\n",
      "172 or my friends took me out for my birthday yesterday ::: his ten month male her aunt bedroom bath prince pair\n",
      "173 they ::: i\n",
      "174 didn't write back ::: i her gate floor enter search my hall entrance second\n",
      "175 living for me sort of came to a halt ::: my back i down then put walk we outside door\n",
      "176 um despite the fact that my life just ::: on in since into east across ago past open next\n",
      "177 went on and on i didn't think about ::: nearly she since in never several i will be direction\n",
      "178 my future i didn't think about my needs ::: since telephone extremely event from education she science condition by\n",
      "179 i didn't think about my sadness i didn't think about ::: not nothing people understand never is very that about sport\n",
      "180 any of it i didn't have to because i had hope ::: not never really heart nothing yet hate very seem is\n",
      "181 everyday whispering in my ear ::: i nothing think not you how like know really never\n",
      "182 don't give up don't walk away you're ::: i want like not how something voice depend hate happen\n",
      "183 almost there don't stop don't ::: not that be sound word can like hate how i\n",
      "184 grow don't develop don't don't ::: are between grave lip boil usually depend steam across rock\n",
      "185 worry just don't make any sudden movements ::: hammer fever soap pour dance sugar iron piano tea wire\n",
      "186 or you'll blow it {sl} ::: \n",
      "187  ::: \n",
      "188  ::: \n",
      "189 uh so six years ::: \n",
      "190 went on like this without a word from them ::: i ten birth silver\n",
      "191 so finally ::: i ten ago birth\n",
      "192 hurt and confused ::: i my may seven ago save hour about ten we\n",
      "193 beyond my ability to ::: never is may not we about week these period true\n",
      "194 hold it in and frankly finding it really ::: this kind nothing yet fact true enemy therefore sometimes many\n",
      "195 difficult to maintain the illusion that this was temporary ::: may this real is i therefore anything yet nothing nature\n",
      "196 i ::: i think should yet seem nothing true extremely useful therefore\n",
      "197 decided to kind of ::: i\n",
      "198 make one more attempt to ::: i nature science understand\n",
      "199 force the issue so ::: i my this why discover picture job am first about\n",
      "200 i i flew home ::: my her i then at ten grandfather am off him\n",
      "201 and showed up unannounced at my mother's ::: her grandfather his join bedroom ten at office middle morning\n",
      "202 office ::: at her grandfather friend on member student shop chair outside\n",
      "203 it was an amazing visit ::: elephant tea telephone forest\n",
      "204 excuse me {sl} ::: \n",
      "205  ::: \n",
      "206  ::: \n",
      "207 i asked ::: \n",
      "208 the receptionist to ::: \n",
      "209 to page my mother and tell her she had a surprise ::: its library their private general class nearly on large office\n",
      "210 visitor ::: provide list\n",
      "211  ::: desk chair gate\n",
      "212 and i stood there in the lobby {cg} ::: my around front hall down back into entrance sit window\n",
      "213 and i remember seeing my mother come ::: his down my front around her bed shoot over hall\n",
      "214 down this long hallway toward me ::: down her shout you hello me your everybody door bed\n",
      "215 and she was walking and then she ::: shout hello down taxi corner go door chair your walk\n",
      "216 sort of looked up ::: i corner hello bed down front pen entrance silver shout\n",
      "217 and she saw me ::: \n",
      "218 and then she recognized ::: you my hello go everybody down back comfortable door friend\n",
      "219 who it was ::: you hello walk shout go bite your slip east down\n",
      "220 and she turned and walked away ::: you down ball shout expect slip hello hear careful bottom\n",
      "221 again ::: \n",
      "222 it was a really amazing ninety second ::: is sentence per will result month trust seven fever week\n",
      "223 visit ::: nine six week period past month result hour square fever\n",
      "224 two and a half weeks later ::: nine month ten last hundred his finish ago arrive from\n",
      "225 a black funeral wreath ::: nine month second his east in sail round bedroom by\n",
      "226 was delivered to me at my office with a note that ::: his the nine is hit my yesterday shirt home her\n",
      "227 said in memory of our ::: nine by his be shirt woman dead eight month hit\n",
      "228 son ::: \n",
      "229 clearly it was time to ::: i therefore she film it her library which will be\n",
      "230 give up hope and take up therapy ::: not i very think she how never much why wonder\n",
      "231  ::: i pleased\n",
      "232 so i ::: i should she yet how though glad wish provide school\n",
      "233 uh i talked to a counselor who asked ::: i she not how never very glad sorry probably am\n",
      "234 me why i had invited ::: never not how should i is since may much very\n",
      "235 this turmoil into my life ::: never not is should she very how may really music\n",
      "236 i uh i talked to a minister who ::: never is she yet music should her may how really\n",
      "237 suggested a christian youth camp ::: city event nearly seven tennis student children group science dance\n",
      "238 i ::: pupil\n",
      "239  ::: \n",
      "240 i talked to a lesbian ::: sport piano born star shirt who tall potato black coat\n",
      "241 who offered to slash my mother's tires ::: his its will free hit put my science coat dance\n",
      "242 if i'd pay for her flight there ::: on pair across its nearly float chair drive entrance coat\n",
      "243 i signed up for ::: chair on coat across dance desk double float hammer wire\n",
      "244 scream therapy where i beat pillows with tennis ::: on dance rubber chair fresh hammer its coat middle iron\n",
      "245 rackets and screamed obscenities and ::: rubber hand dance rock pair green brush double hammer red\n",
      "246 you know pulled a muscle in my shoulder ::: \n",
      "247 mostly i talked to friends and ::: i\n",
      "248 mostly the pain persisted ::: i\n",
      "249 the sheer ::: i my ago none never remember age sorry quite am\n",
      "250 weight of it nearly crushed me or at ::: i husband my wife\n",
      "251 least that's how it felt at the time ::: i friend my wife daughter library family\n",
      "252 since it was my constant companion ::: i know my we try glad ask why someone friend\n",
      "253 i spent ::: ask someone your glad next my call friend back spend\n",
      "254 most of my time turning it over in my mind ::: fill spend decide send hand hammer receive\n",
      "255 fingering it like ::: fold sun forest fill sword winter begin cap increase market\n",
      "256 some sort of psychological worry stone ::: rubber leaf sand flag forest cap fold narrow\n",
      "257 over ::: \n",
      "258 the years it's been eroded ::: pour steam fold clock gate entrance ice\n",
      "259 by so much handling ::: steam fold clock top thin pour float rubber flag century\n",
      "260 all that remains now is ::: pour\n",
      "261 a small ::: entrance pour desk clock library\n",
      "262 hard nearly weightless pebble ::: pour leaf entrance sun narrow its below thin clock floor\n",
      "263 really worn away ::: i pour clock narrow comb\n",
      "264 is most of the anger and ::: i pour across library ten east clock\n",
      "265 much of the hurt but one question ::: ten i across hundred begin half clock size arrive below\n",
      "266 remains how was it possible ::: we much i may how say little possible see here\n",
      "267 that they taught me love and loyalty ::: you know may is why how if think real anything\n",
      "268 in excess of that which they themselves ::: is this always not anything who these we people think\n",
      "269 possessed ::: are is team event most stay since threat order past\n",
      "270  ::: \n",
      "271 i have come ::: therefore may most person anyone i choice useful sentence wish\n",
      "272 to believe that it's not possible to understand ::: know may we why what should my about is i\n",
      "273 what they did not possible ::: you my we where about think know me let here\n",
      "274 for me anyway to understand ::: east my hello daughter visit aunt back home telephone morning\n",
      "275 it ::: aunt hour school boy hello east visit daughter home children\n",
      "276 would seem to indicate that there ::: know why people never not ago may i are anything\n",
      "277 was some justification for it and i know for ::: know not anything something think is really may always why\n",
      "278 certain that there is not ::: know i anything think do if you sorry fun mean\n",
      "279 still there's ::: i know we want think hate sorry\n",
      "280 no escaping my parents this um ::: know i why never we you anything think mean not\n",
      "281 this thing they did this extreme ::: think yet know is never real lot we happen i\n",
      "282 and unfathomable ::: is true kind yet double think enemy nothing amount hurt\n",
      "283 and many layered thing they did ::: enemy happen kind already ground shoulder steam shine double period\n",
      "284 tore a hole in the middle of my life ::: chair gate floor wheel\n",
      "285 i have spent ::: i taxi bedroom pour\n",
      "286 years and a lot of money ::: four their hundred his door in prince coat pink your\n",
      "287 darning that hole ::: his nearly pair shoulder float eight leaf sharp coat drop\n",
      "288 while trying to keep the rest of ::: nearly third float pair coat lamp leaf shoulder nine rubber\n",
      "289 my world from unraveling ::: nine lamp sheet silver float square total per flat cloth\n",
      "290 and yet their influence ::: his ten chair duck i up minute shoulder\n",
      "291 on me is enduring ::: \n",
      "292  ::: \n",
      "293 my parents loved me terribly ::: chair hello heaven goat photograph god wife prince\n",
      "294 i have been ::: know how me woman anything love real someone father understand\n",
      "295 courageous in the face of bullies ::: real she yet uncle life woman friend him he me\n",
      "296 there is ::: never know is uncle nature must why he anything she\n",
      "297 such a thing as too much patience ::: how who never he kind know someone difficult case member\n",
      "298 but ::: who know him person he yet is nature false lesson\n",
      "299 no such thing as too much forgiveness ::: know if me you he anything never why stupid can\n",
      "300  ::: know can never me not really why anything he job\n",
      "301 and love has seen me ::: \n",
      "302 through every misfortune ::: \n",
      "303  ::: \n"
     ]
    }
   ],
   "source": [
    "test_ws = wordseqs[test_stories[0]]\n",
    "test_words = test_ws.chunks()[trim[0]:-trim[1]]\n",
    "\n",
    "for i in xrange(ntrials_test):\n",
    "    nwds = (decoded_words[i,:]>0).sum()\n",
    "    word_ix = np.argsort(decoded_words[i,:])[::-1][:nwds]\n",
    "    print i, ' '.join(test_words[i]), ':::', ' '.join([basis_vocab[i] for i in word_ix][:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "# the challenge\n",
    "\n",
    "Now that you've got an idea for what decoding is and how you might go about it, your challenge is as pretty simple. Design a decoder that considerably outperforms our decoder and you win the prize! Train your decoder using the stimuli and responses for the following stories:\n",
    "- `alternateithicatom`\n",
    "- `avatar`\n",
    "- `howtodraw`\n",
    "- `legacy`\n",
    "- `life`\n",
    "- `myfirstdaywiththeyankees`\n",
    "- `naked`\n",
    "- `odetostepfather`\n",
    "- `souls`\n",
    "- `undertheinfluence`\n",
    "\n",
    "And decode the responses in  a prediction for the story we'll evaluate how well it performs on a held-out test story:\n",
    " - `TEST`\n",
    "\n",
    "**Please only use the test story to generate your submission. Do not peak at it to build your decoder.**\n",
    "\n",
    "# submission format\n",
    "Run your decoder on the responses stored in the `TEST` field of `responses.h5`. Produce a space-delimited text file in which each line contains the word predictions for a single time point. That is, the text file contains one line per row in the `TEST` response array. Include a blank line if your decoder predicts no output for a given time point. For clarity, we've provided an example submission called `GLABEXAMPLE_decoding_results.txt`\n",
    "\n",
    "Send an email to `robertg@berkeley.edu` with the subject \"`<team_name> 2016 IEEE HACKATHON SUBMISSION`\". Include as attachments the following files:\n",
    "- **`<team_name>_decoding_results.txt`**: the aforementioned space-delimited text file\n",
    "- **`README.txt`** (or `.pdf`, `.md` if you want to include images or formatting): a document containing a brief description of your approach\n",
    "\n",
    "# evaluation\n",
    "We'll evaluate the quality of your decoding using one quantitative and one qualitative method.\n",
    "\n",
    "### 1. Mean word embedding vector correlation\n",
    "We'll take the decoded words you submitted for the test story and transform them into the semantic word embedding space. We'll sum the vectors for each time point and compute the correlation coefficient between the summed vector with the semantic vector from the actual story."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2016-10-05T16:41:13.422249",
     "start_time": "2016-10-05T16:41:13.057200"
    },
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEkCAYAAADeqh2wAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XeYJGW5/vHvO8POxiYIEpawggQFA8oxHTy4YD6C6biP\nkWyEYwKOAiq7ICDgmQHEgwoCgoLwACKCigEYFRFEZPkpGVzCsuySl9lZ2DBdvz/eGqjt7Z6unuk0\n0/fnuuaa7uqqt+6qnumnK70VkiRBRERkJF2tDiAiIu1PxUJERKpSsRARkapULEREpCoVCxERqUrF\nQkREqlKx6BAhhLkhhHtanQMghLAghHBkpecyeiGE7hDC2SGEx0MIQyGE3VqdSSYGFYvO0q4X1fwb\ncHIzZhRCWBVC2KcB7d4TQjiq3u2Own8BHwHeA2wGXN/aODJRrNPqACJJkjzR6gzjXQhhUpIkq4Dt\ngYeTJLmxTu2JANqymJBCCJNDCN8LITwdQngihHA6MLnMeB8JIdwSQng23RXUG0KYVjLOwSGE20II\nz4UQloQQLs68NiOE8IMQwqPp6zeFEN5eMv2rQwjXp6/fFUKYUyZHud1SR4cQTknzLw4h9IUQujLj\nTAkhnJFZxv8LIRw/0q62EMIC4t/8OSGEYghhKPPaLiGE34QQBtLluTSEsFXm9c1DCJeEEB5L19e9\nIYRD09euBV4KzB1uNzttSYZzQgi/CyF8KYSwMIQwGELwEMIGtbw3IYRrQwg/DCEcE0JYBDyQ5jgG\n2CbN8a903HVCCCek81uRvp8fLZlfMYTw+RDC+SGEp4HzQgiz0uEfDSFclWa9I4SwWwhhZgjhlyGE\nZWl7by5p74x0HS0PIdwXQjguhNCTeX1uujX23rTNZekybVvSzi4hhF+HEJam780NIYTXZV5/ewjh\nunQ+C0PcBfeiSn8DMgZJkuhngv0Qd+ksBvYkftP8NrAUuDszzn7AE8DHgFnAm4H5wLmZcY4GngE+\nB2wL7AwckXn9YuBfwNuAHYBTgBXA9unrU4CFwBXAK4A3AH8FlgFHZtpZUOb5E8BXiB/CHwJWAvtn\nxvkO8Ahxd8t2wPHA09llLLNeNgJWAf8NbAxsnA7fERgAjkrb2gm4CLgL6EnH+QXwW+CVwFbAW4AP\np69tkK6Hk4bbBUKFDOek78XP0/nuBtwNXFrje3Nt2s7pwMvSzOun7/V9wIuBDdNxvw08BnwwfR+P\nAIaA3TPtFdNxDgK2Ttf7rHT4PcBe6bQ/Axal6+J96bCLgQeA7rStAHyTuHtxK+Lf4cPA3Mz85qZ/\nB78i/l29Evgb8IfMODul4/wEeE2a6cPAG9LX9wAG08zbALsAVwPXtvp/cCL+tDyAfur8hsI04Fng\ngJLhN7FmsVgAfLpknP9IPxzWS9tZDny5wnxemo77zpLhNwM/TB9/klhs1s28vlM6XbVi8fOSdn8F\nnJ9ZxueA/UrG+QsjFIt0nFXAPiXDzgEuKBk2Of0gem/6fD5w1Ajt3jPS6yXzegaYkRn29nSdbJPn\nvUmfXwvcWab9uSXv89R0XX2mZLyfAb/PPC8CZ5SMM1wsPp8Z9m/psC9lhu1MLD47jrDcXwLuKsm5\nEnhRZpgBq3mhQP8YuGWENq8Fji8ZtlWa71XN+H/rpB/thpp4Xgr0ED84s64bfhBC2Ij4QdCXbtoP\nhBAGgF8T/9G2JX6oTwZ+V2E+OxIPmP+pZPgf02kBXg7ckSTJM8MvJklyG/EbcTXzS54vAjZJH28L\nTAJK98uXLnNerwM+ULIuHicu/3bpOKcAX0t3g5wQQviPUc4L4PYkSZZlnv85/b1jlfcmIS77sJtz\nzGt4XZW+T3/ghfdp2E0V2vh/mceL09//KBkWiFtUAIQQPpWuq8Vp/m8RlytrUZIkT2afl7TzWuKW\nQiWvA75Usp5uI66n7UaYTkZBB7gnplDl9eEvCV8A+su8vhB4dY52GmllyfOENY+xBep3dlcX8Vvs\nt1h7mZ8ASJLkRyGEXwPvAnYHfh1C+FmSJPU+syrPezNsMGebed/HSu1lD3QnIwzrAkiPS32XuBvx\nj8QtKQOOLWm33Hv8fDs5dAEnEt+7UovLDJMxULGYeO4j/hP+O3BHZviuww+SJHk0hPAQ8LIkSc4u\n10gI4Xbi7ot3AP8sM8pt6e/dgKsyw3fjhW+8twOfCiGsO7x1EULYibibayzuJS7jm4A7M8PfmGPa\nlUB3ybC/EXdbLBhpwiRJlgDnAuemheOCEMJB6VZCuXYreXkIYUZm62JX4gflbXnemxrdSzyOtBvx\n/Rg2m/Lvaz38B/D3JElOHR4QQth6FO3cDLx1hNf/BuyUJMm/RtG21EjFYoJJkmR5COH7wLEhhEeJ\nB2kPJB6AXpIZ9WvAD9MzXy4nflPcEXhXkiSfTZJkMITQC8wLITxH3B01DXh3kiQnJEnyrxDCJcDp\nIYTPEg9wHkTctTF8ps0FxAOd54cQvpZOfwrxWMhYl/EHmWW8G9iXuNvr0SqTLwB2DyFcBaxM4mm7\nxwM3hhB+ApxKPNC7NfEA7ilJktwfQjiNeNzkLuJxgP8CHsx84C8Adg0hbJku35NJuhO93CIQzzb6\nBrAh8Vv45ZliNeJ7k3M1xRklybMhhO8A3wwhPA7cCswhHrB+Wy1t1eAu4IAQwnuJBWkv4AM5p81u\nBZ0E3BBCuADoBZ4i7pp6KImnBh8F/Cb9Oz2PeJLC9sQTIg5OkmRFPRZGIh2zmJgOJ55tcx5xv/56\nxA+k5yVJ8hPiroH3pOP8lfjPtzAzzjeIH1yfJ+6jvop4VsqwA4HfEHcDzCd+039PkiR3p9M/C7wb\neFE6jx8Dfaz9gV76oZpn99JXiGdZnZ+2vT7wI+LW0EgOJZ41c/9wjiRJ7iRuiU1Pl/E24AfEs7me\nTqcLxLPM/kHcPTQV+M9Mu3PTDHel7W45Qoa/Eo8h/Y5YgG4lrkvSPFXfG2rbBfc14MxM/o8BH0+S\npD9He+WGVxv2A+J7fTbwd+Kxhbk5sz7fTpIk/yRuAW1EXOe3AIcQD6aT5t+DeCbVH4nrsZe420vX\niNRZqPzlpzHMrIu4+bjQ3d9rZi8BLiR+oNwM7O3uq5saSiaEEMLVxG/0a13L0S5CCOcAmydJ8o5W\nZxGpRSu2LL7ImvtOTwR63X174re4A8tOVcLMZtc/Wv0pZ30N5wwhvCKEsE8IYbv08YnEb6FntDIf\njL912e6Us75Gm7OpxcLMtiBuuv8wM3gP4NL08bnk37c5u37JGmp2qwPkNLvVAXKanf5OiBcL/pV4\n6uls4P1JklQ61beZZrc6QE6zWx0gp9mtDpDT7FYHyGn2aCZq9gHuk4H/IT0bxsw2BJ5y92L6+kJg\nZpMzyTiUXq/xplbnqFWSJPu3OoPIaDRty8LM3gMscff5rHnGQyvP5RcRkRyadoDbzI4HPkG8nH8q\nUCCesfMOYFN3L5rZG4G57v7uMtPPJrP55O55z64QEZEMMzs687Tf3furTdP0s6EAzOwtwKHp2VAX\nAT9z94vM7HvAre7+/RzNJIsWLWps0DooFAoMDAy0OkZVylmb3t5eDj300LKvtUvGapSzvsZLzpkz\nZ8Io9ui0w3UWhwOHmNndxNNnz2pxHpGq+vr6Wh1BpKlacgW3u/+B2JEZ7r6A2HW1iIi0qXbYshAR\nkTanYiEiIlWpWIiISFUqFiKjcMghh7Q6gkhTqViIjEKl02ZFJioVCxERqUrFQkREqlKxEBGRqlQs\nRESkKhULkVHo7e1tdQSRplKxEBkF9Q0lnUbFQkREqlKxEBGRqlQsRESkKhULERGpSsVCZBTUN5R0\nGhULkVFQ31DSaVpypzyRVkkWLqB43Cg+6GduRfc3Tql/IJFxQsVCOksCrF5d+3RDQ3WPIjKeNK1Y\nmNlk4I9ATzrfS9z9aDM7B3gLsJT4r7yfu/+/ZuWSDrX5LLq+luPCukUPUDxWxydEmlYs3H2Fme3u\n7svNrBv4s5ldlb58mLv/rFlZRAiBMGlS1dGSbm18i0CTD3C7+/L04WRioSqmz0Mzc4iMlfqGkk7T\n1GJhZl1mdguwGPidu9+UvnSsmc03s14zq/51T6TF1DeUdJpmb1kU3f01wBbA681sR+Bwd3858Dpg\nQ+CrzcwkIiLVtWSHrLs/Y2b9wLvcvS8dtio92F32vEYzmw3MzrRBoVBofNgx6unpUc46GmvOoenT\nGAC6urpytTM0fXrF8StN3ynrslmUs/7MbF7mab+791ebpplnQ20ErHL3pWY2FXg7cIKZberui80s\nAO8H/llu+nRh+jOD5g4MDDQ49dgVCgWUs37GmjMZjIfNisVirnaSwcGK41eavlPWZbMoZ30VCgXc\nfV6t0zVzN9RmwLVmNh+4EfiNu/8KON/MbgVuJe6GOraJmUREJIdmnjr7D+C1ZYa/tVkZROpFfUNJ\np1HfUCKjoL6hpNOoWIiISFUqFiIiUpWKhYiIVKViISIiValYiIyC+oaSTqNiITIK6htKOo2KhYiI\nVKViISIiValYiIhIVSoWIiJSlYqFyCiobyjpNCoWIqOgvqGk06hYiIhIVbm6KDez3YH73X2BmW0G\nnAAUgSPcfXEjA4qISOvl3bI4HRhKH/cCk4jF4oxGhBIRkfaS9+ZHm7v7g2a2DvBOYBawEljUsGQi\nOSRDQ/Dc8vwTPDvYuDAiE1jeYvGMmW0CvAK43d2XmVkPcQtDpHUevp/iN7/c9Nn29vbqILd0lLy7\noU4DbgLOB/4vHbYrcGcjQonULHTBtBn5f6ZMG9Ps1DeUdJpcWxbufqKZXQYMuft96eCHgQPzzsjM\nJgN/BHrS+V7i7keb2UuAC4EXATcDe7v76vyLIAJsMYvuo05tdQqRCSvXloWZXe7ud2cKBe5+N3BM\n3hm5+wpgd3d/DbAz8G4zewNwItDr7tsDT1NDARIRkebIuxtq9wrDZ9cyM3cfPhI5mbh1kaRtX5oO\nPxf4QC1tiohI4424G8rMhrccejKPh20DPFDLzMysi7ir6aXEYx/3AU+7ezEdZSEws5Y2RUSk8aod\ns9gy/d2VeQxxi+AhYF4tM0uLwmvMbF3gMuBltUwv0i7UN5R0mhGLhbvvD2Bm17v7mfWaqbs/Y2b9\nwJuA9c2sKy0kWxAPnK/FzGaT2e3l7hQKhXpFapienh7lrKPSnKunTWcZ0N3d3ZD8Q9OnMwB0dXWt\n0f68efNyZ2xXyllf4yUngJnNyzztd/f+atPkPRvqTDNbD9gBmFHy2jU5w20ErHL3pWY2FXg7sduQ\na4E5wEXAvsDlFTL0A/2ZQXMHBgbyzLqlCoUCylk/pTmT5fEiu6GhoYbkTwZj+8ViMXf743Vdtivl\nrK9CoYC7z6t1urx9Q+1HPMawDMheLpsQj13ksRlwbnrcogu4yN1/ZWZ3ABea2TeBW4CzcrYnIiJN\nkvcK7uOAD7n7r0c7I3f/B/DaMsMXAG8YbbsiItJ4eU+dXQf4bSODiIhI+8pbLE4Evp7uQhLpeL29\nva2OINJUeXdDfRnYFPiKmT2RfcHdt6p7KpE219fXp44EpaPkLRafaGgKERFpa3lPnf1Do4OIiEj7\nqlgszOxr7n5c+rhih4HuflQjgomISPsYactii8zjLSuOJSIiE17FYuHun8s83r85cUTGB/UNJZ0m\n7wFuzGw74KPA5sT+m37q7vc0KphIO9OZUNJp8t78aC9i1+IvA54k9hH1NzN7bwOziYhIm8i7ZXE8\n8D53v3Z4QNoL7HeBXzQgl4iItJG8xWIL4E8lw65jzYPgIhPXY4sZ+vaRuUYd6O6muP6GdB3wpQaH\nEmmevMViPnAosduPYYekw0UmvpUr4O5/5hp1CGAznUAoE0veYvE54Aoz+yLxDnlbAYPAXo0KJtIW\nNtqErsOOW2tw749+zKH77b3W8OTxJSQ/+k4zkok0Vd4ruO80s5cDbyTeI3sRcKO7r2pkOJFWC1Om\nwg6vXGv4yeedz2HfOmntCWasR9KEXCLNVksvsknJT7EhiUREpO3kPXX2VcA9gAP/A1wM3GNmr25g\nNhERaRN5tyzOJt5WdQt3fz3xwrzvpsNFRGSCy1sstgdOcfcEIP19KrBdo4KJiEj7yHs21K+A9wKX\nZYbtBfwy74zMbAvgPGAT4vGOM9z9NDObC3wKeDQd9Uh3vypvuyKtoL6hpNPkLRbdwIVmdjPx1Nkt\ngV2Ay83svOGR3H2fEdpYDRzi7vPNbAZws5n9Ln2tz937ao8v0hrqG0o6Td5i8c/0Z9jtwG9qmZG7\nLwYWp4+XmdkdxGMfAKGWtkREpLnyXmdxdD1namYvAXYGbgTeDBxsZnsDfwMOdfel9ZyfiIiMTe4u\nyusl3QV1CfDFdAvjdOAYd0/M7FigDziwzHSzgdnDz92dQqHQnNBj0NPTo5x1VJpz9bTpLAO6u7vb\nIv/Q9GkMAF3dXW2RZyTj9T1vV+MlJ4CZzcs87Xf3/mrTNLVYmNk6xELxY3e/HMDdH8uMciZwRblp\n04XpzwyaOzAw0JigdVQoFFDO+inNmSwfBGBoaKgt8ieDywEoDhXbIs9Ixut73q7GU053n1frdLVc\nwV0PZwO3u/upwwPMbNPM6x9kzWMjIm2pt7e31RFEmqppWxZmtivwceAfZnYLscuQI4GPmdnOxNNp\n7wc+06xMIqPV19enM6Kko+QqFmYWgE8Sb6u6kbu/ysx2AzZ1d8/Thrv/mXgKbildUyETUlIcqmn8\n0FXu30OkPeTdsjgGeDtwCvD9dNhC4GRif1EikvXIQxQ/84H842+6Bd3fPL1xeUTGKO8xi/2APd39\nQni+B+YFwDaNCCUyroWuGn50iZGMD7Vcwb0sfTxcLGZkhokIEDbfivV/enXus2KSRx6ieNTBDU4l\nMnZ5tyx+BfSZ2WR4/hjGN6lwmqvIRKe+oaTT5C0WhwCbAUuB9YhbFLOArzYol0hb05lQ0mnydvfx\nDPABM9uEeP/th9K+nkREpAPUelHes8DDQJeZzTSzmQ3IJCIibSbvdRZvA84g7nrKnr6RUP7aCRER\nmUDyblmcBRxPPF4xKfPT06BcIiLSRvKeOjsFOMfda7skVWSC6u3t1UFu6Sh5tyxOBr6SnjIr0vH6\n+nRjR+ksebcsLiXeGe8IM3s8+4K76ypuqZvk4Qconv6tiq8/09VFsVh8YcDqlU1IJSJ5i8UlwJ+A\ni4lnRIk0xupV8Oiiii8XK74iIo2Ut1hsDbzG3fW/Ks2x2ZZ0HXTkWoOnT5/O4ODg2uNPmtSEUCKd\nK2+xuBzYA/h9A7OIvGBSD2HTzdca3F0oEMbB3chEJpq8xWIy8Asz+xOwJPuCu+9T91QibU59Q0mn\nyVssbkt/pMMlywdhwd21TzhlKuGlL6t/oBZp1WmzyYrn4N47ap+wZzJhux3rH0g6Rt6+oY5udBAZ\nJ5Y8TPGUubVPN2tbur+u003H7KnHR7f+N96M7uN+UP880jEqFgsz283d/5g+3qPSeO5+TSOCSZub\nMhW22aH6eM8uH92WiIyspwe2zbGlsOI5uO/OxueRCW+kLYvTgVekj8+qME5CzrvlmdkWwHnAJsQz\nIM909++Y2QbARcR+p+4HzN2X5mlTWmjTLej+8jFVR0vuv4ficbrSue42eHG+9b9kEcWvf7YJgWSi\nq1gs3P0Vmcdb12Feq4FD3H2+mc0Abjaz3wL7A79395PM7KvAEcDhdZifiIjUSa7uPszs8grDf5Z3\nRu6+2N3np4+XAXcAWwDvA85NRzsXeH/eNkVapbe3t9URRJoqb99Qu1cYPns0MzWzlwA7AzcAm7j7\nEogFBdh4NG2KNJP6hpJOM+LZUGY2vFO0J/N42DbAA7XOMN0FdQnwRXdfZmZJySilz4enm02mOLk7\nhUKh1tk3XU9Pz4TKuXraNJYB3d3dOcefno7fVWP75cdvp/VZKUctGYeemc4A0NWVb/0MDdQ4/rLK\n47fTuhyJctafmc3LPO139/5q01Q7dXbL9HdX5jHED/SHgHmlE4zEzNYhFoofu/vwrq0lZraJuy8x\ns02BR8tNmy5Mf2bQ3IFxcCVvoVBgIuVMli8HYGhoKOf4g+n4xRrbLz9+O63PSjlqyZikXZcUiznX\nTx3Hb6d1ORLlrK9CoYC7z6t1uhGLhbvvD2Bm17v7maPMlnU2cLu7n5oZ9gtgP+BEYF9i1yIiItJG\n8l6UN+ZCYWa7Ah8H/mFmtxC3To4kFgk3swOIu7VsrPMSEZH6ytvdx5i5+5+pfL/utzUrh0g9qG8o\n6TR5z4YSkQzdUlU6TcViYWbfzjyu2N2HiIhMfCNtWXw68/jnjQ4iIiLta6RjFrea2SXA7cDkMtdZ\nAODuRzUkmYiItI2RisWHiFsXs4DAmtdZDCt7AZ2IiEwsI3Uk+ChwLMSL6YavuRCR2DdUXQ9yDyyl\neOm51ccbbP+LvmRiynudxf5pV+J7AZsDDwNXuvuTjQwn0q76+vrqWywGB0iuurR+7YnUWa5iYWZv\nAn4J3Em8cG5P4BQze4+7/6WB+UQmthnrET44itvYT59R/ywiI8h7Ud4pwEHufuHwADP7MPAd4HWN\nCCbSCUJhXcK7P9TqGCJV5b0ob3vAS4ZdAmxb3zgiItKO8haLe4CPlAybA9xX3zgiItKO8u6G+hJw\npZl9gXjM4iXAdsRjFyIdR31DSafJtWXh7tcDLwW+C9wMnAZsmw4X6TjqG0o6Te5eZ939KeAnDcwi\nIiJtSr3OiohIVSoWIiJSVd6L8rrcvdjoMDKBLXuG4vVXVx/v8SWNz9KJnl2+1vpfOWUKxeeeKzt6\nmDKV8Np/b0YyGSeqFgsz6waWmdn67r6iCZlkInriUZJzTq0+3jhR976hGm1g6Vrrf/kIoycv3pRu\nFQvJqFos3H3IzO4GNgQWNT6STCjTC4Q37V77dBtuUv8sdVT3vqEaZcrUiut/nUmTWL1q1RrDkmef\nhfk3NCOZjDN5z4Y6n3idxanAQjJdk7v7NXkaMLOziNdlLHH3V6XD5gKfAh5NRzvS3a/KmUnGgfDi\nTQkHfLnVMTpWWG+Diut/eqHAwMCavdgmjy6iqGIhZeQtFp9Lf88rGZ4A2+Rs4xzi9RnnlQzvc/e+\nnG2IiEgL5O2ifOuxzsjdrzOzWWVeCmNtW0REGiv3RXlmNgl4IzDT3S8ys+kA7j44xgwHm9newN+A\nQ9196RjbExGROst76uwrgV8AK4AtgIuAtwD7Ah8ew/xPB45x98TMjgX6gAMrZJgNzB5+7u4UCoUx\nzLo5enp6JlTO1dOmsQzo7u5uyXK1y/o8/PDDK+Zol4zVlMs5NDiDAaCrq6ttlmE8r892ZWbzMk/7\n3b2/2jR5tyy+Bxzl7j82s6fSYX8AzqwpYQl3fyzz9EzgihHG7Qf6M4Pmlh6ca0eFMgcR21HenMny\neMLl0NBQS5arXdbn5z//+Yo52iVjNeVyJoPLACgWi22zDON5fbajQqGAu8+rdbq8V3DvxAv9QiXw\n/O6nqTXOL5A5RmFmm2Ze+yDwzxrbExGRJsi7ZXE/sAvxuAIAZvZ64N68MzKzC4i7kTY0sweBucDu\nZrYzUEzn8Zm87YmISPPkLRbfAH5pZt8HeszsCOCzxGskcnH3j5UZfE7e6UVEpHXy3s/iSuBdwIuJ\nxypmAR909982MJuIiLSJWu5ncQtwUAOziIwb465vKJExynvqbA/wdeCjwExiH1EXAse5e/luK0Um\nsHHTN5RIndRy6uwOwPA9uGcBRwKbAwc0JpqIiLSLvMXi/cBL3f3p9PntZnYj8WwoFQsRkQku73UW\ni4FpJcOmAo/UN46IiLSjilsWZrZH5umPgavM7DRiF+VbAgezdg+yIiIyAY20G+qsMsOOLHn+GeDE\n+sURGR8OOeSQVkcQaaqKxaIe3ZKLTFQ6E0o6Td5jFiIi0sHyXmfxauBkYGdgRjo4AIm79zQom4iI\ntIm8p87+FLiUeJ3Fs42LIyIi7ShvsdiUeD+LpJFhRESkPeU9ZnEuUK7XWJGO1Nvb2+oIIk2Vd8vi\nBOAvZnYksCT7grvvUX4SkYlLfUNJp8lbLC4BFgCXoWMWIiIdJ2+x2BnY0N1XNjKMNF/y1BMkf7mG\n5yZPprhiRfUJnnq88aFEpO3kLRZ/AnYE5jcwi7TCk4+RXPZj1M+8iIwkb7FYAPzWzC5j7WMWR9U9\nlTRd2GBDeOPs/BNssFHDsohI+8lbLKYBvwR6iJ0I1szMzgL2BJa4+6vSYRsAFxHvj3E/YO6+dDTt\ny9h0bbgxfHDfVscYN9Q3lHSaXMXC3fevw7zOAU5jzZ5qDwd+7+4nmdlXgSPSYSJtTWdCSafJ293H\nNpVec/d/5WnD3a8zs1klg98HvCV9fC7Qj4qFiEjbybsb6l4gIfYHNWz4au7uMcx/Y3dfAuDui81s\n4zG0JSL18vijDP1PDTsUNnwx3Yef1Lg80nJ5d0OtcaW3mW0KzCWeJVVPFbsTMbPZwOxMJgqFQp1n\nX389PT1tnXP19OksA0IIzGjjnMPafX3C+MgI5XMODc5gACApwtNP5G6rq4HLPJ7XZ7sys3mZp/3u\n3l9tmrxbFmtItwK+BNwNXDCaNlJLzGwTd1+SFqBHR5hnP3E31bC5AwMDY5h1cxQKBdo5ZzI4GH8n\nSVvnHNbu6xPGR0YonzOZMp2uE8/O38iTj1I88XCKxWLDlnk8r892VCgUcPd5tU43lvtZ7MDa9+Wu\nJrDmrqxfAPulj/cFLh9DHpGmmah9Q4XubsKLNsr9w3ovanVkaZK8B7j/xJq7iKYBOwHH5J2RmV1A\n3I20oZk9SNyNdQJwsZkdADwAWN72RFpJfUNJp8m7G+qHJc8HgVvd/Z68M3L3Sr3Wvi1vGyIi0hp5\nD3Cf2+ggIiLSvvLuhuohHlvI3lYVAHffp/6xRESkneTdDXUu8GrgCkr6hhIRkYkvb7F4F7C1uz/d\nyDAi44X6hpJOk/fU2QeByY0MIjKe6Ewo6TR5tyzOAy43s1NZu4vya+qeSp6XPPwAxYtKT0arLszc\niq6PfKoBiUTKWPoUQ33fyD16WHd9uj6pgjue5C0W/53+Pr5keAJU7GRQ6mD5INxxa82TJStz3PVO\npF5WrazmLSHDAAAP6UlEQVTp7zTZUN3AjTd5T53dutFBpIrNtqTrI5+sOlqy6EGSi85qQiARYN0N\n6Pry0blHT5Y+TXL2yQ0MJI0yqr6hpAWmzyDs+Jrq4/VMqdwbo0idhcmTIc/f5bDHl+jvc5waS99Q\nIh1rovYNJVKJtixERkF9Q00MydAQPPZI7RN2dRE2nln/QG1MxUJEOtfA0xS/cVDt0627Pt2951Uf\nbwJRsRARCV2w8WbVxysOwWOLG5+nDalYiIisuz7dx36v6mjJ0qcoHrZvEwK1Hx3gFhGRqlQsREZB\nfUNJp9FuqInqscUUf3pG1dGSpU82IczEozOhpNOoWExUzzxNcs2VrU4hIhOEisVEs9EmhFF0IDh5\nk81Qb1IiUklbFAszux9YChSBVe7++tYmGr/C+i8ivHWvmqfrKRRYMTDQgEQiMhG0RbEgFonZ7v5U\nq4OIiMja2uVsqED7ZBGpSn1DSadply2LBPiNmSXAGe5+ZqsDiYxEfUM1R/LEYzz96ffVPuH6G9J9\n0tn1D9TB2qVY7Oruj5jZi4Hfmdkd7n5ddgQzmw3MHn7u7hQKheamHIWenp4x5Vw9bRrLgO7u7oYu\n71hzNks75ayUo50yjqQVOYeeHWQACCHkmndxxXKeSWrv1DyEyu/PGu2vWsEzQOjKmWdoVRy/TP7x\n8r4DmNm8zNN+d++vNk1bFAt3fyT9/ZiZXQa8HriuZJx+oD8zaO7AODggWygUGEvOZPlyAIaGhsbU\nTjVjzdks7ZSzUo52yjiSVuRMBpfF30mSa97JssH4YION6Dohx+2Fn3qC4uEHkhRztj+cJ+/4yyrn\nH0/vu7vPq3W6lh8nMLNpZjYjfTwdeAfwz9amEpG2EiB0dVX9IYRWJ52w2mHLYhPgsvR4xTrA+e7+\n2xZnEhGRjJYXC3dfAOzc6hwitVDfUGP07HKK/b+qPl66m2i8SlasIPnL1bVPOKmHrl3fVv9AY9Dy\nYiEyHulMqDFavozk/O+3OkXjrXh2dMs5Y11QsRCRjjVlKuEt76ppkkmTelg1qadBgZpkUg/h3/eo\nPt7KlSR/uabxeUZBxUJEmibMWJfwidpuYzptnJxlNKIpU+nKsdzJwNK2LRYtPxtKRETan4qFiIhU\npWIhMgrqG0o6jYqFyCj09fW1OoJIU6lYiIhIVSoWIiJSlYqFiIhUpessSiSDy2DhgtonnDqdsNU2\n9Q8kItIGVCxKPXgfxb5v1D7d9q+g+3+Or38eaUvqG0o6jYpFJVOnw5ZbVx9v+eDotkRkXFPfUNJp\nVCwq2Wobug87rupoyV3/oPi/X2tCIBGR1tEBbhERqUrFQkREqprwu6GShQsoXnFR/gkGnm5cGCBZ\n9CDFyy/IP8GyZxoXRmSiGhxg6HsnVB9v1crRtb982VrtD66zDkOrV9en/TY04YsFA8/A369vdYoX\nLGuzPDIqvb29OsjdzlatbOz/2erVa7W/qnFzawttUSzM7F3AKcTdYme5+4l1n8nms+ja6yP5x5+x\nXt0jrGHTLeh6/8fzjz+90LgsUrO+vj4Vi3Y0vUDXZ79a+3Tr5Ly50rTpFdufMmUqzz33bIX2J9We\nqc20vFiYWRfwXeCtwCLgJjO73N3vrOuMCusRdtm1rk2OSWHd9sojMgGEyZOhgf9XYVJPxfZ7CgVW\njPebNI2gHQ5wvx64x90fcPdVwIXA+1qcSUREMlq+ZQFsDjyUeb6QWEDGl6HVJM88tdbg4tCq2IXI\nsOxjEZFxoh2KRWN1rwOF9WDa9MbO5747KR6671qDdS6TiOQX4ufVjHVbHWQt7VAsHga2yjzfIh22\nBjObDcwefu7uzJw5s3rrM2fC7LePNePI7e/+zsa130SFwvg4iN4OOZMkGfH1dsiYh3LW19hzzoQL\nr65LlpGY2bzM03537686UZIkLf2ZM2dO95w5c+6dM2fOrDlz5vTMmTNn/pw5c16eY7p5rc6ec/mU\ns8NyjoeMyqmctf60/AC3uw8B/w38FrgNuNDd72htKhERyWqH3VC4+1XADq3OISIi5bV8y2IM+lsd\nIKf+VgfIqb/VAXLqb3WAHPpbHSCn/lYHyKm/1QFy6m91gJz6RzNRSJKRD9SJiIiM5y0LERFpEhUL\nERGpqi0OcOdhZhsAFwGzgPsBc/elZcbbEvghsCVQBP7T3R9st5zpuAXgduAyd/9CszKm866a08xe\nDXwPKABDwPHu7k3INmLHkmbWA5wH7AI8Dny4me9xDTm/DHyS2CHpY8AB7v7QWg21OGdmvP8CLgb+\nzd3/3sSIw/OvmtPMDJhL/N++1d0/0dyUud73LYFzgfXTcY5w9183OeNZwJ7AEnd/VYVxvgO8GxgE\n9nP3+SO1OZ62LA4Hfu/uOwDXAEdUGO884ER335HYbcijTco3LG9OgG8Cf2hKqrXlyTkI7O3uryT+\nUZ1iZg29tDTTseQ7gZ2Aj5rZy0pGOxB40t23I/7TntTITOXkzPl3YBd33xm4FPh2c1PmzomZzQC+\nANzQ3ITPz79qTjPbFvgq8Kb0b/JL7ZgT+Dpwkbu/FvgocHpzUwJwDjFjWWb2buCl6f/QZ4DvV2tw\nPBWL9xGrNenv95eOYGYvB7rd/RoAd1/u7s81LyKQIyeAme0CbEy8vqQVquZ093vd/b708SPEwvvi\nBufK07FkNvslxB6Lm61qTnf/Q+bv7wZiP2jNlrejzm8CJwArmhkuI0/OTwH/5+7PALj7403OCPly\nFoHhL1XrU6ZHikZz9+uAtTure8H7iF+scfcbgfXMbJOR2hxPxWJjd18C4O6LiR+0pbYHlprZpWZ2\ns5mdaGahqSlz5Ewz/S9wGNDsfMPyrM/nmdnrgUnDxaOBynUsWfoh+/w46UWdT5vZixqcq1SenFkH\nAk3dFZGqmtPMXgNs0exdJSXyrM/tgR3M7Dozu97MWtHPTp6cRwN7m9lDwJXA55uUrRaly/EwVb7M\ntNUxCzP7HZCtbgFIiJt1pcqd87sO8GZgZ+KKcGA/4iZZO+U8CPiluy+Ku2AbUzDqkHO4nc2I30L2\nrmvA+mlVwc3FzD5BPL7yllZnKZV+cekDsr1gtuv6XAfYFtiN2J/cH83sFcNbGm3ko8A57n6ymb0R\n+Alxl9W41lbFwt0r9vhnZkvMbBN3X2Jmm1L+WMRCYL67P5BO83PgDdS5WNQh55uAN5vZQcSDx5PM\nbMDdj2yznMMH4a8kHqS7qZ75KsjTseRC4gkMi8ysG1jX3Z9sQrasvB1gvo14PGi3dLdFs1XLWSB+\nkPWnhWNT4HIze2+TD3Lnfd9vcPcicL+Z3Q1sB9zcnIhAvpwHkh4vcPcbzGyKmW3Uot1mlTxM/B8a\nVvbvN6utikUVvyBuJZxI/BZ0eZlxbgLWN7MN3f0JYI90WDNVzZk9g8PM9iUeBK1rocihak4zmwT8\nHDjX3S9rUq6bgG3NbBbwCPAR4je1rCuImW8E5hAP0Ddb1Zzp7p3vA+9M/x5bYcSc6bfy53dBmtm1\nwCHufks75Uz9PB12rpltRCwU/2pqynw5HwDeRsz5cmByiwpFoPJW4i+Ag4GL0q2fp4d3S1cyno5Z\nnAi83czuIh7QPAHigWIzOwMg/cZxGHCNmd2aTndmu+VsE3lyGnG33n5mdouZ/d3Myp6GVy+VOpY0\ns6PNbM90tLOAjczsHuIZMYc3MtMYcp4ETAcuTtffz9s0Z1ZCC3ZD5cnp7r8BnjCz24CrgcPcfaSD\nuC3JSfwM+pSZzQfOZ81dfE1hZhcA1wPbm9mDZra/mX3GzD6dLsevgAVmdi/wA+Ku8RGpuw8REalq\nPG1ZiIhIi6hYiIhIVSoWIiJSlYqFiIhUpWIhIiJVqViIiEhVKhYiIlKVioWIiFQ1nrr7kA5nZguA\nA4e7oK9x2n8CB7n7H+ufrP7MbHvizam2Ab5GvGLYga2JHUF+Gji42vK023Kn96R4ZfpzZStusiSj\no2IhE065ouLur2hhpNH4CnCNu78GwMx+CFzt7oemr5+Wp5F6LfdYCnWJvYA/A78ndjPxsbFmk+bQ\nbihpqrSX2KrDhFnEvocqPR+X3P1kd/8rsZfTBa3OI/mpbyipiZltAZwK/Aexw7mfuvsX0t41Tyfe\nS2QhcKS7X5FOs4B4L++PE29gMwO4t2TYdOK9N04j3q9gADjF3U/LzPv5b7dm9lXindM2Bh4Evu7u\nPzez89I2nyPeN/wYd//fkmkrZs3M57vAPsTuqK8C9nX3lXnXR/ray9JlLLdONqu0rGZ2NfHeFyuB\n1cSeTt9CvJf3KuK9MX6bWZ6RMmSXu+I8yyz3LOLNmvYjdsa51jotXRe1MLMjgFPdfflY2pHm0ZaF\n5Jbef/hK4jfCrYh31rrQzNYhdnl8FfG2q18Azjez7TKTf4R4H+/105471xhG7O30CuAWYDNiT7hf\nNLNK9+S4F9jV3dcl3pnsJ+n9OfYhFo893X3d0g+1nFkhdn3+DuIxglcTPzRzrY/MfK4oN5/0vhEV\nl9Xd3wr8CfjvdBmGnx+cPr8nT4aSrCPOs8xyvyRd7n2rrdNamdlexKLUitvMyijpmIXU4vXED5qv\npN3BA1xvZm8Gprv7iemwa83sSmI//8ekw05190Ul7T0/zOJtWzdy9+PS1+5P99N/FPhdaRB3vzTz\n+GIzOzLNN7yFUKmb7TfmyDqcbUma7Qri1kGu9ZFjPlfVsqxVjJQh63U55znScufuutziPea3Az4L\nXEbcWnk/8G/AkcRbjf4BOK5SG9JeVCykFlsCD2Q+lIbNZM37+UK8AUz2m+PCMu1lh80CNjez4Tve\nBeKWb9mzeMxsH+DLxG/AEHdjbVQlf96sANkbwSwnfiCXqrQ+qs2npmWtYqQMWXnnmWe581gJ3A6s\ncvdTzez77r6CuE6adSMtqSMVC6nFQ8BWZtZV8uG0iDVv0Qhxl8hdmeflDo5lhz0E/Mvdd6gWwsy2\nAs4Adnf3v6TDbuGFb74jHYhbxJq3xSyXNa9K62N4PpXWSe5lHWOG0vHGMs+aDm66+z/M7DDgkvT5\nilHOV9qEioXU4q/EW0meYGbziAc7dyHe3nS5mX0F6CPeXW9PYF6NbQ+kbXyHeCD3ZcBUd/9bybjT\ngSLweLrPfl8ge4roEuL1CeVO87wRGCyT9egasmYzr7U+3P16Rl4n99WwrGPJUDreWOa51jo1s3OA\nxN0PqDDNO4DP1LAs0sZ0gFtyS7+57kXcF/0g8duqufuqdPh/Ao8TD17unTkQW22rYrjtPYn7yBcA\njxLPwlm3dBp3vwPoBW4AFgM7AddlxvsW8A0ze9LMDimZtlLWuytlq6TS+qgyn3tqWdZqz0fKUGa8\nWueZVW6dbsma673UFHfX6bEThE6dFZGamdkkYD7wqszZbTKBqViIiEhV2g0lIiJVqViIiEhVKhYi\nIlKVioWIiFSlYiEiIlWpWIiISFUqFiIiUpWKhYiIVKViISIiVf1/UBuG+Hu2RVwAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fd3fa2233d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "_ = plt.hist(decoding_perf[:-3], 25, histtype='step', lw=2)\n",
    "_ = plt.title('decoding test performance')\n",
    "_ = plt.xlabel('correlation coefficient, $r^2$')\n",
    "_ = plt.ylabel('number of time points')\n",
    "_ = plt.axvline(decoding_perf[:-3].mean(), color='k', linestyle='--', lw=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Human rating\n",
    "We'll present human raters with the original passage and output from several submissions including our best and have them judge which of the submissions best matches to the meaning of the original passage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Resources\n",
    "Data and other resources used in this notebook are provided at the following file locations or urls:\n",
    "- [responses](https://berkeley.box.com/shared/static/znhnewml3snwwio48wkup06um9y6vl9y.h5)\n",
    "- stimuli (words) (`data/grids`)\n",
    " - align with responses using files in (`data/trfiles`)\n",
    "- [stimuli (features)](https://berkeley.box.com/shared/static/r0b65ge04ypym4xg4epe3yex9sfb2f53.h5)\n",
    "- 985 common english words vocabulary (`notebooks/basis_vocabulary.txt`)\n",
    "- [word2vec model](https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing)\n",
    "- [example decoding model weights](https://berkeley.box.com/shared/static/7a9bey1qudhv2megh138sf5bzyni5rtm.h5)\n",
    "- [ridge regression code](https://github.com/alexhuth/ridge.git)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}