terrain-automation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overview\n",
    "\n",
    "This notebook provides instructions for using the Terrain API to automate some tasks that require a little more work in the CyVerse Discovery Environment (DE).\n",
    "\n",
    "## Target Audience\n",
    "\n",
    "- Users interested in automating their tasks.\n",
    "- Users who want to integrate CyVerse resources into their applications.\n",
    "- Users looking for an introduction to using APIs in order to make use of other APIs related to CyVerse.\n",
    "\n",
    "Because of time constraints, this tutorial assumes that you have some familiarity with Python and [JavaScript Object Notation](https://en.wikipedia.org/wiki/JSON) (JSON).\n",
    "\n",
    "## Example Web App: Integrated Genome Browser\n",
    "\n",
    "The [Integrated Genome Browser](http://lorainelab.org/), built by the Lorraine lab at UNC Charlotte, has an associated web application called [BioViz Connect](https://connect.bioviz.org) that uses The Terrain API to browse the data store, associate metadata with files, and submit jobs using CyVerse resources. See this introductory video for more information: [BioViz Connect Introductory Video](https://www.youtube.com/watch?v=JpQMtnDoOjA).\n",
    "\n",
    "## What is a Web Application Programming Interface (Web API), anyway?\n",
    "\n",
    "At its most basic level, an [application programming interface](https://en.wikipedia.org/wiki/API) (API) is a way to interact with a piece of software programmatically. APIs are common in operating systems and software libraries. For operating systems and software libraries, an API usually consists of a set of functions that you can call. A [web API](https://en.wikipedia.org/wiki/API) is similar in that it provides a way to interact with a piece of software, but in this case the software is running on a remote server somewhere on the internet. Web APIs allow clients to interact with them using internet protocols such as [HyperText Transfer Protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol) (HTTP) or [Simple Object Access Protocol](https://en.wikipedia.org/wiki/SOAP) (SOAP).\n",
    "\n",
    "The Terrain API uses HTTP to transfer data between itself and its clients (that is, any piece of software that happens to be interacting with Terrain). Terrain uses [JavaScript Object Notation](https://en.wikipedia.org/wiki/JSON) (JSON) to both send and receive complex data. APIs like Terrain are typically called [Representational State Transfer](https://en.wikipedia.org/wiki/Representational_state_transfer) (REST) APIs, although we don't claim that Terrain satisfies all of the constraints of REST architecutres.\n",
    " \n",
    "## Why Use Terrain?\n",
    "\n",
    "The DE itself provides a lot of tools for managing and analyzing data, so one question that may come up is: why bother to use Terrain directly when I can use the DE itself? This is absolutely a valid point, but there are some situations where a graphical user interface can become a bit of a hinderance. Suppose, for example, that you need to launch dozens of analyses that all use the same app with slightly different parameter values. The DE currently provides no way to do this unless the parameters being varied happen to refer to input files. Launching so many similar jobs using a GUI would be tedious and error prone. If you're making calls to Terrain directly, you can write a short script to quickly launch all of the analyses with all of the required parameter variations.\n",
    "\n",
    "Of course, there is a trade-off. Making calls directly to the API does take some effort; it means that you have to perform all of these tasks:\n",
    "\n",
    "1. Authenticate to terrain.\n",
    "1. Identify the app that you want to run.\n",
    "1. Obtain information about the app parameters.\n",
    "1. Launch the app.\n",
    "1. Send a notification when the job is done.\n",
    "\n",
    "The DE manages all of these tasks for you. If the DE suits your needs, by all means, use it. If the tasks that you have to perform become repetitive, however, investing a little time in writing a script to automate job submission might actually save you some time in the long run.\n",
    "\n",
    "## Documentation\n",
    "\n",
    "Terrain has two primary sources of documentation:\n",
    "\n",
    "- Latest and Greatest: https://de.cyverse.org/terrain/docs\n",
    "- Older Documentation: https://cyverse-de.github.io/api\n",
    "\n",
    "In most cases, you'll want to use the latest documentation because some of the older documentation is out of date. The only time that the older documentation is preferable is when the newer documentation hasn't been written for an endpoint that you want to use. If you use the older documentation, the best place to look is the [endpoint index](https://cyverse-de.github.io/api/endpoint-index.html). This page includes a list of links to all of the older DE documentation.\n",
    "\n",
    "# Prerequisites\n",
    "\n",
    "Before actually calling terrain, we'll have to make sure that we have all of the libraries that we need, and that we have credentials that we can use to let Terrain know who we are.\n",
    "\n",
    "## Libraries\n",
    "\n",
    "We'll be making extensive use of the [Requests library](https://requests.readthedocs.io/en/master/), which makes calling APIs quite simple. We'll also need to be able to prompt for a password and pretty-print some data structures. There's also one case where we'll have to serialize some JSON."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import json\n",
    "import pprint\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Authenticating\n",
    "\n",
    "Terrain uses OAuth2 for most endpoints. This works well for the DE because it's already integrated with an identity provider that is capable of providing tokens. For direct API calls, however, we needed something a little more convenient. For this purpose, we created a set of endpoints dedicated to obtaining OAuth2 tokens. These endpoints use HTTP basic authentication (that is, a username and password) and, assuming the credentials are valid, return a token that can be used to call other Terrain endpoints. In Python, supporting HTTP basic authentication means that we have to prompt for a username and password. This is where the `getpass` library comes in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Username: \", end='', flush=True)\n",
    "username = input()\n",
    "print(\"Password: \", end='', flush=True)\n",
    "password = getpass.getpass()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the username and password, we can obtain the authentication token by calling the `/terrain/token` endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(\"https://de.cyverse.org/terrain/token/keycloak\", auth=(username, password))\n",
    "r.raise_for_status()\n",
    "token = r.json()['access_token']\n",
    "auth_headers = {\"Authorization\": \"Bearer \" + token}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 1: VICE Analysis\n",
    "\n",
    "Launching VICE analyses in the DE is actually quite simple, but the simplicity of this task makes it an ideal first example.\n",
    "\n",
    "## Finding the App\n",
    "\n",
    "The first step is to find a VICE app to use. For this task, we'll use the app, `rstudio-chipqc 1.22`. The first step is to get the app ID so that we can get information about how to launch the analysis. The app search endpoint provides a convenient way to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_params = {\"search\": \"rstudio-chipqc 1.22\"}\n",
    "r = requests.get(\"https://de.cyverse.org/terrain/apps\", headers=auth_headers, params=query_params)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining the App Details\n",
    "\n",
    "Armed with some information about the app, we can now obtain the full app description, which contains all of the information necessary to launch an analysis using the app. The first step in doing that is to obtain the information that we need from the app search above. For this step, we need the system ID and the app ID. The system ID refers to the system that is responsible for managing the app. Currently there are two valid system IDs: `de` and `agave`. Apps that use the system ID, `de`, are defined in and managed by the DE itself. Apps that use the system ID, `agave`, are defined in and managed by the [Tapis API](https://tapis-project.org), formerly known as Agave. Of course, the app ID refers to the app itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app_listing = r.json()[\"apps\"][0]\n",
    "system_id = app_listing[\"system_id\"]\n",
    "app_id = app_listing[\"id\"]\n",
    "print(\"System ID: \", system_id)\n",
    "print(\"App ID: \", app_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can obtain the full app description."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://de.cyverse.org/terrain/apps/{0}/{1}\".format(system_id, app_id)\n",
    "r = requests.get(url, headers=auth_headers)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output from this endpoint deserves a little explanation. At the top level, we have the basic app information such as the name, ID, and description of the app. The top level also contains a list labeled `groups`. These groups provide a way to place related parameters on the same panel in the app launch window in the DE. Each group contains a list of parameters, and the parameters themselves provide the information we need to submit the job.\n",
    "\n",
    "The primary piece of information that we're going to need from this file is the parameter ID for the input file name. We may as well grab it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameter_id = r.json()[\"groups\"][0][\"parameters\"][0][\"id\"]\n",
    "print(\"Parameter ID: \", parameter_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Analysis Submission Request Body\n",
    "\n",
    "The analysis submission endpoint is the first endpoint we've encountered so far that has a request body, and this request body needs to be formatted correctly for the analysis submission to succeed. The request body looks something like this:\n",
    "\n",
    "``` json\n",
    "{\n",
    "    \"requirements\": [\n",
    "        {\n",
    "            \"min_cpu_cores\": 1,\n",
    "            \"min_memory_limit\": 2147483648,\n",
    "            \"min_disk_space\": 549755813888,\n",
    "            \"step_number\": 0\n",
    "        }\n",
    "    ],\n",
    "    \"config\": {},\n",
    "    \"name\": \"string\",\n",
    "    \"app_id\": \"string\",\n",
    "    \"system_id\": \"string\",\n",
    "    \"debug\": false,\n",
    "    \"output_dir\": \"string\",\n",
    "    \"notify\": true\n",
    "}\n",
    "```\n",
    "\n",
    "Not all of the available fields are listed in the example JSON above, but I did include all of the required fields and one optional field that I wanted to highlight.\n",
    "\n",
    "| Parameter Name | Description                                                                              |\n",
    "| -------------- | ---------------------------------------------------------------------------------------- |\n",
    "| config         | A map from parameter ID to parameter value.                                              |\n",
    "| name           | The name of the analysis.                                                                |\n",
    "| app_id         | The app ID from the submission information above.                                        |\n",
    "| system_id      | The system ID from the submission information above.                                     |\n",
    "| debug          | This parameter can be used to enable debugging, which isn't necessary.                   |\n",
    "| output_dir     | The path to the folder in the data store where the output files should be placed.        |\n",
    "| notify         | This parameter can be used to enable or disable job status update notifications.         |\n",
    "| requirements   | This parameter is used to specify execution system requirements.                         |\n",
    "\n",
    "So now we have to plug in the values. The example data that we're going to use is in `/iplant/home/shared/workshop_material/terrain_intro/example-data`, so that is the path that we need to use for the input parameter value. In addition to that, it's fairly common for a VICE analysis to require more memory than the default for the app. The reason for this is that VICE apps are interactive; the amount of memory required depends largely upon what is being done with the app. When you're requesting resource requirements, you can optionally specify different requirements for different steps in the analysis. VICE apps always contain exactly one step, so we only have to submit one set of resource requests. For this simple example, we don't need too much memory, so we're going to ask for 4 [Gibibytes](https://en.wikipedia.org/wiki/Gibibyte) (that is, 4 * 2^30 bytes) of memory.\n",
    "\n",
    "Keep in mind that the request body below is written in Python rather than JSON, so it will look slightly different from the JSON request body listed above. The `requests` library will convert this Python object to a JSON object for us before sending the request to terrain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request_body = {\n",
    "    \"config\": {\n",
    "        parameter_id: \"/iplant/home/shared/workshop_material/terrain_intro/example-data\"\n",
    "    },\n",
    "    \"name\": \"terrain-automation-vice\",\n",
    "    \"app_id\": app_id,\n",
    "    \"system_id\": system_id,\n",
    "    \"debug\": False,\n",
    "    \"output_dir\": \"/iplant/home/\" + username + \"/analyses\",\n",
    "    \"notify\": True,\n",
    "    \"requirements\": [\n",
    "        {\n",
    "            \"min_memory_limit\": 4 * 2 ** 30,\n",
    "            \"step_number\": 0\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "pprint.pprint(request_body)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can finally submit the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.post(\"https://de.cyverse.org/terrain/analyses\", headers=auth_headers, json=request_body)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, for a VICE analysis, we need to obtain the URL used to access the running analysis. We can use the analysis listing endpoint to get this information. First, we need to get the analysis ID from the response body of the previous step. With that information we can build and serialize a filter to place in a query parameter and call the endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_params = {\"filter\": json.dumps([{\"field\":\"id\",\"value\":r.json()[\"id\"]}])}\n",
    "r = requests.get(\"https://de.cyverse.org/terrain/analyses\", headers=auth_headers, params=query_params)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 2: Processing Multiple Files\n",
    "\n",
    "This example uses the the DE's batch processing feature to submit multiple jobs at once. This is admittedly a toy example; I want to extract the shebang line, if present, from a set of files in a single directory.\n",
    "\n",
    "## Listing Files\n",
    "\n",
    "The first thing we need to do is to obtain the list of files that we want to process. We can use the [paged directory endpoint](https://de.cyverse.org/terrain/docs/index.html#!/filesystem/get_terrain_secured_filesystem_paged_directory) in Terrain to get that information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_params = {\n",
    "    \"path\": \"/iplant/home/shared/iplantcollaborative/example_data/heuristomancer\",\n",
    "    \"limit\": 500\n",
    "}\n",
    "r = requests.get(\"https://de.cyverse.org/terrain/secured/filesystem/paged-directory\", headers=auth_headers, params=query_params)\n",
    "r.raise_for_status()\n",
    "paths = [f[\"path\"] for f in r.json()[\"files\"]]\n",
    "pprint.pprint(paths)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating a Path List File\n",
    "\n",
    "The next step is to create a high throughput path list file and upload it into the data store. Note: if you have a file in your home directory in the data store called `terrain-automation-paths.htpl`, either remove or rename the file before executing the next code block. Alternately, you can choose to edit the code block to change the file name before executing it. Also note that I'm only using the first eight files in that directory. Otherwise, this code would submit 127 jobs, which is a lot of jobs to submit for a simple demonstration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_list_file = \"/iplant/home/\" + username + \"/terrain-automation-paths.htpl\"\n",
    "request_body = {\n",
    "    \"content\": \"# application/vnd.de.path-list+csv; version=1\\n\" + \"\\n\".join(paths[:8]),\n",
    "    \"dest\": path_list_file\n",
    "}\n",
    "r = requests.post(\"https://de.cyverse.org/terrain/secured/fileio/saveas\", headers=auth_headers, json=request_body)\n",
    "r.raise_for_status()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Obtaining the App Details\n",
    "\n",
    "Now that we have the path list file, the next step is to get the analysis submission information just like we did in the first example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_params = {\"search\": \"Grep 3.1-2\"}\n",
    "r = requests.get(\"https://de.cyverse.org/terrain/apps\", headers=auth_headers, params=query_params)\n",
    "r.raise_for_status()\n",
    "app_listing = r.json()[\"apps\"][0]\n",
    "system_id = app_listing[\"system_id\"]\n",
    "app_id = app_listing[\"id\"]\n",
    "\n",
    "url = \"https://de.cyverse.org/terrain/apps/{0}/{1}\".format(system_id, app_id)\n",
    "r = requests.get(url, headers=auth_headers)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building the Analysis Submission Request Body\n",
    "\n",
    "This time, the submission request body is slightly different because we have three parameters that we need to specify: the parameters for the regular expression, input file, and output file name. First, we'll need to obtain the parameter IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "app = r.json()\n",
    "parameters = app[\"groups\"][0][\"parameters\"]\n",
    "[ regex_id ] = [ p[\"id\"] for p in parameters if p[\"label\"] == \"Text to match\"]\n",
    "[ input_id ] = [ p[\"id\"] for p in parameters if p[\"label\"] == \"Input File\"]\n",
    "[ output_id ] = [ p[\"id\"] for p in parameters if p[\"label\"] == \"Output File Name\"]\n",
    "print(\"Regular Expression Parameter ID:\", regex_id)\n",
    "print(\"Input File Parameter ID:\", input_id)\n",
    "print(\"Output File Name Parameter ID:\", output_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the parameter IDs, we can build the request body."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "request_body = {\n",
    "    \"config\": {\n",
    "        regex_id: \"#!/\",\n",
    "        input_id: path_list_file,\n",
    "        output_id: \"shebang.txt\"\n",
    "    },\n",
    "    \"name\": \"terrain-automation-batch\",\n",
    "    \"app_id\": app_id,\n",
    "    \"system_id\": system_id,\n",
    "    \"debug\": False,\n",
    "    \"output_dir\": \"/iplant/home/\" + username + \"/analyses\",\n",
    "    \"notify\": True\n",
    "}\n",
    "pprint.pprint(request_body)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, finally, we can submit the job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.post(\"https://de.cyverse.org/terrain/analyses\", headers=auth_headers, json=request_body)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())\n",
    "analysis_id = r.json()[\"id\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Listing Analyses in a Batch\n",
    "\n",
    "This time, we want to list all of the analyses that were submitted as part of this batch, so the analysis listing request is going to be just a little bit different. Instead of filtering for the `id` field. We're going to filter for the `parent_id` field."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_params = {\"filter\": json.dumps([{\"field\":\"parent_id\",\"value\":analysis_id}])}\n",
    "r = requests.get(\"https://de.cyverse.org/terrain/analyses\", headers=auth_headers, params=query_params)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example 3: Parameter Sweep\n",
    "\n",
    "Another case where it might be more convenient to use Terrain directly is when you want to submit multiple jobs with slightly different values for a single non-input parameter. This pattern is typically called a parameter sweep. For this example, we're going to use an app called `trim-galore-0.4.1`. Suppose we want to see how the output of this app changes for different error rates.The default error rate is `0.01`, but we're hoping to see what happens for error rates of `0.02`, `0.03` and `0.04` as well. To accomplish this, we're going to submit four different jobs, one for each error rate that we want to try.\n",
    "\n",
    "## Obtaining App Information\n",
    "\n",
    "As usual, the first step is to obtain some information about the app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query_params = {\"search\": \"trim-galore-0.4.1\"}\n",
    "r = requests.get(\"https://de.cyverse.org/terrain/apps\", headers=auth_headers, params=query_params)\n",
    "r.raise_for_status\n",
    "app_listing = r.json()[\"apps\"][0]\n",
    "system_id = app_listing[\"system_id\"]\n",
    "app_id = app_listing[\"id\"]\n",
    "\n",
    "url = \"https://de.cyverse.org/terrain/apps/{0}/{1}\".format(system_id, app_id)\n",
    "r = requests.get(url, headers=auth_headers)\n",
    "r.raise_for_status()\n",
    "pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, I cheated by copying and pasting the parameter IDs simply because the parameter labels are long and easy to mistype. The parameters that we want to specify are the two input parameters (one for each file in the paired read) and the error rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_1_id = \"56bc4ac4-ad24-11e7-8a4a-008cfa5ae621_56c81f34-ad24-11e7-8a4a-008cfa5ae621\"\n",
    "read_2_id = \"56bc4ac4-ad24-11e7-8a4a-008cfa5ae621_56c93ebe-ad24-11e7-8a4a-008cfa5ae621\"\n",
    "error_rate_id = \"56bc4ac4-ad24-11e7-8a4a-008cfa5ae621_56d60db0-ad24-11e7-8a4a-008cfa5ae621\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting the Jobs\n",
    "\n",
    "Since we're going to be submitting multiple very similar jobs, we're going to want to have an easy way to submit each job without having to duplicate a bunch of code. A function is the perfect way to provide that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def submit_parameter_sweep_job(read1, read2, error_rate):\n",
    "    request_body = {\n",
    "        \"config\": {\n",
    "            read_1_id: [read1],\n",
    "            read_2_id: [read2],\n",
    "            error_rate_id: error_rate\n",
    "        },\n",
    "        \"name\": \"terrain-automation-sweep-\" + error_rate,\n",
    "        \"app_id\": app_id,\n",
    "        \"system_id\": system_id,\n",
    "        \"debug\": False,\n",
    "        \"output_dir\": \"/iplant/home/\" + username + \"/analyses\",\n",
    "        \"notify\": True\n",
    "    }\n",
    "    r = requests.post(\"https://de.cyverse.org/terrain/analyses\", headers=auth_headers, json=request_body)\n",
    "    r.raise_for_status()\n",
    "    pprint.pprint(r.json())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look closely at the parameter values for the paired read parameters, you'll notice that we're passing in an array rather than a simple string. The reason for this is that the type of both parameters is `MultiFileSelector`, which requires a list of file paths rather than a single file path. The format of the values for various parameter types is not currently well documented. Please feel free to contact CyVerse with questions about various parameter types.\n",
    "\n",
    "The next step is to call this function for each of the values that we want to try. For these jobs, we're going to use the files in the example data directory for this app."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "read1 = \"/iplant/home/shared/iplantcollaborative/example_data/trim-galore/ATreads.fq\"\n",
    "read2 = \"/iplant/home/shared/iplantcollaborative/example_data/trim-galore/ATreads2.fq\"\n",
    "for error_rate in [\"0.01\", \"0.02\", \"0.03\", \"0.04\"]:\n",
    "    submit_parameter_sweep_job(read1, read2, error_rate)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}