Sama_Databricks_Connector_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "7f5530b2-b65d-45e5-898b-9e0019a5a0be",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Introduction to the Sama SDK and Databricks Connector Tutorial\n",
    "In this tutorial, we'll show you how to set up and use the Sama SDK and Databricks Connector. These tools make it easy to create, monitor, and access annotated tasks within Databricks. We'll guide you through the installation, configuration, and provide practical examples for your understanding. Let's begin simplifying your workflow with these tools!\n",
    "\n",
    "<div class=\"container-image\">\n",
    "<img src=\"https://sama-documentation-assets.s3.amazonaws.com/databricks/images/partnerships/databricksXsama.png\"/>\n",
    "</div>\n",
    "\n",
    "<style>\n",
    ".container-image {\n",
    "    object-fit: scale-down;\n",
    "}\n",
    "</style>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "364587d0-8134-4958-9606-6ffaedab3650",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Step 1: Requirements\n",
    "**1.1 Installation requirements**\n",
    "Before you begin, make sure you install Sama\n",
    "By runing the following cell you will download and install Sama, and their dependencies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "21e85b50-d2a3-4e35-ab63-72658683e6c5",
     "showTitle": false,
     "title": ""
    }
   },
   "outputs": [],
   "source": [
    "%pip install sama"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Importing Sama Client**\n",
    "Now you can import the Sama client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sama.databricks import Client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1.2 Project requirements**\n",
    "\n",
    "You need to specify:\n",
    "*   Project ID\n",
    "*   [API Key](https://accounts.sama.com/account/profile)\n",
    "\n",
    "📘 **Note**: Your Sama Project Manager will provide you with the correct Project ID(s). The following instructions also assume that your Project Manager has already configured all the necessary Sama Project inputs and outputs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Provide API Key and Project ID\n",
    "To begin interacting with the Databricks Connector, you'll need to provide your API key and Project ID. These credentials are essential for establishing a connection.\n",
    "\n",
    "Replace \"YOUR_API_KEY\" and \"YOUR_PROJECT_ID\" with the credentials you've obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "API_KEY = \"YOUR_API_KEY\" #@param {type:\"string\"}\n",
    "PROJECT_ID = \"YOUR_PROJECT_ID\" #@param {type:\"string\"}\n",
    "\n",
    "if not(API_KEY):\n",
    "  raise ValueError(\"API_KEY not set\")\n",
    "if not(PROJECT_ID):\n",
    "  raise ValueError(\"PROJECT_ID not set\")\n",
    "\n",
    "client = Client(API_KEY)\n",
    "client.get_project_information(PROJECT_ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✅ This cell returns the project id, name, state, type, asset_country_code, project_group, name, description and client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Usage with Spark Dataframes\n",
    "\n",
    "Now that you are set up and properly configured, you can start:\n",
    "\n",
    "1. Creating tasks, using data from a Dataframe, in the Sama Platform to be picked up by the annotators and quality teams.\n",
    "2. Monitoring batch, task, and project status.\n",
    "3. Get delivered tasks, into a Dataframe, which have been annotated and reviewed by our quality team.\n",
    "4. Rejecting and deleting tasks.\n",
    "\n",
    "📘 **Note**: As an alternative, [Sama API](https://docs.sama.com/reference/documentation) is also available for all of these and more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3.1 Provide a Sample Dataset**\n",
    "\n",
    "In this step, we'll provide you with a sample JSON dataset that you can use for this demo and testing purposes.\n",
    "You can see json schema of different output types in [sama.helpjuice.com](https://sama.helpjuice.com/en_US/recipes/recipe-json) creation format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "sample_data = [{\n",
    "    'url': 'https://static.wikia.nocookie.net/speedracer/images/9/9a/Speed_Racer_behind_the_wheel.png',\n",
    "    'name': 'speed_racer_img_1',\n",
    "    'client_batch_id': 'speed_racer',\n",
    "    'output_weather_condition': {'rain': '0', 'snow': '0', 'clear': '1'},\n",
    "    'output_vehicle_image_annotation': {\n",
    "        'layers': {\n",
    "            'vector_tagging': [\n",
    "                {\n",
    "                    'shapes': [{\n",
    "                        'tags': {\n",
    "                            'transcription': '',\n",
    "                            'type_of_vehicle': '1'\n",
    "                        },\n",
    "                        'type': 'rectangle',\n",
    "                        'index': 1,\n",
    "                        'points': [[51, 20], [150, 20], [51, 72], [150, 72]]\n",
    "                    }],\n",
    "                    'group_type': None\n",
    "                },\n",
    "                {\n",
    "                    'shapes': [{\n",
    "                        'tags': {\n",
    "                            'transcription': '',\n",
    "                            'type_of_vehicle': '1'\n",
    "                        },\n",
    "                        'type': 'rectangle',\n",
    "                        'index': 2,\n",
    "                        'points': [[160, 71], [199, 71], [160, 83], [199, 83]]\n",
    "                    }],\n",
    "                    'group_type': None\n",
    "                }\n",
    "            ]\n",
    "        }\n",
    "    }\n",
    "},\n",
    "{\n",
    "    'url': 'https://media.comicbook.com/uploads1/2015/05/speed-racer-137552.jpg',\n",
    "    'name': 'speed_racer_img_2'\n",
    "},\n",
    "{\n",
    "    'url': 'https://upload.wikimedia.org/wikipedia/en/8/81/Speed_Racer_Family.jpg',\n",
    "    'name': 'speed_racer_img_3'\n",
    "},\n",
    "{\n",
    "    'url': 'https://upload.wikimedia.org/wikipedia/en/2/25/Speed_Racer_promotional_image.jpg',\n",
    "    'name': 'speed_racer_img_4'\n",
    "}\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3.2 Create a Dataframe using the sample data**\n",
    "\n",
    "📘 **Note**: `spark.createDataFrame` can't convert output_vehicle_image_annotation to a MapType properly without a Dataframe schema. Convert it to a JSON string as an alternative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#CREATE a DataFrame using this modified data\n",
    "df = spark.createDataFrame(sample_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✅ After running the cell above, you can anticipate receiving an output similar to the following:  `{'batch_id': 123456}`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = client.get_task_status_to_table(spark, PROJECT_ID, task_id=\"task_id\")\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✅ This cell returns a dataframe of task data and its current status. See docs for additional filters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = client.get_delivered_tasks_to_table(spark, PROJECT_ID, client_batch_id=\"651b4adaf97fd2713cabbba4\", from_timestamp=\"2023-09-13T00:00:00.000Z\")\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✅ This cell returns a dataframe of delivered(fully annotated) task data and answers. See docs for additional filters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = client.get_delivered_tasks_since_last_call_to_table(spark, PROJECT_ID, consumer=\"consumer value\")\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "✅ This cell returns a dataframe of delivered(fully annotated) task data and answers since the last call to this endpoint with a specific consumer key. See docs for additional filters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "303cf4d9-d8df-489d-935f-18f6f02aa2d4",
     "showTitle": false,
     "title": ""
    }
   },
   "source": [
    "# Other SDK functions\n",
    "1. Get task and delivery schemas\n",
    "1. Get status and cancel batch creation jobs\n",
    "1. Update task priorities\n",
    "1. Reject and delete tasks\n",
    "1. Get project stats and information\n",
    "\n",
    "Please see full documentation at [docs.sama.com](https://docs.sama.com)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "0bc4ac81-d388-4b97-972e-25392eb86f92",
     "showTitle": true,
     "title": "Get creation task schema"
    }
   },
   "outputs": [],
   "source": [
    "#GET task creation schema\n",
    "client.get_creation_task_schema(PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "478fba71-c2e6-48af-b66e-78d14310a6c9",
     "showTitle": true,
     "title": "Get delivery task schema"
    }
   },
   "outputs": [],
   "source": [
    "#GET delivery task schema \n",
    "client.get_delivery_task_schema(PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "52cfd5aa-c187-443b-8171-ed0011c3e404",
     "showTitle": true,
     "title": "Cancel batch creation job"
    }
   },
   "outputs": [],
   "source": [
    "#CANCEL a batch creation job\n",
    "client.cancel_batch_creation_job(PROJECT_ID, \"testbatchid12345\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "a19458fe-1ebb-4f70-bec6-fd9fb2a2bd1e",
     "showTitle": true,
     "title": "Update task priorities"
    }
   },
   "outputs": [],
   "source": [
    "#UPDATE task priorities\n",
    "client.update_task_priorities(PROJECT_ID, [\"testtaskid1\", \"testtaskid2\"], -100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "655ad191-aeab-401f-914e-7b047b08eb42",
     "showTitle": true,
     "title": "Reject a task"
    }
   },
   "outputs": [],
   "source": [
    "#REJECT a task\n",
    "client.reject_task(PROJECT_ID, task_id=\"testtaskid3\", reasons=[\"Not accurate\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {},
     "inputWidgets": {},
     "nuid": "47b2b8c1-f3e9-4913-8ad5-e7ec4e793764",
     "showTitle": true,
     "title": "Delete tasks"
    }
   },
   "outputs": [],
   "source": [
    "#DELETE a task\n",
    "client.delete_tasks(PROJECT_ID, [\"testtaskid4\", \"testtaskid5\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "b428ee3b-ef55-400a-9648-9a4e07f1387e",
     "showTitle": true,
     "title": "Get status of batch creation Job"
    }
   },
   "outputs": [],
   "source": [
    "#GET status of batch creation job\n",
    "statuses = client.get_status_batch_creation_job(PROJECT_ID, \"testbatchid\")\n",
    "\n",
    "for item in statuses:\n",
    "    print(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "42234516-39b8-495c-abf0-f2da392ea433",
     "showTitle": true,
     "title": "Get project information"
    }
   },
   "outputs": [],
   "source": [
    "#GET project information\n",
    "client.get_project_information(PROJECT_ID)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "application/vnd.databricks.v1+cell": {
     "cellMetadata": {
      "byteLimit": 2048000,
      "rowLimit": 10000
     },
     "inputWidgets": {},
     "nuid": "33c9c335-91a3-4d21-a542-eaf69e3b1f5b",
     "showTitle": true,
     "title": "Get project stats"
    }
   },
   "outputs": [],
   "source": [
    "#GET project stats\n",
    "client.get_project_stats(PROJECT_ID)"
   ]
  }
 ],
 "metadata": {
  "application/vnd.databricks.v1+notebook": {
   "dashboards": [],
   "language": "python",
   "notebookMetadata": {
    "pythonIndentUnit": 4
   },
   "notebookName": "Sama Databricks Connector Tutorial",
   "widgets": {}
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}