Anthropic_Lesson_3_ImagesAndStream.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "14143ee5-4013-4cd2-92a6-26c40c2fbb63",
   "metadata": {},
   "source": [
    "# L3: Multimodal Requests & Streaming"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdcf803d-9f5b-40f2-8b34-a45d1d09579e",
   "metadata": {},
   "source": [
    "<p style=\"background-color:#fff6e4; padding:15px; border-width:3px; border-color:#f5ecda; border-style:solid; border-radius:6px\"> ⏳ <b>Note <code>(Kernel Starting)</code>:</b> This notebook takes about 30 seconds to be ready to use. You may start and watch the video while you wait.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e62a8776-8327-4a47-a2b9-a6e736199a61",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "from anthropic import Anthropic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e95d44f-3014-4909-b3b7-79e106e9e028",
   "metadata": {
    "height": 64
   },
   "outputs": [],
   "source": [
    "# Load environment variables\n",
    "from helper import load_env\n",
    "load_env()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30cfcc8d-c946-4216-9dbd-8656746ea1c0",
   "metadata": {},
   "source": [
    "<div style=\"background-color:#fff6ff; padding:13px; border-width:3px; border-color:#efe6ef; border-style:solid; border-radius:6px\">\n",
    "<p> 💻 &nbsp; <b>Access <code>requirements.txt</code> and <code>helper.py</code> files:</b> 1) click on the <em>\"File\"</em> option on the top menu of the notebook and then 2) click on <em>\"Open\"</em>.\n",
    "\n",
    "<p> ⬇ &nbsp; <b>Download Notebooks:</b> 1) click on the <em>\"File\"</em> option on the top menu of the notebook and then 2) click on <em>\"Download as\"</em> and select <em>\"Notebook (.ipynb)\"</em>.</p>\n",
    "\n",
    "<p> 📒 &nbsp; For more help, please see the <em>\"Appendix – Tips, Help, and Download\"</em> Lesson.</p>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d671c069-8a47-4d40-8f81-eb5439f391f2",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "client = Anthropic()\n",
    "MODEL_NAME=\"claude-3-5-sonnet-20241022\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1baca753-df9c-4294-bce0-b48886132d42",
   "metadata": {},
   "source": [
    "## Content Blocks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3512fa68-200f-402d-9faf-251257c4591b",
   "metadata": {
    "height": 234
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"tell me a joke\"\n",
    "    }\n",
    "]\n",
    "\n",
    "response = client.messages.create(\n",
    "    messages=messages,\n",
    "    model=MODEL_NAME,\n",
    "    max_tokens=200\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49959f1a-d636-4de5-b1d8-3db3c98bcb1c",
   "metadata": {},
   "source": [
    "<p style=\"background-color:#f7fff8; padding:15px; border-width:3px; border-color:#e0f0e0; border-style:solid; border-radius:6px\"> 🚨\n",
    "&nbsp; <b>Different Run Results:</b> The output generated by AI chat models can vary with each execution due to their probabilistic nature. Don't be surprised if your results differ from those shown in the video.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9d02dc0-32c7-4e21-bcf9-aa1cbbf9859e",
   "metadata": {
    "height": 268
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"text\", \"text\": \"tell me a joke\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "response = client.messages.create(\n",
    "    messages=messages,\n",
    "    model=MODEL_NAME,\n",
    "    max_tokens=200\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5acbad6e-5681-44b7-bae6-94d8e8ddeadb",
   "metadata": {
    "height": 285
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\"type\": \"text\", \"text\": \"who\"},\n",
    "            {\"type\": \"text\", \"text\": \"made\"},\n",
    "            {\"type\": \"text\", \"text\": \"you?\"},\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "response = client.messages.create(\n",
    "    messages=messages,\n",
    "    model=MODEL_NAME,\n",
    "    max_tokens=200\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd35038b-8bc7-4dc2-838b-43bf0bb60c2e",
   "metadata": {},
   "source": [
    "## Image Prompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e0c8852c-f64c-42a3-a375-7ebbf9f56ff5",
   "metadata": {
    "height": 47
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image\n",
    "Image(filename='images/food.png') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a64ef2ac-c000-4940-ab84-c0b324472bad",
   "metadata": {},
   "source": [
    "## Image Messages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f104c0a0-4171-4eca-8d4d-f38841b2df5c",
   "metadata": {
    "height": 166
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "# opens the image file in \"read binary\" mode\n",
    "with open(\"images/food.png\", \"rb\") as image_file:\n",
    "    #reads the contents of the image as a bytes object\n",
    "    binary_data = image_file.read() \n",
    "    #encodes the binary data using Base64 encoding\n",
    "    base_64_encoded_data = base64.b64encode(binary_data) \n",
    "    #decodes base_64_encoded_data from bytes to a string\n",
    "    base64_string = base_64_encoded_data.decode('utf-8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f4ceae3-fc55-42d8-b20a-1e8f2c975f9d",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "base64_string[:100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0618c76-e91c-414b-9330-7f2b484e320a",
   "metadata": {
    "height": 319
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [{\n",
    "            \"type\": \"image\",\n",
    "            \"source\": {\n",
    "                \"type\": \"base64\",\n",
    "                \"media_type\": \"image/png\",\n",
    "                \"data\": base64_string\n",
    "            },\n",
    "        },\n",
    "        {\n",
    "            \"type\": \"text\",\n",
    "            \"text\": \"\"\"How many to-go containers of each type \n",
    "            are in this image?\"\"\"\n",
    "        }]\n",
    "    }\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e03af7c0-1628-4f60-8e55-c4545c780ffa",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "response = client.messages.create(\n",
    "    messages=messages,\n",
    "    model=MODEL_NAME,\n",
    "    max_tokens=200\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5c693110-9941-4d14-a899-4ebda8b096f8",
   "metadata": {},
   "source": [
    "## Image Block Helper Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "960d513b-f898-44cb-8460-76053a1a1251",
   "metadata": {
    "height": 455
   },
   "outputs": [],
   "source": [
    "import base64\n",
    "import mimetypes\n",
    "\n",
    "def create_image_message(image_path):\n",
    "    # Open the image file in \"read binary\" mode\n",
    "    with open(image_path, \"rb\") as image_file:\n",
    "        # Read the contents of the image as a bytes object\n",
    "        binary_data = image_file.read()\n",
    "    # Encode the binary data using Base64 encoding\n",
    "    base64_encoded_data = base64.b64encode(binary_data)\n",
    "    # Decode base64_encoded_data from bytes to a string\n",
    "    base64_string = base64_encoded_data.decode('utf-8')\n",
    "    # Get the MIME type of the image based on its file extension\n",
    "    mime_type, _ = mimetypes.guess_type(image_path)\n",
    "    # Create the image block\n",
    "    image_block = {\n",
    "        \"type\": \"image\",\n",
    "        \"source\": {\n",
    "            \"type\": \"base64\",\n",
    "            \"media_type\": mime_type,\n",
    "            \"data\": base64_string\n",
    "        }\n",
    "    }\n",
    "    \n",
    "    \n",
    "    return image_block"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c3c3c25-8a0d-4199-9800-66c9b57a24ba",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "Image(\"images/plant.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7ec90fa-adc0-4953-bcaa-3fea9d993801",
   "metadata": {
    "height": 285
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            create_image_message(\"./images/plant.png\"),\n",
    "            {\"type\": \"text\", \"text\": \"What species is this?\"}\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "response = client.messages.create(\n",
    "    model=MODEL_NAME,\n",
    "    max_tokens=2048,\n",
    "    messages=messages\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83e06e1a-5c94-4666-ae19-a72db567da56",
   "metadata": {},
   "source": [
    "## A Real World Use Case"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe0951a8-d035-435b-b010-a324ed85c74f",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": [
    "Image(filename='images/invoice.png') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf2be60b-551a-4403-9755-c162393bcd45",
   "metadata": {
    "height": 387
   },
   "outputs": [],
   "source": [
    "messages = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            create_image_message(\"./images/invoice.png\"),\n",
    "            {\"type\": \"text\", \"text\": \"\"\"\n",
    "                Generate a JSON object representing the contents\n",
    "                of this invoice.  It should include all dates,\n",
    "                dollar amounts, and addresses. \n",
    "                Only respond with the JSON itself.\n",
    "            \"\"\"\n",
    "            }\n",
    "        ]\n",
    "    }\n",
    "]\n",
    "\n",
    "response = client.messages.create(\n",
    "    model=MODEL_NAME,\n",
    "    max_tokens=2048,\n",
    "    messages=messages\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6c1490a-2964-4793-bf45-5934ec70dfa0",
   "metadata": {},
   "source": [
    "## Streaming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eab3f3de-c2c3-4d1d-b787-e8b3a2479f5b",
   "metadata": {
    "height": 115
   },
   "outputs": [],
   "source": [
    "response = client.messages.create(\n",
    "    max_tokens=1024,\n",
    "    messages=[{\"role\": \"user\", \"content\": \"write a poem\"}],\n",
    "    model=MODEL_NAME,\n",
    ")\n",
    "print(response.content[0].text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da7a63f9-fbff-4225-8277-6f55baba26de",
   "metadata": {
    "height": 132
   },
   "outputs": [],
   "source": [
    "with client.messages.stream(\n",
    "    max_tokens=1024,\n",
    "    messages=[{\"role\": \"user\", \"content\": \"write a poem\"}],\n",
    "    model=MODEL_NAME,\n",
    ") as stream:\n",
    "  for text in stream.text_stream:\n",
    "      print(text, end=\"\", flush=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c71183a2-3e3b-471a-b3aa-9bf1424b5aca",
   "metadata": {},
   "source": [
    "<p style=\"background-color:#f7fff8; padding:15px; border-width:3px; border-color:#e0f0e0; border-style:solid; border-radius:6px\"> 🚨\n",
    "&nbsp; <b>Computer Use Code:</b> The computer use code shown at the end of the video is not part of this lab. This code will also be reviewed step by step in the next lessons, and finally, you will see it in detail in the \"Computer Use\" lesson.</p>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3a9122e-4466-42b9-a706-ab2f9dd2654e",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1667fe4b-eb55-4e43-8537-d3fa754acfbc",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68a2f0eb-0aa3-4352-863e-c29c4cf11689",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "abf87871-0cb1-431e-917d-d4fe11c98fa2",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "999cc944-25c3-403a-aeb9-f64fe6c4d2aa",
   "metadata": {
    "height": 30
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}