22_RAG_splitting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9894eef3-c141-4b25-8483-80456a716589",
   "metadata": {},
   "source": [
    "# Comparing splitters for RAG\n",
    "\n",
    "Let's look at sentence splitting in detail."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2843715-cebf-4724-acaf-bce25399ff5a",
   "metadata": {},
   "source": [
    "## But first, let's learn how to learn LlamaIndex.\n",
    "\n",
    "The LlamaIndex documentation is here: https://docs.llamaindex.ai/en/stable/ There are several important tabs:\n",
    "- Component Guides - explains how to use the various components: loaders, splitters, indexers, etc.\n",
    "- API Reference - explains each component in detail\n",
    "- Examples - many examples showcasing different components\n",
    "- Search bar at the upper right - pretty good way to find more information (like examples) about a specific component\n",
    "\n",
    "In addition to the documentation, you can find the source code here:\n",
    "- https://github.com/run-llama/llama_index\n",
    "- Search bar at the top - great way to find the source for a specific component\n",
    "\n",
    "### Let's use the documentation to learn more about Splitters\n",
    "\n",
    "- Start with the Component Guide to learn about various kinds of splitters\n",
    "- Then go to the API reference to learn about a specific splitter (SentenceSplitter)\n",
    "- Go to github to read the full source file\n",
    "- Go back to the docs and search for examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "initial_id",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:20.014116631Z",
     "start_time": "2024-06-02T06:51:20.013126125Z"
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%load_ext dotenv\n",
    "%dotenv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4f82c275efc6aa93",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:27.966680878Z",
     "start_time": "2024-06-02T06:51:20.554743009Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "from llama_index.core import Document\n",
    "from llama_index.core.node_parser import (\n",
    "    SentenceSplitter,\n",
    "    SemanticSplitterNodeParser,\n",
    "    MarkdownNodeParser,\n",
    ")\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ab8255e29444203f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:28.392069312Z",
     "start_time": "2024-06-02T06:51:27.969236719Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# configure\n",
    "filename = 'sleeping_gods.md'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ce661127-494a-4e55-a49d-2c554330a216",
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_nodes(nodes):\n",
    "    for ix, node in enumerate(nodes):\n",
    "        print(f\"\\n\\n>>> Node {ix} length={len(node.text)}\\n\")\n",
    "        print(node.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c36a892-7f1f-4eb9-a917-6fc03c26c41b",
   "metadata": {},
   "source": [
    "## Load document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7188773706598f27",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:38.604999793Z",
     "start_time": "2024-06-02T06:51:34.534023729Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "77034\n"
     ]
    }
   ],
   "source": [
    "with open(f'data/{filename}', 'r', encoding='utf-8') as file:\n",
    "    document = Document(\n",
    "        text = file.read(),\n",
    "        metadata = {\"filename\": filename},\n",
    "    )\n",
    "print(len(document.text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f8aa00a6-6312-420a-b23a-cab2377dc450",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "## Overview (page 1)\n",
      "\n",
      "1-4 players, ages 13+, 1-20 hours\n",
      "\n",
      "\"This is the Wandering Sea. The gods have brought you here, and you must wake them if you wish to return home.\"\n",
      "\n",
      "In Sleeping Gods, you and up to three friends become Captain Sofi Odessa and her crew, lost in a strange world in 1929 on your steamship, the Manticore. You must work together to survive, exploring exotic islands, meeting new characters, and seeking out the totems of the gods so that you can return home.\n",
      "\n",
      "Sleeping Gods is a campaign game. Each session can last as long as you want. When you are ready to take a break, you mark your progress on a journey log sheet, making it easy to return to the same place in the game the next time you play. You can play alone or with friends throughout your campaign— it's easy to swap players in and out at will. Your goal is to find as many totems as possible, which are hidden throughout the world. Like reading a book, you'll complete this journey one or two hours at a time, discovering new lands, stories, and challenges along the way.\n",
      "\n",
      "Sleeping Gods is an atlas game. Each page of the atlas represents only a small portion of the world you can explore. When you reach the edge of a page and you want to continue in the same direction, you simply turn to a new page and sail onward.\n",
      "\n",
      "Sleeping Gods is a storybook game. Each new location holds wild adventure, hidden treasures, and vivid characters. Your choices affect the characters and the plot of the game, and may help or hinder your chances of getting home!\n",
      "\n",
      "Welcome to a vast world. Your journey starts now.\n",
      "\n",
      "## Setup (page 4)\n",
      "\n",
      "Follow these instructions if you are starting a new campaign. If this is your first campaign, we recommend using the quick start guide first. If you are setting up the game to continue a campaign, turn to pg. 32.\n",
      "\n",
      "1. Place the atlas in the center of the table and turn to pg. 2. Place the ship in the sea region that contains the number 2 location.\n",
      "\n",
      "2. Place the ship board near the atlas. Place the correct side face up depending on number of players, 1-2 or 3-4, listed in the top right corner of the ship board. Place the ship action token on the ship board. Place 3 coins and 1 grain on the ship board. You start the game with these resources.\n",
      "\n",
      "3. Place the Captain Sofi Odessa crew board near the ship board, in the center of the table. Distribute the remaining 8 crew boards among the players as evenly as possible. Each player places their crew boards in front of them. Place the synergy tokens nearby.\n",
      "\n",
      "4. Shuffle the ability deck and place it near the ship board. Decide who will be the first player and give them the captain token.\n",
      "\n",
      "5. Shuffle the market deck and place it nearby.\n",
      "\n",
      "6. Separate the event cards by type (mild, perilous, deadly). Shuffle each pile separately. Draw 6 deadly cards and place them face down in a new deck. Draw 6 perilous cards and place them face down on top of the deadly. Draw 6 mild cards and place them face down on top of the perilous. You should now have a deck of 18 event cards. Place this deck face down on its slot on the ship board. Place the unused cards in the box.\n",
      "\n",
      "7. Place the starting adventure cards near the ship board. These cards are: Gloria, Soup, Gear, and Flapjacks (you can find these cards in the market deck). Place the magnetic card box that contains adventure and quest cards nearby IMPORTANT: Do not change the order of cards in the quest card and adventure card decks, and do not look through the cards unless instructed.\n",
      "\n",
      "8. Place the deck of enemy cards near the board, face down. IMPORTANT: Do not change the order of cards in this deck and do not look through this deck unless instructed.\n",
      "\n",
      "9. Place the storybook near the board.\n",
      "\n",
      "10. Shuffle the search tokens and place them in a stack, face down, near the ship board.\n",
      "\n",
      "11. Give each player (not each crew member) 1 command and 1 ability card.\n",
      "\n",
      "12. Place the level cards near the ship board. You may look through these at any time. Place all remaining tokens near the board.\n",
      "\n",
      "13. Retrieve a new journey log sheet for your campaign. Decide if you will be playing normal or brutal mode and check the appropriate box (see \"Defeat\" on pg. 28). Write the date at the top of the sheet and write in all players' names at the bottom of the sheet.\n",
      "\n",
      "## Basics (page 6)\n",
      "\n",
      "Pgs. 6-9 introduce some basics of Sleeping Gods to help you get your sea legs. Turn structure and actions are explained starting on pg. 10.\n",
      "\n",
      "### Ship Board (page 6)\n",
      "\n",
      "#### Ship Room (page 6)\n",
      "\n",
      "There are five rooms on the ship. Each room contains an action that you can take at the start of your turn. Each room also contains spaces for ship damage. A room with all damage spaces filled may not be used for a ship action. Once the ship has 11 damage, you are defeated!\n",
      "\n",
      "After taking a ship action and drawing an event card, perform two of these actions.\n",
      "\n",
      "> The actions that you can perform appear in the lower-right corner of the ship board.\n",
      "\n",
      "Each turn draw and resolve an event card from the event deck, placed here.\n",
      "\n",
      "> The event deck is placed in the lower-left corner of the ship board.\n",
      "\n",
      "### Crew Board (page 7)\n",
      "\n",
      "Each player controls 2 or more crew. Crew take damage and fatigue throughout the game.\n",
      "\n",
      "Each crew member has a unique set of skills.\n",
      "\n",
      "> Crew member skills appear along the upper-right side of the crew board.\n",
      "\n",
      "You may pay command to use unique crew member abilities.\n",
      "\n",
      "> Crew member abilities appear near the bottom-middle of the crew board.\n",
      "\n",
      "Name And Origin\n",
      "\n",
      "> Name and origin appear in the lower-left of the crew board.\n",
      "\n",
      "When you equip ability cards to a crew member, place them beneath these 2 slots.\n",
      "\n",
      "> The two slots are at the bottom left of the crew board.\n",
      "\n",
      "Health: A crew member may only take damage up to their maximum health.\n",
      "\n",
      "> A crew member's maximum health appears in a heart near the bottom-middle of the crew board.\n",
      "\n",
      "Weapon Stats: Accuracy, Damage, Block.\n",
      "\n",
      "> Weapon stats: accuracy, damage, and block appear in the lower-right of the crew board.\n",
      "\n",
      "### Ability Card (page 7)\n",
      "\n",
      "This card has a fate number of 5. When you \"draw fate,\" you draw an ability card and use this number.\n",
      "\n",
      "> The Fate Number appears at the top-left of the card.\n",
      "\n",
      "To equip this card to a crew member, you must pay 2 command.\n",
      "\n",
      "> The commands for equipping the card to a crew member are on the top-right.\n",
      "\n",
      "This card has a savvy skill symbol. When equipped, it gives a crew member +1 savvy.\n",
      "\n",
      "> The symbol for the savvy skill appears on the left-hand side, halfway up the card.\n",
      "\n",
      "When this card is equipped, it gives the crew member the \"Triage\" ability. It costs no command to use once equipped.\n",
      "\n",
      "> The visual representation of the \"Triage\" ability is in the middle at the bottom of the card.\n",
      "\n",
      "Secret: Players must keep their abilities cards in hand secret until they are played. They may not show them or give them to other players.\n",
      "\n",
      "Hand Limit: Each player has a hand limit of 3 ability cards. If after drawing new Ability Cards your hand size has exceeded the limit, you must immediately discard down to 3 cards. You may select the card(s) to discard.\n",
      "\n",
      "### Game Concepts And Terms (page 8)\n",
      "\n",
      "#### Active Player (page 8)\n",
      "\n",
      "When it is your turn, you are the active player.\n",
      "\n",
      "#### Damage (page 8)\n",
      "\n",
      "Damage is physical and mental injury to a crew member or an enemy, represented by damage tokens (the blood drop). If the storybook or a card says \"-1 health,\" it means take 1 damage. When a crew member takes damage, place the indicated amount of damage tokens on the crew board.\n",
      "\n",
      "When you are instructed to \"Restore health\" to a crew member, remove the specified number of damage tokens.\n",
      "\n",
      "This symbol represents the amount of damage dealt by a crew member or enemy card in an attack.\n",
      "\n",
      "#### Health (page 8)\n",
      "\n",
      "Health is the amount of damage a crew member can take and still function normally. If a crew member has damage equal to their health (known as having 0 health), they can no longer attack, participate in challenges, or activate any of their crew board abilities or ability cards until they regain at least 1 health. Thematical\n"
     ]
    }
   ],
   "source": [
    "cutoff = 8000\n",
    "print(document.text[:cutoff])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "51aa19b5-2b2c-4ebe-87db-02950a5e7703",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8000\n"
     ]
    }
   ],
   "source": [
    "# let's just focus on the first part of the document to make this manageable\n",
    "document.text = document.text[:cutoff]\n",
    "print(len(document.text))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88f49a31-7079-43f0-a395-82884763f8a0",
   "metadata": {},
   "source": [
    "## Sentence Splitter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9d5056f1cdaa9510",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:39.802412275Z",
     "start_time": "2024-06-02T06:51:38.607042780Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "chunk_size = 300\n",
    "chunk_overlap = 50\n",
    "splitter = SentenceSplitter(chunk_size=chunk_size, chunk_overlap=chunk_overlap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "ef997d55278a031d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-06-02T06:51:39.856952432Z",
     "start_time": "2024-06-02T06:51:39.800673831Z"
    },
    "collapsed": false,
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8\n"
     ]
    }
   ],
   "source": [
    "nodes = splitter.get_nodes_from_documents([document], show_progress=False)\n",
    "print(len(nodes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "052494f5-ef59-49ae-b07b-cefbc9e65e37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      ">>> Node 0 length=1168\n",
      "\n",
      "## Overview (page 1)\n",
      "\n",
      "1-4 players, ages 13+, 1-20 hours\n",
      "\n",
      "\"This is the Wandering Sea. The gods have brought you here, and you must wake them if you wish to return home.\"\n",
      "\n",
      "In Sleeping Gods, you and up to three friends become Captain Sofi Odessa and her crew, lost in a strange world in 1929 on your steamship, the Manticore. You must work together to survive, exploring exotic islands, meeting new characters, and seeking out the totems of the gods so that you can return home.\n",
      "\n",
      "Sleeping Gods is a campaign game. Each session can last as long as you want. When you are ready to take a break, you mark your progress on a journey log sheet, making it easy to return to the same place in the game the next time you play. You can play alone or with friends throughout your campaign— it's easy to swap players in and out at will. Your goal is to find as many totems as possible, which are hidden throughout the world. Like reading a book, you'll complete this journey one or two hours at a time, discovering new lands, stories, and challenges along the way.\n",
      "\n",
      "Sleeping Gods is an atlas game. Each page of the atlas represents only a small portion of the world you can explore.\n",
      "\n",
      "\n",
      ">>> Node 1 length=1198\n",
      "\n",
      "Sleeping Gods is an atlas game. Each page of the atlas represents only a small portion of the world you can explore. When you reach the edge of a page and you want to continue in the same direction, you simply turn to a new page and sail onward.\n",
      "\n",
      "Sleeping Gods is a storybook game. Each new location holds wild adventure, hidden treasures, and vivid characters. Your choices affect the characters and the plot of the game, and may help or hinder your chances of getting home!\n",
      "\n",
      "Welcome to a vast world. Your journey starts now.\n",
      "\n",
      "## Setup (page 4)\n",
      "\n",
      "Follow these instructions if you are starting a new campaign. If this is your first campaign, we recommend using the quick start guide first. If you are setting up the game to continue a campaign, turn to pg. 32.\n",
      "\n",
      "1. Place the atlas in the center of the table and turn to pg. 2. Place the ship in the sea region that contains the number 2 location.\n",
      "\n",
      "2. Place the ship board near the atlas. Place the correct side face up depending on number of players, 1-2 or 3-4, listed in the top right corner of the ship board. Place the ship action token on the ship board. Place 3 coins and 1 grain on the ship board. You start the game with these resources.\n",
      "\n",
      "3.\n",
      "\n",
      "\n",
      ">>> Node 2 length=1160\n",
      "\n",
      "Place the ship action token on the ship board. Place 3 coins and 1 grain on the ship board. You start the game with these resources.\n",
      "\n",
      "3. Place the Captain Sofi Odessa crew board near the ship board, in the center of the table. Distribute the remaining 8 crew boards among the players as evenly as possible. Each player places their crew boards in front of them. Place the synergy tokens nearby.\n",
      "\n",
      "4. Shuffle the ability deck and place it near the ship board. Decide who will be the first player and give them the captain token.\n",
      "\n",
      "5. Shuffle the market deck and place it nearby.\n",
      "\n",
      "6. Separate the event cards by type (mild, perilous, deadly). Shuffle each pile separately. Draw 6 deadly cards and place them face down in a new deck. Draw 6 perilous cards and place them face down on top of the deadly. Draw 6 mild cards and place them face down on top of the perilous. You should now have a deck of 18 event cards. Place this deck face down on its slot on the ship board. Place the unused cards in the box.\n",
      "\n",
      "7. Place the starting adventure cards near the ship board. These cards are: Gloria, Soup, Gear, and Flapjacks (you can find these cards in the market deck).\n",
      "\n",
      "\n",
      ">>> Node 3 length=1205\n",
      "\n",
      "Place the unused cards in the box.\n",
      "\n",
      "7. Place the starting adventure cards near the ship board. These cards are: Gloria, Soup, Gear, and Flapjacks (you can find these cards in the market deck). Place the magnetic card box that contains adventure and quest cards nearby IMPORTANT: Do not change the order of cards in the quest card and adventure card decks, and do not look through the cards unless instructed.\n",
      "\n",
      "8. Place the deck of enemy cards near the board, face down. IMPORTANT: Do not change the order of cards in this deck and do not look through this deck unless instructed.\n",
      "\n",
      "9. Place the storybook near the board.\n",
      "\n",
      "10. Shuffle the search tokens and place them in a stack, face down, near the ship board.\n",
      "\n",
      "11. Give each player (not each crew member) 1 command and 1 ability card.\n",
      "\n",
      "12. Place the level cards near the ship board. You may look through these at any time. Place all remaining tokens near the board.\n",
      "\n",
      "13. Retrieve a new journey log sheet for your campaign. Decide if you will be playing normal or brutal mode and check the appropriate box (see \"Defeat\" on pg. 28). Write the date at the top of the sheet and write in all players' names at the bottom of the sheet.\n",
      "\n",
      "## Basics (page 6)\n",
      "\n",
      "Pgs.\n",
      "\n",
      "\n",
      ">>> Node 4 length=1277\n",
      "\n",
      "28). Write the date at the top of the sheet and write in all players' names at the bottom of the sheet.\n",
      "\n",
      "## Basics (page 6)\n",
      "\n",
      "Pgs. 6-9 introduce some basics of Sleeping Gods to help you get your sea legs. Turn structure and actions are explained starting on pg. 10.\n",
      "\n",
      "### Ship Board (page 6)\n",
      "\n",
      "#### Ship Room (page 6)\n",
      "\n",
      "There are five rooms on the ship. Each room contains an action that you can take at the start of your turn. Each room also contains spaces for ship damage. A room with all damage spaces filled may not be used for a ship action. Once the ship has 11 damage, you are defeated!\n",
      "\n",
      "After taking a ship action and drawing an event card, perform two of these actions.\n",
      "\n",
      "> The actions that you can perform appear in the lower-right corner of the ship board.\n",
      "\n",
      "Each turn draw and resolve an event card from the event deck, placed here.\n",
      "\n",
      "> The event deck is placed in the lower-left corner of the ship board.\n",
      "\n",
      "### Crew Board (page 7)\n",
      "\n",
      "Each player controls 2 or more crew. Crew take damage and fatigue throughout the game.\n",
      "\n",
      "Each crew member has a unique set of skills.\n",
      "\n",
      "> Crew member skills appear along the upper-right side of the crew board.\n",
      "\n",
      "You may pay command to use unique crew member abilities.\n",
      "\n",
      "> Crew member abilities appear near the bottom-middle of the crew board.\n",
      "\n",
      "\n",
      ">>> Node 5 length=1266\n",
      "\n",
      "Each crew member has a unique set of skills.\n",
      "\n",
      "> Crew member skills appear along the upper-right side of the crew board.\n",
      "\n",
      "You may pay command to use unique crew member abilities.\n",
      "\n",
      "> Crew member abilities appear near the bottom-middle of the crew board.\n",
      "\n",
      "Name And Origin\n",
      "\n",
      "> Name and origin appear in the lower-left of the crew board.\n",
      "\n",
      "When you equip ability cards to a crew member, place them beneath these 2 slots.\n",
      "\n",
      "> The two slots are at the bottom left of the crew board.\n",
      "\n",
      "Health: A crew member may only take damage up to their maximum health.\n",
      "\n",
      "> A crew member's maximum health appears in a heart near the bottom-middle of the crew board.\n",
      "\n",
      "Weapon Stats: Accuracy, Damage, Block.\n",
      "\n",
      "> Weapon stats: accuracy, damage, and block appear in the lower-right of the crew board.\n",
      "\n",
      "### Ability Card (page 7)\n",
      "\n",
      "This card has a fate number of 5. When you \"draw fate,\" you draw an ability card and use this number.\n",
      "\n",
      "> The Fate Number appears at the top-left of the card.\n",
      "\n",
      "To equip this card to a crew member, you must pay 2 command.\n",
      "\n",
      "> The commands for equipping the card to a crew member are on the top-right.\n",
      "\n",
      "This card has a savvy skill symbol. When equipped, it gives a crew member +1 savvy.\n",
      "\n",
      "> The symbol for the savvy skill appears on the left-hand side, halfway up the card.\n",
      "\n",
      "\n",
      ">>> Node 6 length=1274\n",
      "\n",
      "This card has a savvy skill symbol. When equipped, it gives a crew member +1 savvy.\n",
      "\n",
      "> The symbol for the savvy skill appears on the left-hand side, halfway up the card.\n",
      "\n",
      "When this card is equipped, it gives the crew member the \"Triage\" ability. It costs no command to use once equipped.\n",
      "\n",
      "> The visual representation of the \"Triage\" ability is in the middle at the bottom of the card.\n",
      "\n",
      "Secret: Players must keep their abilities cards in hand secret until they are played. They may not show them or give them to other players.\n",
      "\n",
      "Hand Limit: Each player has a hand limit of 3 ability cards. If after drawing new Ability Cards your hand size has exceeded the limit, you must immediately discard down to 3 cards. You may select the card(s) to discard.\n",
      "\n",
      "### Game Concepts And Terms (page 8)\n",
      "\n",
      "#### Active Player (page 8)\n",
      "\n",
      "When it is your turn, you are the active player.\n",
      "\n",
      "#### Damage (page 8)\n",
      "\n",
      "Damage is physical and mental injury to a crew member or an enemy, represented by damage tokens (the blood drop). If the storybook or a card says \"-1 health,\" it means take 1 damage. When a crew member takes damage, place the indicated amount of damage tokens on the crew board.\n",
      "\n",
      "When you are instructed to \"Restore health\" to a crew member, remove the specified number of damage tokens.\n",
      "\n",
      "\n",
      ">>> Node 7 length=649\n",
      "\n",
      "When a crew member takes damage, place the indicated amount of damage tokens on the crew board.\n",
      "\n",
      "When you are instructed to \"Restore health\" to a crew member, remove the specified number of damage tokens.\n",
      "\n",
      "This symbol represents the amount of damage dealt by a crew member or enemy card in an attack.\n",
      "\n",
      "#### Health (page 8)\n",
      "\n",
      "Health is the amount of damage a crew member can take and still function normally. If a crew member has damage equal to their health (known as having 0 health), they can no longer attack, participate in challenges, or activate any of their crew board abilities or ability cards until they regain at least 1 health. Thematical\n"
     ]
    }
   ],
   "source": [
    "print_nodes(nodes[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4226352c-894a-4e14-b1f0-a6d3b7a4c6bb",
   "metadata": {},
   "source": [
    "## SemanticSplitterNodeParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "c246a817-ca13-4e2a-a1ec-1181adf81d27",
   "metadata": {},
   "outputs": [],
   "source": [
    "breakpoint_percentile_threshold=80\n",
    "embed_model = OpenAIEmbedding(model='text-embedding-3-small')\n",
    "splitter = SemanticSplitterNodeParser(breakpoint_percentile_threshold=breakpoint_percentile_threshold, embed_model=embed_model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "38f8d7f9-0aa1-4ee8-bf37-9e7cb570bd59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "26\n"
     ]
    }
   ],
   "source": [
    "nodes = splitter.get_nodes_from_documents([document], show_progress=False)\n",
    "print(len(nodes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "7591713f-5c10-48be-be05-88db54bb882f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      ">>> Node 0 length=554\n",
      "\n",
      "## Overview (page 1)\n",
      "\n",
      "1-4 players, ages 13+, 1-20 hours\n",
      "\n",
      "\"This is the Wandering Sea. The gods have brought you here, and you must wake them if you wish to return home.\"\n",
      "\n",
      "In Sleeping Gods, you and up to three friends become Captain Sofi Odessa and her crew, lost in a strange world in 1929 on your steamship, the Manticore. You must work together to survive, exploring exotic islands, meeting new characters, and seeking out the totems of the gods so that you can return home.\n",
      "\n",
      "Sleeping Gods is a campaign game. Each session can last as long as you want. \n",
      "\n",
      "\n",
      ">>> Node 1 length=357\n",
      "\n",
      "When you are ready to take a break, you mark your progress on a journey log sheet, making it easy to return to the same place in the game the next time you play. You can play alone or with friends throughout your campaign— it's easy to swap players in and out at will. Your goal is to find as many totems as possible, which are hidden throughout the world. \n",
      "\n",
      "\n",
      ">>> Node 2 length=423\n",
      "\n",
      "Like reading a book, you'll complete this journey one or two hours at a time, discovering new lands, stories, and challenges along the way.\n",
      "\n",
      "Sleeping Gods is an atlas game. Each page of the atlas represents only a small portion of the world you can explore. When you reach the edge of a page and you want to continue in the same direction, you simply turn to a new page and sail onward.\n",
      "\n",
      "Sleeping Gods is a storybook game. \n",
      "\n",
      "\n",
      ">>> Node 3 length=80\n",
      "\n",
      "Each new location holds wild adventure, hidden treasures, and vivid characters. \n",
      "\n",
      "\n",
      ">>> Node 4 length=140\n",
      "\n",
      "Your choices affect the characters and the plot of the game, and may help or hinder your chances of getting home!\n",
      "\n",
      "Welcome to a vast world. \n",
      "\n",
      "\n",
      ">>> Node 5 length=259\n",
      "\n",
      "Your journey starts now.\n",
      "\n",
      "## Setup (page 4)\n",
      "\n",
      "Follow these instructions if you are starting a new campaign. If this is your first campaign, we recommend using the quick start guide first. If you are setting up the game to continue a campaign, turn to pg. 32.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      ">>> Node 6 length=65\n",
      "\n",
      "1. Place the atlas in the center of the table and turn to pg. 2. \n",
      "\n",
      "\n",
      ">>> Node 7 length=370\n",
      "\n",
      "Place the ship in the sea region that contains the number 2 location.\n",
      "\n",
      "2. Place the ship board near the atlas. Place the correct side face up depending on number of players, 1-2 or 3-4, listed in the top right corner of the ship board. Place the ship action token on the ship board. Place 3 coins and 1 grain on the ship board. You start the game with these resources.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      ">>> Node 8 length=265\n",
      "\n",
      "3. Place the Captain Sofi Odessa crew board near the ship board, in the center of the table. Distribute the remaining 8 crew boards among the players as evenly as possible. Each player places their crew boards in front of them. Place the synergy tokens nearby.\n",
      "\n",
      "4. \n",
      "\n",
      "\n",
      ">>> Node 9 length=132\n",
      "\n",
      "Shuffle the ability deck and place it near the ship board. Decide who will be the first player and give them the captain token.\n",
      "\n",
      "5. \n"
     ]
    }
   ],
   "source": [
    "print_nodes(nodes[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1b93439-5004-4954-97ad-898436554d8f",
   "metadata": {},
   "source": [
    "## MarkdownNodeParser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "0149c4ab-9a6c-454f-b81c-bf932a7ab4b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "splitter = MarkdownNodeParser()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "923f771b-6b2d-4279-9b9c-412d0a62426e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    }
   ],
   "source": [
    "nodes = splitter.get_nodes_from_documents([document], show_progress=False)\n",
    "print(len(nodes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7917d7c9-563a-488c-869f-9737868df802",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      ">>> Node 0 length=1575\n",
      "\n",
      "Overview (page 1)\n",
      "\n",
      "1-4 players, ages 13+, 1-20 hours\n",
      "\n",
      "\"This is the Wandering Sea. The gods have brought you here, and you must wake them if you wish to return home.\"\n",
      "\n",
      "In Sleeping Gods, you and up to three friends become Captain Sofi Odessa and her crew, lost in a strange world in 1929 on your steamship, the Manticore. You must work together to survive, exploring exotic islands, meeting new characters, and seeking out the totems of the gods so that you can return home.\n",
      "\n",
      "Sleeping Gods is a campaign game. Each session can last as long as you want. When you are ready to take a break, you mark your progress on a journey log sheet, making it easy to return to the same place in the game the next time you play. You can play alone or with friends throughout your campaign— it's easy to swap players in and out at will. Your goal is to find as many totems as possible, which are hidden throughout the world. Like reading a book, you'll complete this journey one or two hours at a time, discovering new lands, stories, and challenges along the way.\n",
      "\n",
      "Sleeping Gods is an atlas game. Each page of the atlas represents only a small portion of the world you can explore. When you reach the edge of a page and you want to continue in the same direction, you simply turn to a new page and sail onward.\n",
      "\n",
      "Sleeping Gods is a storybook game. Each new location holds wild adventure, hidden treasures, and vivid characters. Your choices affect the characters and the plot of the game, and may help or hinder your chances of getting home!\n",
      "\n",
      "Welcome to a vast world. Your journey starts now.\n",
      "\n",
      "\n",
      ">>> Node 1 length=2678\n",
      "\n",
      "Setup (page 4)\n",
      "\n",
      "Follow these instructions if you are starting a new campaign. If this is your first campaign, we recommend using the quick start guide first. If you are setting up the game to continue a campaign, turn to pg. 32.\n",
      "\n",
      "1. Place the atlas in the center of the table and turn to pg. 2. Place the ship in the sea region that contains the number 2 location.\n",
      "\n",
      "2. Place the ship board near the atlas. Place the correct side face up depending on number of players, 1-2 or 3-4, listed in the top right corner of the ship board. Place the ship action token on the ship board. Place 3 coins and 1 grain on the ship board. You start the game with these resources.\n",
      "\n",
      "3. Place the Captain Sofi Odessa crew board near the ship board, in the center of the table. Distribute the remaining 8 crew boards among the players as evenly as possible. Each player places their crew boards in front of them. Place the synergy tokens nearby.\n",
      "\n",
      "4. Shuffle the ability deck and place it near the ship board. Decide who will be the first player and give them the captain token.\n",
      "\n",
      "5. Shuffle the market deck and place it nearby.\n",
      "\n",
      "6. Separate the event cards by type (mild, perilous, deadly). Shuffle each pile separately. Draw 6 deadly cards and place them face down in a new deck. Draw 6 perilous cards and place them face down on top of the deadly. Draw 6 mild cards and place them face down on top of the perilous. You should now have a deck of 18 event cards. Place this deck face down on its slot on the ship board. Place the unused cards in the box.\n",
      "\n",
      "7. Place the starting adventure cards near the ship board. These cards are: Gloria, Soup, Gear, and Flapjacks (you can find these cards in the market deck). Place the magnetic card box that contains adventure and quest cards nearby IMPORTANT: Do not change the order of cards in the quest card and adventure card decks, and do not look through the cards unless instructed.\n",
      "\n",
      "8. Place the deck of enemy cards near the board, face down. IMPORTANT: Do not change the order of cards in this deck and do not look through this deck unless instructed.\n",
      "\n",
      "9. Place the storybook near the board.\n",
      "\n",
      "10. Shuffle the search tokens and place them in a stack, face down, near the ship board.\n",
      "\n",
      "11. Give each player (not each crew member) 1 command and 1 ability card.\n",
      "\n",
      "12. Place the level cards near the ship board. You may look through these at any time. Place all remaining tokens near the board.\n",
      "\n",
      "13. Retrieve a new journey log sheet for your campaign. Decide if you will be playing normal or brutal mode and check the appropriate box (see \"Defeat\" on pg. 28). Write the date at the top of the sheet and write in all players' names at the bottom of the sheet.\n",
      "\n",
      "\n",
      ">>> Node 2 length=156\n",
      "\n",
      "Basics (page 6)\n",
      "\n",
      "Pgs. 6-9 introduce some basics of Sleeping Gods to help you get your sea legs. Turn structure and actions are explained starting on pg. 10.\n",
      "\n",
      "\n",
      ">>> Node 3 length=19\n",
      "\n",
      "Ship Board (page 6)\n",
      "\n",
      "\n",
      ">>> Node 4 length=615\n",
      "\n",
      "Ship Room (page 6)\n",
      "\n",
      "There are five rooms on the ship. Each room contains an action that you can take at the start of your turn. Each room also contains spaces for ship damage. A room with all damage spaces filled may not be used for a ship action. Once the ship has 11 damage, you are defeated!\n",
      "\n",
      "After taking a ship action and drawing an event card, perform two of these actions.\n",
      "\n",
      "> The actions that you can perform appear in the lower-right corner of the ship board.\n",
      "\n",
      "Each turn draw and resolve an event card from the event deck, placed here.\n",
      "\n",
      "> The event deck is placed in the lower-left corner of the ship board.\n",
      "\n",
      "\n",
      ">>> Node 5 length=878\n",
      "\n",
      "Crew Board (page 7)\n",
      "\n",
      "Each player controls 2 or more crew. Crew take damage and fatigue throughout the game.\n",
      "\n",
      "Each crew member has a unique set of skills.\n",
      "\n",
      "> Crew member skills appear along the upper-right side of the crew board.\n",
      "\n",
      "You may pay command to use unique crew member abilities.\n",
      "\n",
      "> Crew member abilities appear near the bottom-middle of the crew board.\n",
      "\n",
      "Name And Origin\n",
      "\n",
      "> Name and origin appear in the lower-left of the crew board.\n",
      "\n",
      "When you equip ability cards to a crew member, place them beneath these 2 slots.\n",
      "\n",
      "> The two slots are at the bottom left of the crew board.\n",
      "\n",
      "Health: A crew member may only take damage up to their maximum health.\n",
      "\n",
      "> A crew member's maximum health appears in a heart near the bottom-middle of the crew board.\n",
      "\n",
      "Weapon Stats: Accuracy, Damage, Block.\n",
      "\n",
      "> Weapon stats: accuracy, damage, and block appear in the lower-right of the crew board.\n",
      "\n",
      "\n",
      ">>> Node 6 length=1068\n",
      "\n",
      "Ability Card (page 7)\n",
      "\n",
      "This card has a fate number of 5. When you \"draw fate,\" you draw an ability card and use this number.\n",
      "\n",
      "> The Fate Number appears at the top-left of the card.\n",
      "\n",
      "To equip this card to a crew member, you must pay 2 command.\n",
      "\n",
      "> The commands for equipping the card to a crew member are on the top-right.\n",
      "\n",
      "This card has a savvy skill symbol. When equipped, it gives a crew member +1 savvy.\n",
      "\n",
      "> The symbol for the savvy skill appears on the left-hand side, halfway up the card.\n",
      "\n",
      "When this card is equipped, it gives the crew member the \"Triage\" ability. It costs no command to use once equipped.\n",
      "\n",
      "> The visual representation of the \"Triage\" ability is in the middle at the bottom of the card.\n",
      "\n",
      "Secret: Players must keep their abilities cards in hand secret until they are played. They may not show them or give them to other players.\n",
      "\n",
      "Hand Limit: Each player has a hand limit of 3 ability cards. If after drawing new Ability Cards your hand size has exceeded the limit, you must immediately discard down to 3 cards. You may select the card(s) to discard.\n",
      "\n",
      "\n",
      ">>> Node 7 length=32\n",
      "\n",
      "Game Concepts And Terms (page 8)\n",
      "\n",
      "\n",
      ">>> Node 8 length=72\n",
      "\n",
      "Active Player (page 8)\n",
      "\n",
      "When it is your turn, you are the active player.\n",
      "\n",
      "\n",
      ">>> Node 9 length=500\n",
      "\n",
      "Damage (page 8)\n",
      "\n",
      "Damage is physical and mental injury to a crew member or an enemy, represented by damage tokens (the blood drop). If the storybook or a card says \"-1 health,\" it means take 1 damage. When a crew member takes damage, place the indicated amount of damage tokens on the crew board.\n",
      "\n",
      "When you are instructed to \"Restore health\" to a crew member, remove the specified number of damage tokens.\n",
      "\n",
      "This symbol represents the amount of damage dealt by a crew member or enemy card in an attack.\n"
     ]
    }
   ],
   "source": [
    "print_nodes(nodes[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3425375-d7ad-4ce7-abd4-6e7a5c3048ba",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}