Web Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b93c8352",
   "metadata": {},
   "source": [
    "# Web Scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60c15e9b",
   "metadata": {},
   "source": [
    "* A **Website** is a collection of related web pages that may contain text, images, audio and video. It can be static or dynamic.\n",
    "\n",
    "\n",
    "* **HTTP** is the data communication protocol used to establish communication between client and server.\n",
    "\n",
    "\n",
    "* **HTTP Requests**\tis the request send by the computer to a web server that contains all sorts of potentially interesting information.\n",
    "\n",
    "\n",
    "* A **Server** is used to manage the network resources (**web server**). Servers are also used for running the program or software that provides services (**application server**).\n",
    "\n",
    "\n",
    "* **API** or **Application Programming Interface** allows interactions between systems by following a set of standards and protocols in order to share features, information and data. API acts as an interface between different applications.\n",
    "\n",
    "\n",
    "* A **REST API** (or **Representational State Transfer API**) is an architecture style to develop web applications. It uses HTTP protocol as a communication inteface for different software systems to communicate with each other through the internet. It transfers data through HTTP methods. There are five main methods used in a REST API:\n",
    "\t* `GET` - retrieves a specific resource or collection of resources\n",
    "\t* `POST` - creates a new resource\n",
    "\t* `PUT` - updates an existing resource\n",
    "\t* `DELETE` - removes a specific resource\n",
    "\t* `PATCH` - partially updates an existing resource"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0230114a",
   "metadata": {},
   "source": [
    "* **GET** request data from a resource (URL) whereas **POST** creates/updates a response.\n",
    "\t* `GET` : `requests.get(\"<url>\")`\n",
    "\t* `POST` : `requests.post(\"<url>\", data={\"key\":\"value\"})`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c46c6038",
   "metadata": {},
   "source": [
    "* The **Response** object is returned from the HTTP request that holds the results of the request, it can either be a success or an error. A success response typically includes the requested information or a message confirming that the requested action was completed. An error response includes a message explaining why the request could not be completed. The Response object contains not only the **page content**, but also many other items about the result such as **HTTP status codes** and **headers**.\n",
    "    * **Content Type** is HTTP header that provides the description about what are we sending to the browser.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97763e04",
   "metadata": {},
   "source": [
    "* Output for common requests attributes:\n",
    "\t* `response.content`: raw bytes response payload\n",
    "    * `response.text`: character encoded (e.g. UTF-8) string payload\n",
    "    * `response.headers`: dictionary-like object which contains header payload as key-value\n",
    "\t* `response.status_code`: status code returned by the external service"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1474f4",
   "metadata": {},
   "source": [
    "* Requests is used only to get the page, it **does not do an parsing**.\n",
    "\n",
    "* We use **Beautiful Soup** to do the parsing of the HTML and also the finding of content within the HTML."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bb6d4d0",
   "metadata": {},
   "source": [
    "## Scrape the upcoming Python events"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "8944e1da",
   "metadata": {},
   "outputs": [],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "847d2c81",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get CSS classes from \"Right Click / Inspect\", and search the element\n",
    "css_class_dict = {'recent_events_class' : 'list-recent-events',\n",
    "                 'event_location_class' : 'event-location'}\n",
    "\n",
    "# URL for website\n",
    "url = 'https://www.python.org/events/python-events/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b2e6e9b7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def _parse_response(response_text, css_class_dict):\n",
    "    \n",
    "    # create a BeautifulSoup object and pass it the HTML text\n",
    "    soup = BeautifulSoup(response_text, 'lxml')\n",
    "    \n",
    "    # find the main <ul> tag for the recent events, and then to get all the <li> tags below it\n",
    "    events = soup.find('ul', {'class': css_class_dict['recent_events_class']}).findAll('li')\n",
    "    \n",
    "    # mapping all results to upcoming_events\n",
    "    upcoming_events = []\n",
    "    for event in events:\n",
    "        event_details = dict()\n",
    "        event_details['name'] = event.find('h3').find(\"a\").text\n",
    "        event_details['location'] = event.find('span', {'class', css_class_dict['event_location_class']}).text\n",
    "        event_details['time'] = event.find('time').text\n",
    "        upcoming_events.append(event_details)\n",
    "\n",
    "    # Creating events dataframe\n",
    "    events_df = pd.DataFrame(upcoming_events)\n",
    "        \n",
    "    return events_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "15711f0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_upcoming_events(response_text, css_class_dict):\n",
    "    events_df = _parse_response(response_text, css_class_dict)\n",
    "    return events_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a09cb1fa",
   "metadata": {},
   "source": [
    "### Using `requests` library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "73ab86a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "57b59cf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>location</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PyCon Italia 2024</td>\n",
       "      <td>Metropolitan City of Florence, Italy</td>\n",
       "      <td>22 May – 25 May  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GeoPython 2024</td>\n",
       "      <td>Basel, Switzerland</td>\n",
       "      <td>27 May – 28 May  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>csv,conf,v8</td>\n",
       "      <td>Puebla, Mexico</td>\n",
       "      <td>29 May – 30 May  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DjangoCon Europe 2024</td>\n",
       "      <td>Vigo, Spain</td>\n",
       "      <td>05 June – 09 June  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PyCon Colombia 2024</td>\n",
       "      <td>Medellín, Colombia</td>\n",
       "      <td>07 June – 09 June  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Wagtail Space NL</td>\n",
       "      <td>Arnhem, The Netherlands</td>\n",
       "      <td>12 June – 14 June  2024</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    name                              location  \\\n",
       "0      PyCon Italia 2024  Metropolitan City of Florence, Italy   \n",
       "1         GeoPython 2024                    Basel, Switzerland   \n",
       "2            csv,conf,v8                        Puebla, Mexico   \n",
       "3  DjangoCon Europe 2024                           Vigo, Spain   \n",
       "4    PyCon Colombia 2024                    Medellín, Colombia   \n",
       "5       Wagtail Space NL               Arnhem, The Netherlands   \n",
       "\n",
       "                      time  \n",
       "0    22 May – 25 May  2024  \n",
       "1    27 May – 28 May  2024  \n",
       "2    29 May – 30 May  2024  \n",
       "3  05 June – 09 June  2024  \n",
       "4  07 June – 09 June  2024  \n",
       "5  12 June – 14 June  2024  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# GET HTTP request using requests library\n",
    "response = requests.get(url)\n",
    "response_text = response.text\n",
    "\n",
    "events_df = get_upcoming_events(response_text, css_class_dict)\n",
    "events_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ccac2e9",
   "metadata": {},
   "source": [
    "### Using `urllib3` library\n",
    "This is another common library for retrieving data from URLs and for other functions involving URLs such as parsing of the parts of the actual URL and handling various encodings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bd54148f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import urllib3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e82415f6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>location</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PyCon Italia 2024</td>\n",
       "      <td>Metropolitan City of Florence, Italy</td>\n",
       "      <td>22 May – 25 May  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GeoPython 2024</td>\n",
       "      <td>Basel, Switzerland</td>\n",
       "      <td>27 May – 28 May  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>csv,conf,v8</td>\n",
       "      <td>Puebla, Mexico</td>\n",
       "      <td>29 May – 30 May  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DjangoCon Europe 2024</td>\n",
       "      <td>Vigo, Spain</td>\n",
       "      <td>05 June – 09 June  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PyCon Colombia 2024</td>\n",
       "      <td>Medellín, Colombia</td>\n",
       "      <td>07 June – 09 June  2024</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Wagtail Space NL</td>\n",
       "      <td>Arnhem, The Netherlands</td>\n",
       "      <td>12 June – 14 June  2024</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    name                              location  \\\n",
       "0      PyCon Italia 2024  Metropolitan City of Florence, Italy   \n",
       "1         GeoPython 2024                    Basel, Switzerland   \n",
       "2            csv,conf,v8                        Puebla, Mexico   \n",
       "3  DjangoCon Europe 2024                           Vigo, Spain   \n",
       "4    PyCon Colombia 2024                    Medellín, Colombia   \n",
       "5       Wagtail Space NL               Arnhem, The Netherlands   \n",
       "\n",
       "                      time  \n",
       "0    22 May – 25 May  2024  \n",
       "1    27 May – 28 May  2024  \n",
       "2    29 May – 30 May  2024  \n",
       "3  05 June – 09 June  2024  \n",
       "4  07 June – 09 June  2024  \n",
       "5  12 June – 14 June  2024  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# urllib3 doesn't apply header encoding automatically\n",
    "pool_manager = urllib3.PoolManager()\n",
    "\n",
    "# GET HTTP request using urllib3 library\n",
    "response = pool_manager.request('GET', url)\n",
    "response_text = response.data\n",
    "\n",
    "events_df = get_upcoming_events(response_text, css_class_dict)\n",
    "events_df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33c3ed4a",
   "metadata": {},
   "source": [
    "`requests` and `urllib3` are very similar in terms of capabilities. It is generally recommended to use Requests when it comes to making HTTP requests. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0390f01a",
   "metadata": {},
   "source": [
    "## Session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f374c236",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"args\": {}, \n",
      "  \"headers\": {\n",
      "    \"Accept\": \"*/*\", \n",
      "    \"Accept-Encoding\": \"gzip, deflate\", \n",
      "    \"Cookie\": \"my-cookie=browser\", \n",
      "    \"Host\": \"httpbin.org\", \n",
      "    \"User-Agent\": \"python-requests/2.28.1\", \n",
      "    \"X-Amzn-Trace-Id\": \"Root=1-6647cfee-64797cb578086cd3274122d3\"\n",
      "  }, \n",
      "  \"origin\": \"96.3.185.160\", \n",
      "  \"url\": \"http://httpbin.org/get\"\n",
      "}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# builds on top of urllib3's connection pooling\n",
    "\n",
    "# session reuses the same TCP connection if requests are made to the same host\n",
    "session = requests.Session()\n",
    "\n",
    "r = session.get('http://httpbin.org/get', cookies={'my-cookie': 'browser'})\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d6e4067",
   "metadata": {},
   "source": [
    "## Streaming"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "776093b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'url': 'http://httpbin.org/stream/5', 'args': {}, 'headers': {'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6647cfee-39327e9270aa47ea73d69c77', 'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*'}, 'origin': '96.3.185.160', 'id': 0}\n",
      "{'url': 'http://httpbin.org/stream/5', 'args': {}, 'headers': {'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6647cfee-39327e9270aa47ea73d69c77', 'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*'}, 'origin': '96.3.185.160', 'id': 1}\n",
      "{'url': 'http://httpbin.org/stream/5', 'args': {}, 'headers': {'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6647cfee-39327e9270aa47ea73d69c77', 'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*'}, 'origin': '96.3.185.160', 'id': 2}\n",
      "{'url': 'http://httpbin.org/stream/5', 'args': {}, 'headers': {'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6647cfee-39327e9270aa47ea73d69c77', 'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*'}, 'origin': '96.3.185.160', 'id': 3}\n",
      "{'url': 'http://httpbin.org/stream/5', 'args': {}, 'headers': {'Host': 'httpbin.org', 'X-Amzn-Trace-Id': 'Root=1-6647cfee-39327e9270aa47ea73d69c77', 'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*'}, 'origin': '96.3.185.160', 'id': 4}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "# Streaming is another nifty feature\n",
    "r = requests.get('http://httpbin.org/stream/5', stream=True)\n",
    "\n",
    "for line in r.iter_lines():\n",
    "    # filter out keep-alive new lines\n",
    "    if line:\n",
    "        decoded_line = line.decode('utf-8')\n",
    "        print(json.loads(decoded_line))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04799b4d",
   "metadata": {},
   "source": [
    "## Scraping with Selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "02631141",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.chrome.service import Service\n",
    "from webdriver_manager.chrome import ChromeDriverManager\n",
    "\n",
    "def get_upcoming_events_with_Selenium(url):\n",
    "    # Configure Chrome options\n",
    "    options = webdriver.ChromeOptions()\n",
    "    options.add_argument('--ignore-certificate-errors')\n",
    "    options.add_argument(\"start-maximized\")\n",
    "    options.add_argument(\"disable-infobars\")\n",
    "    options.add_argument(\"--disable-extensions\")\n",
    "    \n",
    "    # Provide the path to the ChromeDriver executable\n",
    "    driver_path = \"F:/chrome-win64/chromedriver.exe\"\n",
    "    driver = webdriver.Chrome(service=Service(driver_path), options=options)    \n",
    "    driver.maximize_window()\n",
    "    \n",
    "    # Navigate to the URL\n",
    "    driver.get(url)\n",
    "    \n",
    "    try:\n",
    "        # Scrape event details\n",
    "        upcoming_events = []\n",
    "        events = driver.find_elements(\"xpath\",'//ul[contains(@class, \"list-recent-events\")]/li')\n",
    "        for event in events:\n",
    "            event_details = {}\n",
    "            event_details['name'] = event.find_element(\"xpath\",'h3[@class=\"event-title\"]/a').text\n",
    "            event_details['location'] = event.find_element(\"xpath\",'p/span[@class=\"event-location\"]').text\n",
    "            event_details['time'] = event.find_element(\"xpath\",'p/time').text\n",
    "            upcoming_events.append(event_details)\n",
    "        \n",
    "        # Close the WebDriver\n",
    "        driver.quit()\n",
    "        \n",
    "        # Create a DataFrame from the scraped data\n",
    "        events_df = pd.DataFrame(upcoming_events)\n",
    "        return events_df\n",
    "    \n",
    "    except Exception as e:\n",
    "        print(\"An error occurred:\", e)\n",
    "        driver.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "d6e78517",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>location</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>PyCon Italia 2024</td>\n",
       "      <td>Metropolitan City of Florence, Italy</td>\n",
       "      <td>22 May – 25 May</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GeoPython 2024</td>\n",
       "      <td>Basel, Switzerland</td>\n",
       "      <td>27 May – 28 May</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>csv,conf,v8</td>\n",
       "      <td>Puebla, Mexico</td>\n",
       "      <td>29 May – 30 May</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>DjangoCon Europe 2024</td>\n",
       "      <td>Vigo, Spain</td>\n",
       "      <td>05 June – 09 June</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>PyCon Colombia 2024</td>\n",
       "      <td>Medellín, Colombia</td>\n",
       "      <td>07 June – 09 June</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Wagtail Space NL</td>\n",
       "      <td>Arnhem, The Netherlands</td>\n",
       "      <td>12 June – 14 June</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Flask Con 2024</td>\n",
       "      <td>Pittsburgh, USA</td>\n",
       "      <td>17 May</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>PyGrunn 2024</td>\n",
       "      <td>Groningen, Netherlands</td>\n",
       "      <td>17 May</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    name                              location  \\\n",
       "0      PyCon Italia 2024  Metropolitan City of Florence, Italy   \n",
       "1         GeoPython 2024                    Basel, Switzerland   \n",
       "2            csv,conf,v8                        Puebla, Mexico   \n",
       "3  DjangoCon Europe 2024                           Vigo, Spain   \n",
       "4    PyCon Colombia 2024                    Medellín, Colombia   \n",
       "5       Wagtail Space NL               Arnhem, The Netherlands   \n",
       "6         Flask Con 2024                       Pittsburgh, USA   \n",
       "7           PyGrunn 2024                Groningen, Netherlands   \n",
       "\n",
       "                time  \n",
       "0    22 May – 25 May  \n",
       "1    27 May – 28 May  \n",
       "2    29 May – 30 May  \n",
       "3  05 June – 09 June  \n",
       "4  07 June – 09 June  \n",
       "5  12 June – 14 June  \n",
       "6             17 May  \n",
       "7             17 May  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = 'https://www.python.org/events/python-events/'\n",
    "events = get_upcoming_events_with_Selenium(url)\n",
    "events"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d5c86c2",
   "metadata": {},
   "source": [
    "## DOM\n",
    "\n",
    "When the browser displays a web page it builds a model of the content of the page in a representation known as the **document object model** (**DOM**). The DOM is a hierarchical representation of the page's entire content, as well as structural information, style information, scripts, and links to other content."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6137ee78",
   "metadata": {},
   "source": [
    "# Scrapy Fundamentals"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6684050b",
   "metadata": {},
   "source": [
    "Scrapy is a very popular open source Python scraping framework for extracting data. It was originally designed for only scraping, but it is has also evolved into a powerful web crawling solution. Scrapy offers a number of powerful features:\n",
    "\n",
    "* Built-in extensions to make HTTP requests and handle compression, authentication, caching, manipulate user-agents, and HTTP headers\n",
    "\n",
    "* Built-in support for selecting and extracting data with selector languages such as CSS and XPath, as well as support for utilizing regular expressions for selection of content and links\n",
    "\n",
    "* Encoding support to deal with languages and non-standard encoding declarations\n",
    "\n",
    "* Flexible APIs to reuse and write custom middleware and pipelines, which provide a clean and easy way to implement tasks such as automatically downloading assets (for example, images or media) and storing data in storage such as file systems, S3, databases, and others."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50b6f4b0",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d873139f",
   "metadata": {},
   "source": [
    "* https://github.com/PacktPublishing/Python-Web-Scraping-Cookbook/tree/master"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "730c7d5b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "affe2251",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}