notebook/onion/2-Acquire.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Acquire the Data\n",
    "\n",
    "\n",
    "## Finding Data Sources\n",
    "\n",
    "There are three place to get onion price and quantity information by market. \n",
    "\n",
    "1. **[Agmarket](http://agmarknet.nic.in/)** - This is the website run by the Directorate of Marketing & Inspection (DMI), Ministry of Agriculture, Government of India and provides daily price and arrival data for all agricultural commodities at national and state level. Unfortunately, the link to get Market-wise Daily Report for Specific Commodity (Onion for us) leads to a multipage aspx entry form to get data for each date. So it is like to require an involved scraper to get the data. Too much effort - Move on. Here is the best link to go to get what is available - http://agmarknet.nic.in/agnew/NationalBEnglish/SpecificCommodityWeeklyReport.aspx?ss=1\n",
    "\n",
    "\n",
    "2. **[Data.gov.in](https://data.gov.in/)** - This is normally a good place to get government data in a machine readable form like csv or xml. The Variety-wise Daily Market Prices Data of Onion is available for each year as an XML but unfortunately it does not include quantity information that is needed. It would be good to have both price and quantity - so even though this is easy, lets see if we can get both from a different source. Here is the best link to go to get what is available - https://data.gov.in/catalog/variety-wise-daily-market-prices-data-onion#web_catalog_tabs_block_10\n",
    "\n",
    "\n",
    "3. **[NHRDF](http://nhrdf.org/en-us/)** - This is the website of National Horticultural Research & Development Foundation and maintains a database on Market Arrivals and Price, Area and Production and Export Data for three commodities - Garlic, Onion and Potatoes. We are in luck! It also has data from 1996 onwards and has only got one form to fill to get the data in a tabular form. Further it also has production and export data. Excellent. Lets use this. Here is the best link to got to get all that is available - http://nhrdf.org/en-us/DatabaseReports\n",
    "\n",
    "\n",
    "## Scraping the Data\n",
    "\n",
    "\n",
    "### Ways to Scrape Data\n",
    "Now we can do this in two different levels of sophistication\n",
    "\n",
    "1. **Automate the form filling process**: The form on this page looks simple. But viewing source in the browser shows there form to fill with hidden fields and we will need to access it as a browser to get the session fields and then submit the form. This is a little bit more complicated than simple scraping a table on a webpage\n",
    "\n",
    "2. **Manually fill the form**: What if we manually fill the form with the desired form fields and then save the page as a html file. Then we can read this file and just scrape the table from it. Lets go with the simple way for now.\n",
    "\n",
    "\n",
    "### Scraping - Manual Form Filling\n",
    "\n",
    "So let us fill the form to get a small subset of data and test our scraping process.  We will start by getting the [Monthwise Market Arrivals](http://nhrdf.org/en-us/MonthWiseMarketArrivals). \n",
    "\n",
    "- Crop Name: Onion\n",
    "- Month: January\n",
    "- Market: All\n",
    "- Year: 2016\n",
    "\n",
    "The saved webpage is available at [MonthWiseMarketArrivalsJan2016.html](MonthWiseMarketArrivalsJan2016.html)\n",
    "\n",
    "### Understand the HTML Structure\n",
    "\n",
    "We need to scrape data from this html page... So let us try to understand the structure of the page.\n",
    "\n",
    "1. You can view the source of the page - typically Right Click and View Source on any browser and that would give your the source HTML for any page.\n",
    "\n",
    "2. You can open the developer tools in your browser and investigate the structure as you mouse over the page \n",
    "\n",
    "3. We can use a tools like [Selector Gadget](http://selectorgadget.com/) to understand the id's and classes' used in the web page\n",
    "\n",
    "Our data is under the **&lt;table&gt;** tag "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the number of tables in the HTML Structure of [MonthWiseMarketArrivalsJan2016.html](MonthWiseMarketArrivalsJan2016.html)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Find all the Tables "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Import the library we need, which is Pandas\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read all the tables from the html document \n",
    "AllTables = pd.read_html('MonthWiseMarketArrivals2017.htm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "5"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let us find out how many tables has it found?\n",
    "len(AllTables)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(AllTables)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #2\n",
    "Find the exact table of data we want in the list of AllTables?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the exact table\n",
    "To read the exact table we need to pass in an identifier value which would identify the table. We can use the `attrs` parameter in read_html to do so. The parameter we will pass is the `id` variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# So can we read our exact table\n",
    "OneTable = pd.read_html('MonthWiseMarketArrivals2017.htm', \n",
    "                      attrs = {'id' : 'dnn_ctr974_MonthWiseMarketArrivals_GridView1'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# So how many tables have we got now\n",
    "len(OneTable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Market</td>\n",
       "      <td>Month Name</td>\n",
       "      <td>Year</td>\n",
       "      <td>Arrival (q)</td>\n",
       "      <td>Price Minimum (Rs/q)</td>\n",
       "      <td>Price Maximum (Rs/q)</td>\n",
       "      <td>Modal Price (Rs/q)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>ABOHAR(PB)</td>\n",
       "      <td>January</td>\n",
       "      <td>2017</td>\n",
       "      <td>200</td>\n",
       "      <td>750</td>\n",
       "      <td>1000</td>\n",
       "      <td>850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2017</td>\n",
       "      <td>69360</td>\n",
       "      <td>730</td>\n",
       "      <td>871</td>\n",
       "      <td>815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>February</td>\n",
       "      <td>2017</td>\n",
       "      <td>58850</td>\n",
       "      <td>767</td>\n",
       "      <td>918</td>\n",
       "      <td>872</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>March</td>\n",
       "      <td>2017</td>\n",
       "      <td>76370</td>\n",
       "      <td>790</td>\n",
       "      <td>910</td>\n",
       "      <td>855</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            0           1     2            3                     4  \\\n",
       "0      Market  Month Name  Year  Arrival (q)  Price Minimum (Rs/q)   \n",
       "1  ABOHAR(PB)     January  2017          200                   750   \n",
       "2    AGRA(UP)     January  2017        69360                   730   \n",
       "3    AGRA(UP)    February  2017        58850                   767   \n",
       "4    AGRA(UP)       March  2017        76370                   790   \n",
       "\n",
       "                      5                   6  \n",
       "0  Price Maximum (Rs/q)  Modal Price (Rs/q)  \n",
       "1                  1000                 850  \n",
       "2                   871                 815  \n",
       "3                   918                 872  \n",
       "4                   910                 855  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the table of data identifed by pandas with just the first five rows\n",
    "OneTable[0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, we have not got the header correctly in our dataframe. Let us see if we can fix this.\n",
    "\n",
    "To get help on any function just use `??` before the function to help. Run this function and see what additional parameter you need to define to get the header correctly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "??pd.read_html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise #3\n",
    "Read the html file again and ensure that the correct header is identifed by pandas?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "OneTable = pd.read_html('MonthWiseMarketArrivalsJan2016.html', header= 0,\n",
    "                      attrs = {'id' : 'dnn_ctr974_MonthWiseMarketArrivals_GridView1'})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the top five rows of the dataframe you have read to ensure the headers are now correct."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "      <th>Arrival (q)</th>\n",
       "      <th>Price Minimum (Rs/q)</th>\n",
       "      <th>Price Maximum (Rs/q)</th>\n",
       "      <th>Modal Price (Rs/q)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>134200</td>\n",
       "      <td>1039</td>\n",
       "      <td>1443</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AHMEDABAD(GUJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>198390</td>\n",
       "      <td>646</td>\n",
       "      <td>1224</td>\n",
       "      <td>997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AHMEDNAGAR(MS)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>208751</td>\n",
       "      <td>175</td>\n",
       "      <td>1722</td>\n",
       "      <td>1138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AJMER(RAJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>4247</td>\n",
       "      <td>722</td>\n",
       "      <td>1067</td>\n",
       "      <td>939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALIGARH(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>12350</td>\n",
       "      <td>1219</td>\n",
       "      <td>1298</td>\n",
       "      <td>1257</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Market Month Name  Year  Arrival (q) Price Minimum (Rs/q)  \\\n",
       "0        AGRA(UP)    January  2016       134200                 1039   \n",
       "1  AHMEDABAD(GUJ)    January  2016       198390                  646   \n",
       "2  AHMEDNAGAR(MS)    January  2016       208751                  175   \n",
       "3      AJMER(RAJ)    January  2016         4247                  722   \n",
       "4     ALIGARH(UP)    January  2016        12350                 1219   \n",
       "\n",
       "  Price Maximum (Rs/q) Modal Price (Rs/q)  \n",
       "0                 1443               1349  \n",
       "1                 1224                997  \n",
       "2                 1722               1138  \n",
       "3                 1067                939  \n",
       "4                 1298               1257  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "OneTable[0].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "###  Dataframe Viewing  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Let us store the dataframe in a df variable. You will see that as a very common convention in data science pandas use\n",
    "df = OneTable[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(84, 7)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Shape of the dateset - number of rows & number of columns in the dataframe\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Market', 'Month Name', 'Year', 'Arrival (q)', 'Price Minimum (Rs/q)',\n",
       "       'Price Maximum (Rs/q)', 'Modal Price (Rs/q)'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Get the names of all the columns \n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RangeIndex(start=0, stop=84, step=1)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "      <th>Arrival (q)</th>\n",
       "      <th>Price Minimum (Rs/q)</th>\n",
       "      <th>Price Maximum (Rs/q)</th>\n",
       "      <th>Modal Price (Rs/q)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>134200</td>\n",
       "      <td>1039</td>\n",
       "      <td>1443</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AHMEDABAD(GUJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>198390</td>\n",
       "      <td>646</td>\n",
       "      <td>1224</td>\n",
       "      <td>997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AHMEDNAGAR(MS)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>208751</td>\n",
       "      <td>175</td>\n",
       "      <td>1722</td>\n",
       "      <td>1138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AJMER(RAJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>4247</td>\n",
       "      <td>722</td>\n",
       "      <td>1067</td>\n",
       "      <td>939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALIGARH(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>12350</td>\n",
       "      <td>1219</td>\n",
       "      <td>1298</td>\n",
       "      <td>1257</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ALWAR(RAJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>9788</td>\n",
       "      <td>625</td>\n",
       "      <td>1200</td>\n",
       "      <td>912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>AMRITSAR(PB)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>24800</td>\n",
       "      <td>913</td>\n",
       "      <td>1308</td>\n",
       "      <td>1160</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>BALLIA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>600</td>\n",
       "      <td>1400</td>\n",
       "      <td>1500</td>\n",
       "      <td>1460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>BANGALORE</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>507223</td>\n",
       "      <td>200</td>\n",
       "      <td>1943</td>\n",
       "      <td>1448</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>BAREILLY(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>18435</td>\n",
       "      <td>1149</td>\n",
       "      <td>1149</td>\n",
       "      <td>1149</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Market Month Name  Year  Arrival (q) Price Minimum (Rs/q)  \\\n",
       "0        AGRA(UP)    January  2016       134200                 1039   \n",
       "1  AHMEDABAD(GUJ)    January  2016       198390                  646   \n",
       "2  AHMEDNAGAR(MS)    January  2016       208751                  175   \n",
       "3      AJMER(RAJ)    January  2016         4247                  722   \n",
       "4     ALIGARH(UP)    January  2016        12350                 1219   \n",
       "5      ALWAR(RAJ)    January  2016         9788                  625   \n",
       "6    AMRITSAR(PB)    January  2016        24800                  913   \n",
       "7      BALLIA(UP)    January  2016          600                 1400   \n",
       "8       BANGALORE    January  2016       507223                  200   \n",
       "9    BAREILLY(UP)    January  2016        18435                 1149   \n",
       "\n",
       "  Price Maximum (Rs/q) Modal Price (Rs/q)  \n",
       "0                 1443               1349  \n",
       "1                 1224                997  \n",
       "2                 1722               1138  \n",
       "3                 1067                939  \n",
       "4                 1298               1257  \n",
       "5                 1200                912  \n",
       "6                 1308               1160  \n",
       "7                 1500               1460  \n",
       "8                 1943               1448  \n",
       "9                 1149               1149  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can we see sample rows - the top 5 rows\n",
    "df.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "      <th>Arrival (q)</th>\n",
       "      <th>Price Minimum (Rs/q)</th>\n",
       "      <th>Price Maximum (Rs/q)</th>\n",
       "      <th>Modal Price (Rs/q)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>UDAIPUR(RAJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>6456</td>\n",
       "      <td>386</td>\n",
       "      <td>1307</td>\n",
       "      <td>846</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>VANI(MS)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>60983</td>\n",
       "      <td>767</td>\n",
       "      <td>1323</td>\n",
       "      <td>1007</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>VARANASI(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>28900</td>\n",
       "      <td>1460</td>\n",
       "      <td>1503</td>\n",
       "      <td>1484</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>YEOLA(MS)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>437432</td>\n",
       "      <td>437</td>\n",
       "      <td>1272</td>\n",
       "      <td>1034</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Total</td>\n",
       "      <td>9307923</td>\n",
       "      <td>751(Avg)</td>\n",
       "      <td>1490(Avg)</td>\n",
       "      <td>1186(Avg)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Market Month Name   Year  Arrival (q) Price Minimum (Rs/q)  \\\n",
       "79  UDAIPUR(RAJ)    January   2016         6456                  386   \n",
       "80      VANI(MS)    January   2016        60983                  767   \n",
       "81  VARANASI(UP)    January   2016        28900                 1460   \n",
       "82     YEOLA(MS)    January   2016       437432                  437   \n",
       "83           NaN        NaN  Total      9307923             751(Avg)   \n",
       "\n",
       "   Price Maximum (Rs/q) Modal Price (Rs/q)  \n",
       "79                 1307                846  \n",
       "80                 1323               1007  \n",
       "81                 1503               1484  \n",
       "82                 1272               1034  \n",
       "83            1490(Avg)          1186(Avg)  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can we see sample rows - the bottom 5 rows\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          AGRA(UP)\n",
       "1    AHMEDABAD(GUJ)\n",
       "2    AHMEDNAGAR(MS)\n",
       "3        AJMER(RAJ)\n",
       "4       ALIGARH(UP)\n",
       "Name: Market, dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Can we access a specific columns\n",
    "df[\"Market\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          AGRA(UP)\n",
       "1    AHMEDABAD(GUJ)\n",
       "2    AHMEDNAGAR(MS)\n",
       "3        AJMER(RAJ)\n",
       "4       ALIGARH(UP)\n",
       "Name: Market, dtype: object"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Using the dot notation\n",
    "df.Market.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "      <th>Arrival (q)</th>\n",
       "      <th>Price Minimum (Rs/q)</th>\n",
       "      <th>Price Maximum (Rs/q)</th>\n",
       "      <th>Modal Price (Rs/q)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>134200</td>\n",
       "      <td>1039</td>\n",
       "      <td>1443</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AHMEDABAD(GUJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>198390</td>\n",
       "      <td>646</td>\n",
       "      <td>1224</td>\n",
       "      <td>997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AHMEDNAGAR(MS)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>208751</td>\n",
       "      <td>175</td>\n",
       "      <td>1722</td>\n",
       "      <td>1138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AJMER(RAJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>4247</td>\n",
       "      <td>722</td>\n",
       "      <td>1067</td>\n",
       "      <td>939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALIGARH(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>12350</td>\n",
       "      <td>1219</td>\n",
       "      <td>1298</td>\n",
       "      <td>1257</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Market Month Name  Year  Arrival (q) Price Minimum (Rs/q)  \\\n",
       "0        AGRA(UP)    January  2016       134200                 1039   \n",
       "1  AHMEDABAD(GUJ)    January  2016       198390                  646   \n",
       "2  AHMEDNAGAR(MS)    January  2016       208751                  175   \n",
       "3      AJMER(RAJ)    January  2016         4247                  722   \n",
       "4     ALIGARH(UP)    January  2016        12350                 1219   \n",
       "\n",
       "  Price Maximum (Rs/q) Modal Price (Rs/q)  \n",
       "0                 1443               1349  \n",
       "1                 1224                997  \n",
       "2                 1722               1138  \n",
       "3                 1067                939  \n",
       "4                 1298               1257  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AHMEDABAD(GUJ)</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AHMEDNAGAR(MS)</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AJMER(RAJ)</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALIGARH(UP)</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Market  Year\n",
       "0        AGRA(UP)  2016\n",
       "1  AHMEDABAD(GUJ)  2016\n",
       "2  AHMEDNAGAR(MS)  2016\n",
       "3      AJMER(RAJ)  2016\n",
       "4     ALIGARH(UP)  2016"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Selecting specific column and rows\n",
    "df[0:5][[\"Market\", \"Year\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing Data\n",
    "- By direct labels\n",
    "- By location - index\n",
    "- By location - labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          AGRA(UP)\n",
       "1    AHMEDABAD(GUJ)\n",
       "2    AHMEDNAGAR(MS)\n",
       "3        AJMER(RAJ)\n",
       "4       ALIGARH(UP)\n",
       "Name: Market, dtype: object"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[\"Market\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "      <th>Arrival (q)</th>\n",
       "      <th>Price Minimum (Rs/q)</th>\n",
       "      <th>Price Maximum (Rs/q)</th>\n",
       "      <th>Modal Price (Rs/q)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>134200</td>\n",
       "      <td>1039</td>\n",
       "      <td>1443</td>\n",
       "      <td>1349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>AHMEDABAD(GUJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>198390</td>\n",
       "      <td>646</td>\n",
       "      <td>1224</td>\n",
       "      <td>997</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>AHMEDNAGAR(MS)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>208751</td>\n",
       "      <td>175</td>\n",
       "      <td>1722</td>\n",
       "      <td>1138</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>AJMER(RAJ)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>4247</td>\n",
       "      <td>722</td>\n",
       "      <td>1067</td>\n",
       "      <td>939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>ALIGARH(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "      <td>12350</td>\n",
       "      <td>1219</td>\n",
       "      <td>1298</td>\n",
       "      <td>1257</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Market Month Name  Year  Arrival (q) Price Minimum (Rs/q)  \\\n",
       "0        AGRA(UP)    January  2016       134200                 1039   \n",
       "1  AHMEDABAD(GUJ)    January  2016       198390                  646   \n",
       "2  AHMEDNAGAR(MS)    January  2016       208751                  175   \n",
       "3      AJMER(RAJ)    January  2016         4247                  722   \n",
       "4     ALIGARH(UP)    January  2016        12350                 1219   \n",
       "\n",
       "  Price Maximum (Rs/q) Modal Price (Rs/q)  \n",
       "0                 1443               1349  \n",
       "1                 1224                997  \n",
       "2                 1722               1138  \n",
       "3                 1067                939  \n",
       "4                 1298               1257  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2016\n",
       "1    2016\n",
       "2    2016\n",
       "3    2016\n",
       "4    2016\n",
       "5    2016\n",
       "Name: Year, dtype: object"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.loc[0:5,\"Year\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Market</th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>AGRA(UP)</td>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     Market Month Name  Year\n",
       "0  AGRA(UP)    January  2016"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[0:4,0:3].head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['2016', 'Total'], dtype=object)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Year.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 84 entries, 0 to 83\n",
      "Data columns (total 7 columns):\n",
      "Market                  83 non-null object\n",
      "Month Name              83 non-null object\n",
      "Year                    84 non-null object\n",
      "Arrival (q)             84 non-null int64\n",
      "Price Minimum (Rs/q)    84 non-null object\n",
      "Price Maximum (Rs/q)    84 non-null object\n",
      "Modal Price (Rs/q)      84 non-null object\n",
      "dtypes: int64(1), object(6)\n",
      "memory usage: 4.7+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Arrival (q)</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>8.400000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>2.216172e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.017445e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.000000e+01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.140450e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.593000e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.379715e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>9.307923e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Arrival (q)\n",
       "count  8.400000e+01\n",
       "mean   2.216172e+05\n",
       "std    1.017445e+06\n",
       "min    4.000000e+01\n",
       "25%    1.140450e+04\n",
       "50%    3.593000e+04\n",
       "75%    1.379715e+05\n",
       "max    9.307923e+06"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "x = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'iloc' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m-----------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-30-262e0ff8c8d9>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Access 10 - 15 row, Year & Arrival columns\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0;34m-\u001b[0m \u001b[0miloc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0mloc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'iloc' is not defined"
     ]
    }
   ],
   "source": [
    "# Access 10 - 15 row, Year & Arrival columns\n",
    "- iloc\n",
    "- loc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Month Name</th>\n",
       "      <th>Year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>January</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Month Name  Year\n",
       "10    January  2016\n",
       "11    January  2016\n",
       "12    January  2016\n",
       "13    January  2016\n",
       "14    January  2016"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.iloc[10:15, 1:3 ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0          AGRA(UP)\n",
       "1    AHMEDABAD(GUJ)\n",
       "2    AHMEDNAGAR(MS)\n",
       "3        AJMER(RAJ)\n",
       "4       ALIGARH(UP)\n",
       "Name: Market, dtype: object"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Works both ways\n",
    "df[\"Market\"][0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['AGRA(UP)', 'AHMEDABAD(GUJ)', 'AHMEDNAGAR(MS)', 'AJMER(RAJ)',\n",
       "       'ALIGARH(UP)', 'ALWAR(RAJ)', 'AMRITSAR(PB)', 'BALLIA(UP)',\n",
       "       'BANGALORE', 'BAREILLY(UP)', 'BELGAUM(KNT)', 'BHATINDA(PB)',\n",
       "       'BHAVNAGAR(GUJ)', 'BHUBNESWER(OR)', 'BIJAPUR(KNT)', 'BURDWAN(WB)',\n",
       "       'CHAKAN(MS)', 'CHANDIGARH', 'CHANDVAD(MS)', 'CHENNAI', 'DEESA(GUJ)',\n",
       "       'DEHRADOON(UTT)', 'DELHI', 'DEVALA(MS)', 'DHAVANGERE(KNT)',\n",
       "       'DHULIA(MS)', 'GONDAL(GUJ)', 'GUWAHATI', 'HASSAN(KNT)',\n",
       "       'HOSHIARPUR(PB)', 'HUBLI(KNT)', 'HYDERABAD', 'INDORE(MP)', 'JAIPUR',\n",
       "       'JALANDHAR(PB)', 'JALGAON(MS)', 'JAMMU', 'JAMNAGAR(GUJ)',\n",
       "       'JODHPUR(RAJ)', 'KALVAN(MS)', 'KANPUR(UP)', 'KARNAL(HR)',\n",
       "       'KHANNA(PB)', 'KOLHAPUR(MS)', 'KOLKATA', 'KOPERGAON(MS)',\n",
       "       'KOTA(RAJ)', 'KURNOOL(AP)', 'LASALGAON(MS)', 'LONAND(MS)',\n",
       "       'LUCKNOW', 'MAHUVA(GUJ)', 'MALEGAON(MS)', 'MANMAD(MS)', 'MUMBAI',\n",
       "       'NAGPUR', 'NEWASA(MS)', 'NIPHAD(MS)', 'PALAYAM(KER)', 'PATIALA(PB)',\n",
       "       'PATNA', 'PHALTAN (MS)', 'PIMPALGAON(MS)', 'PUNE(MS)',\n",
       "       'PURULIA(WB)', 'RAHATA(MS)', 'RAHURI(MS)', 'RAICHUR(KNT)',\n",
       "       'RAIPUR(CHGARH)', 'RAJKOT(GUJ)', 'SAGAR(MP)', 'SAIKHEDA(MS)',\n",
       "       'SANGALI(MS)', 'SANGAMNER(MS)', 'SATANA(MS)', 'SHRIRAMPUR(MS)',\n",
       "       'SINNAR(MS)', 'SOLAPUR(MS)', 'SURAT(GUJ)', 'UDAIPUR(RAJ)',\n",
       "       'VANI(MS)', 'VARANASI(UP)', 'YEOLA(MS)', nan], dtype=object)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Getting unique values of State\n",
    "pd.unique(df['Market'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.frame.DataFrame"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df.Market)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Market                  object\n",
       "Month Name              object\n",
       "Year                    object\n",
       "Arrival (q)              int64\n",
       "Price Minimum (Rs/q)    object\n",
       "Price Maximum (Rs/q)    object\n",
       "Modal Price (Rs/q)      object\n",
       "dtype: object"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['AGRA(UP)', 'AHMEDABAD(GUJ)', 'AHMEDNAGAR(MS)', 'AJMER(RAJ)',\n",
       "       'ALIGARH(UP)', 'ALWAR(RAJ)', 'AMRITSAR(PB)', 'BALLIA(UP)',\n",
       "       'BANGALORE', 'BAREILLY(UP)', 'BELGAUM(KNT)', 'BHATINDA(PB)',\n",
       "       'BHAVNAGAR(GUJ)', 'BHUBNESWER(OR)', 'BIJAPUR(KNT)', 'BURDWAN(WB)',\n",
       "       'CHAKAN(MS)', 'CHANDIGARH', 'CHANDVAD(MS)', 'CHENNAI', 'DEESA(GUJ)',\n",
       "       'DEHRADOON(UTT)', 'DELHI', 'DEVALA(MS)', 'DHAVANGERE(KNT)',\n",
       "       'DHULIA(MS)', 'GONDAL(GUJ)', 'GUWAHATI', 'HASSAN(KNT)',\n",
       "       'HOSHIARPUR(PB)', 'HUBLI(KNT)', 'HYDERABAD', 'INDORE(MP)', 'JAIPUR',\n",
       "       'JALANDHAR(PB)', 'JALGAON(MS)', 'JAMMU', 'JAMNAGAR(GUJ)',\n",
       "       'JODHPUR(RAJ)', 'KALVAN(MS)', 'KANPUR(UP)', 'KARNAL(HR)',\n",
       "       'KHANNA(PB)', 'KOLHAPUR(MS)', 'KOLKATA', 'KOPERGAON(MS)',\n",
       "       'KOTA(RAJ)', 'KURNOOL(AP)', 'LASALGAON(MS)', 'LONAND(MS)',\n",
       "       'LUCKNOW', 'MAHUVA(GUJ)', 'MALEGAON(MS)', 'MANMAD(MS)', 'MUMBAI',\n",
       "       'NAGPUR', 'NEWASA(MS)', 'NIPHAD(MS)', 'PALAYAM(KER)', 'PATIALA(PB)',\n",
       "       'PATNA', 'PHALTAN (MS)', 'PIMPALGAON(MS)', 'PUNE(MS)',\n",
       "       'PURULIA(WB)', 'RAHATA(MS)', 'RAHURI(MS)', 'RAICHUR(KNT)',\n",
       "       'RAIPUR(CHGARH)', 'RAJKOT(GUJ)', 'SAGAR(MP)', 'SAIKHEDA(MS)',\n",
       "       'SANGALI(MS)', 'SANGAMNER(MS)', 'SATANA(MS)', 'SHRIRAMPUR(MS)',\n",
       "       'SINNAR(MS)', 'SOLAPUR(MS)', 'SURAT(GUJ)', 'UDAIPUR(RAJ)',\n",
       "       'VANI(MS)', 'VARANASI(UP)', 'YEOLA(MS)', nan], dtype=object)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.Market.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df.Market)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(df.Market.values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Downloading the Entire Month Wise Arrival Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "AllTable = pd.read_html('MonthWiseMarketArrivals.html', header = 0,\n",
    "                      attrs = {'id' : 'dnn_ctr974_MonthWiseMarketArrivals_GridView1'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AllTable[0].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "??pd.DataFrame.to_csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AllTable[0].columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Change the column names to simpler ones\n",
    "AllTable[0].columns = ['market', 'month', 'year', 'quantity', 'priceMin', 'priceMax', 'priceMod']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "AllTable[0].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Save the dataframe to a csv file\n",
    "AllTable[0].to_csv('MonthWiseMarketArrivals.csv', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}