Pandas/Best-Way-to-Iterate-DataFrame.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "311a40eb",
   "metadata": {},
   "source": [
    "# Daily-Dose-of-Data-Science\n",
    "\n",
    "[Daily Dose of Data Science](https://avichawla.substack.com) is a publication on Substack that brings together intriguing frameworks, libraries, technologies, and tips that make the life cycle of a Data Science project effortless. \n",
    "\n",
    "Author: Avi Chawla\n",
    "\n",
    "[Medium](https://medium.com/@avi_chawla) | [LinkedIn](https://www.linkedin.com/in/avi-chawla/)\n",
    "\n",
    "# If You Are Not Able To Code A Vectorized Approach, Try This.\n",
    "\n",
    "Post Link: [Substack](https://avichawla.substack.com/p/if-you-are-not-able-to-code-a-vectorized)\n",
    "\n",
    "LinkedIn Post: [LinkedIn](https://www.linkedin.com/feed/update/urn:li:activity:7040630377685618688/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d851a9ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_parquet(\"../_Datasets/employee_dataset.parquet\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "ce8e10cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.reset_index(drop = True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ce09b65a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Company_Name</th>\n",
       "      <th>Employee_Job_Title</th>\n",
       "      <th>Employee_City</th>\n",
       "      <th>Employee_Country</th>\n",
       "      <th>Employee_Salary</th>\n",
       "      <th>Employment_Status</th>\n",
       "      <th>Employee_Rating</th>\n",
       "      <th>Credits</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Johnny Maynard</td>\n",
       "      <td>White, Mcclain and Cobb</td>\n",
       "      <td>Equities trader</td>\n",
       "      <td>New Cindychester</td>\n",
       "      <td>Mongolia</td>\n",
       "      <td>764340</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>3.3</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Michael Williams</td>\n",
       "      <td>Scott Inc</td>\n",
       "      <td>Patent examiner</td>\n",
       "      <td>Ricardomouth</td>\n",
       "      <td>Guinea-Bissau</td>\n",
       "      <td>929230</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>4.6</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Laura Flynn</td>\n",
       "      <td>Andrade LLC</td>\n",
       "      <td>Equities trader</td>\n",
       "      <td>North Melissafurt</td>\n",
       "      <td>Tonga</td>\n",
       "      <td>606900</td>\n",
       "      <td>Intern</td>\n",
       "      <td>0.3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Stefanie Archer</td>\n",
       "      <td>James and Sons</td>\n",
       "      <td>Retail merchandiser</td>\n",
       "      <td>Ricardomouth</td>\n",
       "      <td>San Marino</td>\n",
       "      <td>775380</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>2.8</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sierra Garcia</td>\n",
       "      <td>Matthews Inc</td>\n",
       "      <td>Armed forces logistics/support/administrative ...</td>\n",
       "      <td>Whitakerbury</td>\n",
       "      <td>Nepal</td>\n",
       "      <td>944480</td>\n",
       "      <td>Full Time</td>\n",
       "      <td>4.5</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               Name             Company_Name  \\\n",
       "0    Johnny Maynard  White, Mcclain and Cobb   \n",
       "1  Michael Williams                Scott Inc   \n",
       "2       Laura Flynn              Andrade LLC   \n",
       "3   Stefanie Archer           James and Sons   \n",
       "4     Sierra Garcia             Matthews Inc   \n",
       "\n",
       "                                  Employee_Job_Title      Employee_City  \\\n",
       "0                                    Equities trader   New Cindychester   \n",
       "1                                    Patent examiner       Ricardomouth   \n",
       "2                                    Equities trader  North Melissafurt   \n",
       "3                                Retail merchandiser       Ricardomouth   \n",
       "4  Armed forces logistics/support/administrative ...       Whitakerbury   \n",
       "\n",
       "  Employee_Country  Employee_Salary Employment_Status  Employee_Rating  \\\n",
       "0         Mongolia           764340         Full Time              3.3   \n",
       "1    Guinea-Bissau           929230         Full Time              4.6   \n",
       "2            Tonga           606900            Intern              0.3   \n",
       "3       San Marino           775380         Full Time              2.8   \n",
       "4            Nepal           944480         Full Time              4.5   \n",
       "\n",
       "   Credits  \n",
       "0        1  \n",
       "1        4  \n",
       "2        3  \n",
       "3        5  \n",
       "4        4  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df[df.index<100000]\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1bebb3c",
   "metadata": {},
   "source": [
    "#### 1) iterrows()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "20d39a3e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.62 s ± 23.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit [row.Employee_Salary*(1 + row.Credits) for index, row in df.iterrows()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12536830",
   "metadata": {},
   "source": [
    "#### 2) apply()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "def14e52",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "901 ms ± 5.06 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df.apply(lambda row: row.Employee_Salary*(1 + row.Credits), axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af90f1be",
   "metadata": {},
   "source": [
    "#### 3) itertuples()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "94413716",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "87.3 ms ± 486 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit [row.Employee_Salary*(1 + row.Credits) for row in df.itertuples()]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c582a717",
   "metadata": {},
   "source": [
    "#### 4) to_numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "6c25f5f5",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "28.3 ms ± 311 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
     ]
    }
   ],
   "source": [
    "%timeit np_arr = df.to_numpy(); [row[5]*(1 + row[-1]) for row in np_arr]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a0a846b",
   "metadata": {},
   "source": [
    "#### 5) to_dict()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "3535a334",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "429 ms ± 5.71 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
     ]
    }
   ],
   "source": [
    "%timeit df_dict = df.to_dict(\"records\"); [row[\"Employee_Salary\"]*(1 + row[\"Credits\"]) for row in df_dict]"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}