weather_datasets.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "weather_datasets.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2WBjL8NChrxi"
      },
      "source": [
        "#PURPOSE:\n",
        "The purpose of this notebook is to download dataframes used for visualizations and analysis. The important dataframes are\n",
        "\n",
        "df_master_weather : The complilation of raw pod measurement, just concatenating them.\n",
        "\n",
        "df_weather_master_outliers_removed: The same as df_weather_master but SMRO4 data is removed.\n",
        "\n",
        "df_complaints: A dataframe that groups the total complaints by hour and are aggregate by sum. \n",
        "\n",
        "df_merged : A dataframe that groups measurements by hour. Weather values are aggregated by average and complaint totals are aggregated by sum.\n",
        "\n",
        "df_model_analysis: a dataframe used for logistic regression analysis\n",
        "\n",
        "The files downloaded will be '.csv' files that can be opened on programs like microsoft excel.\n",
        "\n",
        "Running this notebook will not affect the analyses for any other notebook.\n",
        "\n",
        "This notebook will reflect any updates made to the github repo 'data' and 'weather_data directories'\n",
        "\n",
        "Running this entire notebook should automatically download the files as a csv to your computer. Your browser may prompt you to allow multiple downloads from this notebook - please select allow.\n",
        "\n",
        "If you encounder download issues, you may directly download each file individusally by running the code cells below. Please ensure that this whole notebook has been run at least once before proceeding to download the files indiviudally, if necessary\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ASvUwS7Jiwp_"
      },
      "source": [
        "##***IMPORTANT: If the weather_data/pod_raw_data is updated, please use this notebook to generate the files listed above and replace/update the files in the weather_data/weather_intermediary_files directory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pFFaYGSOVPbR",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "import pandas as pd\n",
        "import shutil\n",
        "import numpy as np \n",
        "import seaborn as sns \n",
        "import glob\n",
        "import os \n",
        "from datetime import *\n",
        "import seaborn as sns\n",
        "import copy\n",
        "import matplotlib.pyplot as plt\n",
        "import pandas as pd\n",
        "from datetime import datetime\n",
        "import plotly.express as px\n",
        "from google.colab import files"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KHnOmMprT1g-",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "#above string is used to hide code block\n",
        "'''\n",
        "Purpose: Adds date_only and time_only columns to dataframe\n",
        "Column values are filled based on 'DateTime Column'\n",
        "This DIRECTLY changes the dataframe, it does NOT return a copy\n",
        "@param df : a dataframe, this will be a file consisting of data\n",
        "  from each pod pooled together\n",
        "'''\n",
        "def add_dates_times_columns(df):\n",
        "  # insert date only, time only, and DateTime Columns\n",
        "  # adding a filler value x or y that will be replaced later\n",
        "  df.insert(1, 'date_only', 'x')\n",
        "  df.insert(2, 'time_only', 'y')\n",
        "  \n",
        "  # change name of Time column to datetime for clarity that it is a datetime\n",
        "  df.rename(columns={'Time' : 'DateTime'}, inplace=True)\n",
        "  #copy string of DateTime to time and date only columns\n",
        "  df['date_only'] = df['DateTime']\n",
        "  df['time_only'] = df['DateTime']\n",
        "\n",
        "  # convert strings to appropriate data type - DateTime, date, and time\n",
        "  df['DateTime'] = pd.to_datetime(df['DateTime'], format=\"%Y-%m-%d %H:%M:%S\")\n",
        "  df['date_only'] = pd.to_datetime(df['date_only']).dt.date\n",
        "  df['time_only'] = pd.to_datetime(df['time_only']).dt.time\n",
        "\n",
        "  # resetting index to make it look neat\n",
        "  df.reset_index(drop=True, inplace=True)\n",
        "\n",
        "def cardinal_direction(df): \n",
        "  condition_list = [(df['Wind Direction'] > 0) & (df['Wind Direction'] < 45), (df['Wind Direction'] == 45), \n",
        "                    (df['Wind Direction'] > 45) & (df['Wind Direction'] <= 90), (df['Wind Direction'] > 90) & (df['Wind Direction'] < 135), (df['Wind Direction'] == 135), \n",
        "                    (df['Wind Direction'] > 135) & (df['Wind Direction'] <= 180), (df['Wind Direction'] > 180) & (df['Wind Direction'] < 225), (df['Wind Direction'] == 225), \n",
        "                    (df['Wind Direction'] > 225) & (df['Wind Direction'] <= 270), (df['Wind Direction'] > 270) & (df['Wind Direction'] < 315), (df['Wind Direction'] == 315), \n",
        "                    (df['Wind Direction'] > 315) & (df['Wind Direction'] <= 360) ]\n",
        "  choicelist = ['North','North-East', 'East', 'East', 'South-East', 'South', 'South', 'South-West', 'West', 'West', 'North-West', 'North']\n",
        "  \n",
        "  df['Cardinal Direction'] = np.select(condition_list, choicelist) #adding the cardinal direction column\n",
        "  direction_values=df['Cardinal Direction'].values #to get the values either as array or a series\n",
        "  \n",
        "  df.drop(['Cardinal Direction'], axis=1, inplace=True) # we need to drop the column to relocate to the position we want\n",
        "  df.insert(15,'Cardinal Direction',value=direction_values)\n",
        "  return df # to return the dataframe when the function is called\\\n",
        "\n",
        "'''\n",
        "Purpose: The get_weather_master function more than just the necessary\n",
        "  weather data and adds them to the google colab local files.\n",
        "  This function will remove unnecssary files in order to save \n",
        "  memory/space in the local colab session.\n",
        "'''\n",
        "def clean_up():\n",
        "  main_dir_path = '/content/stinky/'\n",
        "  list_files = os.listdir(main_dir_path)\n",
        "\n",
        "  for i in list_files:\n",
        "    path = main_dir_path + i\n",
        "    try:\n",
        "      os.remove(path)\n",
        "    except:\n",
        "      pass\n",
        "\n",
        "  try:\n",
        "    shutil.rmtree('/content/stinky/vessels_data_raw', ignore_errors=True)\n",
        "    shutil.rmtree('/content/stinky/weather_data_raw', ignore_errors=True)\n",
        "    shutil.rmtree('/content/stinky/weather_data/weather_intermediary_files',\n",
        "                  ignore_errors=True)\n",
        "  except:\n",
        "    pass\n",
        " \n",
        "\n",
        "\n",
        "def get_weather_master():\n",
        "  shutil.rmtree('/content/stinky', ignore_errors=True)\n",
        "  !git clone https://github.com/ds5110/stinky\n",
        "\n",
        "  path = 'stinky/weather_data/pod_raw_data/'\n",
        "  lst = os.listdir(path)\n",
        "\n",
        "  df= pd.DataFrame()\n",
        "  df_new= pd.DataFrame()\n",
        "\n",
        "  for i in lst:\n",
        "    url = path+i\n",
        "    if i[-4:]==\".csv\":\n",
        "      if df.empty==True :\n",
        "        df = pd.read_csv(url)\n",
        "        df[\"Pod Name\"] = i[:5]\n",
        "      else:\n",
        "        df_new= pd.read_csv(url)\n",
        "        df_new[\"Pod Name\"]= i[:5]\n",
        "        df= df.append(df_new)\n",
        "\n",
        "  add_dates_times_columns(df)\n",
        "  df = cardinal_direction(df)\n",
        "\n",
        "  df = df.sort_values(by=['Pod Name','DateTime'])\n",
        "  df.reset_index(drop=True, inplace=True)\n",
        "  df.rename(columns={'DateTime':'Date & time (hour rounded)', \n",
        "                     'date_only':'date', 'time_only':'time (hour rounded)'}\n",
        "            , inplace=True)\n",
        "  \n",
        "  clean_up()\n",
        "\n",
        "  return df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UW3lNFrFbLVv",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "#PLEASE SEE FOLLOWING TEXT BLOCK FOR EXPLANATION OF THIS CODE CELL!!!\n",
        "'''\n",
        "Purpose : A helper function to weather_df_interval_groupby_for_plot.\n",
        "    This remove columns with all values that are NA\n",
        "@param df_weather_master : dataframe, the weather master dataframe\n",
        "    i.e. the return value of get_weather_master()\n",
        "return: dataframe, a processed copy of the weather master\n",
        "'''\n",
        "def column_process_weather_df(df_weather_master):\n",
        "  df_copy = df_weather_master.copy()\n",
        "  df_copy =df_copy.dropna(axis=1, how='all')\n",
        "  df_copy.pop('date')\n",
        "  df_copy.pop('time (hour rounded)')\n",
        "\n",
        "  return df_copy\n",
        "\n",
        "\n",
        "'''\n",
        "Purpose: A helper function for aggregation in weather_df_process_for_plotting\n",
        "    This is used to get mode of cardinal directions for any time interval\n",
        "@param: a pandas series containing strings of cardinal directions\n",
        "@return: a string representation of the mode\n",
        "'''\n",
        "def determine_direction_mode(df_column):\n",
        "  df_mode = df_column.mode()\n",
        "  if (len(df_mode.index) > 1):\n",
        "    return 'None'\n",
        "  else:\n",
        "    return df_mode\n",
        "\n",
        "\n",
        "'''\n",
        "Purpose: To create a datetime object from date values as integers\n",
        "This is a helper function to get_weather_info_helper\n",
        "@param day_int : an integer representing the numerical day\n",
        "@param month_int : an integer representing the numerical month i.e. January is 1\n",
        "@param year_int : the year represented as a 4 digit integer i.e. 2020\n",
        "@param time : A boolean, set to False by default. Set true if looking at a particular hour\n",
        "@param hour_int : an integer representing the hour in military time (0-23) where 23 = 11:OO PM\n",
        "@ return : a datetime object representing the data values entered\n",
        "\n",
        "'''\n",
        "def convert_val_to_dt(day_int, month_int, year_int, time = False, hour_int = 0):\n",
        "    if time == False:\n",
        "        x = datetime(year = year_int, month = month_int, day = day_int, hour = 0, minute = 0, second = 0, microsecond = 0)\n",
        "    else:\n",
        "        x = datetime(year = year_int, month = month_int, day = day_int, hour = hour_int, minute = 0, second = 0, microsecond = 0)\n",
        "    return x\n",
        "\n",
        "\n",
        "'''\n",
        "Purpose: A helper function to weather_df_interval_groupby_for_plot\n",
        "    This groups the dataframe by key - weekly, monthly, or daily\n",
        "    * daily actually does not meaure in months but in chunks of 30 days\n",
        "    which is almost a month\n",
        "@param df: a dataframe, weather master \n",
        "    i.e. the return value of get_weather_master()\n",
        "@param key: a string of the grouping requested, \n",
        "    must be 'hours','days','weeks', or'months'\n",
        "@return: dataframe, a processed dataframe that removes columns with all NA\n",
        "and is grouped by user choice\n",
        "'''\n",
        "def weather_df_process_for_plotting(df : pd.DataFrame, key : str, interval = 1):\n",
        "  options = ['hours','days','weeks','months']\n",
        "  \n",
        "  if interval <=0 or not isinstance(interval, int):\n",
        "    raise ValueError(\"Interval must be a positive non-zero integer\")\n",
        "  \n",
        "  key = key.lower()\n",
        "\n",
        "  if key not in options:\n",
        "    raise ValueError(\"Key must be a string: hours, days, weeks, or months\")\n",
        "  \n",
        "  week_interval = str(interval * 7) + 'D'\n",
        "  month_interval = str(interval * 30) + 'D'\n",
        "  day_interval = str(interval * 1) + 'D'\n",
        "  hour_interval = str(interval * 1) + 'H'\n",
        "\n",
        "  dict_options = {'hours': hour_interval, 'days' : day_interval, 'weeks' : week_interval, 'months' : month_interval}\n",
        "\n",
        "  \n",
        "  df_copy = df.copy()\n",
        "\n",
        "  interval_key = dict_options[key]\n",
        "  \n",
        "  try:\n",
        "    df_copy.insert(loc=1, column='DateTime interval',value= 5)\n",
        "  except:\n",
        "    exception = 1\n",
        "  \n",
        "  df_copy['DateTime interval'] = df_copy['Date & time (hour rounded)'].dt.floor(interval_key)\n",
        "\n",
        "  df_copy['Cardinal Direction'] = df_copy['Cardinal Direction'].replace('0', 'Not-measured')\n",
        "  df_copy['Pod Name'] = df_copy['Pod Name'].astype(str)\n",
        "\n",
        "  try:\n",
        "    df_copy.pop(item='Date & time (hour rounded)')\n",
        "  except:\n",
        "    exception = 1\n",
        "\n",
        "  list_col = list(df_copy.columns)\n",
        "\n",
        "  dict_agg = {}\n",
        "\n",
        "  for i in list_col:\n",
        "    if i == 'DateTime interval':\n",
        "       continue\n",
        "    elif i == 'Cardinal Direction':\n",
        "      dict_agg[i] = lambda x: determine_direction_mode(x)\n",
        "    elif i == 'Pod Name':\n",
        "      dict_agg[i] = lambda y : y.mode()\n",
        "    else:\n",
        "      dict_agg[i] = 'mean'\n",
        "\n",
        "  all_pods = list(df_copy['Pod Name'].unique())\n",
        "  df_list = []\n",
        "\n",
        "  for pod in all_pods:\n",
        "    df_iter = df_copy[df_copy['Pod Name'] == pod]\n",
        "    df_iter = df_iter.groupby(by=['DateTime interval']).agg(dict_agg)\n",
        "    df_list.append(df_iter)\n",
        "    \n",
        "  final = pd.concat(df_list)\n",
        "  return final\n",
        "\n",
        "'''\n",
        "Purpose: A helper function to display_weather_EDA \n",
        "    Groups the master weather dataframe by user choice\n",
        "    by key - weekly, monthly, or daily. Keeps the pods separate though.\n",
        "@param df : a dataframe, this should be the master weather dataframe.\n",
        "    i.e. the return value of get_weather_master()\n",
        "@param key : a string value, the way to aggregate the data.\n",
        "    Must be 'hours','days','weeks', or 'months'.\n",
        "@ param interval : an integer, the quantity for aggregation.\n",
        "    i.e. if you want to aggregate by intervals of 3 days then key = 'days', interval = 3\n",
        "'''\n",
        "def weather_df_interval_groupby_for_plot(df : pd.DataFrame, key : str, interval = 1)->pd.DataFrame:\n",
        "  df = column_process_weather_df(df)\n",
        "  return weather_df_process_for_plotting(df, key, interval=interval)\n",
        "\n",
        "'''\n",
        "Purpose: A helper function to display_weather_EDA\n",
        "    Creates two graphs index vs. EDA to visualize and \n",
        "    datetime vs. EDA to visualize\n",
        "@param df_weather_processed : a dataframe, the processed weather dataframe\n",
        "    i.e. it should be the return value of weather_df_interval_groupby_for_plot\n",
        "@param plot_type : a string representing the kind of plot to graph \n",
        "    'scatter' and 'plot' are the only valid plot_type values\n",
        "@return: void, will print out the graphs\n",
        "'''\n",
        "def weather_EDA(df_weather_processed, plot_type : str):\n",
        "\n",
        "  valid_plot_types = ['line', 'scatter']\n",
        "\n",
        "  if (plot_type not in valid_plot_types):\n",
        "    raise ValueError(\"plot_type must be 'line' or 'scatter'\")\n",
        "\n",
        "  sns.set()\n",
        "  sns.set(rc={'figure.figsize':(20,10)})\n",
        "\n",
        "  EDA_list_visualize = ['Temp Avg', 'Hum Avg','Baro Avg','Windspeed','Gust','Interval Precip','Dew Point','Heat Index','Wind Chill']\n",
        "\n",
        "  index_key = 'DateTime interval'\n",
        "  df_weather_processed = df_weather_processed.sort_values(by=['Pod Name'])\n",
        "  df_weather_processed_reset = df_weather_processed.reset_index(drop=True).index\n",
        "  id = df_weather_processed.reset_index(drop=True).index\n",
        "\n",
        "  if plot_type == 'line':\n",
        "    for i in EDA_list_visualize:\n",
        "      fig, ax =plt.subplots(1,2)\n",
        "      sns.lineplot(data = df_weather_processed.reset_index(drop=True), x=id , y = i, hue='Pod Name',  ax=ax[0] )\n",
        "      sns.lineplot(data = df_weather_processed, x = index_key  , y = i, hue='Pod Name',  ax=ax[1] )\n",
        "      plt.xticks(rotation=90)        \n",
        "      fig.show()\n",
        "  elif plot_type == 'scatter':\n",
        "    for i in EDA_list_visualize:\n",
        "      fig, ax =plt.subplots(1,2)\n",
        "      sns.scatterplot(data = df_weather_processed.reset_index(drop=True), x = id  , y = i, hue='Pod Name',  ax=ax[0] )\n",
        "      sns.scatterplot(data = df_weather_processed, x = index_key  , y = i, hue='Pod Name',  ax=ax[1] )\n",
        "      plt.xticks(rotation=90)\n",
        "      fig.show()\n",
        "\n",
        "'''\n",
        "Purpose: Calls the function that will created weather/data visualization\n",
        "@param df_weather_master : the master weather dataframe\n",
        "@param plot_type : a string representing the type of plot to visualize - \n",
        "  only 'scatter' and 'line' are valid options for now\n",
        "  **In the future allow for different plot types\n",
        "@param key : a string - must be 'hours', 'days', 'weeks', 'months'\n",
        "@param interval : an integer, a multiplier for the key interval chosen, defaults to 1\n",
        "@return : void, displays graph\n",
        "'''\n",
        "def display_weather_EDA(df_weather_master, plot_type : str, key : str,interval_qty = 1):\n",
        "  df_to_plot = weather_df_interval_groupby_for_plot(df_weather_master, key, interval_qty)\n",
        "  weather_EDA(df_to_plot, plot_type)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BbZIqucUZ9ek",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "# All the code in this block(besides the github path access) come from \n",
        "# the smell group. It was just packaged into functions\n",
        "'''\n",
        "Purpose: A helper function to get_smell_df_master\n",
        "    This imports smell my city data csv\n",
        "    and adds datetime columns.\n",
        "@return: a dataframe of the smell my city data\n",
        "'''\n",
        "def get_smc():\n",
        "  directory_path = 'stinky/smell_data/smell_raw_data/'\n",
        "  lst = os.listdir(directory_path)\n",
        "\n",
        "  for i in lst:\n",
        "    if i == 'smc.csv':\n",
        "      file_path = directory_path+i\n",
        "      df_smc = pd.read_csv(file_path)\n",
        "    else: continue\n",
        "\n",
        "  # Chnaging the date and time column to datetime format\n",
        "  df_smc['date & time']=df_smc['date & time'].str[0:20]\n",
        "  df_smc['date & time']=pd.to_datetime(df_smc['date & time'])\n",
        "  # Creating separate columns for date and time\n",
        "  df_smc['date'] = [d.date() for d in df_smc['date & time']]\n",
        "  df_smc['time'] = [d.time() for d in df_smc['date & time']]  \n",
        "  \n",
        "  # Creating a date and hour column\n",
        "  df_smc['Date & time (hour rounded)'] = df_smc['date & time'].dt.strftime(\"%Y-%m-%d %H:00:00\")\n",
        "\n",
        "  # renaming columns\n",
        "  df_smc.rename(columns={'skewed latitude':'Latitude', 'skewed longitude':'Longitude'}, inplace=True)\n",
        "\n",
        "  return df_smc\n",
        "\n",
        "'''\n",
        "Purpose: A helper function to get_smell_df_master\n",
        "    This imports see clickfix data csv\n",
        "    and add datetime columns.\n",
        "@return: a dataframe of the see click fix data\n",
        "'''\n",
        "def get_scf():\n",
        "  directory_path = 'stinky/smell_data/smell_raw_data/'\n",
        "  lst = os.listdir(directory_path)\n",
        "\n",
        "  for i in lst:\n",
        "    if i == 'scf.csv':\n",
        "      file_path = directory_path+i\n",
        "      df_scf = pd.read_csv(file_path)\n",
        "    else: continue\n",
        "\n",
        "  # Changing the date and time column to datetime format\n",
        "  df_scf['Created at local']=pd.to_datetime(df_scf['Created at local'])\n",
        "  df_scf['Closed at local']=pd.to_datetime(df_scf['Closed at local'])\n",
        "\n",
        "  # Creating separate columns for date and time\n",
        "  df_scf['date'] = [d.date() for d in df_scf['Created at local']]\n",
        "  df_scf['time'] = [d.time() for d in df_scf['Created at local']]\n",
        "\n",
        "  # Creating a date and hour column (using the 'Created at local' column)\n",
        "  df_scf['Date & time (hour rounded)'] = df_scf['Created at local'].dt.strftime(\"%Y-%m-%d %H:00:00\") \n",
        "\n",
        "  # renaming columns\n",
        "  df_scf.rename(columns={'Description':'smell description', 'Lat':'Latitude', 'Lng':'Longitude'}, inplace=True) \n",
        "  return df_scf\n",
        "\n",
        "'''\n",
        "Purpose: A heplper function to get_smell_df_master\n",
        "    It merges two dataframes. These parameters\n",
        "    should be the return values of get_smc() and get_scf().\n",
        "@param smc: a dataframe, the return value of get_smc()\n",
        "@param sfc : a dataframe, the return value of get_scf()\n",
        "@return a dataframe that merged both dataframes.\n",
        "'''\n",
        "def merge_and_process_smell(smc, scf):\n",
        "  df_stinky = scf.append(smc, sort=False)\n",
        "\n",
        "  #get a rounded hour column\n",
        "  df_stinky['time (hour rounded)']= pd.to_datetime(df_stinky['Date & time (hour rounded)']).dt.time\n",
        "\n",
        "  # change column positions\n",
        "  date_hour_col = df_stinky.pop('Date & time (hour rounded)')\n",
        "  date_only_col = df_stinky.pop('date')\n",
        "  time_hour_col = df_stinky.pop('time (hour rounded)')\n",
        "\n",
        "  df_stinky.insert(0,column = 'Date & time (hour rounded)', value= date_hour_col)\n",
        "  df_stinky.insert(1,column = 'date', value = date_only_col)\n",
        "  df_stinky.insert(2,column ='time (hour rounded)', value= time_hour_col)\n",
        "  df_stinky.insert(loc=3,column='Complaint total', value= 1)\n",
        "\n",
        "  return df_stinky\n",
        "\n",
        "'''\n",
        "Purpose: get the smell master dataframe\n",
        "This dataframe (df) will be used in many later functions.\n",
        "@return : a dataframe of merged smell my city data and see clickfix data\n",
        "'''\n",
        "def get_smell_df_master():\n",
        "  df_smc = get_smc()\n",
        "  df_scf = get_scf()\n",
        "  return merge_and_process_smell(df_smc, df_scf)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cGFhRnJZZJYG",
        "cellView": "form"
      },
      "source": [
        "#PLEASE SEE FOLLOWING TEXT BLOCK FOR EXPLANATION OF THIS CODE CELL!!!\n",
        "#@title \n",
        "'''\n",
        "Purpose: Helper function to find_min_max_dates\n",
        "@param list_datetime : a list composed of 2 datetime objects\n",
        "@param key: a string, either max or min\n",
        "@return the minimum or maximum of the two datetime objects\n",
        "from list_datetime based on key param\n",
        "'''\n",
        "def date_comp_helper(list_datetime, key):\n",
        "  comp = list_datetime[0]\n",
        "\n",
        "  if key == 'min':\n",
        "    return (pd.Series(list_datetime)).max()\n",
        "  elif key == 'max':\n",
        "    return (pd.Series(list_datetime)).min()\n",
        "\n",
        "'''\n",
        "Purpose: Helper function to trim_dfs_by_datetime.\n",
        "@param list_df: a list of dataframes.\n",
        "Each dataframe must have a 'Date & time (hour rounded)' column\n",
        "with a datatype of datetime\n",
        "@return : two datetime objects, the minimum and maximum dates\n",
        "that fit within the date ranges of two dataframes\n",
        "\n",
        "'''\n",
        "def find_min_max_dates(list_df : list):\n",
        "\n",
        "  list_copy = copy.deepcopy(list_df)\n",
        "  min_list = []\n",
        "  max_list = []\n",
        "\n",
        "  for df in list_copy:\n",
        "    series = df['Date & time (hour rounded)']\n",
        "    min_datetime_i = series.min()\n",
        "    max_datetime_i = df['Date & time (hour rounded)'].max()\n",
        "    min_list.append(min_datetime_i)\n",
        "    max_list.append(max_datetime_i)\n",
        "\n",
        "  min_datetime = date_comp_helper(min_list, 'min')\n",
        "  max_datetime = date_comp_helper(max_list, 'max')\n",
        "\n",
        "\n",
        "  return min_datetime, max_datetime\n",
        "\n",
        "\n",
        "'''\n",
        "Purpose: Helper function to trim_dfs_by_datetime\n",
        "It removes every column except for date & time and Complaint total -\n",
        "Essentially keeping only the columns we are interested in looking at.\n",
        "@param df_smell_master: the smell master df (dataframe)\n",
        "@return: a df with only the 'Date & time (hour rounded)' and 'Complaint total'\n",
        "column. This is copy. This (and most functions) do not\n",
        "affect the parameter passed in.\n",
        "'''\n",
        "def remove_smell_columns_unused(df_smell_master_param):\n",
        "    df_copy = df_smell_master_param.copy()\n",
        "\n",
        "    list_col_keep = ['Date & time (hour rounded)', 'Complaint total']\n",
        "\n",
        "    for i in (list(df_copy.columns)):\n",
        "        if i not in list_col_keep:\n",
        "            df_copy.pop(i)\n",
        "        else:\n",
        "            continue\n",
        "\n",
        "    df_copy = df_copy.groupby(by=['Date & time (hour rounded)']).sum()\n",
        "    df_copy = df_copy.reset_index()\n",
        "\n",
        "    return df_copy\n",
        "\n",
        "'''\n",
        "Purpose: Groups the weather data by datetime hours. \n",
        "The weather data contains multiple measurements for the same datetime hour\n",
        "becuase multiple pods are measuring the data. This function will get \n",
        "all of measurements from each pod for that particular time interval and \n",
        "average them together so that the resulting df will only have 1 set of \n",
        "measurements per datetime interval.\n",
        "@param df_weather_master: the weather master df\n",
        "@ return: a df with the weather values aggregated/grouped by datetime interval\n",
        "'''\n",
        "def flatten_weather_hourly(df_weather_master_param):\n",
        "    # Be aware this does not remove any outliers in the data!!!\n",
        "    df_copy = df_weather_master_param.copy()\n",
        "    df_copy['DateTime interval'] = df_copy['Date & time (hour rounded)'].dt.floor('H')\n",
        "    df_copy = column_process_weather_df(df_copy)\n",
        "\n",
        "    df_copy.pop(item='Pod Name')\n",
        "    df_copy.pop(item='DateTime interval')\n",
        "\n",
        "    list_col = list(df_copy.columns)\n",
        "\n",
        "    dict_agg = {}\n",
        "\n",
        "    for i in list_col:\n",
        "        if i == 'Date & time (hour rounded)':\n",
        "            continue\n",
        "        elif i == 'Cardinal Direction':\n",
        "            dict_agg[i] = lambda x: determine_direction_mode(x)\n",
        "        else:\n",
        "            dict_agg[i] = 'mean'\n",
        "\n",
        "    df_agg = df_copy.groupby(by=['Date & time (hour rounded)']).agg(dict_agg)\n",
        "    df_agg = df_agg.reset_index()\n",
        "    return df_agg\n",
        "\n",
        "\n",
        "'''\n",
        "Purpose: This function will trim each dataframe passed so that\n",
        "they match each others time ranges. The smell data was being collected\n",
        "much earlier than the weather data. Therefore, we have no weather event values\n",
        "that go as far back. We want to analyze weather and complaints together and we cannot \n",
        "analyze them together completely becuase they do not have the same datetime ranges.\n",
        "So we trim the data such that the dataframe return contains the values\n",
        "for the matching time ranges.\n",
        "@param df_smell_master : the master smell df\n",
        "@param df_weather_master : the weather master df\n",
        "@return : a list consisting of two df's (weather and smell) whose\n",
        "min and max date times match each other\n",
        "'''\n",
        "def trim_dfs_by_datetime(df_smell_master, df_weather_master_param):\n",
        "    df_smell_master_copy = df_smell_master.copy()\n",
        "    df_weather_master_copy = df_weather_master_param.copy()\n",
        "\n",
        "    min_datetime, max_datetime = find_min_max_dates([df_smell_master_copy, df_weather_master_copy])\n",
        "\n",
        "    df_smell_master_copy['Date & time (hour rounded)'] = pd.to_datetime(\n",
        "        df_smell_master_copy['Date & time (hour rounded)'])\n",
        "    df_smell = df_smell_master_copy[df_smell_master_copy['Date & time (hour rounded)'] >= min_datetime]\n",
        "    df_weather = df_weather_master_copy[df_weather_master_copy['Date & time (hour rounded)'] >= min_datetime]\n",
        "    df_smell = df_smell[df_smell['Date & time (hour rounded)'] <= max_datetime]\n",
        "    df_weather = df_weather[df_weather['Date & time (hour rounded)'] <= max_datetime]\n",
        "\n",
        "    # Need this second check do not remove\n",
        "    min_datetime, max_datetime = find_min_max_dates([df_smell, df_weather])\n",
        "    df_smell = df_smell[df_smell['Date & time (hour rounded)'] >= min_datetime]\n",
        "    df_weather = df_weather[df_weather['Date & time (hour rounded)'] >= min_datetime]\n",
        "    df_smell = df_smell[df_smell['Date & time (hour rounded)'] <= max_datetime]\n",
        "    df_weather = df_weather[df_weather['Date & time (hour rounded)'] <= max_datetime]\n",
        "\n",
        "    df_weather = df_weather[df_weather['Date & time (hour rounded)'] >= df_smell['Date & time (hour rounded)'].min()]\n",
        "    df_smell = remove_smell_columns_unused(df_smell)\n",
        "    df_smell = df_smell.reset_index(drop=True)\n",
        "\n",
        "    df_weather = flatten_weather_hourly(df_weather)\n",
        "\n",
        "    return df_weather, df_smell\n",
        "\n",
        "'''\n",
        "Purpose: This function will merge two dataframes together in a full outer join.\n",
        "Duplicate values will be dropped. The idea is to call this function on the \n",
        "return value of trim_dfs_by_datetime.\n",
        "@param df_weather : the first item from the list that is returned by trim_dfs_by_datetime\n",
        "@param df_smell : the seocnd item from the list that is returned by trim_dfs_by_datetime\n",
        "@return : a df representing the merged and trimmed data of the weather master df and smell master df\n",
        "'''\n",
        "def merge_trimmed_data(df_weather, df_smell):\n",
        "  df_merged = pd.merge(df_weather,df_smell, how='outer', on='Date & time (hour rounded)') \n",
        "  df_merged = df_merged.reset_index()\n",
        "  df_merged = df_merged.drop_duplicates(subset=['Date & time (hour rounded)'])\n",
        "  df_merged.pop('index')\n",
        "  df_merged = df_merged.reset_index()\n",
        "  df_merged.pop('index')\n",
        "  return df_merged\n",
        "\n",
        "'''\n",
        "Purpose: This is meant to be called on the return value of merge_trimmed_data\n",
        "It will aggregate/group the data by the interval you want to look at.. i.e.\n",
        "if key = days and interval = 3 it will aggregate/group the data in \n",
        "intervals of 3 days (it will perform the correct aggregation type per column\n",
        "so complaint totals are summed up while while weather features like temperature \n",
        "become averaged)\n",
        "@param df : This should be the return value merge_trimmed_data\n",
        "@\n",
        "'''\n",
        "def groupby_interval_merged_trimmed_df(df, key : str, interval = 1):\n",
        "  df_copy = df.copy()\n",
        "\n",
        "  options = ['hours','days','weeks','months']\n",
        "  \n",
        "  if interval <=0 or not isinstance(interval, int):\n",
        "    raise ValueError(\"Interval must be a positive non-zero integer\")\n",
        "  \n",
        "  key = key.lower()\n",
        "\n",
        "  if key not in options:\n",
        "    raise ValueError(\"Key must be a string: hours, days, weeks, or months\")\n",
        "  \n",
        "  week_interval = str(interval * 7) + 'D'\n",
        "  month_interval = str(interval * 30) + 'D'\n",
        "  day_interval = str(interval * 1) + 'D'\n",
        "  hour_interval = str(interval * 1) + 'H'\n",
        "\n",
        "  dict_options = {'hours': hour_interval, 'days' : day_interval, 'weeks' : week_interval, 'months' : month_interval}\n",
        "\n",
        "\n",
        "  interval_key = dict_options[key]\n",
        "\n",
        "  try:\n",
        "    df_copy.insert(loc=1, column='DateTime interval',value= 5)\n",
        "  except:\n",
        "    exception = 1\n",
        "  \n",
        "  df_copy['DateTime interval'] = df_copy['Date & time (hour rounded)'].dt.floor(interval_key)\n",
        "\n",
        "  try:\n",
        "    df_copy.pop(item='Date & time (hour rounded)')\n",
        "  except:\n",
        "    exception = 1\n",
        "\n",
        "  list_col = list(df_copy.columns)\n",
        "\n",
        "  dict_agg = {}\n",
        "\n",
        "  for i in list_col:\n",
        "    if i == 'DateTime interval':\n",
        "       continue\n",
        "    elif i == 'Cardinal Direction':\n",
        "      dict_agg[i] = lambda x: determine_direction_mode(x)\n",
        "    elif i == 'Complaint total':\n",
        "      dict_agg[i] = 'sum'\n",
        "    else:\n",
        "      dict_agg[i] = 'mean'\n",
        "  \n",
        "  df_agg = df_copy.groupby(by=['DateTime interval']).agg(dict_agg)\n",
        "  df_agg = df_agg.reset_index()\n",
        "  return df_agg\n",
        "\n",
        "def create_season_dict():\n",
        "  all_months = [\"December\", \"January\", \"February\", \"March\", \"April\", \n",
        "                \"May\", \"June\", \"July\", \n",
        "                \"August\", \"September\", \"October\", \"November\"]\n",
        "\n",
        "  all_seasons = ['Winter', 'Spring', 'Summer', 'Fall']\n",
        "\n",
        "  season_tracker = 0\n",
        "  month_tracker = 0\n",
        "\n",
        "  season_dict = {}\n",
        "  for i in range(len(all_months)):\n",
        "    season_dict[all_months[i]] = all_seasons[season_tracker]\n",
        "    month_tracker+=1\n",
        "    \n",
        "    if month_tracker == 3:\n",
        "      season_tracker +=1\n",
        "      month_tracker = 0\n",
        "  return season_dict\n",
        "\n",
        "\n",
        "  \n",
        "'''\n",
        "Purpose: To combine the master weather and master smell data set into dataframe\n",
        "param @ df_weather_master : the master weather dataframe\n",
        "param @  df_smell_master : the smell master dataframe\n",
        "param @ key : the type of interval to use, must be 'hours', 'days', 'weeks', or 'months'\n",
        "parman @interval : the qty of interval to use, defaults to 1\n",
        "Return: A dataframe that combined the smell master df and the weather master df\n",
        "\n",
        "For example get_merged_trimmed_df(df_weather_master, df_smell_master, key = 'hours', interval = 12)\n",
        "will return a dataframe that groups the intervals into 12 hour blocks.\n",
        "\n",
        "New Column Explanations: \n",
        "*** 'Hour Group' - can be used to group analysis or visualization by hour blocks\n",
        "*** 'Day_of_week' - can be used to group analysis or visualizaiton by weekday, i.e. Monday, Teusday, ..etc.\n",
        "*** 'Month_of_year' - can be used to group analysis by categorical month i.e. January, February etc.\n",
        "*** 'Season' - can be used to group analysis by categorical season 'Winter', 'Spring', 'Summer', 'Fall'\n",
        "\n",
        "Different keys will provide different extra columns:\n",
        "*** key : Hours -  will add the columns 'Hour Group', 'Day_of_week', 'Month_of_year', and 'Season'\n",
        "*** key : days -  will add the columns 'Day_of_week', 'Month_of_year', and 'Season'\n",
        "*** key : month - will add the columns 'Month_of_year', and 'Season'\n",
        "\n",
        "'''\n",
        "def get_merged_trimmed_df(df_weather_master_param, df_smell_master, key, interval=1):\n",
        "    trimmed_weather, trimmed_smell = trim_dfs_by_datetime(df_smell_master, df_weather_master_param)\n",
        "    merged_df = merge_trimmed_data(trimmed_weather, trimmed_smell)\n",
        "    merged_grouped_df = groupby_interval_merged_trimmed_df(merged_df, key, interval)\n",
        "\n",
        "    merged_grouped_df['DateTime interval'] = pd.to_datetime(merged_grouped_df['DateTime interval'])\n",
        "    seasons_dict = create_season_dict()\n",
        "\n",
        "    if ((key == 'days' and interval == 1) or key == 'hours'):\n",
        "        merged_grouped_df.insert(loc=1, column='Day_of_week', value=5)\n",
        "        merged_grouped_df.insert(loc=2, column='Month_of_year', value=5)\n",
        "        merged_grouped_df.insert(loc=3, column='Season', value=5)\n",
        "        merged_grouped_df['Day_of_week'] = merged_grouped_df['DateTime interval'].dt.day_name()\n",
        "\n",
        "        merged_grouped_df['Month_of_year'] = merged_grouped_df['DateTime interval'].apply(lambda x: x.strftime('%B'))\n",
        "        merged_grouped_df['Season'] = merged_grouped_df['Month_of_year'].apply(lambda y: seasons_dict[y])\n",
        "\n",
        "    elif ((key == 'weeks' and interval == 1) or (key == 'months' and interval == 1)):\n",
        "        merged_grouped_df.insert(loc=1, column='Month_of_year', value=5)\n",
        "        merged_grouped_df.insert(loc=2, column='Season', value=5)\n",
        "\n",
        "        merged_grouped_df['Month_of_year'] = merged_grouped_df['DateTime interval'].apply(lambda x: x.strftime('%B'))\n",
        "        merged_grouped_df['Season'] = merged_grouped_df['Month_of_year'].apply(lambda y: seasons_dict[y])\n",
        "\n",
        "    if (key == 'hours'):\n",
        "        merged_grouped_df.insert(loc=1, column='Hour Group', value=5)\n",
        "        merged_grouped_df['Hour Group'] = merged_grouped_df['DateTime interval'].dt.time\n",
        "\n",
        "    col_complaint = merged_grouped_df.pop(item='Complaint total')\n",
        "    merged_grouped_df.insert(loc=1, column='Complaint total', value=col_complaint)\n",
        "\n",
        "    return merged_grouped_df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hBEw255A5noj",
        "cellView": "form"
      },
      "source": [
        "#@title \n",
        "'''\n",
        "Purpose: A helper function to get_complaint_df \n",
        "    Fills the missing time intervals in the master smell df.\n",
        "    The missing time intervals will have pd.Na for all columns\n",
        "@param smell_master_df : the master smell df\n",
        "@return df with the missing time intervals\n",
        "'''\n",
        "def smell_fill_dt_gaps(smell_master):\n",
        "    df_copy = smell_master.copy()\n",
        "    df_copy['Date & time (hour rounded)'] = pd.to_datetime(df_copy['Date & time (hour rounded)'])\n",
        "    min_dt = df_copy['Date & time (hour rounded)'].min()\n",
        "    max_dt = df_copy['Date & time (hour rounded)'].max()\n",
        "  \n",
        "    #make array from one hour to the next in date range\n",
        "    array_all_dt = np.arange(min_dt, max_dt, timedelta(hours=1)).astype(datetime)\n",
        "    df = pd.DataFrame(array_all_dt, columns = ['Date & time (hour rounded)'] )\n",
        "    \n",
        "    df_merged = pd.merge(df_copy,df, how='outer', on='Date & time (hour rounded)') \n",
        "    df_merged = df_merged.drop_duplicates(subset=['Date & time (hour rounded)'])\n",
        "    df_merged = df_merged.sort_values(by=['Date & time (hour rounded)'])\n",
        "    df_merged = df_merged.reset_index(drop=True)\n",
        "    \n",
        "    return df_merged\n",
        "\n",
        "'''\n",
        "Purpose: The smell master data contains only datetimes where total complaint = 1\n",
        "    Since there isn't a complaint every hour, we are missing datetimes where \n",
        "    there are no complaints.\n",
        "    Therefore we need to fill in the gaps in the data ranges so we can get\n",
        "    the datetime rows where there were no complaints. This function\n",
        "    will fill the missing dates in the smell master df and return a df\n",
        "    containing only Datetime interval, Hour Group, Day_of_week, Month_of_year,\n",
        "    Season, and Complaint total.\n",
        "@param smell_master : the master smell df\n",
        "@return: a df with the only the columns: Datetime interval, \n",
        "    Hour Group, Day_of_week, Month_of_year, Season, and Complaint total\n",
        "'''\n",
        "def get_complaint_df(smell_master_df):\n",
        "    smell_copy = smell_fill_dt_gaps(smell_master_df)\n",
        "    \n",
        "    seasons_dict = create_season_dict()\n",
        "    \n",
        "    smell_copy.insert(loc=1, column='Day_of_week',value= 5)\n",
        "    smell_copy.insert(loc=2, column='Month_of_year',value= 5)\n",
        "    smell_copy.insert(loc=3, column='Season',value= 5)\n",
        "    smell_copy['Day_of_week'] = smell_copy['Date & time (hour rounded)'].dt.day_name()\n",
        " \n",
        "    smell_copy['Month_of_year'] = smell_copy['Date & time (hour rounded)'].apply(lambda x: x.strftime('%B'))\n",
        "    smell_copy['Season'] = smell_copy['Month_of_year'].apply(lambda y : seasons_dict[y])\n",
        "    \n",
        "    smell_copy.insert(loc=1, column = 'Hour Group', value=5)\n",
        "    smell_copy['Hour Group'] = smell_copy['Date & time (hour rounded)'].dt.time\n",
        "    \n",
        "    col_keep = ['Date & time (hour rounded)','Hour Group', 'Day_of_week', 'Month_of_year','Season', 'Complaint total']\n",
        "    total_col = list(smell_copy.columns)\n",
        "    \n",
        "    for i in total_col:\n",
        "        if i not in col_keep:\n",
        "            smell_copy.pop(i)\n",
        "    \n",
        "    smell_copy['Complaint total'].fillna(value=0, inplace=True)\n",
        "    \n",
        "    smell_copy['Complaint total'] = smell_copy['Complaint total'].apply(lambda x : int(x))\n",
        "    \n",
        "    smell_copy = smell_copy.sort_values(by=['Date & time (hour rounded)'])\n",
        "    \n",
        "    smell_copy = smell_copy.reset_index(drop=True)\n",
        "    \n",
        "    smell_copy =smell_copy.rename(columns={'Date & time (hour rounded)': 'DateTime interval'})\n",
        "    return smell_copy"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mfeA4oBni5b9"
      },
      "source": [
        "The code block below will produce a sneak peak of what the the master weather dataframe file will look like. This dataframe contains all the measurements of\n",
        "every pod. Every row represents the measurement for a particular hour. This dataframe is sorted in terms of time and then by pod name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 966
        },
        "id": "pGJ7xEFkVLmL",
        "outputId": "53fc3aa9-7971-4686-d5b0-8e3118daf42d"
      },
      "source": [
        "df_weather_master = get_weather_master()\n",
        "df_weather_master"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Cloning into 'stinky'...\n",
            "remote: Enumerating objects: 668, done.\u001b[K\n",
            "remote: Counting objects: 100% (272/272), done.\u001b[K\n",
            "remote: Compressing objects: 100% (264/264), done.\u001b[K\n",
            "remote: Total 668 (delta 131), reused 0 (delta 0), pack-reused 396\u001b[K\n",
            "Receiving objects: 100% (668/668), 17.39 MiB | 13.34 MiB/s, done.\n",
            "Resolving deltas: 100% (336/336), done.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Date &amp; time (hour rounded)</th>\n",
              "      <th>date</th>\n",
              "      <th>time (hour rounded)</th>\n",
              "      <th>Temp Avg</th>\n",
              "      <th>Temp Low</th>\n",
              "      <th>Temp High</th>\n",
              "      <th>Hum Avg</th>\n",
              "      <th>Hum Low</th>\n",
              "      <th>Hum High</th>\n",
              "      <th>Baro Avg</th>\n",
              "      <th>Baro Low</th>\n",
              "      <th>Baro High</th>\n",
              "      <th>Windspeed</th>\n",
              "      <th>Gust</th>\n",
              "      <th>Wind Direction</th>\n",
              "      <th>Cardinal Direction</th>\n",
              "      <th>Interval Precip</th>\n",
              "      <th>Leaf Wetness (Minutes)</th>\n",
              "      <th>Solar Radiation Avg</th>\n",
              "      <th>Inside Temp Avg</th>\n",
              "      <th>Inside Temp Low</th>\n",
              "      <th>Inside Temp High</th>\n",
              "      <th>Heat Index</th>\n",
              "      <th>Wind Chill</th>\n",
              "      <th>Dew Point</th>\n",
              "      <th>UV Index</th>\n",
              "      <th>Station Voltage</th>\n",
              "      <th>Station Voltage 2</th>\n",
              "      <th>t1ia</th>\n",
              "      <th>t1il</th>\n",
              "      <th>t1ih</th>\n",
              "      <th>t2ia</th>\n",
              "      <th>t2il</th>\n",
              "      <th>t2ih</th>\n",
              "      <th>wetbulb</th>\n",
              "      <th>s2ia</th>\n",
              "      <th>smia</th>\n",
              "      <th>lvl1</th>\n",
              "      <th>lvl2</th>\n",
              "      <th>lvl3</th>\n",
              "      <th>Pod Name</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2020-09-02 12:00:00</td>\n",
              "      <td>2020-09-02</td>\n",
              "      <td>12:00:00</td>\n",
              "      <td>75.00</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>60.0</td>\n",
              "      <td>60.0</td>\n",
              "      <td>60.0</td>\n",
              "      <td>29.97</td>\n",
              "      <td>29.97</td>\n",
              "      <td>29.98</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>157.0</td>\n",
              "      <td>South</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>66.51</td>\n",
              "      <td>65.0</td>\n",
              "      <td>69.0</td>\n",
              "      <td>75.00</td>\n",
              "      <td>75.0</td>\n",
              "      <td>60.2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.530</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65.7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2020-09-02 13:00:00</td>\n",
              "      <td>2020-09-02</td>\n",
              "      <td>13:00:00</td>\n",
              "      <td>75.16</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.2</td>\n",
              "      <td>59.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>60.0</td>\n",
              "      <td>29.96</td>\n",
              "      <td>29.95</td>\n",
              "      <td>29.98</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>157.0</td>\n",
              "      <td>South</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>73.92</td>\n",
              "      <td>69.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.16</td>\n",
              "      <td>75.2</td>\n",
              "      <td>59.8</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.504</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65.3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2020-09-02 14:00:00</td>\n",
              "      <td>2020-09-02</td>\n",
              "      <td>14:00:00</td>\n",
              "      <td>75.41</td>\n",
              "      <td>75.2</td>\n",
              "      <td>75.5</td>\n",
              "      <td>58.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>29.94</td>\n",
              "      <td>29.93</td>\n",
              "      <td>29.95</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>74.98</td>\n",
              "      <td>74.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.41</td>\n",
              "      <td>75.4</td>\n",
              "      <td>59.6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.500</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65.3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2020-09-02 15:00:00</td>\n",
              "      <td>2020-09-02</td>\n",
              "      <td>15:00:00</td>\n",
              "      <td>75.32</td>\n",
              "      <td>75.2</td>\n",
              "      <td>75.4</td>\n",
              "      <td>59.0</td>\n",
              "      <td>58.0</td>\n",
              "      <td>59.0</td>\n",
              "      <td>29.91</td>\n",
              "      <td>29.90</td>\n",
              "      <td>29.93</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>74.95</td>\n",
              "      <td>74.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.32</td>\n",
              "      <td>75.3</td>\n",
              "      <td>59.9</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.500</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65.4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2020-09-02 16:00:00</td>\n",
              "      <td>2020-09-02</td>\n",
              "      <td>16:00:00</td>\n",
              "      <td>75.47</td>\n",
              "      <td>75.2</td>\n",
              "      <td>75.7</td>\n",
              "      <td>59.0</td>\n",
              "      <td>59.0</td>\n",
              "      <td>59.0</td>\n",
              "      <td>29.89</td>\n",
              "      <td>29.88</td>\n",
              "      <td>29.90</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>0.0</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>74.90</td>\n",
              "      <td>74.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.47</td>\n",
              "      <td>75.5</td>\n",
              "      <td>60.1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.500</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>65.6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23522</th>\n",
              "      <td>2021-07-23 20:00:00</td>\n",
              "      <td>2021-07-23</td>\n",
              "      <td>20:00:00</td>\n",
              "      <td>64.32</td>\n",
              "      <td>63.7</td>\n",
              "      <td>64.7</td>\n",
              "      <td>75.0</td>\n",
              "      <td>70.0</td>\n",
              "      <td>81.0</td>\n",
              "      <td>30.14</td>\n",
              "      <td>30.13</td>\n",
              "      <td>30.15</td>\n",
              "      <td>2.03667</td>\n",
              "      <td>7.2</td>\n",
              "      <td>111.0</td>\n",
              "      <td>East</td>\n",
              "      <td>0.01</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>82.77</td>\n",
              "      <td>82.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>64.32</td>\n",
              "      <td>64.3</td>\n",
              "      <td>56.3</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.866</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>59.4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23523</th>\n",
              "      <td>2021-07-23 21:00:00</td>\n",
              "      <td>2021-07-23</td>\n",
              "      <td>21:00:00</td>\n",
              "      <td>63.51</td>\n",
              "      <td>62.7</td>\n",
              "      <td>64.0</td>\n",
              "      <td>81.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>30.13</td>\n",
              "      <td>30.12</td>\n",
              "      <td>30.15</td>\n",
              "      <td>3.15167</td>\n",
              "      <td>9.3</td>\n",
              "      <td>283.0</td>\n",
              "      <td>West</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>82.28</td>\n",
              "      <td>82.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>63.51</td>\n",
              "      <td>63.5</td>\n",
              "      <td>57.5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.766</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>59.8</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23524</th>\n",
              "      <td>2021-07-23 22:00:00</td>\n",
              "      <td>2021-07-23</td>\n",
              "      <td>22:00:00</td>\n",
              "      <td>62.94</td>\n",
              "      <td>62.7</td>\n",
              "      <td>63.1</td>\n",
              "      <td>81.0</td>\n",
              "      <td>79.0</td>\n",
              "      <td>83.0</td>\n",
              "      <td>30.15</td>\n",
              "      <td>30.14</td>\n",
              "      <td>30.15</td>\n",
              "      <td>3.17167</td>\n",
              "      <td>9.5</td>\n",
              "      <td>315.0</td>\n",
              "      <td>North-West</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>82.00</td>\n",
              "      <td>82.0</td>\n",
              "      <td>82.0</td>\n",
              "      <td>62.94</td>\n",
              "      <td>62.9</td>\n",
              "      <td>57.1</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.685</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>59.4</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23525</th>\n",
              "      <td>2021-07-23 23:00:00</td>\n",
              "      <td>2021-07-23</td>\n",
              "      <td>23:00:00</td>\n",
              "      <td>62.57</td>\n",
              "      <td>62.0</td>\n",
              "      <td>63.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>79.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>30.14</td>\n",
              "      <td>30.14</td>\n",
              "      <td>30.15</td>\n",
              "      <td>4.81667</td>\n",
              "      <td>11.0</td>\n",
              "      <td>317.0</td>\n",
              "      <td>North</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>82.00</td>\n",
              "      <td>82.0</td>\n",
              "      <td>82.0</td>\n",
              "      <td>62.57</td>\n",
              "      <td>62.6</td>\n",
              "      <td>56.2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.603</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>58.7</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO7</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23526</th>\n",
              "      <td>2021-07-24 00:00:00</td>\n",
              "      <td>2021-07-24</td>\n",
              "      <td>00:00:00</td>\n",
              "      <td>61.70</td>\n",
              "      <td>61.5</td>\n",
              "      <td>62.0</td>\n",
              "      <td>78.0</td>\n",
              "      <td>77.0</td>\n",
              "      <td>80.0</td>\n",
              "      <td>30.14</td>\n",
              "      <td>30.13</td>\n",
              "      <td>30.14</td>\n",
              "      <td>5.11167</td>\n",
              "      <td>11.4</td>\n",
              "      <td>318.0</td>\n",
              "      <td>North</td>\n",
              "      <td>0.00</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>81.82</td>\n",
              "      <td>81.0</td>\n",
              "      <td>82.0</td>\n",
              "      <td>61.70</td>\n",
              "      <td>61.7</td>\n",
              "      <td>54.8</td>\n",
              "      <td>NaN</td>\n",
              "      <td>6.538</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>57.6</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>SMRO7</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>23527 rows × 41 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      Date & time (hour rounded)        date  ... lvl3  Pod Name\n",
              "0            2020-09-02 12:00:00  2020-09-02  ...  NaN     SMRO3\n",
              "1            2020-09-02 13:00:00  2020-09-02  ...  NaN     SMRO3\n",
              "2            2020-09-02 14:00:00  2020-09-02  ...  NaN     SMRO3\n",
              "3            2020-09-02 15:00:00  2020-09-02  ...  NaN     SMRO3\n",
              "4            2020-09-02 16:00:00  2020-09-02  ...  NaN     SMRO3\n",
              "...                          ...         ...  ...  ...       ...\n",
              "23522        2021-07-23 20:00:00  2021-07-23  ...  NaN     SMRO7\n",
              "23523        2021-07-23 21:00:00  2021-07-23  ...  NaN     SMRO7\n",
              "23524        2021-07-23 22:00:00  2021-07-23  ...  NaN     SMRO7\n",
              "23525        2021-07-23 23:00:00  2021-07-23  ...  NaN     SMRO7\n",
              "23526        2021-07-24 00:00:00  2021-07-24  ...  NaN     SMRO7\n",
              "\n",
              "[23527 rows x 41 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MY36D81xjFF-"
      },
      "source": [
        "Please run the code block below manually if you only want to download the df_weather_master file on it's own. By default runnning this entire notebook will download this file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "d4gWxDOPXMXu",
        "outputId": "d0ae0c15-43f2-4435-abc7-bfa8ef632529"
      },
      "source": [
        "df_weather_master.to_csv('df_weather_master.csv', index=False) \n",
        "files.download('df_weather_master.csv')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_1c9bfe7b-ae5a-4684-b173-5627a525922a\", \"df_weather_master.csv\", 4349442)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "cellView": "form",
        "id": "zbBuqpW63Foo"
      },
      "source": [
        "#@title \n",
        "\n",
        "#This codeblock is written to remove the outliers like allof SMRO4\n",
        "\n",
        "#remove SMRO4\n",
        "df_weather_outlier = df_weather_master.copy()\n",
        "df_weather_outlier = df_weather_outlier[df_weather_outlier['Pod Name'] != 'SMRO4']\n",
        "\n",
        "# SMRO6 and SMRO7 also had 1 or 2 outliers that we remove here. \n",
        "# SMRO6 and SMRo7 had two clusters of data when plotte over time. The two clusters\n",
        "# were separated by a gap in time and we will be removing the smaller of the two clusters\n",
        "#which are composed of 2-3 measurements. These were likely measured for some preliminary testing \n",
        "# of the pod and this will only hinder our data as they look like outliers as weel. \n",
        "# Manually remove SMRO6 outliers\n",
        "SMRO6_only = df_weather_outlier[df_weather_outlier['Pod Name'] == 'SMRO6']\n",
        "df_weather_outlier= df_weather_outlier[df_weather_outlier['Pod Name'] != 'SMRO6']\n",
        "SMRO6_outlier_date = convert_val_to_dt(22,1,2021,time=True, hour_int=10) \n",
        "SMRO6_removed_outlier = SMRO6_only[SMRO6_only['Date & time (hour rounded)'] > SMRO6_outlier_date ]\n",
        "df_weather_outlier = df_weather_outlier.append(SMRO6_removed_outlier)\n",
        "\n",
        "# remove SMRO7\n",
        "SMRO7_only = df_weather_outlier[df_weather_outlier['Pod Name'] == 'SMRO7']\n",
        "df_weather_outlier = df_weather_outlier[df_weather_outlier['Pod Name'] != 'SMRO7']\n",
        "SMRO7_outlier_date = convert_val_to_dt(25,11,2020,time=True,hour_int=23) \n",
        "SMRO7_only = SMRO7_only[SMRO7_only['Date & time (hour rounded)'] > SMRO7_outlier_date]\n",
        "\n",
        "df_weather_outlier = df_weather_outlier.append(SMRO7_only)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cWIEAESh4T33"
      },
      "source": [
        "Please run the code block below manually if you only want to download the df_weather_masterr_outliers_removed file on it's own. By default runnning this entire notebook will download this file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "4BeMrxnE4G11",
        "outputId": "af1c9808-de26-4ad2-d34d-6dd53b4f1e49"
      },
      "source": [
        "df_weather_outlier.to_csv('df_weather_master_outliers_removed.csv', index=False) \n",
        "files.download('df_weather_master_outliers_removed.csv')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_f93e7388-fef6-4b6a-8a14-2f4717f31cd2\", \"df_weather_master_outliers_removed.csv\", 3807792)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-aD80kOi6lnh"
      },
      "source": [
        "The code block below provides a sneak peak to the complaint only data set"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "id": "xLNSk7Nb6hlA",
        "outputId": "7a74f1eb-5736-4933-c680-e31671d3c8fe"
      },
      "source": [
        "complaint_data_only = get_complaint_df(get_smell_df_master())\n",
        "complaint_data_only"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>DateTime interval</th>\n",
              "      <th>Hour Group</th>\n",
              "      <th>Day_of_week</th>\n",
              "      <th>Month_of_year</th>\n",
              "      <th>Season</th>\n",
              "      <th>Complaint total</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2019-05-22 09:00:00</td>\n",
              "      <td>09:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>May</td>\n",
              "      <td>Spring</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2019-05-22 10:00:00</td>\n",
              "      <td>10:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>May</td>\n",
              "      <td>Spring</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2019-05-22 11:00:00</td>\n",
              "      <td>11:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>May</td>\n",
              "      <td>Spring</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2019-05-22 12:00:00</td>\n",
              "      <td>12:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>May</td>\n",
              "      <td>Spring</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2019-05-22 13:00:00</td>\n",
              "      <td>13:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>May</td>\n",
              "      <td>Spring</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19027</th>\n",
              "      <td>2021-07-23 04:00:00</td>\n",
              "      <td>04:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19028</th>\n",
              "      <td>2021-07-23 05:00:00</td>\n",
              "      <td>05:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19029</th>\n",
              "      <td>2021-07-23 06:00:00</td>\n",
              "      <td>06:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19030</th>\n",
              "      <td>2021-07-23 07:00:00</td>\n",
              "      <td>07:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19031</th>\n",
              "      <td>2021-07-23 08:00:00</td>\n",
              "      <td>08:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>19032 rows × 6 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "        DateTime interval Hour Group  ...  Season Complaint total\n",
              "0     2019-05-22 09:00:00   09:00:00  ...  Spring               1\n",
              "1     2019-05-22 10:00:00   10:00:00  ...  Spring               0\n",
              "2     2019-05-22 11:00:00   11:00:00  ...  Spring               0\n",
              "3     2019-05-22 12:00:00   12:00:00  ...  Spring               0\n",
              "4     2019-05-22 13:00:00   13:00:00  ...  Spring               0\n",
              "...                   ...        ...  ...     ...             ...\n",
              "19027 2021-07-23 04:00:00   04:00:00  ...  Summer               0\n",
              "19028 2021-07-23 05:00:00   05:00:00  ...  Summer               0\n",
              "19029 2021-07-23 06:00:00   06:00:00  ...  Summer               0\n",
              "19030 2021-07-23 07:00:00   07:00:00  ...  Summer               0\n",
              "19031 2021-07-23 08:00:00   08:00:00  ...  Summer               1\n",
              "\n",
              "[19032 rows x 6 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LIMx3jOX6KCD"
      },
      "source": [
        "Please run the code block manually if you want to download the the df_coomplaints file on it's own. By default runnning this entire notebook will download this file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "yu6f390D5XvH",
        "outputId": "38fb14e3-4da8-45d4-aa34-9ee3bdb1343b"
      },
      "source": [
        "complaint_data_only.to_csv('df_complaints.csv', index=False) \n",
        "files.download('df_complaints.csv')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_744c27a9-0b4c-4207-81fb-ea9b025ce7f9\", \"df_complaints.csv\", 1002228)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lXg30JmrdLOr"
      },
      "source": [
        "The code block below will produce a sneak peak of what the merged dataframe will look like. This dataframe represents an aggregation of data from the pods - essentially averaging out all the values for any matching particular time frame. It also contains the number of complaints made at any particular time frame. Each measurement is done by the hour, but if you want to get aggregattions in different intervals, please see \"Optional Section\""
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uWxeb3AjaNHQ",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 786
        },
        "outputId": "25038e7d-31ac-4a93-bab5-279f23de0958"
      },
      "source": [
        "merged_data = get_merged_trimmed_df(df_weather_master,get_smell_df_master(),\"hours\")\n",
        "merged_data"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>DateTime interval</th>\n",
              "      <th>Complaint total</th>\n",
              "      <th>Hour Group</th>\n",
              "      <th>Day_of_week</th>\n",
              "      <th>Month_of_year</th>\n",
              "      <th>Season</th>\n",
              "      <th>Temp Avg</th>\n",
              "      <th>Temp Low</th>\n",
              "      <th>Temp High</th>\n",
              "      <th>Hum Avg</th>\n",
              "      <th>Hum Low</th>\n",
              "      <th>Hum High</th>\n",
              "      <th>Baro Avg</th>\n",
              "      <th>Baro Low</th>\n",
              "      <th>Baro High</th>\n",
              "      <th>Windspeed</th>\n",
              "      <th>Gust</th>\n",
              "      <th>Wind Direction</th>\n",
              "      <th>Cardinal Direction</th>\n",
              "      <th>Interval Precip</th>\n",
              "      <th>Inside Temp Avg</th>\n",
              "      <th>Inside Temp Low</th>\n",
              "      <th>Inside Temp High</th>\n",
              "      <th>Heat Index</th>\n",
              "      <th>Wind Chill</th>\n",
              "      <th>Dew Point</th>\n",
              "      <th>Station Voltage</th>\n",
              "      <th>wetbulb</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2020-09-02 19:00:00</td>\n",
              "      <td>3.0</td>\n",
              "      <td>19:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>September</td>\n",
              "      <td>Fall</td>\n",
              "      <td>75.590</td>\n",
              "      <td>75.20</td>\n",
              "      <td>75.70</td>\n",
              "      <td>61.0</td>\n",
              "      <td>61.0</td>\n",
              "      <td>61.0</td>\n",
              "      <td>29.820</td>\n",
              "      <td>29.810</td>\n",
              "      <td>29.830</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.0</td>\n",
              "      <td>75.000</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.590</td>\n",
              "      <td>75.60</td>\n",
              "      <td>61.20</td>\n",
              "      <td>6.5000</td>\n",
              "      <td>65.60</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2020-09-02 20:00:00</td>\n",
              "      <td>1.0</td>\n",
              "      <td>20:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>September</td>\n",
              "      <td>Fall</td>\n",
              "      <td>75.130</td>\n",
              "      <td>75.00</td>\n",
              "      <td>75.40</td>\n",
              "      <td>61.0</td>\n",
              "      <td>61.0</td>\n",
              "      <td>61.0</td>\n",
              "      <td>29.800</td>\n",
              "      <td>29.800</td>\n",
              "      <td>29.810</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.0</td>\n",
              "      <td>74.430</td>\n",
              "      <td>74.0</td>\n",
              "      <td>75.0</td>\n",
              "      <td>75.130</td>\n",
              "      <td>75.10</td>\n",
              "      <td>60.70</td>\n",
              "      <td>6.5000</td>\n",
              "      <td>65.80</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2020-09-02 21:00:00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>21:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>September</td>\n",
              "      <td>Fall</td>\n",
              "      <td>74.720</td>\n",
              "      <td>74.50</td>\n",
              "      <td>75.00</td>\n",
              "      <td>61.0</td>\n",
              "      <td>61.0</td>\n",
              "      <td>62.0</td>\n",
              "      <td>29.790</td>\n",
              "      <td>29.790</td>\n",
              "      <td>29.800</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.0</td>\n",
              "      <td>74.000</td>\n",
              "      <td>74.0</td>\n",
              "      <td>74.0</td>\n",
              "      <td>74.720</td>\n",
              "      <td>74.70</td>\n",
              "      <td>60.50</td>\n",
              "      <td>6.5000</td>\n",
              "      <td>65.50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2020-09-02 22:00:00</td>\n",
              "      <td>1.0</td>\n",
              "      <td>22:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>September</td>\n",
              "      <td>Fall</td>\n",
              "      <td>74.330</td>\n",
              "      <td>74.20</td>\n",
              "      <td>74.50</td>\n",
              "      <td>62.0</td>\n",
              "      <td>62.0</td>\n",
              "      <td>62.0</td>\n",
              "      <td>29.780</td>\n",
              "      <td>29.760</td>\n",
              "      <td>29.790</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.0</td>\n",
              "      <td>74.000</td>\n",
              "      <td>74.0</td>\n",
              "      <td>74.0</td>\n",
              "      <td>74.330</td>\n",
              "      <td>74.30</td>\n",
              "      <td>60.40</td>\n",
              "      <td>6.5000</td>\n",
              "      <td>65.40</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2020-09-02 23:00:00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>23:00:00</td>\n",
              "      <td>Wednesday</td>\n",
              "      <td>September</td>\n",
              "      <td>Fall</td>\n",
              "      <td>74.110</td>\n",
              "      <td>74.00</td>\n",
              "      <td>74.20</td>\n",
              "      <td>62.0</td>\n",
              "      <td>62.0</td>\n",
              "      <td>62.0</td>\n",
              "      <td>29.760</td>\n",
              "      <td>29.760</td>\n",
              "      <td>29.770</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>South-East</td>\n",
              "      <td>0.0</td>\n",
              "      <td>74.000</td>\n",
              "      <td>74.0</td>\n",
              "      <td>74.0</td>\n",
              "      <td>74.110</td>\n",
              "      <td>74.10</td>\n",
              "      <td>60.20</td>\n",
              "      <td>6.4970</td>\n",
              "      <td>65.10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7736</th>\n",
              "      <td>2021-07-23 04:00:00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>04:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>60.778</td>\n",
              "      <td>60.32</td>\n",
              "      <td>61.20</td>\n",
              "      <td>87.8</td>\n",
              "      <td>87.2</td>\n",
              "      <td>89.0</td>\n",
              "      <td>30.082</td>\n",
              "      <td>30.082</td>\n",
              "      <td>30.086</td>\n",
              "      <td>1.319002</td>\n",
              "      <td>4.14</td>\n",
              "      <td>256.4</td>\n",
              "      <td>None</td>\n",
              "      <td>0.0</td>\n",
              "      <td>68.524</td>\n",
              "      <td>68.2</td>\n",
              "      <td>69.0</td>\n",
              "      <td>60.778</td>\n",
              "      <td>60.78</td>\n",
              "      <td>57.10</td>\n",
              "      <td>6.4690</td>\n",
              "      <td>58.50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7737</th>\n",
              "      <td>2021-07-23 05:00:00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>05:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>60.214</td>\n",
              "      <td>59.86</td>\n",
              "      <td>60.48</td>\n",
              "      <td>89.2</td>\n",
              "      <td>88.0</td>\n",
              "      <td>89.6</td>\n",
              "      <td>30.084</td>\n",
              "      <td>30.082</td>\n",
              "      <td>30.090</td>\n",
              "      <td>1.253668</td>\n",
              "      <td>4.04</td>\n",
              "      <td>274.6</td>\n",
              "      <td>West</td>\n",
              "      <td>0.0</td>\n",
              "      <td>68.102</td>\n",
              "      <td>67.8</td>\n",
              "      <td>68.6</td>\n",
              "      <td>60.214</td>\n",
              "      <td>60.20</td>\n",
              "      <td>57.00</td>\n",
              "      <td>6.4588</td>\n",
              "      <td>58.24</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7738</th>\n",
              "      <td>2021-07-23 06:00:00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>06:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>60.084</td>\n",
              "      <td>59.78</td>\n",
              "      <td>60.44</td>\n",
              "      <td>89.0</td>\n",
              "      <td>88.4</td>\n",
              "      <td>89.6</td>\n",
              "      <td>30.094</td>\n",
              "      <td>30.088</td>\n",
              "      <td>30.104</td>\n",
              "      <td>1.076666</td>\n",
              "      <td>3.76</td>\n",
              "      <td>266.4</td>\n",
              "      <td>None</td>\n",
              "      <td>0.0</td>\n",
              "      <td>67.872</td>\n",
              "      <td>67.2</td>\n",
              "      <td>68.2</td>\n",
              "      <td>60.084</td>\n",
              "      <td>60.08</td>\n",
              "      <td>56.88</td>\n",
              "      <td>6.4502</td>\n",
              "      <td>58.14</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7739</th>\n",
              "      <td>2021-07-23 07:00:00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>07:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>60.732</td>\n",
              "      <td>59.92</td>\n",
              "      <td>61.84</td>\n",
              "      <td>87.6</td>\n",
              "      <td>85.6</td>\n",
              "      <td>89.0</td>\n",
              "      <td>30.108</td>\n",
              "      <td>30.104</td>\n",
              "      <td>30.114</td>\n",
              "      <td>0.862002</td>\n",
              "      <td>3.76</td>\n",
              "      <td>278.4</td>\n",
              "      <td>North</td>\n",
              "      <td>0.0</td>\n",
              "      <td>67.864</td>\n",
              "      <td>67.2</td>\n",
              "      <td>69.0</td>\n",
              "      <td>60.732</td>\n",
              "      <td>60.74</td>\n",
              "      <td>57.00</td>\n",
              "      <td>6.4898</td>\n",
              "      <td>58.46</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7740</th>\n",
              "      <td>2021-07-23 08:00:00</td>\n",
              "      <td>2.0</td>\n",
              "      <td>08:00:00</td>\n",
              "      <td>Friday</td>\n",
              "      <td>July</td>\n",
              "      <td>Summer</td>\n",
              "      <td>65.366</td>\n",
              "      <td>61.84</td>\n",
              "      <td>69.02</td>\n",
              "      <td>77.4</td>\n",
              "      <td>69.2</td>\n",
              "      <td>86.0</td>\n",
              "      <td>30.114</td>\n",
              "      <td>30.110</td>\n",
              "      <td>30.120</td>\n",
              "      <td>1.430664</td>\n",
              "      <td>5.72</td>\n",
              "      <td>270.8</td>\n",
              "      <td>West</td>\n",
              "      <td>0.0</td>\n",
              "      <td>71.048</td>\n",
              "      <td>68.2</td>\n",
              "      <td>73.2</td>\n",
              "      <td>65.366</td>\n",
              "      <td>65.34</td>\n",
              "      <td>57.84</td>\n",
              "      <td>6.8094</td>\n",
              "      <td>60.70</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>7741 rows × 28 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "       DateTime interval  Complaint total  ... Station Voltage wetbulb\n",
              "0    2020-09-02 19:00:00              3.0  ...          6.5000   65.60\n",
              "1    2020-09-02 20:00:00              1.0  ...          6.5000   65.80\n",
              "2    2020-09-02 21:00:00              0.0  ...          6.5000   65.50\n",
              "3    2020-09-02 22:00:00              1.0  ...          6.5000   65.40\n",
              "4    2020-09-02 23:00:00              0.0  ...          6.4970   65.10\n",
              "...                  ...              ...  ...             ...     ...\n",
              "7736 2021-07-23 04:00:00              0.0  ...          6.4690   58.50\n",
              "7737 2021-07-23 05:00:00              0.0  ...          6.4588   58.24\n",
              "7738 2021-07-23 06:00:00              0.0  ...          6.4502   58.14\n",
              "7739 2021-07-23 07:00:00              0.0  ...          6.4898   58.46\n",
              "7740 2021-07-23 08:00:00              2.0  ...          6.8094   60.70\n",
              "\n",
              "[7741 rows x 28 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BOyC9qgslh7z"
      },
      "source": [
        "Please run the code block below manually if you only want to download the df_merged file on it's own. By default runnning this entire notebook will download this file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "8S6cJWFBbejK",
        "outputId": "e2d2ad58-94ce-422a-8418-ff7be1eae42b"
      },
      "source": [
        "merged_data.to_csv('df_merged.csv', index=False) \n",
        "files.download('df_merged.csv')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_1c8e2679-5744-4721-b0c3-564e45927c07\", \"df_merged.csv\", 1728980)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5I7Imnb-8fIQ"
      },
      "source": [
        "The codeblock below shows the datframe used for logistic modeling."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "cellView": "form",
        "id": "Z6U4cq7H7F9i",
        "outputId": "f48204d6-e643-4208-c884-1989a9c289f5"
      },
      "source": [
        "#@title\n",
        "pd.options.mode.chained_assignment = None  # default='warn'\n",
        "def complaint_or_not(value : int):\n",
        "  if value > 0:\n",
        "    return 1\n",
        "  else:\n",
        "    return 0\n",
        "\n",
        "df_model_analysis = merged_data[['Complaint total','Temp Avg', 'Hum Avg',  'Baro Avg', 'Windspeed', 'Gust', 'Wind Direction', \n",
        " 'Heat Index', 'Wind Chill', 'Dew Point']]\n",
        "df_model_analysis['Complaint total'] = df_model_analysis['Complaint total'].apply(lambda x : complaint_or_not(x))\n",
        "df_model_analysis"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Complaint total</th>\n",
              "      <th>Temp Avg</th>\n",
              "      <th>Hum Avg</th>\n",
              "      <th>Baro Avg</th>\n",
              "      <th>Windspeed</th>\n",
              "      <th>Gust</th>\n",
              "      <th>Wind Direction</th>\n",
              "      <th>Heat Index</th>\n",
              "      <th>Wind Chill</th>\n",
              "      <th>Dew Point</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>75.590</td>\n",
              "      <td>61.0</td>\n",
              "      <td>29.820</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>75.590</td>\n",
              "      <td>75.60</td>\n",
              "      <td>61.20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>75.130</td>\n",
              "      <td>61.0</td>\n",
              "      <td>29.800</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>75.130</td>\n",
              "      <td>75.10</td>\n",
              "      <td>60.70</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0</td>\n",
              "      <td>74.720</td>\n",
              "      <td>61.0</td>\n",
              "      <td>29.790</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>74.720</td>\n",
              "      <td>74.70</td>\n",
              "      <td>60.50</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1</td>\n",
              "      <td>74.330</td>\n",
              "      <td>62.0</td>\n",
              "      <td>29.780</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>74.330</td>\n",
              "      <td>74.30</td>\n",
              "      <td>60.40</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0</td>\n",
              "      <td>74.110</td>\n",
              "      <td>62.0</td>\n",
              "      <td>29.760</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00</td>\n",
              "      <td>135.0</td>\n",
              "      <td>74.110</td>\n",
              "      <td>74.10</td>\n",
              "      <td>60.20</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>...</th>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "      <td>...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7736</th>\n",
              "      <td>0</td>\n",
              "      <td>60.778</td>\n",
              "      <td>87.8</td>\n",
              "      <td>30.082</td>\n",
              "      <td>1.319002</td>\n",
              "      <td>4.14</td>\n",
              "      <td>256.4</td>\n",
              "      <td>60.778</td>\n",
              "      <td>60.78</td>\n",
              "      <td>57.10</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7737</th>\n",
              "      <td>0</td>\n",
              "      <td>60.214</td>\n",
              "      <td>89.2</td>\n",
              "      <td>30.084</td>\n",
              "      <td>1.253668</td>\n",
              "      <td>4.04</td>\n",
              "      <td>274.6</td>\n",
              "      <td>60.214</td>\n",
              "      <td>60.20</td>\n",
              "      <td>57.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7738</th>\n",
              "      <td>0</td>\n",
              "      <td>60.084</td>\n",
              "      <td>89.0</td>\n",
              "      <td>30.094</td>\n",
              "      <td>1.076666</td>\n",
              "      <td>3.76</td>\n",
              "      <td>266.4</td>\n",
              "      <td>60.084</td>\n",
              "      <td>60.08</td>\n",
              "      <td>56.88</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7739</th>\n",
              "      <td>0</td>\n",
              "      <td>60.732</td>\n",
              "      <td>87.6</td>\n",
              "      <td>30.108</td>\n",
              "      <td>0.862002</td>\n",
              "      <td>3.76</td>\n",
              "      <td>278.4</td>\n",
              "      <td>60.732</td>\n",
              "      <td>60.74</td>\n",
              "      <td>57.00</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7740</th>\n",
              "      <td>1</td>\n",
              "      <td>65.366</td>\n",
              "      <td>77.4</td>\n",
              "      <td>30.114</td>\n",
              "      <td>1.430664</td>\n",
              "      <td>5.72</td>\n",
              "      <td>270.8</td>\n",
              "      <td>65.366</td>\n",
              "      <td>65.34</td>\n",
              "      <td>57.84</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>7741 rows × 10 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "      Complaint total  Temp Avg  Hum Avg  ...  Heat Index  Wind Chill  Dew Point\n",
              "0                   1    75.590     61.0  ...      75.590       75.60      61.20\n",
              "1                   1    75.130     61.0  ...      75.130       75.10      60.70\n",
              "2                   0    74.720     61.0  ...      74.720       74.70      60.50\n",
              "3                   1    74.330     62.0  ...      74.330       74.30      60.40\n",
              "4                   0    74.110     62.0  ...      74.110       74.10      60.20\n",
              "...               ...       ...      ...  ...         ...         ...        ...\n",
              "7736                0    60.778     87.8  ...      60.778       60.78      57.10\n",
              "7737                0    60.214     89.2  ...      60.214       60.20      57.00\n",
              "7738                0    60.084     89.0  ...      60.084       60.08      56.88\n",
              "7739                0    60.732     87.6  ...      60.732       60.74      57.00\n",
              "7740                1    65.366     77.4  ...      65.366       65.34      57.84\n",
              "\n",
              "[7741 rows x 10 columns]"
            ]
          },
          "metadata": {},
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iqv8Sbsh804N"
      },
      "source": [
        "Please run the code block below manually if you only want to download the df_model_analysis file on it's own. By default runnning this entire notebook will download this file."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        },
        "id": "C9Tw9LnK8N2w",
        "outputId": "9b9efbb7-26ad-491e-fdea-deda43dea37a"
      },
      "source": [
        "df_model_analysis.to_csv('df_model_analysis.csv', index=False) \n",
        "files.download('df_model_analysis.csv')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "\n",
              "    async function download(id, filename, size) {\n",
              "      if (!google.colab.kernel.accessAllowed) {\n",
              "        return;\n",
              "      }\n",
              "      const div = document.createElement('div');\n",
              "      const label = document.createElement('label');\n",
              "      label.textContent = `Downloading \"${filename}\": `;\n",
              "      div.appendChild(label);\n",
              "      const progress = document.createElement('progress');\n",
              "      progress.max = size;\n",
              "      div.appendChild(progress);\n",
              "      document.body.appendChild(div);\n",
              "\n",
              "      const buffers = [];\n",
              "      let downloaded = 0;\n",
              "\n",
              "      const channel = await google.colab.kernel.comms.open(id);\n",
              "      // Send a message to notify the kernel that we're ready.\n",
              "      channel.send({})\n",
              "\n",
              "      for await (const message of channel.messages) {\n",
              "        // Send a message to notify the kernel that we're ready.\n",
              "        channel.send({})\n",
              "        if (message.buffers) {\n",
              "          for (const buffer of message.buffers) {\n",
              "            buffers.push(buffer);\n",
              "            downloaded += buffer.byteLength;\n",
              "            progress.value = downloaded;\n",
              "          }\n",
              "        }\n",
              "      }\n",
              "      const blob = new Blob(buffers, {type: 'application/binary'});\n",
              "      const a = document.createElement('a');\n",
              "      a.href = window.URL.createObjectURL(blob);\n",
              "      a.download = filename;\n",
              "      div.appendChild(a);\n",
              "      a.click();\n",
              "      div.remove();\n",
              "    }\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/javascript": [
              "download(\"download_0b1c8540-e24e-407d-a293-860295868710\", \"df_model_analysis.csv\", 591032)"
            ],
            "text/plain": [
              "<IPython.core.display.Javascript object>"
            ]
          },
          "metadata": {}
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cos4eGGmmXE4"
      },
      "source": [
        "###OPTIONAL\n",
        "\n",
        "By default the code block below will not download this optional file\n",
        "\n",
        "If you would like to aggregate the data in different time intervals for the merged_file Please see instructions below\n",
        "\n",
        "1.) In the code block below please change optional = 0 to optional = 1\n",
        "\n",
        "2.) Please change key = 'hours' to the appropriate key. Key choices include:\n",
        "  'days', 'weeks', 'months'\n",
        "\n",
        "3.) If you would like to aggregate in a specific quanitity of intervals please change interval = 1 to interval = quantity \n",
        "\n",
        "4.) Then run the code block below. \n",
        "\n",
        "An example is shown in the text cell titled 'OPTIONAL EXAMPLE'"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QMxlxE5tnIPX"
      },
      "source": [
        "optional = 0\n",
        "key = 'hours' \n",
        "interval = 1\n",
        "file_name = 'user_choice_merged_by_'+ str(interval) + 'key' + '.csv'\n",
        "if (optional == 1):\n",
        "  user_choice_merged = get_merged_trimmed_df(df_weather_master, get_smell_df_master(), key = key, interval=interval)\n",
        "  display(user_choice_merged)\n",
        "  user_choice_merged.to_csv(file_name, index = False)\n",
        "  files.download('user_choice_merged.csv')"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J9raveMVn8EJ"
      },
      "source": [
        "### OPTIONAL EXAMPLE\n",
        "***Please note this is a text block and NOT a code block, so this cell block can NOT be run.\n",
        "\n",
        "If I wanted to download the merged file which aggregated the data in intervals of 3 days, then I would need to make the following changes to the code block above:\n",
        "\n",
        "```python\n",
        "optional = 1 #change optional to = 1\n",
        "key = 'days'  # change key = 'hours' to key = 'days'\n",
        "interval = 3 # change interval = 1 to interval = 3\n",
        "if (optional == 1):\n",
        "  user_choice_merged = get_merged_trimmed_df(df_weather_master, get_smell_df_master(), key = key, interval=interval)\n",
        "  display(user_choice_merged)\n",
        "  user_choice_merged.to_csv('user_choice_merged.csv', index = False)\n",
        "  files.download('user_choice_merged.csv')\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MhrgBp3Bi_G2"
      },
      "source": [
        "##END \n",
        "Code and analyses by Ramzi Adil, Syed Haider, and Karen Paresh for DS5110 Summer 2021 Semester.\n",
        "\n",
        "Many thanks to Prof. Bogden for the guidance and resources important for this project!"
      ]
    }
  ]
}