PPG&BP- Method 6- Pre-processing (2).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3: Pre-processing\n",
    "Pre-processing of PPG signal segments includes smoothing the signal, finding cycles using the peak points in each cycle, two-dimensional normalization of PPG signals. Also, rejecting low-quality segments and cutting the segments so that they contain complete waves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Plot examples original PPG signals (before pre-processing)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_raws_PPG_signal(PPG, num):\n",
    "    \"\"\"\n",
    "    Plot examples of raws PPG signal.\n",
    "\n",
    "    Parameters:\n",
    "    - PPG (DataFrame): The PPG data containing multiple segments.\n",
    "    - num (int): The number of segment examples to plot.\n",
    "\n",
    "    Returns:\n",
    "    None\n",
    "    \"\"\"\n",
    "    \n",
    "    for i in range(0, num):\n",
    "        # Extract the data for the current example\n",
    "        data = (PPG.iloc[i * 700: (i + 1) * 700].values).copy()\n",
    "        \n",
    "        # Plot the PPG data \n",
    "        plt.figure(figsize=(7, 3))\n",
    "        plt.title(f'Raw PPG')\n",
    "        plt.plot(data)\n",
    "        plt.xlabel('Number of samples')\n",
    "        plt.ylabel('Amplitude (volts)')\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the train data before pre-proccesing \n",
    "# plot_raws_PPG_signal(train_df['PLETH'], 50)\n",
    "\n",
    "# Plot the test data before pre-proccesing \n",
    "# plot_raws_PPG_signal(test_df['PLETH'], 30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A: Noise filtering\n",
    "Noise filtering by a fourth-order Savitzky–Golay filter with a window size of 19. This filter is a moving average filter to smooth the PPG signal in order to reduce noise and capture trends or patterns in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Filter_Savitzky_Golay(data):\n",
    "    \"\"\"\n",
    "    Filter the PLETH signal by segments using the Savitzky-Golay filter.\n",
    "\n",
    "    Parameters:\n",
    "    - data (DataFrame): The DataFrame containing the PLETH signal and segments numbers.\n",
    "\n",
    "    Returns:\n",
    "    None\n",
    "\n",
    "    Modifies the input DataFrame by adding a new column 'PLETH_filtered' containing the filtered signal.\n",
    "    \"\"\"\n",
    "\n",
    "    # Group the PLETH signal by segments\n",
    "    grouped_df = data['PLETH'].groupby(data['Part_Number'])\n",
    "    PLETH_filtered = np.array([])\n",
    "\n",
    "    # Apply the Savitzky-Golay filter to each segment of the PLETH signal\n",
    "    for part in grouped_df:\n",
    "        part_filtered = savgol_filter(part[1].values, window_length=19, polyorder=4)\n",
    "        PLETH_filtered = np.concatenate((PLETH_filtered.ravel(), part_filtered.ravel()))\n",
    "\n",
    "    # Add a new column to the DataFrame with the filtered signal\n",
    "    data['PLETH_filtered'] = PLETH_filtered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the filter on the train & test segments\n",
    "Filter_Savitzky_Golay(train_df)\n",
    "Filter_Savitzky_Golay(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### A.1: Plot examples of segments after filter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_filtered_PPG_signal(PPG, num):\n",
    "    \"\"\"\n",
    "    Plot examples of filtered PPG signal.\n",
    "\n",
    "    Parameters:\n",
    "    - PPG (DataFrame): The PPG data containing multiple segments.\n",
    "    - num (int): The number of segment examples to plot.\n",
    "\n",
    "    Returns:\n",
    "    None\n",
    "    \"\"\"\n",
    "    \n",
    "    for i in range(0, num):\n",
    "        # Extract the data for the current example\n",
    "        data = (PPG.iloc[i * 700: (i + 1) * 700]).copy()\n",
    "        \n",
    "        # Plot the PPG data \n",
    "        plt.figure(figsize=(10, 5))\n",
    "        \n",
    "        plt.subplot(121)\n",
    "        plt.title(f'Raw PPG segment')\n",
    "        plt.plot(data['PLETH'].values)\n",
    "        plt.xlabel('Number of samples')\n",
    "        plt.ylabel('Amplitude (volts)')\n",
    "        \n",
    "        plt.subplot(122)\n",
    "        plt.title(f'Filterded PPG segment')\n",
    "        plt.plot(data['PLETH_filtered'].values)\n",
    "        plt.xlabel('Number of samples')\n",
    "        plt.ylabel('Amplitude (volts)')\n",
    "        \n",
    "        plt.tight_layout()\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot train example segments before & after noise filtering \n",
    "# plot_filtered_PPG_signal(train_df, 10)\n",
    "\n",
    "# Plot test example segments before & after noise filtering \n",
    "# plot_filtered_PPG_signal(test_df, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### B: Trend removel\n",
    "The removal of the trend line caused by the breathing activity has also been removed from the segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Trend_Removel(data):\n",
    "    \"\"\"\n",
    "    Apply trend removal to the PLETH signal by parts.\n",
    "\n",
    "    Parameters:\n",
    "    - data (DataFrame): The DataFrame containing the PLETH signal and part numbers.\n",
    "\n",
    "    Returns:\n",
    "    None\n",
    "\n",
    "    Modifies the input DataFrame by updating the 'PLETH_filtered' column with the trend-removed signal.\n",
    "    \"\"\"\n",
    "\n",
    "    # Group the filtered PLETH signal by part number\n",
    "    grouped_df = data['PLETH_filtered'].groupby(data['Part_Number'])\n",
    "    PLETH_trend_removal = np.array([])\n",
    "\n",
    "    # Apply trend removal to each part of the filtered PLETH signal\n",
    "    for part in grouped_df:\n",
    "        part_detrend = signal.detrend(part[1])\n",
    "        PLETH_trend_removal = np.concatenate((PLETH_trend_removal.ravel(), part_detrend.ravel()))\n",
    "\n",
    "    # Update the 'PLETH_filtered' column with the trend-removed signal\n",
    "    data['PLETH_trend_removel'] = PLETH_trend_removal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the trend removel on the train & test segments\n",
    "Trend_Removel(train_df)\n",
    "Trend_Removel(test_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### B.1: Plot examples of segments after trend removel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_trend_removel_PPG_signal(PPG, num):\n",
    "    \"\"\"\n",
    "    Plot examples of trend removel PPG signal compared to the filtered signal.\n",
    "\n",
    "    Parameters:\n",
    "    - PPG (DataFrame): The PPG data containing multiple segments.\n",
    "    - num (int): The number of segment examples to plot.\n",
    "\n",
    "    Returns:\n",
    "    None\n",
    "    \"\"\"\n",
    "    \n",
    "    for i in range(0, num):\n",
    "        # Extract the data for the current example\n",
    "        data = (PPG.iloc[i * 700: (i + 1) * 700]).copy()\n",
    "        \n",
    "        # Plot the PPG data \n",
    "        plt.figure(figsize=(7, 3))\n",
    "        \n",
    "        plt.title(f'PPG segment')\n",
    "        plt.plot(data['PLETH_filtered'].values, label='Filterded PPG')\n",
    "        plt.plot(data['PLETH_trend_removel'].values, label='Trend removel PPG')\n",
    "        plt.xlabel('Number of samples')\n",
    "        plt.ylabel('Amplitude (volts)')\n",
    "        \n",
    "        plt.legend(loc='lower right', fontsize=8)\n",
    "        plt.tight_layout()\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot train example segments before & after trend removel\n",
    "# plot_trend_removel_PPG_signal(train_df, 10)\n",
    "\n",
    "# Plot test example segments before & after trend removel\n",
    "# plot_trend_removel_PPG_signal(test_df, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### C: Cycle detection\n",
    "Using the heartpy library, for each segment the cycles were defined based on finding the peak points of each cycle in the single."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_max_peaks(data):\n",
    "    \"\"\"\n",
    "    Extract the indices of the maximum peaks from the input data using the heartpy library.\n",
    "\n",
    "    Parameters:\n",
    "    - data (array-like): The input data for peak extraction.\n",
    "\n",
    "    Returns:\n",
    "    - max_peaks (numpy array): An array of indices representing the positions of the maximum peaks.\n",
    "    \"\"\"\n",
    "    \n",
    "    # The input data is processed using the heartpy library's `process()` function with a sampling rate of 100.0 (100 Hz).\n",
    "    working_data, measures = hp.process(data, 100.0)\n",
    "    \n",
    "    # Extract the 'peaklist' from the 'working_data' dictionary.\n",
    "    max_peaks = np.array(working_data['peaklist'])\n",
    "    \n",
    "    return max_peaks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_min_peaks(data, max_peaks):\n",
    "    \"\"\"\n",
    "    Extract the indices of the minimum peaks from the input data based on the indices of the maximum peaks.\n",
    "\n",
    "    Parameters:\n",
    "    - data (array-like): The input data for peak extraction.\n",
    "    - max_peaks (array-like): An array of indices representing the positions of the maximum peaks.\n",
    "\n",
    "    Returns:\n",
    "    - min_peaks (numpy array): An array of indices representing the positions of the minimum peaks.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Initialized 'min_peaks' list to store the indices of the minimum peaks.\n",
    "    min_peaks = []\n",
    "\n",
    "    # Check if the minimum value between the first maximum peak and the preceding data point is less than the first data point. \n",
    "    if np.min(data[1:max_peaks[0]]) < data[0]:\n",
    "        # If true, the index of the minimum value is appended to 'min_peaks'.\n",
    "        min_peaks.append(np.argmin(data[0:max_peaks[0]]))\n",
    "\n",
    "    # Iterates over pairs of adjacent maximum peaks using a loop. \n",
    "    for peak0, peak1 in zip(max_peaks[:-1], max_peaks[1:]):\n",
    "        # Find the index of the minimum value within the corresponding data range between the two maximum peaks and appends it to 'min_peaks'.\n",
    "        min_peaks.append(peak0 + np.argmin(data[peak0:peak1]))\n",
    "        \n",
    "    # 'min_peaks' list is converted to a numpy array.\n",
    "    min_peaks = np.array(min_peaks)\n",
    "    \n",
    "    return min_peaks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_peaks(data):\n",
    "    \"\"\"\n",
    "    Calculate the indices of the minimum and maximum peaks in the input data using the `get_max_peaks()` and `get_min_peaks()` functions.\n",
    "\n",
    "    Parameters:\n",
    "    - data (array-like): The input data for peak extraction.\n",
    "\n",
    "    Returns:\n",
    "    - max_peaks (numpy array): An array of indices representing the positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): An array of indices representing the positions of the minimum peaks.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Use `get_max_peaks()` funtion to calculate the indices of the maximum peaks in the input data.\n",
    "    max_peaks = get_max_peaks(data)\n",
    "   \n",
    "    # Use `get_min_peaks()` with the input data and the indices of the maximum peaks to calculate the indices of the corresponding minimum peaks.\n",
    "    min_peaks = get_min_peaks(data, max_peaks)\n",
    "    \n",
    "    # Check the sizes of the `max_peaks` and `min_peaks` arrays to ensure they have the same number of elements. If the sizes are not equal, it adjusts the `max_peaks` array by removing the first element.\n",
    "    if max_peaks.size > min_peaks.size:\n",
    "        max_peaks = max_peaks[1:]\n",
    "    elif max_peaks.size < min_peaks.size:\n",
    "        max_peaks = max_peaks[1:]\n",
    "\n",
    "    return max_peaks, min_peaks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### C.1: Plot examples of segments for which peaks were not found"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_examples_without_peaks(df, num):\n",
    "    \"\"\"\n",
    "    Plots PPG signal examples without any peaks found.\n",
    "\n",
    "    Parameters:\n",
    "        df (DataFrame): DataFrame containing the PPG signal data.\n",
    "        num (int): Number of examples to plot.\n",
    "    \"\"\"\n",
    "\n",
    "    for i in range(0, num):\n",
    "        data = (df['PLETH_trend_removel'].iloc[i*700:(i+1)*700].values).copy()\n",
    "        try:\n",
    "            max_peaks, min_peaks = get_peaks(data)\n",
    "        except:\n",
    "            # Plot the PPG signal with no peaks found\n",
    "            plt.figure(figsize=(7, 3))\n",
    "            plt.title(f'Bad PPG segment (no peaks found)')\n",
    "            plt.plot(data)\n",
    "            plt.xlabel('Number of samples')\n",
    "            plt.ylabel('Amplitude (volts)')\n",
    "            pass\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot examples of segments using plot_examples_without_peaks function\n",
    "# plot_examples_without_peaks(train_df, 1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### D: Disqualification of segments\n",
    "Artifacts were filtered from the training data set according to two requirements defined by learning the best parameters, so that only high-quality signals were retained while leaving enough data for training. The segments were rejected by defining the coefficient of variation for time (X) and amplitude (Y)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "def Conditions_PPG(data, max_peaks, min_peaks, amp_cv=4, time_cv=12):\n",
    "    \"\"\"\n",
    "    Apply filtering conditions to identify \"good\" segments of the signal based on amplitude and time criteria.\n",
    "\n",
    "    Parameters:\n",
    "    - data (array-like): The input PPG signal data.\n",
    "    - max_peaks (numpy array): An array of indices representing the positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): An array of indices representing the positions of the minimum peaks.\n",
    "    - amp_std (float): The threshold value for the amplitude coefficient of variation. Default is 4.\n",
    "    - time_std (float): The threshold value for the time coefficient of variation. Default is 12.\n",
    "\n",
    "    Returns:\n",
    "    - amp_standart_score (float): The normalized amplitude standard score.\n",
    "    - time_standart_score (float): The normalized time standard score.\n",
    "    - is_good_part (bool): Indicates whether the part of the signal satisfies the filtering conditions.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Calculate the time differences between consecutive minimum and maximum peaks in the PPG signal.\n",
    "    time_min = min_peaks[1:] - min_peaks[:-1]\n",
    "    time_max = max_peaks[1:] - max_peaks[:-1]\n",
    "\n",
    "    # Calculate the mean time difference and calculate the amplitude coefficient of variation using the mean and standard deviation.\n",
    "    mean_time = (time_min + time_max) / 2\n",
    "    time_cv_score = mean_time.mean() / mean_time.std()\n",
    "\n",
    "    # Calculate the amplitude differences between corresponding maximum and minimum peaks and calculate the time coefficient of variation using the mean and standard deviation.\n",
    "    amp = data[max_peaks] - data[min_peaks]\n",
    "    amp_cv_score = amp.mean() / amp.std()\n",
    "\n",
    "    # Checks if the normalized amplitude coefficient of variation is greater than amp_std and the normalized time standard score is greater than time_std.\n",
    "    if (amp_cv_score > amp_cv) and (time_cv_score > time_cv):\n",
    "        # If both conditions are satisfied, the segment of the signal is considered \"good\".\n",
    "        return amp_cv_score, time_cv_score, True\n",
    "    \n",
    "    else:\n",
    "        # If the conditions are not met,the segment of the signal is considered \" not good\".\n",
    "        return amp_cv_score, time_cv_score, False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### D.1: Plot examples of valid & invalid segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 212,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_examples_valid_and_invalid_segments(df):\n",
    "    \"\"\"\n",
    "    Plots examples of PPG signals that are valid & invalid segments.\n",
    "\n",
    "    Args:\n",
    "        df (DataFrame): DataFrame containing the PPG signal data.\n",
    "        num (int): Number of examples to plot.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Grouped the data frame by segments\n",
    "    grouped_df = df.groupby(df['Part_Number'])\n",
    "\n",
    "    # Iterate over each segment in the grouped data frame\n",
    "    for part in grouped_df:\n",
    "        # Extract the PPG signal for the segment\n",
    "        data = (part[1]['PLETH_trend_removel'].values).copy()\n",
    "        \n",
    "        try:\n",
    "            # Attempt to find the max and min peaks using the 'get_peaks' function\n",
    "            max_peaks, min_peaks = get_peaks(data)\n",
    "        except:\n",
    "            pass\n",
    "        else:\n",
    "            # Evaluate the conditions on the PPG data using the 'Conditions_PPG' function\n",
    "            amp_cv_score, time_cv_score, booli = Conditions_PPG(data, max_peaks, min_peaks)\n",
    "            if booli:\n",
    "                # Plot the original and the normalized signals with peaks\n",
    "                plt.figure(figsize=(7, 4))\n",
    "                plt.title(f'Valid PPG segment')\n",
    "                plt.plot(data)\n",
    "                plt.plot(max_peaks, data[max_peaks], 'or')\n",
    "                plt.plot(min_peaks, data[min_peaks], 'oy')\n",
    "                plt.ylabel('Amplitude (volts)')\n",
    "                plt.xlabel('Number of samples')\n",
    "                plt.show()\n",
    "                \n",
    "            else: \n",
    "                plt.figure(figsize=(7, 4))\n",
    "                plt.title(f'Invalid PPG segment')\n",
    "                plt.ylabel('Amplitude (volts)')\n",
    "                plt.xlabel('Number of samples')\n",
    "                plt.plot(data)\n",
    "                plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot train example valid & invalid segments\n",
    "# plot_examples_valid_and_invalid_segments(train_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### E: 2-dimensional normalization\n",
    "For the signal segments that passed the above conditions, normalization was performed in width and amplitude."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def normalize_data(data, max_peaks, min_peaks):\n",
    "    \"\"\"\n",
    "    Normalize the PPG signal data between peaks by performing linear normalization on the corresponding data points \n",
    "    within each peak window.\n",
    "\n",
    "    Parameters:\n",
    "    - data (array-like): The input PPG signal data.\n",
    "    - max_peaks (numpy array): An array of indices representing the positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): An array of indices representing the positions of the minimum peaks.\n",
    "\n",
    "    Returns:\n",
    "    - norm_data (numpy array): The normalized amplitude (y) coordinates between peaks.\n",
    "    - norm_time (numpy array): The normalized time (x) coordinates between peaks.\n",
    "    - all_peaks (numpy array): An array of indices representing the positions of all peaks.\n",
    "    - max_peaks (numpy array): The original positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): The original positions of the minimum peaks.\n",
    "    \"\"\"\n",
    "\n",
    "    # Initializes arrays for storing the normalized time and amplitude values.\n",
    "    norm_data = np.zeros_like(data)\n",
    "    norm_time = norm_data.copy()\n",
    "    \n",
    "    # Concatenates the positions of maximum and minimum peaks into a single array and sorts them in ascending order.\n",
    "    all_peaks = np.sort(np.concatenate([max_peaks, min_peaks], axis=0))\n",
    "\n",
    "    # Iterates over eacSh pair of consecutive peaks and performs linear normalization on the corresponding data points \n",
    "    # within the window.\n",
    "    for i, (peak0, peak1) in enumerate(zip(all_peaks[:-1], all_peaks[1:])):\n",
    "        window = data[peak0:peak1].copy()\n",
    "        if peak0 == peak1:\n",
    "            window = np.append(window, peak0)\n",
    "        window -= window.min()\n",
    "        window /= window.max()\n",
    "        norm_data[peak0:peak1] = window\n",
    "        norm_time[peak0:peak1] = np.linspace(0.5 * i, 0.5 * (i + 1), window.size + 1)[1:]\n",
    "    \n",
    "    return norm_data, norm_time, all_peaks, max_peaks, min_peaks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### F: Cutting into whole cycles\n",
    "Cutting of the normal and normalized segments was performed so that each segment contains only complete cycles. That is, each section now contains a sequence of complete cycles only, from the first high point to the last high point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def cutting_into_whole_cycles(data, time, max_peaks, min_peaks):\n",
    "    \"\"\"\n",
    "    Cuts the data into whole cycles by trimming the time and amplitude arrays to exclude values outside the range of the concatenated peaks.\n",
    "\n",
    "    Parameters:\n",
    "    - data (numpy array): The amplitude (y) coordinates.\n",
    "    - time (numpy array): The time (x) coordinates.\n",
    "    - all_peaks (numpy array): An array of indices representing the positions of all peaks.\n",
    "    - max_peaks (numpy array): The original positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): The original positions of the minimum peaks.\n",
    "\n",
    "    Returns:\n",
    "    - time (numpy array): The trimmed time (x) coordinates.\n",
    "    - data (numpy array): The trimmed amplitude (y) coordinates.\n",
    "    - fix_min_peaks (numpy array): The adjusted positions of the fixed minimum peaks.\n",
    "    - fix_max_peaks (numpy array): The adjusted positions of the fixed maximum peaks.\n",
    "    \"\"\"\n",
    "\n",
    "    # Trim the normalized time and amplitude arrays to exclude any values outside the range of the concatenated peaks.\n",
    "    all_peaks = np.sort(np.concatenate([max_peaks, min_peaks], axis=0))\n",
    "    data = data[all_peaks[0]:all_peaks[-1]]\n",
    "    time = time[all_peaks[0]:all_peaks[-1]]\n",
    "    time[0] = 0\n",
    "    \n",
    "    # Adjusts the positions of the normalized maximum and minimum peaks to match the trimmed arrays.\n",
    "    fix_max_peaks = max_peaks - all_peaks[0]\n",
    "    fix_min_peaks = min_peaks - all_peaks[0]\n",
    "\n",
    "    # If the last position of the normalized maximum peaks is greater than the last position of the normalized minimum peaks, it decrements the last position of the normalized maximum peaks by 1.\n",
    "    if fix_max_peaks[-1] > fix_min_peaks[-1]:\n",
    "        fix_max_peaks[-1] -= 1\n",
    "        \n",
    "    # If the last position of the normalized maximum peaks is less than the last position of the normalized minimum peaks, it decrements the last position of the normalized minimum peaks by 1.\n",
    "    elif fix_max_peaks[-1] < fix_min_peaks[-1]:\n",
    "        fix_min_peaks[-1] -= 1\n",
    "\n",
    "    return time, data, fix_min_peaks, fix_max_peaks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### F.1: Plot examples of processed segments & normalized processed segments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_examples_with_peaks_and_norm(df, num):\n",
    "    \"\"\"\n",
    "    Plots examples of PPG signals with peaks and their corresponding normalized segments.\n",
    "\n",
    "    Args:\n",
    "        df (DataFrame): DataFrame containing the PPG signal data.\n",
    "        num (int): Number of examples to plot.\n",
    "    \"\"\"\n",
    "    \n",
    "    # Grouped the data frame by segments\n",
    "    grouped_df = df.groupby(df['Part_Number'])\n",
    "\n",
    "    # Iterate over each segment in the grouped data frame\n",
    "    for part in grouped_df:\n",
    "        # Extract the PPG signal for the segment\n",
    "        data = (part[1]['PLETH_trend_removel'].values).copy()\n",
    "        \n",
    "        try:\n",
    "            # Attempt to find the max and min peaks using the 'get_peaks' function\n",
    "            max_peaks, min_peaks = get_peaks(data)\n",
    "        except:\n",
    "            pass\n",
    "        else:\n",
    "            # Evaluate the conditions on the PPG data using the 'Conditions_PPG' function\n",
    "            amp_cv_score, time_cv_score, booli = Conditions_PPG(data, max_peaks, min_peaks)\n",
    "            if booli:\n",
    "                # Plot the original and the normalized signals with peaks\n",
    "                plt.figure(figsize=(10, 5))\n",
    "        \n",
    "                plt.subplot(121)\n",
    "                plt.title(f'Processed original PPG signal')\n",
    "                plt.plot(data)\n",
    "                plt.plot(max_peaks, data[max_peaks], 'or')\n",
    "                plt.plot(min_peaks, data[min_peaks], 'oy')\n",
    "\n",
    "                norm_data, norm_time, all_peaks, max_peaks, min_peaks = normalize_data(data, max_peaks, min_peaks)\n",
    "                norm_time, norm_data, norm_min_peaks, norm_max_peaks = cutting_into_whole_cycles(norm_data, norm_time, max_peaks, min_peaks)\n",
    "                plt.subplot(122)\n",
    "                plt.title(f'Processed normalized PPG segment')\n",
    "                plt.plot(norm_time, norm_data)\n",
    "                plt.plot(norm_time[norm_max_peaks], norm_data[norm_max_peaks], 'or')\n",
    "                plt.plot(norm_time[norm_min_peaks], norm_data[norm_min_peaks], 'oy')\n",
    "                plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the train data after pre-proccesing   \n",
    "# plot_examples_with_peaks_and_norm(train_df, 10)\n",
    "\n",
    "# Plot the test data after pre-proccesing  \n",
    "# plot_examples_with_peaks_and_norm(test_df, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All the pre-processing \n",
    "Display all the pre-process steps on the data (train & test) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [],
   "source": [
    "def norm_df_with_peaks(data, max_peaks, min_peaks):\n",
    "    \"\"\"\n",
    "    Creates a normalized data frame with the peaks by applying normalization and cutting into whole cycles.\n",
    "\n",
    "    Parameters:\n",
    "    - data (numpy array): The original PPG signal data.\n",
    "    - max_peaks (numpy array): The positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): The positions of the minimum peaks.\n",
    "\n",
    "    Returns:\n",
    "    - norm_part_df (pandas DataFrame): The normalized data frame with the peaks.\n",
    "    \"\"\"\n",
    "\n",
    "    # Call the normalize_data function to normalize the data based on the maximum and minimum peaks.\n",
    "    norm_data, norm_time, all_peaks, max_peaks, min_peaks = normalize_data(data, max_peaks, min_peaks)\n",
    "    \n",
    "    # Call the cutting_into_whole_cycles function to trim the normalized data to include only whole cycles.\n",
    "    norm_time, norm_data, norm_min_peaks, norm_max_peaks = cutting_into_whole_cycles(norm_data, norm_time, all_peaks, max_peaks, min_peaks)\n",
    "    \n",
    "    # Create an empty DataFrame to store the normalized data.\n",
    "    norm_part_df = pd.DataFrame()\n",
    "    \n",
    "    # Add columns for normalized time and data.\n",
    "    norm_part_df['norm_time(X)'] = norm_time\n",
    "    norm_part_df['norm_data(y)'] = norm_data\n",
    "    \n",
    "    # Assign 'Min' and 'Max' labels to the corresponding peaks.\n",
    "    norm_part_df.loc[norm_min_peaks, 'peak'] = 'Min'\n",
    "    norm_part_df.loc[norm_max_peaks, 'peak'] = 'Max'\n",
    "    \n",
    "    return norm_part_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "def df_with_peaks(data, max_peaks, min_peaks):\n",
    "    \"\"\"\n",
    "    Creates the original data frame with the peaks by extracting the peaks from the original data.\n",
    "\n",
    "    Parameters:\n",
    "    - data (pandas DataFrame): The original PPG data frame.\n",
    "    - max_peaks (numpy array): The positions of the maximum peaks.\n",
    "    - min_peaks (numpy array): The positions of the minimum peaks.\n",
    "\n",
    "    Returns:\n",
    "    - new_part_df (pandas DataFrame): The original data frame with the peaks.\n",
    "    \"\"\"\n",
    "\n",
    "    # Create a copy of the second part of the original data frame to store the new data frame with the peaks.\n",
    "    new_part_df = data[1].copy()\n",
    "\n",
    "    # Reset the index of the new data frame to ensure sequential index values.\n",
    "    new_part_df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "    # Concatenate the positions of the maximum and minimum peaks into a single array and sort them in ascending order.\n",
    "    all_peaks = np.sort(np.concatenate([max_peaks, min_peaks], axis=0))\n",
    "\n",
    "    # Assign 'Min' and 'Max' labels to the rows in the new data frame corresponding to the positions of the minimum and maximum peaks, respectively.\n",
    "    new_part_df.loc[min_peaks, 'peak'] = 'Min'\n",
    "    new_part_df.loc[max_peaks, 'peak'] = 'Max'\n",
    "    \n",
    "    # Extracts the data between the first and last peak positions to include only the relevant data within the peaks.\n",
    "    new_part_df = new_part_df[all_peaks[0]:all_peaks[-1]]\n",
    "\n",
    "    # Calculate the number of cycles in the segment by 2 and subtracting 1 (since the first peak is not considered a complete cycle).\n",
    "    cycle_nums = (len(all_peaks) / 2) - 1\n",
    "    \n",
    "    # Add a new column 'Num_of_Cycles_in_part' to the new data frame, indicating the number of cycles in each part.\n",
    "    new_part_df['Num_of_Cycles_in_part'] = cycle_nums\n",
    "\n",
    "    return new_part_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_processing1(df):\n",
    "    \"\"\"\n",
    "    Performs the first stage of pre-processing on the given PPG data frame.\n",
    "    Its include peak extraction, condition evaluation, and creating the normalized and processed data frames.\n",
    "    \n",
    "    Parameters:\n",
    "    - df (pandas DataFrame): The input PPG data frame.\n",
    "\n",
    "    Returns:\n",
    "    - norm_df (pandas DataFrame): The normalized data frame with the peaks.\n",
    "    - processed_df (pandas DataFrame): The processed data frame with the peaks and additional information.\n",
    "    - parts_with_peaks (int): The number of parts that contain peaks.\n",
    "    - parts_after_conditions (int): The number of parts that satisfy the conditions for further processing.\n",
    "    \"\"\"\n",
    "\n",
    "    # Group the data frame by the 'Part_Number' column\n",
    "    grouped_df = df.groupby(df['Part_Number'])\n",
    "\n",
    "    # Initialize variables to track part numbers, parts with peaks, and parts after applying conditions\n",
    "    part_num = 0\n",
    "    parts_with_peaks = 0\n",
    "    parts_after_conditions = 0\n",
    "\n",
    "    # Create empty data frames for the normalized data and processed data\n",
    "    norm_df = pd.DataFrame()\n",
    "    processed_df = pd.DataFrame()\n",
    "    processed_df['Num_of_Cycles_in_part'] = np.nan\n",
    "\n",
    "    # Iterate over each segment in the grouped data frame\n",
    "    for part in grouped_df:\n",
    "        # Extract the PPG signal for the segment\n",
    "        data = (part[1]['PLETH_trend_removel'].values).copy()\n",
    "        \n",
    "        try:\n",
    "            # Attempt to find the max and min peaks using the 'get_peaks' function\n",
    "            max_peaks, min_peaks = get_peaks(data)\n",
    "            parts_with_peaks += 1\n",
    "        except:\n",
    "            pass\n",
    "        else:\n",
    "            # Evaluate the conditions on the PPG data using the 'Conditions_PPG' function\n",
    "            amp_coefficient_variation_score, time_coefficient_variation_score, booli = Conditions_PPG(data, max_peaks, min_peaks)\n",
    "            \n",
    "            if booli:\n",
    "                parts_after_conditions += 1\n",
    "                # Create the normalized data frame with the peaks\n",
    "                norm_part_df = norm_df_with_peaks(data, max_peaks, min_peaks)\n",
    "                part_num += 1\n",
    "                norm_part_df['Part_Number'] = part_num\n",
    "                norm_df = pd.concat([norm_df, norm_part_df])\n",
    "\n",
    "                # Create the original data frame with the peaks\n",
    "                new_part_df = df_with_peaks(part, max_peaks, min_peaks)\n",
    "                new_part_df['Part_Number'] = part_num\n",
    "                processed_df = pd.concat([processed_df, new_part_df])\n",
    "\n",
    "    # Reset the index of the normalized data frame and processed data frame\n",
    "    norm_df.reset_index(drop=True, inplace=True)\n",
    "    processed_df.reset_index(drop=True, inplace=True)\n",
    "\n",
    "    return norm_df, processed_df, parts_with_peaks, parts_after_conditions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 155,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the first pre-processing function on the train & test data\n",
    "norm_train_df1, train_df1, parts_with_peaks_train, parts_after_conditions_train = pre_proccesing1(train_df)\n",
    "norm_test_df1, test_df1, parts_with_peaks_test, parts_after_conditions_test = pre_proccesing1(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train parts that found peaks for them: 5016, which are: 79.58%\n",
      "Train parts that passed the conditions: 3421, which are: 68.2%\n",
      "\n",
      "\n",
      "Test parts that found peaks for them: 1245, which are: 20.88%\n",
      "Test parts that passed the conditions: 930, which are: 74.7%\n"
     ]
    }
   ],
   "source": [
    "# Find the maximum value that represent the number of segments in each data frame\n",
    "train_parts = train_df['Part_Number'].values[-1]\n",
    "test_parts = test_df['Part_Number'].values[-1]\n",
    "\n",
    "# Print the percentage of train segments with peaks and passing the conditions\n",
    "print(f'Train parts that found peaks for them: {parts_with_peaks_train}, which are: {np.round((parts_with_peaks_train/train_parts)*100, 2)}%')\n",
    "print(f'Train parts that passed the conditions: {parts_after_conditions_train}, which are: {np.round((parts_after_conditions_train/parts_with_peaks_train)*100, 2)}%')\n",
    "print('\\n')\n",
    "\n",
    "# Print the percentage of test segments with peaks and passing the conditions\n",
    "print(f'Test parts that found peaks for them: {parts_with_peaks_test}, which are: {np.round((parts_with_peaks_test/test_parts)*100, 2)}%')\n",
    "print(f'Test parts that passed the conditions: {parts_after_conditions_test}, which are: {np.round((parts_after_conditions_test/parts_with_peaks_test)*100, 2)}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train - total parts: 3421, invasive BP: 3000, non-invasive BP: 322\n",
      "test - total parts: 930, invasive BP: 855, non-invasive BP: 35\n"
     ]
    }
   ],
   "source": [
    "# Use the count_parts function to count the number of segments in this step\n",
    "count_parts(train_df1, 'train')\n",
    "count_parts(test_df1, 'test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_processing2(norm_df, df):\n",
    "    \"\"\"\n",
    "    Pre-processes the normalized and original data frames by matching the MAP values to the corresponding segments.\n",
    "\n",
    "    Parameters:\n",
    "        norm_df (DataFrame): Normalized data frame.\n",
    "        df (DataFrame): Original data frame.\n",
    "\n",
    "    Returns:\n",
    "        Tuple: A tuple containing the pre-processed normalized data frame, pre-processed original data frame, and the number of parts with blood pressure (MAP) values.\n",
    "    \"\"\"\n",
    "\n",
    "    # Group the original data frame by segments.\n",
    "    grouped_df = df.groupby('Part_Number')\n",
    "\n",
    "    # Initialize variables.\n",
    "    parts_with_BP = 0\n",
    "    processed_df_list = []\n",
    "    processed_norm_df_list = []\n",
    "\n",
    "    # Iterate over each segment in the original data frame.\n",
    "    for part_number, data in grouped_df:\n",
    "        \n",
    "        # Check if there is only one non-null MAP value in the segment. If there is label of BP, save this segment.\n",
    "        if data['MAP'].notna().sum() == 1:\n",
    "            parts_with_BP += 1  \n",
    "            data['Part_Number'] = parts_with_BP  \n",
    "            processed_df_list.append(data)  \n",
    "\n",
    "            # Find the matching part in the normalized data frame.\n",
    "            data_norm = norm_df[norm_df['Part_Number'] == part_number].copy()\n",
    "            data_norm['Part_Number'] = parts_with_BP \n",
    "            \n",
    "            # Extract the MAP value from the original data frame.\n",
    "            MAP_val = data['MAP'].dropna().values[0]  \n",
    "            data_norm['MAP'] = MAP_val  \n",
    "            processed_norm_df_list.append(data_norm)  \n",
    "\n",
    "    # Concatenate the processed normalized segmwnts and processed original segments into separate data frames\n",
    "    processed_norm_df = pd.concat(processed_norm_df_list).reset_index(drop=True)\n",
    "    processed_df = pd.concat(processed_df_list).reset_index(drop=True)\n",
    "\n",
    "    return processed_norm_df, processed_df, parts_with_BP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-193-48cb32e24bb3>:10: SettingWithCopyWarning: \n",
      "A value is trying to be set on a copy of a slice from a DataFrame.\n",
      "Try using .loc[row_indexer,col_indexer] = value instead\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n",
      "  data['Part_Number'] = parts_with_BP\n"
     ]
    }
   ],
   "source": [
    "# Apply the second pre-processing function on the train & test data\n",
    "norm_train_df2, train_df2, parts_with_BP_train = pre_proccesing2(norm_train_df1, train_df1)\n",
    "norm_test_df2, test_df2, parts_with_BP_test = pre_proccesing2(norm_test_df1, test_df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train parts with BP: 3322, which are: 97.11\n",
      "Test parts with BP: 890, which are: 95.7\n"
     ]
    }
   ],
   "source": [
    "# Print the percentage of train segments that passed all the pre-processing\n",
    "print(f'Train parts with BP: {parts_with_BP_train}, which are: {np.round((parts_with_BP_train/parts_after_conditions_train)*100,2)}')\n",
    "\n",
    "# Print the percentage of test segments that passed all the pre-processing\n",
    "print(f'Test parts with BP: {parts_with_BP_test}, which are: {np.round((parts_with_BP_test/parts_after_conditions_test)*100,2)}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 196,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "train - total parts: 3322, invasive BP: 3000, non-invasive BP: 322\n",
      "test - total parts: 890, invasive BP: 855, non-invasive BP: 35\n"
     ]
    }
   ],
   "source": [
    "# Use the count_parts function to count the number of segments in this step\n",
    "count_parts(train_df2, 'train')\n",
    "count_parts(test_df2, 'test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the proccessed df & proccessed norm df to the desktop "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df = train_df2.copy()\n",
    "train_norm_df = norm_train_df2.copy()\n",
    "\n",
    "test_df = test_df2.copy()\n",
    "test_norm_df = norm_test_df2.copy()\n",
    "\n",
    "run = '8'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the train proccessed df to the desktop in 2 different files (beacause its too big)\n",
    "file_path1 = f'data/train_proccessed_df1_run{run}.xlsx'\n",
    "file_path2 = f'data/train_proccessed_df2_run{run}.xlsx'\n",
    "\n",
    "cut_ind = train_df.shape[0]//2\n",
    "\n",
    "train_df[:cut_ind].to_excel(file_path1, index=False)\n",
    "train_df[cut_ind:].to_excel(file_path2, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the train proccessed norm df to the desktop in 2 different files (beacause its too big)\n",
    "file_path1_norm = f'data/train_proccessed_norm_df1_run{run}.xlsx'\n",
    "file_path2_norm = f'data/train_proccessed_norm_df2_run{run}.xlsx'\n",
    "\n",
    "cut_ind_norm = train_norm_df.shape[0]//2\n",
    "\n",
    "train_norm_df[:cut_ind_norm].to_excel(file_path1_norm, index=False)\n",
    "train_norm_df[cut_ind_norm:].to_excel(file_path2_norm, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 200,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the test proccessed df to the desktop\n",
    "test_file_path = f'C:data/test_proccessed_df_run{run}.xlsx'\n",
    "\n",
    "test_df.to_excel(test_file_path, index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Download the test proccessed norm df to the desktop\n",
    "test_file_path_norm = f'C:data/test_proccessed_norm_df_run{run}.xlsx'\n",
    "\n",
    "test_norm_df.to_excel(test_file_path_norm, index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}