ccsc_FlarePredict.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1f0aea4a",
   "metadata": {},
   "source": [
    "# Predicting Solar Flares Using a Long Short-term Memory Network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ec079e6",
   "metadata": {},
   "source": [
    "### Hao Liu, Chang Liu, Jason T. L. Wang and Haimin Wang"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f20f257a",
   "metadata": {},
   "source": [
    "## 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73c9d7b7",
   "metadata": {},
   "source": [
    "Solar flares, the largest explosive events in our solar system, are intense bursts of radiation that occur in the Sun's atmosphere and release massive amounts of energy into space. They last from minutes to hours and are often seen as bright chromospheric ribbons and hot coronal loops on the Sun. Some flares are small and innocent while others can be tremendous and violent. Powerful flares and the often accompanied coronal mass ejections (CMEs) can cause severe influences on the near-Earth environment, resulting in potentially life-threatening consequences (Daglis et al. 2004). Therefore, substantial efforts are being invested on solar-flare research including forecast and mitigation plans.\n",
    "\n",
    "In this notebook, we attempt to use SDO/HMI vector magnetic field data together with flaring history to predict solar flares that would occur in an AR within 24 hr of a given time point, with a deep-learning method, named long short-term memory (LSTM; Hochreiter & Schmidhuber 1997)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64bde3dd",
   "metadata": {},
   "source": [
    "## 2. LSTM Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8a9da94",
   "metadata": {},
   "source": [
    "### 2.1 Data Prepration & Loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deac8a76",
   "metadata": {},
   "source": [
    "The data folder includes three sub-directories: LSTM_C_sample_run, LSTM_M_sample_run, and LSTM_M5_sample_run.\n",
    "* The LSTM_C_sample_run includes a CSV training data file that is used to train the model and a CSV test data file that is used to predict the C category and higher flares.\n",
    "* The LSTM_M_sample_run includes a CSV training data file that is used to train the model and a CSV test data file that is used to predict the M category and higher flares.\n",
    "* The LSTM_M5_sample_run includes a CSV training data file that is used to train the model and a CSV test data file that is used to predict the M5 category and higher flares.\n",
    "\n",
    "The files are loaded and used during the testing and training process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "70175507",
   "metadata": {},
   "source": [
    "### 2.2 C Flare Model Training and Testing\n",
    "You may train the model with your own data or train the model with the default data (see Sections 2.2.1 and 2.2.2)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb36dd2",
   "metadata": {},
   "source": [
    "#### 2.2.1 C Flare Model Training with Default Data\n",
    "Here, we show how to train the model with default data.\n",
    "To train the model with your own data:\n",
    "1. You should first upload your file to the data directory (in the left hand side file list).\n",
    "2. Edit the path to the training file:<br> 'train_data_file':'data/LSTM_C_sample_run/normalized_training.csv' <br>and replace the value 'data/LSTM_C_sample_run/normalized_training.csv' with your new file name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3295900a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Loading the train_model function...')\n",
    "from flarepredict_train import train_model\n",
    "args = {'train_data_file':'data/LSTM_C_sample_run/normalized_training.csv',\n",
    "        'flare': 'C',\n",
    "       'modelid': 'custom_model_id'\n",
    "      }\n",
    "train_model(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b007ca70",
   "metadata": {},
   "source": [
    "#### 2.2.2 Predicting with Your C Flare Model\n",
    "To predict the testing data using the model you trained above, make sure the modelid value in the args variable in the following code is set exactly as the one used in the training, for example: 'custom_model_id'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e930d791",
   "metadata": {},
   "outputs": [],
   "source": [
    "from flarepredict_test import test_model\n",
    "args =  {'test_data_file': 'data/LSTM_C_sample_run/normalized_testing.csv', \n",
    "         'flare': 'C', \n",
    "         'modelid': 'custom_model_id'}\n",
    "result_file_name = test_model(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a4616d9",
   "metadata": {},
   "source": [
    "#### 2.2.3 Reading the Results\n",
    "The prediction result can be shown by passing the result variable to the pandas function read_csv as shown in the following example. The result shows the label your model predicted for each case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5749cbaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(result_file_name)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf6f175c",
   "metadata": {},
   "source": [
    "### 2.3 M Flare Model Training and Testing\n",
    "You may train the model with your own data or train the model with the default data (see Sections 2.3.1 and 2.3.2)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d70a41b6",
   "metadata": {},
   "source": [
    "#### 2.3.1 M Flare Model Training with Default Data\n",
    "Here, we show how to train the model with default data.\n",
    "To train the model with your own data:\n",
    "1. You should first upload your file to the data directory (in the left hand side file list).\n",
    "2. Edit the path to the training file:<br> 'train_data_file':'data/LSTM_M_sample_run/normalized_training.csv' <br>and replace the value 'data/LSTM_M_sample_run/normalized_training.csv' with your new file name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61a9178a",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Loading the train_model function...')\n",
    "from flarepredict_train import train_model\n",
    "args = {'train_data_file':'data/LSTM_M_sample_run/normalized_training.csv',\n",
    "        'flare': 'M',\n",
    "       'modelid': 'custom_model_id'\n",
    "      }\n",
    "train_model(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4af9abc3",
   "metadata": {},
   "source": [
    "#### 2.3.2 Predicting with Your M Flare Model\n",
    "To predict the testing data using the model you trained above, make sure the modelid value in the args variable in the following code is set exactly as the one used in the training, for example: 'custom_model_id'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52d45d07",
   "metadata": {},
   "outputs": [],
   "source": [
    "from flarepredict_test import test_model\n",
    "args =  {'test_data_file': 'data/LSTM_M_sample_run/normalized_testing.csv', \n",
    "         'flare': 'M', \n",
    "         'modelid': 'custom_model_id'}\n",
    "result_file_name = test_model(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "74bbecc1",
   "metadata": {},
   "source": [
    "#### 2.3.3 Reading the Results\n",
    "The prediction result can be shown by passing the result variable to the pandas function read_csv as shown in the following example. The result shows the label your model predicted for each case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8124cabe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(result_file_name)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3393ad46",
   "metadata": {},
   "source": [
    "### 2.4 M5 Flare Model Training and Testing\n",
    "You may train the model with your own data or train the model with the default data (see Sections 2.4.1 and 2.4.2)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63c5feae",
   "metadata": {},
   "source": [
    "#### 2.4.1 M5 Flare Model Training with Default Data\n",
    "Here, we show how to train the model with default data.\n",
    "To train the model with your own data:\n",
    "1. You should first upload your file to the data directory (in the left hand side file list).\n",
    "2. Edit the path to the training file:<br> 'train_data_file':'data/LSTM_M5_sample_run/normalized_training.csv' <br>and replace the value 'data/LSTM_M5_sample_run/normalized_training.csv' with your new file name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11ea06c7",
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Loading the train_model function...')\n",
    "from flarepredict_train import train_model\n",
    "args = {'train_data_file':'data/LSTM_M5_sample_run/normalized_training.csv',\n",
    "        'flare': 'M5',\n",
    "       'modelid': 'custom_model_id'\n",
    "      }\n",
    "train_model(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b364ba8f",
   "metadata": {},
   "source": [
    "#### 2.4.2 Predicting with Your M5 Flare Model\n",
    "To predict the testing data using the model you trained above, make sure the modelid value in the args variable in the following code is set exactly as the one used in the training, for example: 'custom_model_id'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e263c7b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from flarepredict_test import test_model\n",
    "args =  {'test_data_file': 'data/LSTM_M5_sample_run/normalized_testing.csv', \n",
    "         'flare': 'M5', \n",
    "         'modelid': 'custom_model_id'}\n",
    "result_file_name = test_model(args)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60f762cb",
   "metadata": {},
   "source": [
    "#### 2.4.3 Reading the Results\n",
    "The prediction result can be shown by passing the result variable to the pandas function read_csv as shown in the following example. The result shows the label your model predicted for each case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a84ff40e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df = pd.read_csv(result_file_name)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78689232",
   "metadata": {},
   "source": [
    "## 3. Acknowledgment\n",
    "We thank the team of SDO/HMI for producing vector magnetic data products. The flare catalogs were prepared by and made available through NOAA NCEI. This work was supported by U.S. NSF grants AGS-1927578 and AGS-1954737.\n",
    "\n",
    "## 4. References\n",
    "DeepSun: Predicting Solar Flares Using a Long Short-term Memory Network\n",
    "\n",
    "Hao Liu, Chang Liu, Jason T. L. Wang and Haimin Wang\n",
    "\n",
    "https://iopscience.iop.org/article/10.3847/1538-4357/ab1b3c\n",
    "\n",
    "https://github.com/deepsuncode/LSTM-flare-prediction"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}