tutorials/06. Integration of RNA-seq and DNA methylation data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e4e0226-62e3-42e3-a092-dd14339c8198",
   "metadata": {},
   "source": [
    "# 06. Integration of RNA-seq and DNA methylation data\n",
    "\n",
    "Simultaneous profiling of RNA-seq and DNA methylation from cells broadens our understanding about the interaction between transcriptomics and epigenomics. In this tutorial, we introduce how to integrate scRNA-seq and scBS-seq (DNA methylation) using scMaui by using the mouse embryo scNMT-seq data set from [GSE121708](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE121708).\n",
    "\n",
    "### Data Preprocessing\n",
    "The data set provides a detailed pipeline of data processing on their [Github repo](https://github.com/rargelaguet/scnmt_gastrulation). RNA-seq count matrix can be downloaded [here](https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE121650). However, scBS-seq data is not provided as a matrix but in .tsv file with BED format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c504e2bf-cf42-451e-974f-7e9772f98605",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>chr</th>\n",
       "      <th>pos</th>\n",
       "      <th>met_reads</th>\n",
       "      <th>nonmet_reads</th>\n",
       "      <th>rate</th>\n",
       "      <th>end</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>chr1</td>\n",
       "      <td>3003379</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3003380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>chr1</td>\n",
       "      <td>3008546</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3008547</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>chr1</td>\n",
       "      <td>3017509</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>3017510</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>chr1</td>\n",
       "      <td>3017624</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3017625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>chr1</td>\n",
       "      <td>3020144</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3020145</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    chr      pos  met_reads  nonmet_reads  rate      end\n",
       "0  chr1  3003379          1             0     1  3003380\n",
       "1  chr1  3008546          1             0     1  3008547\n",
       "2  chr1  3017509          2             1     1  3017510\n",
       "3  chr1  3017624          1             0     1  3017625\n",
       "4  chr1  3020144          2             0     1  3020145"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "pd.read_csv(\"../../GSE121650_scRNA_methyl/met/GSM3443369_E4.5-5.5_new_Plate1_A02_met.tsv\", \n",
    "            sep=\"\\t\", nrows=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6510bf22-e170-4b2e-8a05-9f9da1d4722d",
   "metadata": {},
   "source": [
    "Therefore, we used an R package [Methrix](https://www.bioconductor.org/packages/release/bioc/html/methrix.html) to convert the BED-formatted files into a cell x region matrix containing methylation beta-values (but it only has 0 or 1 since the data is from single-cells) following [the tutorial](https://www.bioconductor.org/packages/release/bioc/vignettes/methrix/inst/doc/methrix.html#Reading_bedgraph_files). After reading the .tsv files with ```methrix::read_bedgraphs```, you can extract the beta-value matrix using ```methrix::get_matrix```. \n",
    "\n",
    "Once you have a matrix, you can follow the [AnnData tutorial](https://anndata.readthedocs.io/en/latest/tutorials/notebooks/getting-started.html) to create an anndata object with the data. **Please note that GSE121708 also provides quality control (QC) results. We only used data which passed the QC**\n",
    "\n",
    "### Load Data \n",
    "We have 939 cells with 10,000 features for RNA-seq and 1,174 features for BS-seq. We selected top 10,000 highly variable genes for the RNA-seq modality, whereas we chose promoter and enhancer regions for the BS-seq modality. Please find the details about promoter and enhancer regions in the [Methods](https://www.biorxiv.org/content/10.1101/2023.01.18.524506v1) section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b6b1e444-1c2c-471c-b942-4ff4be97d4a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from scmaui.utils import init_model_params\n",
    "from scmaui.data import load_data, SCDataset\n",
    "from scmaui.ensembles import EnsembleVAE\n",
    "import os \n",
    "import warnings\n",
    "warnings.filterwarnings('ignore') # to remove warnings\n",
    "\n",
    "adatas = load_data([\"../data/scNMT-seq/rna.h5ad\", \n",
    "                    \"../data/scNMT-seq/methyl.h5ad\"], names=['gex', 'methyl'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "a2c463d5-70ed-45da-aa91-9e8ac33b8c91",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[AnnData object with n_obs × n_vars = 939 × 10000\n",
       "     obs: 'id_rna', 'id_met', 'id_acc', 'embryo', 'plate', 'stage', 'lineage10x', 'lineage10x_2'\n",
       "     var: 'n_cells', 'highly_variable', 'means', 'dispersions', 'dispersions_norm', 'view'\n",
       "     uns: 'embryo_colors', 'hvg', 'lineage10x_2_colors', 'neighbors', 'stage_colors', 'umap', 'view'\n",
       "     obsm: 'X_pca', 'X_umap', 'mask'\n",
       "     obsp: 'connectivities', 'distances',\n",
       " AnnData object with n_obs × n_vars = 939 × 1174\n",
       "     obs: 'id_rna', 'id_met', 'id_acc', 'embryo', 'plate', 'stage', 'lineage10x', 'lineage10x_2'\n",
       "     var: 'view'\n",
       "     uns: 'view'\n",
       "     obsm: 'mask']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adatas[\"input\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "345c8ec1-d09f-4eec-91a9-1c91944ad205",
   "metadata": {},
   "source": [
    "scBS-seq data has much higher sparsity than other single-cell omics assays. **Missing values can be given as NaN**, so that scMaui does not calculate reconstruction losses using the missing values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "b65e8c24-905f-4ccc-a348-5d0cb529f9e0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>chr7_17201476-17205476</th>\n",
       "      <th>chr7_17148280-17152280</th>\n",
       "      <th>chr7_16673706-16677706</th>\n",
       "      <th>chr7_16257541-16261541</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sample</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>E4.5-5.5_new_Plate3_E09</th>\n",
       "      <td>0.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E4.5-5.5_new_Plate3_H09</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>E4.5-5.5_new_Plate4_F01</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         chr7_17201476-17205476  chr7_17148280-17152280  \\\n",
       "sample                                                                    \n",
       "E4.5-5.5_new_Plate3_E09                     0.0                     NaN   \n",
       "E4.5-5.5_new_Plate3_H09                     NaN                     NaN   \n",
       "E4.5-5.5_new_Plate4_F01                     NaN                     NaN   \n",
       "\n",
       "                         chr7_16673706-16677706  chr7_16257541-16261541  \n",
       "sample                                                                   \n",
       "E4.5-5.5_new_Plate3_E09                     NaN                     NaN  \n",
       "E4.5-5.5_new_Plate3_H09                     NaN                     NaN  \n",
       "E4.5-5.5_new_Plate4_F01                     NaN                     NaN  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adatas[\"input\"][1].to_df().iloc[:3,212:216]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b91ccef-6560-4be0-94c2-ed9aa61f6278",
   "metadata": {},
   "source": [
    "### Run scMaui\n",
    "\n",
    "Now we run scMaui to integrate scRNA-seq and scBS-seq modalities. Details about running scMaui is described in [Tutorial 02](https://github.com/BIMSBbioinfo/scmaui-experiments/blob/5347ce3673602dd8d0ff89760ebb212d8707aa38/tutorials/02.%20Run%20scMaui%20on%20single-cell%20multiomics%20dataset.ipynb). **Please, note that we use binary loss for the scBS-seq modality considering the binary values in the modality**. Embryo information is assigned as a batch effect factor. Downstream analysis can be done following [Tutorial 03](https://github.com/BIMSBbioinfo/scmaui-experiments/blob/main/tutorials/03.%20Single-cell%20Multiomics%20data%20analysis%20based%20on%20scMaui%20latents.ipynb) and [Tutorial 04](https://github.com/BIMSBbioinfo/scmaui-experiments/blob/main/tutorials/04.%20scMaui%20latent%20analysis.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "13c53a43-67bb-49bd-a40c-c9a0eb7f7d45",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using vae\n",
      "Run model 1\n",
      "Epoch 1/100\n",
      "7/7 - 1s - adv: 1561.8699 - kl: 510.3268 - recon: 16871.5737 - loss: 15820.0305 - val_adv: 225.7889 - val_kl: 920.3942 - val_recon: 15197.1201 - val_loss: 15891.7256\n",
      "Epoch 2/100\n",
      "7/7 - 0s - adv: 1727.3707 - kl: 951.8890 - recon: 14547.2672 - loss: 13771.7860 - val_adv: 152.4576 - val_kl: 795.0279 - val_recon: 13257.5752 - val_loss: 13900.1455\n",
      "Epoch 3/100\n",
      "7/7 - 0s - adv: 1479.9218 - kl: 615.3281 - recon: 13466.6271 - loss: 12602.0334 - val_adv: 151.0450 - val_kl: 352.7277 - val_recon: 12816.9883 - val_loss: 13018.6709\n",
      "Epoch 4/100\n",
      "7/7 - 0s - adv: 1439.6411 - kl: 300.7198 - recon: 13198.2538 - loss: 12059.3324 - val_adv: 140.4270 - val_kl: 211.2634 - val_recon: 12629.2246 - val_loss: 12700.0615\n",
      "Epoch 5/100\n",
      "7/7 - 1s - adv: 1477.0756 - kl: 209.7215 - recon: 13146.9377 - loss: 11879.5834 - val_adv: 158.3611 - val_kl: 155.2321 - val_recon: 12542.0137 - val_loss: 12538.8848\n",
      "Epoch 6/100\n",
      "7/7 - 1s - adv: 1414.2125 - kl: 173.7845 - recon: 12988.3706 - loss: 11747.9426 - val_adv: 134.0301 - val_kl: 141.3140 - val_recon: 12523.2783 - val_loss: 12530.5625\n",
      "Epoch 7/100\n",
      "7/7 - 0s - adv: 1371.0117 - kl: 165.4964 - recon: 13118.0426 - loss: 11912.5272 - val_adv: 135.6183 - val_kl: 160.0133 - val_recon: 12507.9980 - val_loss: 12532.3936\n",
      "Epoch 8/100\n",
      "7/7 - 0s - adv: 1296.0250 - kl: 163.4808 - recon: 12987.8395 - loss: 11855.2952 - val_adv: 129.2089 - val_kl: 157.9816 - val_recon: 12458.7090 - val_loss: 12487.4814\n",
      "Epoch 9/100\n",
      "7/7 - 0s - adv: 1348.8949 - kl: 146.1159 - recon: 12926.9994 - loss: 11724.2203 - val_adv: 113.7034 - val_kl: 175.1861 - val_recon: 12491.3857 - val_loss: 12552.8691\n",
      "Epoch 10/100\n",
      "7/7 - 1s - adv: 1319.3081 - kl: 171.1030 - recon: 12970.2196 - loss: 11822.0146 - val_adv: 127.3456 - val_kl: 171.6339 - val_recon: 12394.0010 - val_loss: 12438.2891\n",
      "Epoch 11/100\n",
      "7/7 - 0s - adv: 1263.5977 - kl: 189.0158 - recon: 12930.4602 - loss: 11855.8783 - val_adv: 130.6761 - val_kl: 164.3835 - val_recon: 12355.1650 - val_loss: 12388.8730\n",
      "Epoch 12/100\n",
      "7/7 - 0s - adv: 1235.4105 - kl: 179.6111 - recon: 12827.6146 - loss: 11771.8152 - val_adv: 155.6862 - val_kl: 149.2011 - val_recon: 12356.6514 - val_loss: 12350.1660\n",
      "Epoch 13/100\n",
      "7/7 - 1s - adv: 1262.1622 - kl: 180.3109 - recon: 12704.5892 - loss: 11622.7379 - val_adv: 143.0500 - val_kl: 157.5549 - val_recon: 12293.3760 - val_loss: 12307.8809\n",
      "Epoch 14/100\n",
      "7/7 - 1s - adv: 1284.0421 - kl: 188.1135 - recon: 12661.9884 - loss: 11566.0598 - val_adv: 159.4786 - val_kl: 134.6209 - val_recon: 12278.5137 - val_loss: 12253.6562\n",
      "Epoch 15/100\n",
      "7/7 - 1s - adv: 1220.0173 - kl: 260.7887 - recon: 12663.1487 - loss: 11703.9199 - val_adv: 131.8300 - val_kl: 147.4387 - val_recon: 12253.6250 - val_loss: 12269.2334\n",
      "Epoch 16/100\n",
      "7/7 - 0s - adv: 1334.8243 - kl: 214.0804 - recon: 12667.9585 - loss: 11547.2147 - val_adv: 167.0368 - val_kl: 256.3799 - val_recon: 12246.9629 - val_loss: 12336.3057\n",
      "Epoch 17/100\n",
      "7/7 - 0s - adv: 1472.7733 - kl: 179.7978 - recon: 12625.7504 - loss: 11332.7750 - val_adv: 152.0497 - val_kl: 152.4596 - val_recon: 12224.6611 - val_loss: 12225.0713\n",
      "Epoch 18/100\n",
      "7/7 - 0s - adv: 1416.9339 - kl: 188.0425 - recon: 12656.5969 - loss: 11427.7057 - val_adv: 161.1364 - val_kl: 101.3861 - val_recon: 12176.7266 - val_loss: 12116.9756\n",
      "Epoch 19/100\n",
      "7/7 - 0s - adv: 1564.5028 - kl: 136.6765 - recon: 12594.4860 - loss: 11166.6598 - val_adv: 158.8192 - val_kl: 105.5553 - val_recon: 12166.9268 - val_loss: 12113.6631\n",
      "Epoch 20/100\n",
      "7/7 - 1s - adv: 1554.7934 - kl: 111.4656 - recon: 12562.5386 - loss: 11119.2108 - val_adv: 210.3008 - val_kl: 77.2108 - val_recon: 12158.7686 - val_loss: 12025.6787\n",
      "Epoch 21/100\n",
      "7/7 - 0s - adv: 1511.2238 - kl: 98.0411 - recon: 12522.8151 - loss: 11109.6321 - val_adv: 193.8386 - val_kl: 76.1761 - val_recon: 12142.7148 - val_loss: 12025.0518\n",
      "Epoch 22/100\n",
      "7/7 - 0s - adv: 1475.4819 - kl: 91.5982 - recon: 12463.2168 - loss: 11079.3334 - val_adv: 183.9609 - val_kl: 67.6800 - val_recon: 12098.1357 - val_loss: 11981.8545\n",
      "Epoch 23/100\n",
      "7/7 - 0s - adv: 1468.8845 - kl: 101.8087 - recon: 12496.9189 - loss: 11129.8431 - val_adv: 204.5932 - val_kl: 74.7927 - val_recon: 12117.1035 - val_loss: 11987.3037\n",
      "Epoch 24/100\n",
      "7/7 - 0s - adv: 1515.5727 - kl: 94.3957 - recon: 12435.0996 - loss: 11013.9222 - val_adv: 191.5639 - val_kl: 69.6247 - val_recon: 12062.2373 - val_loss: 11940.2988\n",
      "Epoch 25/100\n",
      "7/7 - 0s - adv: 1487.5816 - kl: 89.8854 - recon: 12283.1304 - loss: 10885.4340 - val_adv: 212.6402 - val_kl: 66.7500 - val_recon: 12054.8105 - val_loss: 11908.9199\n",
      "Epoch 26/100\n",
      "7/7 - 0s - adv: 1535.0295 - kl: 86.0045 - recon: 12356.8047 - loss: 10907.7797 - val_adv: 216.7668 - val_kl: 65.2152 - val_recon: 12045.0811 - val_loss: 11893.5293\n",
      "Epoch 27/100\n",
      "7/7 - 0s - adv: 1532.6673 - kl: 85.0183 - recon: 12285.4883 - loss: 10837.8394 - val_adv: 204.1407 - val_kl: 69.7441 - val_recon: 12023.3936 - val_loss: 11888.9971\n",
      "Epoch 28/100\n",
      "7/7 - 0s - adv: 1566.0867 - kl: 86.5406 - recon: 12324.3850 - loss: 10844.8390 - val_adv: 207.1474 - val_kl: 62.6254 - val_recon: 12011.4277 - val_loss: 11866.9053\n",
      "Epoch 29/100\n",
      "7/7 - 0s - adv: 1560.6500 - kl: 90.6152 - recon: 12359.9048 - loss: 10889.8699 - val_adv: 208.2439 - val_kl: 78.2962 - val_recon: 12008.9961 - val_loss: 11879.0479\n",
      "Epoch 30/100\n",
      "7/7 - 0s - adv: 1525.0581 - kl: 106.0989 - recon: 12252.0006 - loss: 10833.0415 - val_adv: 205.9644 - val_kl: 79.7504 - val_recon: 11986.2168 - val_loss: 11860.0020\n",
      "Epoch 31/100\n",
      "7/7 - 0s - adv: 1557.0090 - kl: 100.4375 - recon: 12277.1919 - loss: 10820.6204 - val_adv: 212.7809 - val_kl: 77.8845 - val_recon: 11983.8896 - val_loss: 11848.9932\n",
      "Epoch 32/100\n",
      "7/7 - 0s - adv: 1480.3736 - kl: 97.8374 - recon: 12162.8882 - loss: 10780.3522 - val_adv: 209.8588 - val_kl: 68.6992 - val_recon: 11970.7422 - val_loss: 11829.5830\n",
      "Epoch 33/100\n",
      "7/7 - 0s - adv: 1516.6913 - kl: 91.5605 - recon: 12111.9767 - loss: 10686.8457 - val_adv: 206.4487 - val_kl: 73.4294 - val_recon: 11938.7627 - val_loss: 11805.7441\n",
      "Epoch 34/100\n",
      "7/7 - 1s - adv: 1489.5651 - kl: 103.4344 - recon: 12222.6761 - loss: 10836.5457 - val_adv: 212.8465 - val_kl: 82.0667 - val_recon: 11935.9678 - val_loss: 11805.1875\n",
      "Epoch 35/100\n",
      "7/7 - 1s - adv: 1512.8202 - kl: 102.0185 - recon: 12146.7590 - loss: 10735.9574 - val_adv: 208.8751 - val_kl: 69.9533 - val_recon: 11903.4365 - val_loss: 11764.5146\n",
      "Epoch 36/100\n",
      "7/7 - 0s - adv: 1577.7406 - kl: 86.7509 - recon: 12096.0426 - loss: 10605.0529 - val_adv: 207.4155 - val_kl: 64.5848 - val_recon: 11914.9160 - val_loss: 11772.0850\n",
      "Epoch 37/100\n",
      "7/7 - 0s - adv: 1621.5991 - kl: 84.0101 - recon: 12127.6747 - loss: 10590.0854 - val_adv: 240.9614 - val_kl: 68.0948 - val_recon: 11893.3926 - val_loss: 11720.5254\n",
      "Epoch 38/100\n",
      "7/7 - 0s - adv: 1668.4492 - kl: 87.0344 - recon: 12044.5558 - loss: 10463.1411 - val_adv: 242.1588 - val_kl: 68.5760 - val_recon: 11903.9570 - val_loss: 11730.3740\n",
      "Epoch 39/100\n",
      "7/7 - 1s - adv: 1716.6683 - kl: 88.0452 - recon: 12108.6813 - loss: 10480.0583 - val_adv: 244.9111 - val_kl: 63.9947 - val_recon: 11897.4121 - val_loss: 11716.4961\n",
      "Epoch 40/100\n",
      "7/7 - 0s - adv: 1684.5632 - kl: 85.7591 - recon: 12074.2568 - loss: 10475.4525 - val_adv: 234.3232 - val_kl: 59.0105 - val_recon: 11859.5830 - val_loss: 11684.2705\n",
      "Epoch 41/100\n",
      "7/7 - 0s - adv: 1673.9854 - kl: 77.1598 - recon: 11972.3320 - loss: 10375.5063 - val_adv: 238.2724 - val_kl: 61.7596 - val_recon: 11862.5586 - val_loss: 11686.0459\n",
      "Epoch 42/100\n",
      "7/7 - 0s - adv: 1703.1123 - kl: 74.9499 - recon: 12021.9481 - loss: 10393.7859 - val_adv: 247.3058 - val_kl: 55.8101 - val_recon: 11851.7266 - val_loss: 11660.2305\n",
      "Epoch 43/100\n",
      "7/7 - 0s - adv: 1764.5930 - kl: 72.0882 - recon: 12042.9064 - loss: 10350.4019 - val_adv: 249.7959 - val_kl: 51.8498 - val_recon: 11873.9082 - val_loss: 11675.9619\n",
      "Epoch 44/100\n",
      "7/7 - 0s - adv: 1771.6310 - kl: 70.5708 - recon: 12038.0619 - loss: 10337.0015 - val_adv: 241.0456 - val_kl: 58.9116 - val_recon: 11895.5361 - val_loss: 11713.4014\n",
      "Epoch 45/100\n",
      "7/7 - 1s - adv: 1692.1153 - kl: 73.6261 - recon: 12098.2874 - loss: 10479.7980 - val_adv: 230.9296 - val_kl: 59.0672 - val_recon: 11892.5283 - val_loss: 11720.6660\n",
      "Epoch 46/100\n",
      "7/7 - 0s - adv: 1631.7819 - kl: 77.3806 - recon: 11985.9692 - loss: 10431.5677 - val_adv: 236.8502 - val_kl: 66.0374 - val_recon: 11845.0127 - val_loss: 11674.1992\n",
      "Epoch 47/100\n",
      "7/7 - 0s - adv: 1650.2389 - kl: 84.4019 - recon: 11998.4011 - loss: 10432.5643 - val_adv: 232.0319 - val_kl: 77.9281 - val_recon: 11818.0303 - val_loss: 11663.9258\n",
      "Epoch 48/100\n",
      "7/7 - 0s - adv: 1664.6486 - kl: 91.0016 - recon: 11880.2249 - loss: 10306.5779 - val_adv: 223.3185 - val_kl: 73.3013 - val_recon: 11801.9941 - val_loss: 11651.9775\n",
      "Epoch 49/100\n",
      "7/7 - 0s - adv: 1645.7782 - kl: 88.4920 - recon: 11952.8369 - loss: 10395.5509 - val_adv: 218.9125 - val_kl: 70.0485 - val_recon: 11802.8320 - val_loss: 11653.9688\n",
      "Epoch 50/100\n",
      "7/7 - 0s - adv: 1619.7413 - kl: 82.4725 - recon: 11924.3201 - loss: 10387.0514 - val_adv: 237.2040 - val_kl: 60.7655 - val_recon: 11800.0020 - val_loss: 11623.5635\n",
      "Epoch 51/100\n",
      "7/7 - 1s - adv: 1647.1364 - kl: 76.1585 - recon: 11912.5469 - loss: 10341.5690 - val_adv: 245.6100 - val_kl: 61.0033 - val_recon: 11799.2832 - val_loss: 11614.6758\n",
      "Epoch 52/100\n",
      "7/7 - 0s - adv: 1740.4569 - kl: 74.2343 - recon: 11822.8330 - loss: 10156.6101 - val_adv: 228.4178 - val_kl: 63.2377 - val_recon: 11816.5938 - val_loss: 11651.4131\n",
      "Epoch 53/100\n",
      "7/7 - 0s - adv: 1667.1356 - kl: 78.7117 - recon: 11961.8068 - loss: 10373.3828 - val_adv: 223.1767 - val_kl: 61.2877 - val_recon: 11776.5078 - val_loss: 11614.6191\n",
      "Epoch 54/100\n",
      "7/7 - 0s - adv: 1691.7571 - kl: 76.7394 - recon: 11925.0216 - loss: 10310.0040 - val_adv: 215.0291 - val_kl: 57.0357 - val_recon: 11770.8408 - val_loss: 11612.8477\n",
      "Epoch 55/100\n",
      "7/7 - 0s - adv: 1649.9418 - kl: 73.4779 - recon: 11921.2932 - loss: 10344.8293 - val_adv: 216.3103 - val_kl: 57.7489 - val_recon: 11778.9648 - val_loss: 11620.4033\n",
      "Epoch 56/100\n",
      "7/7 - 0s - adv: 1629.7456 - kl: 68.8502 - recon: 11859.4431 - loss: 10298.5480 - val_adv: 209.9624 - val_kl: 60.2522 - val_recon: 11784.9795 - val_loss: 11635.2686\n",
      "Epoch 57/100\n",
      "7/7 - 1s - adv: 1623.4584 - kl: 75.5027 - recon: 11872.6176 - loss: 10324.6619 - val_adv: 216.2158 - val_kl: 61.3423 - val_recon: 11758.9326 - val_loss: 11604.0596\n",
      "Epoch 58/100\n",
      "7/7 - 0s - adv: 1625.4145 - kl: 78.3734 - recon: 11845.5494 - loss: 10298.5085 - val_adv: 233.4599 - val_kl: 62.2989 - val_recon: 11748.7314 - val_loss: 11577.5703\n",
      "Epoch 59/100\n",
      "7/7 - 0s - adv: 1612.1958 - kl: 76.9893 - recon: 11948.1228 - loss: 10412.9163 - val_adv: 216.6599 - val_kl: 61.2794 - val_recon: 11741.9219 - val_loss: 11586.5410\n",
      "Epoch 60/100\n",
      "7/7 - 0s - adv: 1685.9872 - kl: 75.5214 - recon: 11779.3085 - loss: 10168.8427 - val_adv: 239.7788 - val_kl: 61.5102 - val_recon: 11752.3984 - val_loss: 11574.1289\n",
      "Epoch 61/100\n",
      "7/7 - 0s - adv: 1703.2387 - kl: 72.4969 - recon: 11799.4960 - loss: 10168.7544 - val_adv: 217.1982 - val_kl: 60.5950 - val_recon: 11749.6182 - val_loss: 11593.0146\n",
      "Epoch 62/100\n",
      "7/7 - 0s - adv: 1719.4012 - kl: 71.1689 - recon: 11862.7288 - loss: 10214.4965 - val_adv: 233.9064 - val_kl: 58.1071 - val_recon: 11755.1504 - val_loss: 11579.3516\n",
      "Epoch 63/100\n",
      "7/7 - 0s - adv: 1738.1504 - kl: 70.6966 - recon: 11861.5613 - loss: 10194.1077 - val_adv: 236.6474 - val_kl: 52.1564 - val_recon: 11739.6191 - val_loss: 11555.1279\n",
      "Epoch 64/100\n",
      "7/7 - 0s - adv: 1705.2606 - kl: 64.9227 - recon: 11784.8258 - loss: 10144.4879 - val_adv: 220.6684 - val_kl: 49.6136 - val_recon: 11743.5850 - val_loss: 11572.5303\n",
      "Epoch 65/100\n",
      "7/7 - 0s - adv: 1654.5176 - kl: 65.3392 - recon: 11841.5315 - loss: 10252.3530 - val_adv: 229.7695 - val_kl: 48.4030 - val_recon: 11743.8164 - val_loss: 11562.4502\n",
      "Epoch 66/100\n",
      "7/7 - 0s - adv: 1659.9936 - kl: 63.6229 - recon: 11797.7000 - loss: 10201.3292 - val_adv: 219.7183 - val_kl: 47.8690 - val_recon: 11741.4746 - val_loss: 11569.6250\n",
      "Epoch 67/100\n",
      "7/7 - 0s - adv: 1671.3741 - kl: 62.2853 - recon: 11851.2986 - loss: 10242.2098 - val_adv: 244.0232 - val_kl: 47.3302 - val_recon: 11724.0557 - val_loss: 11527.3623\n",
      "Epoch 68/100\n",
      "7/7 - 0s - adv: 1761.6407 - kl: 60.3862 - recon: 11726.6503 - loss: 10025.3959 - val_adv: 233.5981 - val_kl: 46.5843 - val_recon: 11711.5645 - val_loss: 11524.5508\n",
      "Epoch 69/100\n",
      "7/7 - 0s - adv: 1760.7829 - kl: 60.8887 - recon: 11760.3228 - loss: 10060.4287 - val_adv: 223.5418 - val_kl: 45.5998 - val_recon: 11710.8174 - val_loss: 11532.8750\n",
      "Epoch 70/100\n",
      "7/7 - 0s - adv: 1658.6402 - kl: 60.7720 - recon: 11780.8524 - loss: 10182.9841 - val_adv: 215.3505 - val_kl: 45.8958 - val_recon: 11707.1289 - val_loss: 11537.6738\n",
      "Epoch 71/100\n",
      "7/7 - 0s - adv: 1644.2681 - kl: 59.9291 - recon: 11727.8336 - loss: 10143.4944 - val_adv: 226.7578 - val_kl: 46.7481 - val_recon: 11711.1572 - val_loss: 11531.1475\n",
      "Epoch 72/100\n",
      "7/7 - 0s - adv: 1651.2292 - kl: 60.8647 - recon: 11698.2506 - loss: 10107.8861 - val_adv: 214.1166 - val_kl: 45.3104 - val_recon: 11707.5879 - val_loss: 11538.7822\n",
      "Epoch 73/100\n",
      "7/7 - 0s - adv: 1675.9885 - kl: 60.3330 - recon: 11769.7314 - loss: 10154.0759 - val_adv: 212.9624 - val_kl: 45.6469 - val_recon: 11696.8809 - val_loss: 11529.5654\n",
      "Epoch 74/100\n",
      "7/7 - 0s - adv: 1588.0754 - kl: 59.5352 - recon: 11739.6394 - loss: 10211.0994 - val_adv: 171.8728 - val_kl: 44.3826 - val_recon: 11690.5723 - val_loss: 11563.0820\n",
      "Epoch 75/100\n",
      "7/7 - 0s - adv: 1596.9508 - kl: 58.0653 - recon: 11698.6115 - loss: 10159.7261 - val_adv: 147.6323 - val_kl: 44.4584 - val_recon: 11693.7207 - val_loss: 11590.5469\n",
      "Epoch 76/100\n",
      "7/7 - 0s - adv: 1471.8838 - kl: 57.7920 - recon: 11802.2064 - loss: 10388.1147 - val_adv: 136.7036 - val_kl: 44.1015 - val_recon: 11694.5908 - val_loss: 11601.9883\n",
      "Epoch 77/100\n",
      "7/7 - 0s - adv: 1482.8431 - kl: 57.0743 - recon: 11580.5891 - loss: 10154.8203 - val_adv: 135.3512 - val_kl: 43.3556 - val_recon: 11686.4297 - val_loss: 11594.4336\n",
      "Epoch 78/100\n",
      "7/7 - 0s - adv: 1492.9310 - kl: 55.1327 - recon: 11710.3618 - loss: 10272.5636 - val_adv: 141.1812 - val_kl: 44.9839 - val_recon: 11690.8887 - val_loss: 11594.6904\n",
      "Epoch 79/100\n",
      "7/7 - 0s - adv: 1549.8340 - kl: 55.9801 - recon: 11654.8845 - loss: 10161.0304 - val_adv: 148.9291 - val_kl: 40.5758 - val_recon: 11685.2197 - val_loss: 11576.8672\n",
      "Epoch 80/100\n",
      "7/7 - 0s - adv: 1545.9444 - kl: 56.4178 - recon: 11733.1724 - loss: 10243.6459 - val_adv: 168.5276 - val_kl: 44.0890 - val_recon: 11705.9131 - val_loss: 11581.4746\n",
      "Epoch 81/100\n",
      "7/7 - 0s - adv: 1555.5761 - kl: 57.1660 - recon: 11664.5939 - loss: 10166.1840 - val_adv: 167.1143 - val_kl: 46.5533 - val_recon: 11697.9785 - val_loss: 11577.4180\n",
      "Epoch 82/100\n",
      "7/7 - 0s - adv: 1630.1928 - kl: 57.6147 - recon: 11747.5778 - loss: 10174.9998 - val_adv: 188.3273 - val_kl: 46.0831 - val_recon: 11682.8672 - val_loss: 11540.6230\n",
      "Epoch 83/100\n",
      "7/7 - 0s - adv: 1636.0216 - kl: 58.7629 - recon: 11723.2611 - loss: 10146.0026 - val_adv: 200.3535 - val_kl: 48.1741 - val_recon: 11679.9668 - val_loss: 11527.7871\n",
      "Epoch 84/100\n",
      "7/7 - 0s - adv: 1700.0052 - kl: 60.2031 - recon: 11678.3586 - loss: 10038.5566 - val_adv: 179.3431 - val_kl: 46.7433 - val_recon: 11667.5068 - val_loss: 11534.9072\n",
      "Epoch 85/100\n",
      "7/7 - 0s - adv: 1687.9093 - kl: 61.9972 - recon: 11638.0114 - loss: 10012.0994 - val_adv: 173.5614 - val_kl: 46.5388 - val_recon: 11670.8311 - val_loss: 11543.8086\n",
      "Epoch 86/100\n",
      "7/7 - 0s - adv: 1631.9423 - kl: 60.3911 - recon: 11664.8575 - loss: 10093.3065 - val_adv: 167.5234 - val_kl: 43.8071 - val_recon: 11674.1758 - val_loss: 11550.4590\n",
      "Epoch 87/100\n",
      "7/7 - 0s - adv: 1623.2407 - kl: 60.4186 - recon: 11569.0814 - loss: 10006.2592 - val_adv: 173.7817 - val_kl: 45.0954 - val_recon: 11659.9453 - val_loss: 11531.2598\n",
      "Epoch 88/100\n",
      "7/7 - 0s - adv: 1550.5303 - kl: 58.9333 - recon: 11583.2761 - loss: 10091.6791 - val_adv: 177.6024 - val_kl: 42.8080 - val_recon: 11655.8027 - val_loss: 11521.0078\n",
      "Epoch 89/100\n",
      "7/7 - 0s - adv: 1611.9523 - kl: 57.7111 - recon: 11611.2820 - loss: 10057.0406 - val_adv: 151.1363 - val_kl: 42.2511 - val_recon: 11671.4268 - val_loss: 11562.5410\n",
      "Epoch 90/100\n",
      "7/7 - 0s - adv: 1656.1992 - kl: 57.4470 - recon: 11613.9486 - loss: 10015.1964 - val_adv: 193.3404 - val_kl: 41.5568 - val_recon: 11716.1943 - val_loss: 11564.4102\n",
      "Epoch 91/100\n",
      "7/7 - 0s - adv: 1645.1582 - kl: 58.7862 - recon: 11679.2189 - loss: 10092.8469 - val_adv: 190.0059 - val_kl: 41.5874 - val_recon: 11679.4502 - val_loss: 11531.0322\n",
      "Epoch 92/100\n",
      "7/7 - 0s - adv: 1583.2309 - kl: 57.6475 - recon: 11552.5798 - loss: 10026.9966 - val_adv: 158.2977 - val_kl: 41.0823 - val_recon: 11665.6436 - val_loss: 11548.4277\n",
      "Epoch 93/100\n",
      "7/7 - 0s - adv: 1503.0301 - kl: 56.5546 - recon: 11623.9536 - loss: 10177.4779 - val_adv: 137.6024 - val_kl: 40.8508 - val_recon: 11655.9766 - val_loss: 11559.2246\n",
      "Epoch 94/100\n",
      "7/7 - 0s - adv: 1397.3519 - kl: 55.4060 - recon: 11620.3114 - loss: 10278.3654 - val_adv: 146.3663 - val_kl: 44.1725 - val_recon: 11693.1328 - val_loss: 11590.9395\n",
      "Epoch 95/100\n",
      "7/7 - 1s - adv: 1353.4040 - kl: 56.1807 - recon: 11639.5642 - loss: 10342.3409 - val_adv: 144.3999 - val_kl: 39.8476 - val_recon: 11646.3184 - val_loss: 11541.7666\n",
      "Epoch 96/100\n",
      "7/7 - 0s - adv: 1312.3521 - kl: 54.9840 - recon: 11618.1982 - loss: 10360.8302 - val_adv: 130.2652 - val_kl: 40.3217 - val_recon: 11645.1650 - val_loss: 11555.2207\n",
      "Epoch 97/100\n",
      "7/7 - 0s - adv: 1295.3684 - kl: 54.8603 - recon: 11637.2987 - loss: 10396.7908 - val_adv: 149.1576 - val_kl: 40.2275 - val_recon: 11655.7598 - val_loss: 11546.8301\n",
      "Epoch 98/100\n",
      "7/7 - 0s - adv: 1304.0279 - kl: 55.3076 - recon: 11550.5845 - loss: 10301.8640 - val_adv: 132.0147 - val_kl: 40.4755 - val_recon: 11660.0098 - val_loss: 11568.4707\n",
      "Epoch 99/100\n",
      "7/7 - 0s - adv: 1272.5686 - kl: 55.5364 - recon: 11586.2648 - loss: 10369.2325 - val_adv: 127.5250 - val_kl: 42.3072 - val_recon: 11655.5498 - val_loss: 11570.3320\n",
      "Epoch 100/100\n",
      "7/7 - 0s - adv: 1232.8251 - kl: 56.0384 - recon: 11563.1045 - loss: 10386.3181 - val_adv: 120.5499 - val_kl: 42.7489 - val_recon: 11639.9561 - val_loss: 11562.1553\n"
     ]
    }
   ],
   "source": [
    "dataset = SCDataset(adatas, losses=[\"negbinom\", \"binary\"], # Binary loss for scBS-seq\n",
    "                    union=True, adversarial=[\"embryo\"],\n",
    "                    conditional=[\"embryo\"])#\n",
    "\n",
    "params = init_model_params()\n",
    "params.update({'losses': dataset.losses})\n",
    "params['kl_weight'] = 0.3\n",
    "modalities = dataset.modalities()\n",
    "params['input_modality'] = modalities[0]\n",
    "params['output_modality'] = modalities[1]\n",
    "params[\"nlatent\"] = 30\n",
    "params.update(dataset.adversarial_config())\n",
    "params.update(dataset.conditional_config())\n",
    "params\n",
    "\n",
    "# Create a model\n",
    "ensemble = EnsembleVAE(params=params, ensemble_size=1)\n",
    "os.environ[\"CUDA_VISIBLE_DEVICES\"]=\"-1\" # This line is for avoiding tensorflow models to be allocated to GPUs\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' # This line disables Tensorflow debugging information\n",
    "train_loss = ensemble.fit(dataset, epochs=100, \n",
    "                          learning_rate=0.001, batch_size=128)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:anaconda3-scmaui]",
   "language": "python",
   "name": "conda-env-anaconda3-scmaui-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}