Data module update

CONTRIBUTING.md

@@ -4,7 +4,7 @@ Thanks for your interest in contributing MiV-OS project.
 
 The following is a set of guidelines how to contributes. These are mostly guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.
 
-#### Table Of Contents
+**Table Of Contents**
 
 [TLTR! I need three-line summary!!](#three-line-summary)
 
@@ -36,10 +36,15 @@ The following is a set of guidelines how to contributes. These are mostly guidel
 ### Installation and packages
 
 First **create the fork repository and clone** to your local machine.
-We provide [requirements.txt](requirements.txt) to include all the dependencies.
+We provide [requirements.txt](https://github.com/GazzolaLab/MiV-OS/blob/main/requirements.txt) to include all the dependencies that is required to develop. You can either install using `pip install -r requirements.txt` or
 ```bash
-$ pip install -r requirements.txt
+$ pip install miv-os[dev]
 ```
+If you are more interested in working for documentation, use
+```bash
+$ pip install miv-os[docs]
+```
+More details are included [here](https://github.com/GazzolaLab/MiV-OS/blob/main/docs/README.md).
 
 ### Pre-Commit
 

Makefile

@@ -7,5 +7,5 @@ mypy:
 coverage:
 	@pytest --cov=miv tests/
 
-all:test mypy
+all:test mypy coverage
 ci: test mypy

docs/api/io.rst

@@ -1,6 +1,8 @@
-*********************
-Input / Output Module
-*********************
+********************
+Data Managing Module
+********************
+
+.. automodule:: miv.io.data
 
 .. automodule:: miv.io.binary
    :members:

docs/api/signal.rst

@@ -1,34 +1,7 @@
-*************************
-Signal Processing Modules
-*************************
+*********************
+Signal Pre-Processing
+*********************
 
+.. automodule:: miv.signal.filter.filter_collection
 
-Filter
-######
-
-.. currentmodule:: miv.signal.filter
-
-.. automodule:: miv.signal.filter
-
-   .. autosummary::
-       :nosignatures:
-       :toctree: _toctree/FilterAPI
-
-       FilterProtocol
-       ButterBandpass
-       FilterCollection
-
-Spike Detection
-###############
-
-.. automodule:: miv.signal.spike
-
-   .. autosummary::
-       :nosignatures:
-       :toctree: _toctree/DetectionAPI
-
-       SpikeDetectionProtocol
-       ThresholdCutoff
-
-Spike Sorting
-#############
+.. automodule:: miv.signal.spike.detection

docs/guide/data_management.md

@@ -0,0 +1,86 @@
+---
+jupytext:
+  text_representation:
+    extension: .md
+    format_name: myst
+    format_version: 0.13
+    jupytext_version: 1.13.8
+kernelspec:
+  display_name: Python 3 (ipykernel)
+  language: python
+  name: python3
+---
+
+# Data Management
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+
+import os
+import numpy as np
+import quantities as pq
+import matplotlib.pyplot as plt
+
+```
+
+## 1. Data Load
+
+```{code-cell} ipython3
+:tags: [hide-cell]
+
+from miv.io import load_data
+from miv.io.data import Data, Dataset
+```
+
+```{code-cell} ipython3
+# Load dataset from OpenEphys recording
+folder_path: str = "~/Open Ephys/2022-03-10-16-19-09"  # Data Path
+# Provide the path of experimental recording tree to the DataSet class
+# Data set class will load the data and create a list of objects for each data
+# dataset = load_data(folder_path, device="OpenEphys")
+dataset = Dataset(data_folder_path=folder_path,
+                  device="OpenEphys",
+                  channels=32,
+                  sampling_rate=30E3,
+                  timestamps_npy="", # We can read similar to continuous.dat
+
+                  )
+#TODO: synchornized_timestamp what for shifted ??
+# Masking channels for data set. Channels can be a list.
+# Show user the tree. Implement representation method. filter_collection.html#FilterCollection.insert
+# An example code to get the tree https://github.com/skim0119/mindinvitro/blob/master/utility/common.py
+# Trimming the tree??
+```
+
+### 1.1. Meta Data Structure
+
+```{code-cell} ipython3
+# Get signal and rate(hz)
+record_node: int = dataset.get_nodes[0]
+recording = dataset[record_node]["experiment1"]["recording1"]   # Returns the object for recording 1
+# TODO: does openephys returns the timestamp??
+timestamp = recording.timestamp # returns the time stamp for the recording.
+
+signal, _, rate = recording.continuous["100"]
+# time = recording.continuous["100"].timestamp / rate
+num_channels = signal.shape[1]
+```
+
+### 1.2 Raw Data
+
++++
+
+If the data is provided in single `continuous.dat` instead of meta-data, user must provide number of channels and sampling rate in order to import data accurately.
+
+> **WARNING** The size of the raw datafile can be _large_ depending on sampling rate and the amount of recorded duration. We highly recommand using meta-data structure to handle datafiles, since it only loads the data during the processing and unloads once the processing is done.
+
+```{code-cell} ipython3
+from miv.io import load_continuous_data_file
+
+datapath = 'continuous.dat'
+rate = 30_000
+num_channel = 64
+timestamps, signal = load_continuous_data_file(datapath, num_channel, rate)
+```
+
+## 2. Instant Visualization

docs/guide/signal_processing.md

@@ -104,7 +104,7 @@ from miv.signal.filter import FilterCollection, ButterBandpass
 
 [Here](../api/signal.html#filter) is the list of provided filters.
 All filters are `Callable`, taking `signal` and `sampling_rate` as parameters.
-To define a multiple filters together, we provide [`FilterCollection`](../api/_toctree/FilterAPI/miv.signal.filter.FilterCollection) that execute multiple filters in a series.
+To define a multiple filters together, we provide [`FilterCollection`](miv.signal.filter.FilterCollection) that execute multiple filters in a series.
 
 ```{code-cell} ipython3
 # Butter bandpass filter

docs/index.rst

@@ -1,7 +1,5 @@
 .. MiV-OS documentation master file, created by
    sphinx-quickstart on Thu Mar 24 23:35:49 2022.
-   You can adapt this file completely to your liking, but it should at least
-   contain the root `toctree` directive.
 
 MiV-OS documentation!
 =====================
@@ -17,19 +15,26 @@ Installation Instruction
 
 You can also download the source code from `GitHub <https://github.com/GazzolaLab/MiV-OS>`_ directly.
 
+Contribution
+------------
+
+Any contribution to this project is welcome! If you are interested or have any questions, please don't hesitate to contact us.
+If you are interested in contributing to this project, we prepared contribution guideline :ref:`here <overview/contribution:Contributing to MiV-OS>`.
+
 
 .. toctree::
    :maxdepth: 2
-   :caption: Contents:
+   :caption: Overview
 
    overview/about
-   overview/dependencies
    overview/references
+   overview/contribution
 
 .. toctree::
    :maxdepth: 2
    :caption: User Guide
 
+   guide/data_management
    guide/signal_processing
    guide/spike_cutout
    guide/spike_sorting

docs/overview/contribution.md

@@ -0,0 +1,3 @@
+```{include} ../../CONTRIBUTING.md
+:relative-images:
+```

docs/overview/references.rst

@@ -1,3 +1,15 @@
 **********
 References
 **********
+
+Neural Ensemble
+###############
+
+- Python-Neo [1]_
+- Elephant/Viziphant [2]_
+
+---------------
+
+.. [1] Garcia S., Guarino D., Jaillet F., Jennings T.R., Pröpper R., Rautenberg P.L., Rodgers C., Sobolev A.,Wachtler T., Yger P. and Davison A.P. (2014) Neo: an object model for handling electrophysiology data in multiple formats. Frontiers in Neuroinformatics 8:10: doi:10.3389/fninf.2014.00010
+
+.. [2] Denker M, Yegenoglu A, Grün S (2018) Collaborative HPC-enabled workflows on the HBP Collaboratory using the Elephant framework. Neuroinformatics 2018, P19. doi:10.12751/incf.ni2018.0019

miv/io/__init__.py

@@ -1 +1,2 @@
+from miv.io.data import *
 from miv.io.binary import *