Merge pull request #869 from MouseLand/refactor

Update documentation to reflect default parameters

docs/inputs.rst

@@ -59,6 +59,8 @@ this array into a numpy array called ``bad_frames.npy``:
 Put this file into the first folder in your ops['data_path'] (the first
 folder you choose in the GUI).
 
+.. _inputs-diff-file-types:
+
 Different file types
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -173,7 +175,7 @@ Elements of these ``BinaryRWFile`` instances can be accessed similar to how one
    f_input.shape # returns shape of your input (num_frames, Ly, Lx)
    f_input[0] # returns the first frame with shape (Ly, Lx)
 
-Also, ``BinaryRWFile`` instances can be directly passed to the several wrapper functions ``suite2p`` offers (e.g., ``suite2p.detection_wrapper``, ``suite2p.extraction_wrapper``, etc.). If you'd like to run only specific modules, you will have to use the ``BinaryRWFile`` class. For example, this is how you can run the detection module on an input file that has already been registered. 
+Also, ``BinaryRWFile`` instances can be directly passed to the several wrapper functions ``suite2p`` offers (e.g., ``suite2p.detection_wrapper``, ``suite2p.extraction_wrapper``, etc.). These wrapper functions can  also directly work with Numpy arrays so feel free to pass them as inputs. If you'd like to run only specific modules, you will have to use the ``BinaryRWFile`` class. For example, this is how you can run the detection module on an input file that has already been registered. 
 
 ::
 

docs/settings.rst

@@ -1,8 +1,7 @@
 Settings (ops.npy)
 ------------------
 
-Here is a summary of all the parameters that the pipeline takes, and its
-default value.
+Suite2p can be run with different configurations using the ``ops`` dictionary. The ``ops`` dictionary will describe the settings used for a particular run of the pipeline. Here is a summary of all the parameters that the pipeline takes and their default values. 
 
 Main settings
 ~~~~~~~~~~~~~
@@ -27,6 +26,8 @@ These are the essential settings that are dataset-specific.
    -  1.0 for GCaMP6m
    -  1.25-1.5 for GCaMP6s
 
+-  **force_sktiff**: (*boolean, default: False*) specifies whether or not to use scikit-image for reading in tiffs
+
 -  **fs**: (*float, default: 10.0*) Sampling rate (per plane). For
    instance, if you have a 10 plane recording acquired at 30Hz, then the
    sampling rate per plane is 3Hz, so set ops['fs'] = 3.
@@ -42,16 +43,60 @@ These are the essential settings that are dataset-specific.
    line scanning (set by user). If set to any value besides 0, then this
    offset is used and applied to all frames in the recording.
 
+- **bidi_corrected**: (*bool, default: False*) Specifies whether to do bidi correction. 
+
 - **frames_include**: (*int, default: -1*) if greater than zero, only *frames_include* frames are processed. useful for testing parameters on a subset of data.
 
+- **multiplane_parallel**: (*boolean, default: False*) specifies whether or not to run pipeline on server 
+
+- **ignore_flyback**: (*list[ints], default: empty list*) specifies which planes will be ignored as flyback planes by the pipeline. 
+
+File input/output settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Suite2p can accomodate many different file formats. Refer to this
+:ref:`page <inputs-diff-file-types>` for a detailed list of formats suite2p can work with. 
+
+- **fast_disk**: (*list[str], default: empty list*) specifies location where temporary binary file will be stored. Defaults to ``save_path0`` if no directory is provided by user. 
+
+- **delete_bin** (*bool, default:False*) specifies whether to delete binary file created during registration stage. 
+
+- **mesoscan** (*bool, default: False*) specifies whether file being read in is a scanimage mesoscope recording
+
+- **bruker** (*bool, default: False*) specifies whether provided tif files are single page BRUKER tiffs
+
+- **bruker_bidirectional** (*bool, default: False*) specifies whether BRUKER files are bidirectional multiplane recordings. The ``True`` setting corresponds to the following plane order (first plane is indexed as zero): [0,1,2,2,1,0]. ``False`` corresponds to [0,1,2,0,1,2]. 
+
+- **h5py** (*list[str], default: empty list*) specifies path to h5py file that will be used as inputs. Keep in mind the pathname provided here overwrites the pathname specified in ``ops[data_path]``. 
+
+- **h5py_key** (*str, default: 'data'*) key used to access data array in h5py file. Only use this when the ``h5py`` setting is set to ``True``. 
+
+- **nwb_file** (*str, default: ''*) specifies path to NWB file you use to use as input
+
+- **nwb_driver** (*str, default: ''*) location of driver for NWB file. Leave this empty if the pathname refers to a local file.
+
+- **nwb_series** (*str, default: ''*) Name of TwoPhotonSeries values you wish to retrieve from your NWB file. 
+
+- **save_path0** (*list[str], default: empty list*) List containing pathname of where you'd like to save your pipeline results. If list is empty, the first element of ``ops['data_path']`` is used. 
+
+- **save_folder** (*list[str], default: empty list*) List containing directory name you'd like results to be saved under. Defaults to ``"suite2p"``. 
+
+- **look_one_level_down**: (*bool, default: False*) specifies whether to look in all subfolders when searching for tiffs. Make sure to specify subfolders in the ``subfolders`` parameter below. 
+
+- **subfolders** (*list[str], default: empty list*) Specifies subfolders you'd like to look through. Make sure to have the above parameter ``ops[look_one_level_down] = True`` when using this parameter.
+
+- **move_bin** (*bool, default: False*) If True and ``ops['fast_disk']`` is different from ``ops[save_disk]``, the created binary file is moved to ``ops['save_disk']``. 
+
 Output settings
 ~~~~~~~~~~~~~~~
 
--  **preclassify**: (*float, default: 0.3*) (**new**) apply classifier
+-  **preclassify**: (*float, default: 0.0*) (**new**) apply classifier
    before signal extraction with probability threshold of "preclassify".
    If this is set to 0.0, then all detected ROIs are kept and signals
    are computed.
 
+- **save_nwb**: (*bool, default: False*) whether to save output as NWB file 
+
 -  **save_mat**: (*bool, default: False*) whether to save the results in
    matlab format in file "Fall.mat". NOTE the cells you click in the GUI
    will NOT change "Fall.mat". But there is a **new** button in the GUI
@@ -69,8 +114,11 @@ Output settings
    a timing dictionary for each plane. Timing dictionary will contain keys
    corresponding to stages and values corresponding to the duration of that stage.
 
-Registration
-~~~~~~~~~~~~
+
+Registration settings
+~~~~~~~~~~~~~~~~~~~~~
+
+These settings are specific to the registration module of suite2p.
 
 - **do_registration**: (*bool, default: True*) whether or not to run
   registration
@@ -81,7 +129,7 @@ Registration
   expression, you may want to use this channel for alignment rather
   than the functional channel.
 
-- **nimg_init**: (*int, default: 200*) how many frames to use to
+- **nimg_init**: (*int, default: 300*) how many frames to use to
   compute reference image for registration
 
 - **batch_size**: (*int, default: 500*) how many frames to register
@@ -119,13 +167,24 @@ Registration
 - **reg_tif_chan2**: (*bool, default: False*) whether or not to write
   the registered binary of the non-functional channel to tiff files
 
-**1P registration settings**
+- **subpixel**: (*int, default:10*) Precision of Subpixel Registration (1/subpixel steps)
+
+- **th_badframes**: (*float, default: 1.0*) Involved with setting threshold for excluding frames for cropping. Set this smaller to exclude more frames. 
+
+- **norm_frames**: (*bool, default: True*) Normalize frames when detecting shifts
+
+- **force_refImg**: (*bool, default: False*) Specifies whether to use refImg stored in ``ops``. Make sure that ``ops['refImg']`` has a valid file pathname. 
+
+- **pad_fft**: (*bool, default: False*) Specifies whether to pad image or not during FFT portion of registration. 
+
+1P registration
+^^^^^^^^^^^^^^^
 
 - **1Preg**: (*bool, default: False*) whether to perform high-pass
   spatial filtering and tapering (parameters set below), which help
   with 1P registration
 
-- **spatial_hp**: (*int, default: 42*) window in pixels for spatial
+- **spatial_hp_reg**: (*int, default: 42*) window in pixels for spatial
   high-pass filtering before registration
 
 - **pre_smooth**: (*float, default: 0*) if > 0, defines stddev of
@@ -136,7 +195,8 @@ Registration
   on edges - they are set to zero (important for vignetted windows, for
   FFT padding do not set BELOW 3*ops['smooth_sigma'])
 
-**Non-rigid registration**
+Non-rigid registration
+^^^^^^^^^^^^^^^^^^^^^^
 
 - **nonrigid**: (*bool, default: True*) whether or not to perform
   non-rigid registration, which splits the field of view into blocks
@@ -156,13 +216,13 @@ Registration
 - **maxregshiftNR**: (*float, default: 5.0*) maximum shift in pixels of
   a block relative to the rigid shift
 
-ROI detection
-~~~~~~~~~~~~~
+ROI detection settings 
+~~~~~~~~~~~~~~~~~~~~~~
 
 - **roidetect**: (*bool, default: True*) whether or not to run ROI
   detect and extraction
 
-- **sparse_mode**: (*bool, default: False*) whether or not to use sparse_mode cell detection
+- **sparse_mode**: (*bool, default: True*) whether or not to use sparse_mode cell detection
 
 - **spatial_scale**: (*int, default: 0*), what the optimal scale of the
   recording is in pixels. if set to 0, then the algorithm determines it
@@ -172,12 +232,14 @@ ROI detection
 - **connected**: (*bool, default: True*) whether or not to require ROIs
   to be fully connected (set to *0* for dendrites/boutons)
 
-- **threshold_scaling**: (*float, default: 5.0*) this controls the
+- **threshold_scaling**: (*float, default: 1.0*) this controls the
   threshold at which to detect ROIs (how much the ROIs have to stand
   out from the noise to be detected). if you set this higher, then
   fewer ROIs will be detected, and if you set it lower, more ROIs will
   be detected.
 
+- **spatial_hp_detect**: (*int, default: 25*) window for spatial high-pass filtering for neuropil subtracation before ROI detection takes place.
+
 - **max_overlap**: (*float, default: 0.75*) we allow overlapping ROIs
   during cell detection. After detection, ROIs with more than
   ops['max_overlap'] fraction of their pixels overlapping with other
@@ -200,8 +262,33 @@ ROI detection
 - **nbinned**: (*int, default: 5000*) maximum number of binned frames
   to use for ROI detection.
 
-Signal extraction
-~~~~~~~~~~~~~~~~~
+- **denoise**: (*bool, default: False*) Whether or not binned movie should be denoised before cell detection in sparse_mode. If True, make sure to set ``ops['sparse_mode']`` is also set to True. 
+
+Cellpose Detection 
+^^^^^^^^^^^^^^^^^^
+These settings are only used if ``ops['anatomical_only']`` is set to an integer greater than 0. 
+
+- **anatomical_only**: (*int, default: 0*) If greater than 0, specifies what to use `Cellpose <https://cellpose.readthedocs.io/>`_ on.
+
+    - 1: Will find masks on max projection image divided by mean image.
+    - 2: Will find masks on mean image
+    - 3: Will find masks on enhanced mean image
+    - 4: Will find masks on maximum projection image 
+
+- **diameter**: (*int, default: 0*) Diameter that will be used for cellpose. If set to zero, diameter is estimated. 
+
+- **cellprob_threshold**: (*float, default: 0.0*) specifies threshold for cell detection that will be used by cellpose. 
+
+- **flow_threshold**: (*float, default: 1.5*) specifies flow threshold that will be used for cellpose.
+
+- **spatial_hp_cp**: (*int, default: 0*) Window for spatial high-pass filtering of image to be used for cellpose. 
+
+- **pretrained_model**: (*str, default: 'cyto'*) Path to pretrained model or string for model type (can be user's model ).
+
+Signal extraction settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+- **neuropil_extract**: (*bool, default: True*) Whether or not to extract signal from neuropil. If False, Fneu is set to zero. 
 
 - **allow_overlap**: (*bool, default: False*) whether or not to extract
   signals from pixels which belong to two ROIs. By default, any pixels
@@ -214,39 +301,58 @@ Signal extraction
 - **inner_neuropil_radius**: (*int, default: 2*) number of pixels to
   keep between ROI and neuropil donut
 
-Spike deconvolution
-~~~~~~~~~~~~~~~~~~~
+- **lam_percentile**: (*int, default: 50*)Percentile of Lambda within area to ignore when excluding cell pixels for neuropil extraction
 
-We neuropil-correct the trace Fout = F - ops['neucoeff'] \* Fneu, and
-then baseline-correct these traces with an ops['baseline'] filter, and
+Spike deconvolution settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We neuropil-correct the trace ``Fout = F - ops['neucoeff'] * Fneu``, and
+then baseline-correct these traces with an ``ops['baseline']`` filter, and
 then detect spikes.
 
+- **spikedetect**: (*bool, default: True*) Whether or not to run spike_deconvolution
+
 - **neucoeff**: (*float, default: 0.7*) neuropil coefficient for all ROIs.
 
 - **baseline**: (*string, default 'maximin'*) how to compute the
   baseline of each trace. This baseline is then subtracted from each
   cell. *'maximin'* computes a moving baseline by filtering the data
-  with a Gaussian of width ops['sig_baseline'] \* ops['fs'], and then
-  minimum filtering with a window of ops['win_baseline'] \* ops['fs'],
+  with a Gaussian of width ``ops['sig_baseline'] * ops['fs']``, and then
+  minimum filtering with a window of ``ops['win_baseline'] * ops['fs']``,
   and then maximum filtering with the same window. *'constant'*
   computes a constant baseline by filtering with a Gaussian of width
-  ops['sig_baseline'] \* ops['fs'] and then taking the minimum value of
+  ``ops['sig_baseline'] * ops['fs']`` and then taking the minimum value of
   this filtered trace. *'constant_percentile'* computes a constant
-  baseline by taking the ops['prctile_baseline'] percentile of the
+  baseline by taking the ``ops['prctile_baseline']`` percentile of the
   trace.
 
 - **win_baseline**: (*float, default: 60.0*) window for maximin filter
   in seconds
 
 - **sig_baseline**: (*float, default: 10.0*) Gaussian filter width in
   seconds, used before maximin filtering or taking the minimum value of
-  the trace, ops['baseline'] = 'maximin' or 'constant'.
+  the trace, ``ops['baseline'] = 'maximin'`` or ``'constant'``.
 
 - **prctile_baseline**: (*float, optional, default: 8*) percentile of
-  trace to use as baseline if ops['baseline'] = 'constant_percentile'.
+  trace to use as baseline if ``ops['baseline'] = 'constant_percentile'``.
+
+Classification settings
+~~~~~~~~~~~~~~~~~~~~~~~
 
-Channel 2 settings
-~~~~~~~~~~~~~~~~~~
+- **soma_crop**: (*bool, default: True*) Specifies whether to crop dendrites for cell classification stats (e.g., compactness)
+
+- **use_builtin_classifier**: (*bool, default: False*) Specifies whether or not to use built-in classifier for cell detection. This will override classifier specified in ``ops['classifier_path']`` if set to True. 
+
+- **classifier_path**: (*str, default: ''*) Path to classifier file you want to use for cell classification
+
+Channel 2 specific settings
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 - **chan2_thres**: threshold for calling an ROI "detected" on a second
   channel
+
+
+Miscellaneous settings
+~~~~~~~~~~~~~~~~~~~~~~
+
+- **suite2p_version**: specifies version of suite2p pipeline that was run with these settings. Changing this parameter will NOT change the version of suite2p used. 

scripts/generate_test_data.py

@@ -32,14 +32,6 @@ def generate_1p1c1500_expected_data(ops):
 		suite2p.run_s2p(ops=test_ops)
 		rename_output_dir('1plane1chan1500')
 
-	# def generate_1p2c_expected_data(ops):
-	#   """
-	#   Generates expected output for test_1plane_2chan_sourcery of test_full_pipeline.py.
-	#   """
-	#   test_ops = FullPipelineTestUtils.initialize_ops_test_1plane_2chan_sourcery(ops.copy())
-	#   suite2p.run_s2p(ops=test_ops)
-	#   rename_output_dir('1plane2chan')
-
 	def generate_2p2c1500_expected_data(ops):
 		"""
 		Generates expected output for test_2plane_2chan_with_batches of test_full_pipeline.py.
@@ -78,17 +70,18 @@ def generate_detection_1plane1chan_test_data(ops):
 			[[Path(ops['data_path'][0]).joinpath('detection/pre_registered.npy')]],
 			(404, 360)
 		)
-		ops, stat = suite2p.detection.detect(ops[0])
-		ops['neuropil_extract'] = True
-		cell_masks, neuropil_masks = masks.create_masks(stat, ops['Ly'], ops['Lx'], ops=ops)
-		output_dict = {
-			'stat': stat,
-			'cell_masks': cell_masks,
-			'neuropil_masks': neuropil_masks
-		}
-		np.save('expected_detect_output_1p1c0.npy', output_dict)
-		# Remove suite2p directory generated by prepare function
-		shutil.rmtree(os.path.join(test_data_dir_path, 'suite2p'))
+		with suite2p.io.BinaryRWFile(Ly = ops[0]['Ly'], Lx = ops[0]['Lx'], filename=ops[0]['reg_file']) as f_reg:
+			ops, stat = suite2p.detection.detection_wrapper(f_reg, ops=ops[0])
+			ops['neuropil_extract'] = True
+			cell_masks, neuropil_masks = masks.create_masks(stat, ops['Ly'], ops['Lx'], ops=ops)
+			output_dict = {
+				'stat': stat,
+				'cell_masks': cell_masks,
+				'neuropil_masks': neuropil_masks
+			}
+			np.save('expected_detect_output_1p1c0.npy', output_dict)
+			# Remove suite2p directory generated by prepare function
+			shutil.rmtree(os.path.join(test_data_dir_path, 'suite2p'))
 
 	def generate_detection_2plane2chan_test_data(ops):
 		"""
@@ -114,16 +107,17 @@ def generate_detection_2plane2chan_test_data(ops):
 		two_plane_ops[1]['meanImg_chan2'] = np.load(detection_dir.joinpath('meanImg_chan2p1.npy'))
 		for i in range(len(two_plane_ops)):
 			op = two_plane_ops[i]
-			# Neuropil_masks are later needed for extraction test data step
-			op['neuropil_extract'] = True
-			op, stat = suite2p.detection.detect(ops=op)
-			cell_masks, neuropil_masks = masks.create_masks(stat, op['Ly'], op['Lx'], ops=op)
-			output_dict = {
-				'stat': stat,
-				'cell_masks': cell_masks,
-				'neuropil_masks': neuropil_masks
-			}
-			np.save('expected_detect_output_%ip%ic%i.npy' % (ops['nchannels'], ops['nplanes'], i), output_dict)
+			with suite2p.io.BinaryRWFile(Ly = op['Ly'], Lx = op['Lx'], filename=op['reg_file']) as f_reg:
+				# Neuropil_masks are later needed for extraction test data step
+				op['neuropil_extract'] = True
+				op, stat = suite2p.detection.detection_wrapper(f_reg, ops=op)
+				cell_masks, neuropil_masks = masks.create_masks(stat, op['Ly'], op['Lx'], ops=op)
+				output_dict = {
+					'stat': stat,
+					'cell_masks': cell_masks,
+					'neuropil_masks': neuropil_masks
+				}
+				np.save('expected_detect_output_%ip%ic%i.npy' % (ops['nchannels'], ops['nplanes'], i), output_dict)
 		# Get rid of registered binary files that were created for detection module in 
 		# DetectionTestUtils.prepare
 		remove_binary_file(test_data_dir_path, 0, '')