Add Jump documentation (RCAL-115)

docs/roman/jump/arguments.rst

@@ -0,0 +1,48 @@
+Arguments
+=========
+
+The ``jump`` step has five optional arguments that can be set by the user:
+
+* ``--rejection_threshold``: A floating-point value that sets the sigma
+  threshold for jump detection for ramps having 5 or more groups.. In the code
+  sigma is determined using the read noise from the read noise reference file
+  and the Poisson noise (based on the median difference between samples, and the
+  gain reference file). Note that any noise source beyond these two that may be
+  present in the data will lead to an increase in the false positive rate and
+  thus may require an increase in the value of this parameter. The default value
+  of 4.0 for the rejection threshold will yield 6200 false positives for every
+  million pixels, if the noise model is correct.
+
+* ``--three_group_rejection_threshold``:  A floating-point value that sets the
+  sigma threshold for jump detection for ramps having exactly 3 groups. The
+  default value is 6.0
+
+* ``--four_group_rejection_threshold``:  A floating-point value that sets the
+  sigma threshold for jump detection for ramps having exactly 4 groups. The
+  default value is 5.0
+
+* ``--maximum_cores``: The fraction of available cores that will be
+  used for multi-processing in this step. The default value is 'none' which does not use
+  multi-processing. The other options are 'quarter', 'half', and 'all'. Note that these
+  fractions refer to the total available cores and on most CPUs these include physical
+  and virtual cores. The clock time for the step is reduced
+  almost linearly by the number of physical cores used on all machines. For example, on an Intel CPU with
+  six real cores and 6 virtual cores setting maximum_cores to 'half' results in a
+  decrease of a factor of six in the clock time for the step to run. Depending on the system
+  the clock time can also decrease even more with maximum_cores is set to 'all'.
+
+* ``--flag_4_neighbors``: If set to True (default is True) it will cause the four perpendicular
+  neighbors of all detected jumps to be flagged as a jump. This is needed because of
+  the inter-pixel capacitance (IPC) causing a small jump in the neighbors. The small jump
+  might be below the rejection threshold but will affect the slope determination of
+  the pixel. The step will take about 40% longer to run when this is set to True.
+
+* ``--max_jump_to_flag_neighbors``: A floating point value in units of sigma that limits
+  the flagging of neighbors. Any jump above this cutoff will not have its neighbors flagged.
+  The concept is that the jumps in neighbors will be above the rejection-threshold and thus
+  be flagged as primary jumps. The default value is 1000.
+
+* ``--min_jump_to_flag_neighbors``: A floating point value in units of sigma that limits
+  the flagging of neighbors of marginal detections. Any primary jump below this value will
+  not have its neighbors flagged. The goal is to prevent flagging jumps that would be too
+  small to significantly affect the slope determination.  The default value is 10.

docs/roman/jump/description.rst

@@ -0,0 +1,77 @@
+Description
+===========
+
+Assumptions
+-----------
+We assume that the ``saturation`` step has already been applied to the input
+science exposure, so that saturated values are appropriately flagged in the
+input GROUPDQ array. We also assume that steps such as the reference pixel
+correction (``refpix``) and non-linearity correction (``linearity``) have been applied, so
+that the input data ramps do not have any non-linearities or noise above the modeled Poission
+and read noise due to instrumental effects. The absence of any of these preceding corrections
+or residual non-linearities or noise can lead to the false detection of jumps in the ramps,
+due to departure from linearity.
+
+The ``jump`` step will automatically skip execution if the input data contain fewer
+than 3 groups in the integration, because the baseline algorthim requires two first
+differences to work.
+
+Algorithm
+---------
+This routine detects jumps in an exposure by looking for outliers
+in the up-the-ramp signal for each pixel in the integration within
+an input exposure. On output, the GROUPDQ array is updated with the DQ flag
+"JUMP_DET" to indicate the location of each jump that was found.
+In addition, any pixels that have non-positive or NaN values in the gain
+reference file will have DQ flags "NO_GAIN_VALUE" and "DO_NOT_USE" set in the
+output PIXELDQ array.
+The SCI and ERR arrays of the input data are not modified.
+
+The current implementation uses the two-point difference method described
+in Anderson&Gordon2011_.
+
+Two-Point Difference Method
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+The two-point difference method is applied to the integration as follows:
+
+* Compute the first differences for each pixel (the difference between
+  adjacent groups)
+* Compute the clipped (dropping the largest difference) median of the first differences for each pixel.
+* Use the median to estimate the Poisson noise for each group and combine it
+  with the read noise to arrive at an estimate of the total expected noise for
+  each difference.
+* Compute the "difference ratio" as the difference between the first differences
+  of each group and the median, divided by the expected noise.
+* If the largest "difference ratio" is greater than the rejection threshold,
+  flag the group corresponding to that ratio as having a jump.
+* If a jump is found in a given pixel, iterate the above steps with the
+  jump-impacted group excluded, looking for additional lower-level jumps
+  that still exceed the rejection threshold.
+* Stop iterating on a given pixel when no new jumps are found or only one difference remains.
+* If the there are only three differences (four groups), the standard median is used rather than 
+  the clipped median.
+* If there are only two differences (three groups), the smallest one is compared to the larger
+  one and if the larger one is above a threshold, it is flagged as a jump.
+
+Note that any ramp values flagged as SATURATED in the input GROUPDQ array
+are not used in any of the above calculations and hence will never be
+marked as containing a jump.
+
+Multiprocessing
+===============
+This step has the option of running in multiprocessing mode. In that mode it will
+split the input data cube into a number of row slices based on the number of available
+cores on the host computer and the value of the max_cores input parameter. By
+default the step runs on a single processor. At the other extreme if max_cores is
+set to 'all', it will use all available cores (real and virtual). Testing has shown
+a reduction in the elapsed time for the step proportional to the number of real
+cores used. Using the virtual cores also reduces the elasped time but at a slightly
+lower rate than the real cores.
+
+If multiprocessing is requested the input cube will be divided into a number of
+slices in the row dimension (with the last slice being slightly larger, if needed).
+The slices are then sent to twopoint_difference.py by detect_jumps. After all the
+slices have finished processing, detect_jumps assembles the output group_dq cube
+from the slices.
+
+.. _Anderson&Gordon2011: https://ui.adsabs.harvard.edu/abs/2011PASP..123.1237A

docs/roman/jump/index.rst

@@ -0,0 +1,14 @@
+.. _jump_step:
+
+==============
+Jump Detection
+==============
+
+.. toctree::
+   :maxdepth: 2
+
+   description.rst
+   arguments.rst
+   reference_files.rst
+
+.. automodapi:: romancal.jump

docs/roman/jump/reference_files.rst

@@ -0,0 +1,15 @@
+Reference File Types
+=====================
+
+The ``jump`` step uses two reference files: :ref:`GAIN <gain_reffile>`
+and :ref:`READNOISE <readnoise_reffile>`.
+The GAIN reference file is used to temporarily convert pixel values in
+the ``jump`` step from units of DN to electrons.
+The READNOISE reference file is used in estimating the expected noise
+in each pixel.
+Both are necessary for proper computation of noise estimates within the
+``jump`` step.
+
+:ref:`GAIN <gain_reffile>`
+
+:ref:`READNOISE <readnoise_reffile>`

docs/roman/package_index.rst

@@ -8,4 +8,5 @@ Package Index
    dark_current/index.rst
    flatfield/index.rst
    gain_scale/index.rst
+   jump/index.rst
    references_general/index.rst