JP-3005: Pixel replacement for flagged pixels before spectral extraction

CHANGES.rst

@@ -186,6 +186,12 @@ pipeline
   calling the ``resample_spec`` and ``extract_1d`` steps, to avoid issues with the
   input data accidentally getting modified by those steps. [#7451]
 
+pixel_replace
+-------------
+
+- Add ``pixel_replace`` step to ``Spec2Pipeline``, which uses a weighted interpolation
+  to estimate flux values for pixels flagged as ``DO_NOT_USE``. [#7398]
+
 ramp_fitting
 ------------
 

docs/jwst/pipeline/calwebb_spec2.rst

@@ -77,6 +77,8 @@ TSO exposures. The instrument mode abbreviations used in the table are as follow
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+------+------+--------+-----+
 | :ref:`residual_fringe <residual_fringe_step>` \ :sup:`2` |     |     |     |     |     | |c| |      |      |        |     |
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+------+------+--------+-----+
+| :ref:`pixel_replace <pixel_replace_step>` \ :sup:`2`     | |c| | |c| |     | |c| | |c| |     |      | |c|  |  |c|   |     |
++----------------------------------------------------------+-----+-----+-----+-----+-----+-----+------+------+--------+-----+
 | :ref:`resample_spec <resample_step>`                     | |c| | |c| |     | |c| |     |     |      |      |        |     |
 +----------------------------------------------------------+-----+-----+-----+-----+-----+-----+------+------+--------+-----+
 | :ref:`cube_build <cube_build_step>`                      |     |     | |c| |     |     | |c| |      |      |        |     |
@@ -90,7 +92,7 @@ For NIRISS and NIRCam WFSS, as well as NIRCam TSO grism exposures, the order is
 flat_field, extract_2d, and srctype (no wavecorr).
 For all other modes the order is extract_2d, srctype, wavecorr, and flat_field.
 
-:sup:`2`\ By default the :ref:`residual_fringe <residual_fringe_step>` is skipped in the ``calwebb_spec2`` pipeline. 
+:sup:`2`\ By default this step is skipped in the ``calwebb_spec2`` pipeline.
 
 Notice that NIRSpec MOS is the only mode to receive master background subtraction
 in the ``calwebb_spec2`` pipeline. All other spectral modes have master background

docs/jwst/pixel_replace/arguments.rst

@@ -0,0 +1,15 @@
+Step Arguments
+==============
+
+The ``pixel_replace`` step has the following step-specific arguments:
+
+``--algorithm`` (str, default='fit_profile')
+  This sets the method used to estimate flux values for bad pixels. Only one
+  option has been implemented so far, using a profile fit to adjacent column values.
+
+``--n_adjacent_cols`` (int, default=3)
+  Number of adjacent columns (on either side of column containing a bad pixel) to use in
+  creation of the source profile, in cross-dispersion direction. The total number of
+  columns used in the profile will be twice this number; on array edges, the total number
+  of columns contributing to the source profile will be less than ``2 * n_adjacent_cols``.
+

docs/jwst/pixel_replace/index.rst

@@ -0,0 +1,14 @@
+.. _pixel_replace_step:
+
+===============
+Pixel Replacement
+===============
+
+.. toctree::
+   :maxdepth: 2
+
+   main.rst
+   arguments.rst
+   reference_files.rst
+
+.. automodapi:: jwst.pixel_replace.pixel_replace_step

docs/jwst/pixel_replace/main.rst

@@ -0,0 +1,34 @@
+Description
+===========
+
+:Classes: `jwst.pixel_replace.PixelReplaceStep`
+:Alias: pixel_replace
+
+During spectral extraction, pixels flagged as bad are ignored in the summation process.
+If a bad pixel is part of the point-spread function (PSF) at a given wavelength, the
+absence of the signal in the flagged pixel will lead to a divot at that wavelength in
+the extracted spectrum.
+
+To avoid this defect in the 1-D spectrum, this step estimates the flux values of pixels
+flagged as ``DO_NOT_USE`` in 2-D extracted spectra, prior to rectification in the
+``resample_spec`` step. ``pixel_replace`` inserts these estimates into the data array,
+unsets the ``DO_NOT_USE`` flag and sets the ``FLUX_ESTIMATED`` flag for each pixel affected.
+
+Algorithms
+==========
+
+Currently, one algorithm has been tested to estimate the missing fluxes.
+
+Adjacent Profile Approximation
+------------------------------
+
+First, the input 2-d spectral cutout is scanned across the dispersion axis to determine
+which cross-dispersion vectors (column or row, depending on dispersion direction) contain
+at least one flagged pixel. Next, for each affected vector, a median normalized profile is created.
+
+First, the adjacent arrays (the number of which is set by the step argument
+``n_adjacent_cols``) are individually normalized. Next, each pixel in the profile is set to
+the median of the normalized values. This results in a median of normalized values filling the vector.
+
+Finally, this profile is scaled to the vector containing a missing pixel, and the value is
+estimated from the scaled profile.

docs/jwst/pixel_replace/reference_files.rst

@@ -0,0 +1,4 @@
+Reference File
+==============
+
+This step does not use any reference file.

docs/jwst/references_general/references_general.rst

@@ -602,7 +602,7 @@ Bit  Value         Name              Description
 4    16            OUTLIER           Flagged by outlier detection
 5    32            PERSISTENCE       High persistence
 6    64            AD_FLOOR          Below A/D floor
-7    128           RESERVED
+7    128           UNDERSAMP         Undersampling correction
 8    256           UNRELIABLE_ERROR  Uncertainty exceeds quoted error
 9    512           NON_SCIENCE       Pixel not on science portion of detector
 10   1024          DEAD              Dead pixel
@@ -623,7 +623,7 @@ Bit  Value         Name              Description
 25   33554432      UNRELIABLE_FLAT   Flat variance large
 26   67108864      OPEN              Open pixel (counts move to adjacent pixels)
 27   134217728     ADJ_OPEN          Adjacent to open pixel
-28   268435456     UNRELIABLE_RESET  Sensitive to reset anomaly
+28   268435456     FLUX_ESTIMATED    Pixel flux estimated due to missing/bad data
 29   536870912     MSA_FAILED_OPEN   Pixel sees light from failed-open shutter
 30   1073741824    OTHER_BAD_PIXEL   A catch-all flag
 31   2147483648    REFERENCE_PIXEL   Pixel is a reference pixel

jwst/lib/suffix.py

@@ -190,6 +190,7 @@
     'wavecorrstep',
     'wfsscontamstep',
     'undersamplingcorrectionstep',
+    'pixelreplacestep',
 }
 
 

jwst/pipeline/calwebb_spec2.py

@@ -25,6 +25,7 @@
 from ..msaflagopen import msaflagopen_step
 from ..pathloss import pathloss_step
 from ..photom import photom_step
+from ..pixel_replace import pixel_replace_step
 from ..resample import resample_spec_step
 from ..srctype import srctype_step
 from ..straylight import straylight_step
@@ -78,6 +79,7 @@ class Spec2Pipeline(Pipeline):
         'barshadow': barshadow_step.BarShadowStep,
         'wfss_contam': wfss_contam_step.WfssContamStep,
         'photom': photom_step.PhotomStep,
+        'pixel_replace': pixel_replace_step.PixelReplaceStep,
         'resample_spec': resample_spec_step.ResampleSpecStep,
         'cube_build': cube_build_step.CubeBuildStep,
         'extract_1d': extract_1d_step.Extract1dStep
@@ -274,6 +276,9 @@ def process_exposure_product(
         calibrated.meta.asn.table_name = op.basename(asn_file)
         calibrated.meta.filename = self.make_output_path(suffix=suffix)
 
+        # Replace bad pixels before rectification
+        calibrated = self.pixel_replace(calibrated)
+
         # Produce a resampled product, either via resample_spec for
         # "regular" spectra or cube_build for IFU data. No resampled
         # product is produced for time-series modes.

jwst/pixel_replace/__init__.py

@@ -0,0 +1,3 @@
+from .pixel_replace_step import PixelReplaceStep
+
+__all__ = ["PixelReplaceStep"]