JP-3005: Pixel replacement for flagged pixels before spectral extraction

[tapastro]

Resolves JP-3005

Closes #

This PR implements an algorithm to estimate values for flagged DO_NOT_USE pixels before spectral extraction takes place. The algorithm creates a median profile from neighboring slices in the dispersed 2d spectrum, then scales the median profile to slice with a bad pixel to estimate its value.

WORK IN PROGRESS: Currently implemented for ImageModel inputs (LRS-FS, others?), and possibly working for MultiSlitModels as well. Testing and coding in progress.

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

Patch coverage: 18.82% and project coverage change: -0.28 ⚠️

Comparison is base (5026e1c) 77.86% compared to head (ec38675) 77.58%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7398      +/-   ##
==========================================
- Coverage   77.86%   77.58%   -0.28%     
==========================================
  Files         452      455       +3     
  Lines       36197    36367     +170     
==========================================
+ Hits        28184    28216      +32     
- Misses       8013     8151     +138     

Flag	Coverage Δ		*Carryforward flag
nightly	77.64% <ø> (ø)		Carriedforward from 5026e1c

*This pull request uses carry forward flags. Click here to find out more.

Impacted Files	Coverage Δ
jwst/lib/suffix.py	92.00% <ø> (ø)
jwst/stpipe/integration.py	100.00% <ø> (ø)
jwst/pixel_replace/pixel_replace.py	14.81% <14.81%> (ø)
jwst/pixel_replace/pixel_replace_step.py	26.66% <26.66%> (ø)
jwst/pipeline/calwebb_spec2.py	93.98% <50.00%> (-0.34%)	⬇️
jwst/pixel_replace/__init__.py	100.00% <100.00%> (ø)
jwst/step.py	100.00% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[hbushouse]

This will need rebasing to account for the move of datamodels to stdatamodels.

[tapastro]

This PR and the associated stdatamodels PR have been updated to use the DQ bit proposed by the working group - bit 28, 'UNRELIABLE_RESET' is being changed to 'FLUX_ESTIMATED', pending reviews on both PRs. No other holds on these updates otherwise, to my knowledge. This PR may need to increment the minimum stdatamodels version once a release is built with the DQflags update.

[hbushouse]

Looks really nice overall. Just some minor questions and comments.

[hbushouse]

I don't see any new docs for the pixel_replace step itself, or mention of it in the list of steps in the calwebb_spec2 docs.

[hbushouse]

One more (hopefully final!) regtest run started at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/675/, which uses stdatamodels=1.4.0 containing the DQ bit and step status updates.

[hbushouse]

All but 2 of the failures in the regtest run are due to the new S_PXREPL keyword appearing in output headers, with the value 'SKIPPED'. The remaining 2 are due to the use of the now obsolete UNRELIABLE_RESET dq value in some old MIRI dark ref files. I'm satisfied and am merging.

[mcara]

All docstring tests on new PRs now are failing due to this PR. Specifically:

/home/docs/checkouts/readthedocs.org/user_builds/jwst-pipeline/checkouts/7398/docs/jwst/pixel_replace/index.rst:3:
WARNING: Title overline too short.

===============
Pixel Replacement
===============
looking for now-outdated files... none found
pickling environment... done
checking consistency... /home/docs/checkouts/readthedocs.org/user_builds/jwst-pipeline/checkouts/7398/docs/jwst/pixel_replace/index.rst:
WARNING: document isn't included in any toctree

The two problems are in jwst/pixel_replace/index.rst: Title overline too short AND the file itself is not included anywhere.

[hbushouse]

@tapastro Can you fix the doc file issues?