JP-3667: Run nsclean step on imprint and background association members

[hayescr]

Resolves JP-3667

Closes #8591

Run nsclean on imprint and background association members in the spec2 pipeline.

The spec2 pipeline will loop through the imprint and background members in an input association and run each through the nsclean step (if it is not skipped) before performing imprint and background subtraction. This PR is dependent on the changes to add the clean_flicker_noise step in #8669.

Tasks

• request a review from someone specific, to avoid making the maintainers review every PR
• add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• Does this PR change user-facing code / API?
  • add an entry to CHANGES.rst within the relevant release section (otherwise add the no-changelog-entry-needed label to this PR)
  • update or add relevant tests
  • update relevant docstrings and / or docs/ page
  • start a regression test and include a link to the running job (click here for instructions)
    • Do truth files need to be updated ("okified")?
      • after the reviewer has approved these changes, run okify_regtests to update the truth files
• if a JIRA ticket exists, make sure it is resolved properly

[codecov]

Codecov Report

Attention: Patch coverage is 25.00000% with 6 lines in your changes missing coverage. Please review.

Project coverage is 61.75%. Comparing base (3d7f8f5) to head (2ff8573).

Files with missing lines	Patch %	Lines
jwst/pipeline/calwebb_spec2.py	25.00%	6 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8786      +/-   ##
==========================================
- Coverage   61.76%   61.75%   -0.01%     
==========================================
  Files         377      377              
  Lines       38743    38750       +7     
==========================================
+ Hits        23929    23931       +2     
- Misses      14814    14819       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[melanieclarke]

@hayescr - is this ready for review now? If so, I can merge in the latest and start some regtests.

[hayescr]

@melanieclarke Yes, it's ready for review, I went ahead and took it out of draft.

[melanieclarke]

This looks good to me! I'm running regression tests here:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1721/

[melanieclarke]

One thing though - we should probably add a note to the docs for nsclean that if background/imprint images are present, they will be corrected with the same parameters as for the science.

[hayescr]

Ah yes, I've added a note to the nsclean documentation pointing this out and explicitly saying that if a user-supplied mask is provided it will be used for all of the association members, but otherwise the masks will be calculated on-the-fly separately for each image. Let me know if that sounds alright.

[melanieclarke]

Docs look fine, one small fix suggested.

I found a small issue in testing, investigating now.

[hayescr]

Thanks for the catch! I had checked the other intermediate file save options and they were saving to different file names, but I had missed checking the --steps.nsclean.save_results option. I tested your suggestion as well and that does the trick. I've incorporated your suggestions now.

[melanieclarke]

Everything looks good to me now. Local tests on an association with background exposures are succeeding, and the background subtracted image looks much better.

I'll run regtests one more time before merging when some unrelated things are okified, but I think this is good to go.

[hayescr]

Thanks Melanie! Sounds good 👍

[melanieclarke]

Regtests restarted here:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1728/

Looking at results from some local runs, I expect to see low-level changes in results downstream of nsclean in the NIRSpec spec2 runs, since the regtests include background members. I see new nsclean files made for the background members, and changes in the science products look like low-level improvements to the 1/f noise, as expected.

[melanieclarke]

Regtests look as expected: there are some low level changes for NIRSpec spec2 FS and MOS tests at background subtraction and subsequent steps. The remainder of the failures are unrelated.