Cube Build JP 3016 allow custom suffix and other improvements

[jemorrison]

Resolves JP-3016

Closes #7367

This PR addresses allowing the user to provide a custom suffix. Some improvements were made to manage the memory more efficiently. A few arrays that were not being used any longer were removed. In addition, it removed the option for users with MIRI data to use the coord_system='internal_cal'. 'internal_cal' was designed for NIRSpec and never tested in commissioning. Instead miri users wanting an IFUcube aligned with the ifu plane should use 'coord_system=ifualign'.

A very large effort was made to see if having an option 'in_memory' the way outlier_detection uses such an option was tested. Basically setting in_memory=False would write out out IFU cubes and single IFUCubes if doing outlier detection to disk instead of storing them in memory. When just build typical IFUCube this did not save much memory. So I have put adding this option to cube_build until we better know how outlier detection for IFU is going to be re-designed.

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[jemorrison]

There are changes in cube_build_step at the end of the program that pull out how we deal with the single IFU cube case the typical IFUs. This was done when I was writing out the output files for testing in_memory=False option. Depending on how we update the outlier detection code I might need to write out the files to disk - so I just kept these changes.

I decided to remove these changes because it might complicate the review. If these changes are needed after we decide how output detection will be changed I will add them back in

[drlaw1558]

Can't say as it's obvious to me what the majority of those changes do, but presumably the various test cases have unchanged results?

[hbushouse]

Regression tests started at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/626

[hbushouse]

A lot of tests are failing. See the results of the full regtest run at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/626/testReport/junit/. It's a mixture of both unit tests and regression tests that are all throwing errors.

[codecov]

Codecov Report

Patch coverage: 85.71% and project coverage change: -0.03 ⚠️

Comparison is base (2d9af7c) 77.56% compared to head (0dcc726) 77.54%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7521      +/-   ##
==========================================
- Coverage   77.56%   77.54%   -0.03%     
==========================================
  Files         452      452              
  Lines       36211    36232      +21     
==========================================
+ Hits        28087    28095       +8     
- Misses       8124     8137      +13     

Flag	Coverage Δ
nightly	77.55% <86.66%> (-0.03%)	⬇️

Impacted Files	Coverage Δ
jwst/cube_build/cube_build_step.py	75.81% <73.68%> (-0.75%)	⬇️
jwst/cube_build/ifu_cube.py	89.56% <86.66%> (-2.02%)	⬇️
jwst/cube_build/file_table.py	97.05% <88.46%> (-2.95%)	⬇️
jwst/cube_build/data_types.py	87.17% <90.90%> (+0.81%)	⬆️
jwst/cube_build/cube_build.py	80.09% <100.00%> (-0.10%)	⬇️
jwst/cube_build/cube_build_io_util.py	95.80% <100.00%> (+0.12%)	⬆️

... and 6 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[hbushouse]

Another regtest run started at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/628

[hbushouse]

Still getting some hard errors in regression tests: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/628/testReport/

[hbushouse]

Latest regtest run has solved all errors but one: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/630/testReport/jwst.regtest/test_miri_cubebuild/_stable_deps__test_cube_build_miri_internal_cal/
It's a file not found error, perhaps due to the fact that the new suffix handling might be creating an output file with a different name?

@jemorrison Try running that one regtest locally in your PR branch and see what the new output file name is.

[jemorrison]

Ha. We do not allow miri internal_cal any more. That was never confirmed in commissioning. Instead for these types of cubes the team used coord_system='ifu_align'.
I can update the test and create a truth file for this type.
We do allow NIRSPEC to use internal_cal.

[hbushouse]

Yay! We FINALLY got an error-free regtest run at https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/631/. Just shows 1 minor difference in a keyword comment field, which is expected. So I think this is good to go!

[jemorrison]

Hallelujah !