use scipy directly instead of astropy modeling for residual fringe fit

[braingram]

While looking at the regression tests. I noticed that test_residual_fringe_cal is one of the slower tests.

Running just that test took ~2566 seconds:
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/817

Profiling the test locally revealed that the majority of the time is spent in 2 functions:

• new_fit_1d_fringes_bayes_evidence
• fit_1d_background_complex

This PR aims to improve the second function fit_1d_background_complex by:

• replacing the ChiSqOutlierRejectionFitter class (defined in fitter.py shown in the above plot) with an equivalent function
• replacing the use of splines from astropy modeling with direct use of scipy.interpolate.BSpline

As mentioned in the astropy Spline1D docs much of the functionality of Spline1D is handled by scipy.interpolate.BSpline. The spline fitting here boiled down to:

• a call to LSQUnivariateSpline (which astropy wraps with SplineExactKnotsFitter)
• repeated fits with outlier rejection using the ChiSqOutlierRejectionFitter

The repeated fits are particularly costly with Spline1D due in large part to it's calls to inspect which are avoided by using BSpline directly. Here's a zoomed in portion of the plot showing what occurs during Spline1D.__init__:

Running test_residual_fringe_cal with the changes in this PR took 1562 seconds (~60% the previous time):
https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/818

A similar plot shows the relative speed-up of fit_1d_background_complex (and the remaining large contribution of new_fit_1d_fringes_bayes_evidence to the total runtime):

Checklist for maintainers

• added entry in CHANGES.rst within the relevant release section
• updated or added relevant tests
• updated relevant documentation
• added relevant milestone
• added relevant label(s)
• ran regression tests, post a link to the Jenkins job below.
  How to run regression tests on a PR
• Make sure the JIRA ticket is resolved properly

[codecov]

Codecov Report

Patch coverage: 82.75% and project coverage change: -0.01% ⚠️

Comparison is base (12030b5) 76.58% compared to head (031e4b2) 76.58%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7764      +/-   ##
==========================================
- Coverage   76.58%   76.58%   -0.01%     
==========================================
  Files         456      456              
  Lines       36960    36920      -40     
==========================================
- Hits        28307    28275      -32     
+ Misses       8653     8645       -8     

Flag	Coverage Δ		*Carryforward flag
nightly	77.39% <ø> (-0.03%)	⬇️	Carriedforward from 12030b5

*This pull request uses carry forward flags. Click here to find out more.

Files Changed	Coverage Δ
jwst/residual_fringe/utils.py	55.13% <40.00%> (+0.31%)	⬆️
jwst/residual_fringe/fitter.py	92.00% <91.66%> (-2.34%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nden]

Could you start a regression test run?

[braingram]

Could you start a regression test run?

Regression tests all pased: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/820/

The devdeps failure appears related to numpy 2.0 and nep51 and pyyaml/asdf not playing nice together. I think it should be a simple fix in asdf. Here is a draft PR with tests running to see if the issue is addressed:
asdf-format/asdf#1605

[braingram]

Regression tests passed without any errors. For this run (820), test_residual_fringe_cal took ~48 minutes. For the previous run on the server (819), the same test took 1 hour and 6 minutes.

I marked this as ready for review even though it does not address the other slow function. new_fit_1d_fringes_bayes_evidence. Hopefully it can be addressed in a separate PR.

[hbushouse]

Looks good to me, but I'd like to get some MIRI MRS eyes on it, including testing on a few more datasets than what we have in the regtests, just to verify compatibility with the previous results.

[hbushouse]

@jemorrison @drlaw1558 could you take this PR code on a shake down run using any appropriate test data you have?

[WilliamJamieson]

This is clearly a performance bug in modeling. It is likely one of the sources of its poor performance at scale as the Parameter object is used throughout the code.

Please open an issue on modeling to describe this.

[drlaw1558]

Looks ok to me; in the test cases I ran it on the applied fringe correction was identical and unchanged.

[braingram]

Thanks @drlaw1558 for taking a look.

I rebased the branch off the main branch to pull in the changelog updates (and fix the conflicts in CHANGES.rst).