Jump step long TSO runtimes

[stscijgbot-jp]

Issue JP-3877 was created on JIRA by David Law:

https://jira.stsci.edu/browse/JP-3697 worked multiple different improvements to the jump step that improved runtimes and memory usage for a variety of different observations.  Testing of those changes has turned up an example in which almost no gains were seen and the jump step takes over an hour to run.

This interesting test case is based on unrelated tests triggered by #9160

jw03077002001_02101_00001-seg002_mirimage_uncal.fits is a MIRI imaging TSO with 512x512 pixels, 45 groups, and 50 ints (in this segment).  Jump step runtime is 4047 seconds with 1.17.1, nearly all of it spent in the twopoint difference code.  Surprisingly, that's essentially unchanged with the latest dev jwst/stcal as of today (i.e., with https://jira.stsci.edu/browse/JP-3697 merged) with a current runtime of 3888 seconds.

It would be useful to see if we can identify why this file (50 x 45 x 512 x 512) takes 10x longer than jw01283001001_03101_00001_mirimage (10 x 100 x 1024 x 1032) which nominally has twice the number of elements.

Timothy Brandt this is the example that I mentioned this morning in passing.

[stscijgbot-jp]

Comment by David Law on JIRA:

Update: Timothy Brandt looked at this quickly and found that this file was falling into a section of the jump code that didn't make sense for this kind of file, which reminded me of the existing known issue about negative usable group numbers.  Indeed, it looks like spacetelescope/stcal#338 fixes this, dropping the jump step runtime from 3888 seconds to 68 seconds.

[stscijgbot-jp]

Comment by David Law on JIRA:

Adding a note for future reference about another test case; the NIRSpec BOTS pipeline notebook uses PID 1366 Obs 3.  In 1.17.1 each of the segments in the uncal file takes about 30 minutes to run through the jump step.  After this and other recent improvements for build 11.3 that drops to about 30 seconds each.