Use all groups to flag bad IRS2 reference pixels

[stscijgbot-jp]

Issue JP-3510 was created on JIRA by Melanie Clarke:

In JP-3437, it was pointed out that the current bad reference pixel flagging algorithm frequently misses intermittent bad pixels because it only checks the last group of the last integration.  If a reference pixel happens to be okay in that group but bad in another, it will not be caught, but it will still impact the outcome.

In addition, bad pixels marked in the DQ table in the IRS2 reference file are replaced with 0.0, to be interpolated over later.  Bad pixels marked by the intermittent flagging algorithm are replaced with their nearest good value, no matter how far away the replacement is.  These replacement options should be tested to see which is better – the better one should be used for both kinds of bad reference pixels.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I have been investigating this issue in parallel with JP-3437, and have some draft code I've been testing, on my fork:

https://github.com/melanieclarke/jwst/tree/bad_refpix_flagging

The difference between this version and the prior flagging algorithm are:

• Flagging happens in the datamodel before calling any of the reference pixel corrections, so that the mean offset subtraction code proposed in PR 8143 can be called in between flagging and IRS2 correction, if/when we decide to use it.
• Statistics are compiled across all integrations and groups instead of just the last one.
• Both readouts of reference pixels are combined for computing statistics, and always flagged in pairs, instead of being treated separately.
• Statistics for the rejection test are robustly computed, with sigma clipping instead of a straight mean.
• Bad values may be replaced by values from the nearest reference group within the same amplifier, respecting parity (even/oddness).  Replacement value is the average of upper and lower values if both are good, or directly using the upper or lower values if only one is good. If there are no nearest good values available, but there is a good adjacent neighbor that does not match parity, that value is used.  If there are no good replacement values, the bad pixel is set to 0.0 to be interpolated over.
• Any bad values previously flagged as BAD_REF_PIXEL in the PIXELDQ extension are also replaced with nearest values.  There are currently none flagged in the existing mask reference files, but if the reference file processes change (JP-3508), it will be naturally handled.

 
Flagging from the DQ table in the IRS2 reference file is still done, but instead of replacing with 0, the same replacement rules are applied.  I tested the opposite method, replacing all flagged bad pixels with 0 instead of neighboring values, but the results were very poor, so I think the interpolation method should be a last resort. See attached, *compare_replace.png.
 
In testing, this change fixes at least one case I know of with an egregious bad reference pixel artifact (attached, *compare_flagging.png, testing with jw02593001001_03101_00002_nrs2_rateints). 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #8197

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Here's a test case dataset from the Garden science program. Both detectors, but there are known missed bad interleaved refpix in the nrs2 dataset:

jw02123001001_11101_00001_nrs2_uncal.fits

This is readpatt NRSIRS2 (i.e. group averaged), as is all the Garden data, but would also be useful to check the efficacy on NRSIRS2RAPID to be sure there are no issues.

Attached figures show the effect of unmasked bad refpix that are slipping through the old on-the-fly detection (slow jumpers that did not jump in the last group of the last integration). Melanie's new fix should catch these. Also an unfinished powerpoint I made for the Garden collaboration with various figures on the irs2 badpix, channel-biasing and even/odd. Slides 11-14 are most relevant to these issues.

In these examples I have manually flagged the missed slow-jumpers to demonstrate what the data should look like when all bad refpix are detected. Each Garden dataset has 3 integrations per exposure, so you can simply take the difference between the rateins to find cases of missed bad refpix that jumped in one int but not the others. This is how the Garden team is finding them.

 

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Thanks for the test case, Eddie Bergeron.

Interestingly, the fix as implemented does not catch these two bad pixels with the default threshold.  Since they jumped only in one integration, and the effect is pretty low-level, the flagging checks don't catch them.  We could turn up the threshold, but that tends to flag too many pixels that are okay to use.

Modifying the code slightly to gather stats across all groups, but separately for each integration, does catch the jumpers appropriately without flagging too many extra pixels – see attached, flag_by_int.png.  Bad pixels flagged in one integration are still replaced in all integrations.

Testing with a couple of my other examples, it looks like this improves the correction in a couple other less-obvious cases too, so I think this is probably a generally better approach.

Howard Bushouse - I'll send another PR in a minute for this change.  Sorry for the extra churn!

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

I like that idea (stats and thresh per int), since it only matters on each int separately. Integrations with lots of groups might still have the same problem. We still have the master DQ exten or new masking algorithm to permanently flag anything that looks like it will sneak under the radar, so just look for a sweet spot to get the right balance for OTF and we can throw the rest into the permanent DQ.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

I agree, flagging on the fly can only do so much - getting known bad pixels into the DQ image will still be needed.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

The only other thing would be a full-on jump detection instead of a sigma clip. The jump detection algorithm in the pipeline is robust - good for finding faint CRs. These slow jumpers are alot like faint CRs, except they can be + or - and can "revert" all in the same int. Full jump detection also might be useful for detecting even/odd changes within frames using these interleaved refpix samples. In that case you can average all the even/odd pix within a channel to make it even easier.

But adding that robust jump detection here might be a heavy lift for limited benefit (just a couple of pixels). Sticking them in the permanent DQ will also work.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed (more) by #8214

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Tested with additional test case (jw01914002002_03101_00001_nrs2_rate.fits) with severe ACN caused by a bad reference pixel in the first integration, reported via helpdesk ticket INC0196671. This change to reference pixel flagging catches the bad pixel and greatly improves the image.

Fix is working as expected.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Nice! Yeah, that's a nasty one. I've uploaded a median column from the difference between the 1st and 2nd rateints images. Classic sinc function on the bad pixel, and huge channel background pullup. Its pretty clear the variable channel backgrounds are being caused by this same effect. Also interesting here that the sinc residuals wrap within the channel. I guess this is expected, but this is the first case I've seen with the badpix bright enough and close enough to the channel edge to show this.

[stscijgbot-jp]

Comment by Melanie Clarke on JIRA:

Closing: fix is working as expected.