Cannot open linearity file containing NaN values

[stscijgbot-rstdms]

Issue RCAL-327 was created on JIRA by Tyler Desjardins:

Using the linearity reference file on CRDS, the linearity step runs just fine.  But while trying to test a new linearity reference file that contains NaN values in the coeffs array, I got this:
(rcaltest05) desjard@dlrrtb1[█████████████████████████████████████████████████ strun romancal.step.LinearityStep test_in.asdf --override_linearity='roman_linearity_WFI01.asdf'
2022-04-08 14:24:02,076 - CRDS - ERROR - Error determining best reference for 'pars-linearitystep' = Unknown reference type 'pars-linearitystep'
2022-04-08 14:24:02,077 - stpipe.LinearityStep - INFO - LinearityStep instance created.
2022-04-08 14:24:02,120 - stpipe.LinearityStep - INFO - Step LinearityStep running with args ('test_in.asdf',).
2022-04-08 14:24:02,120 - stpipe.LinearityStep - INFO - Step LinearityStep parameters are: {'pre_hooks': [], 'post_hooks': [], 'output_file': None, 'output_dir': None, 'output_ext': '.asdf', 'output_use_model': False, 'output_use_index': True, 'save_results': True, 'skip': False, 'suffix': None, 'search_output_file': True, 'input_dir': ''}
2022-04-08 14:24:02,225 - stpipe.LinearityStep - INFO - Using Linearity reference file ███████████████████████████████████████████████████████████████████████████

ERROR RUNNING STEP 'LinearityStep':
assignment destination is read-only

Traceback (most recent call last):
File "/home/desjard/miniconda3/envs/rcaltest05/bin/strun", line 28, in
step = Step.from_cmdline(sys.argv[1:])
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/stpipe/step.py", line 179, in from_cmdline
return cmdline.step_from_cmdline(args)
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/stpipe/cmdline.py", line 343, in step_from_cmdline
step.run(*positional)
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/stpipe/step.py", line 430, in run
step_result = self.process(*args)
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/romancal/linearity/linearity_step.py", line 55, in process
new_data, new_pdq = linearity_correction(output_model.data, gdq, pdq,
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/stcal/linearity/linearity.py", line 64, in linearity_correction
lin_coeffs = correct_for_flag(lin_coeffs, lin_dq, dqflags)
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/stcal/linearity/linearity.py", line 219, in correct_for_flag
lin_coeffs[:, yf[ii], xf[ii]] = ben_cor
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/asdf/tags/core/ndarray.py", line 365, in setitem
raise e from None
File "/home/desjard/miniconda3/envs/rcaltest05/lib/python3.9/site-packages/asdf/tags/core/ndarray.py", line 362, in setitem
self.make_array()._setitem(*args)
ValueError: assignment destination is read-only

 

Note: The reference file is opened read-only but he coefficient array needs NAN's replaced with 0's. Update the roman linearity step line 43 of linearity step needs to be
lin_coeffs = lin_model.coeffs.copy()

to allow updates.

[stscijgbot-rstdms]

Comment by Nadia Dencheva on JIRA:

Tyler Desjardins Do you expect linearity files to have NaNs in the future? How should we handle them in the pipeline?

 

[stscijgbot-rstdms]

Comment by Tyler Desjardins on JIRA:

We're going to talk about this in RTB and make a decision on Tuesday. I think despite what the JWST documentation says, my preference right now is to not use NaNs, but use zeros and flag the DQ array in the reference file. But we'll let you know next week for sure!

[stscijgbot-rstdms]

Comment by Tyler Desjardins on JIRA:

Nadia Dencheva , we made a decision about this. We'd like to have the code work as it is described on the JWST documentation. So if a NaN is present in any of the coefficients for a given pixel in the linearity file, then no linearity correction should be applied to that pixel, and the pixeldq array should be updated with the flag for no linearity correction.

[stscijgbot-rstdms]

Comment by Nadia Dencheva on JIRA:

#484