MIRI LRS specwcs datamodel

[jemorrison]

CLOSING PR in favor of #393

Helps to resolveJP-3848

Closes #

This PR adds a datamodel for the reference file type, specwcs, for the LRS. It can be used for Fixed slit and slitless.
The reference file for Fixed slit contains the V2, V3 corners of the slit. These values are not in the slitless reference file
and default to None when the reference files are read in using datamodels.

Tasks

• update or add relevant tests
• update relevant docstrings and / or docs/ page
• Does this PR change any API used downstream? (if not, label with no-changelog-entry-needed)
  • write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • run jwst regression tests with this branch installed ("git+https://github.com/<fork>/stdatamodels@<branch>")

news fragment change types...

• changes/<PR#>.feature.rst: new feature
• changes/<PR#>.bugfix.rst: fixes an issue
• changes/<PR#>.doc.rst: documentation change
• changes/<PR#>.removal.rst: deprecation or removal of public API
• changes/<PR#>.misc.rst: infrastructure or miscellaneous change

[jemorrison]

@drlaw1558 @melanieclarke

Here is the first attempt at a data model for specwcs for MIRI slit data. This data model can be used for slit or slitless.
It is rather simple. If it used on slitless data the v2, v3 corners of the slit do not exist in the reference file and default to be None.

[codecov]

Codecov Report

Attention: Patch coverage is 12.76596% with 41 lines in your changes missing coverage. Please review.

Project coverage is 77.58%. Comparing base (788f7bd) to head (233ab1e).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
src/stdatamodels/jwst/datamodels/wcs_ref_models.py	12.76%	41 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #382      +/-   ##
==========================================
+ Coverage   67.59%   77.58%   +9.98%     
==========================================
  Files         115      115              
  Lines        5932     5193     -739     
==========================================
+ Hits         4010     4029      +19     
+ Misses       1922     1164     -758     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nden]

It'd be helpful to add description (or title) for each of these fields. These can come as a minimum from the names in the FITS table.

[jemorrison]

@nden The names in the fits file are exactly these names and no more information in fits file is provided. I will give a shot at a description and run that by Greg Sloan who I think made these reference files.

[nden]

Yes, you are right. The names are the same. I sent you a document with the description. I think it's good to add the descriptions because the meaning of the names is not obvious.

[jemorrison]

done

[nden]

If you want to be able to programmatically create a MIRILRSModel then you need to implement the __init__ method.

A populate_meta method may be useful to make sure the required meta keywords are present.

Would yo consider renaming the model to MiriLRSSpecwcs to be consistent with the other simialr models?

[jemorrison]

@nden I will add init and populate_meta. I will follow what the other models do in wcs_ref_models.py. I have never made a model like this. Could you explain the reason these methods are in the model. Is it to able to create
a model in python - on the fly and not use a reference file.

On the name I tried to name the model in a way that was consistent with the other models defined in wcs_ref_models.py.
All the models had the instrument capitalized and did not have Specwcs in the model name. Am I missing something ?

[nden]

Is it to able to create a model in python - on the fly and not use a reference file.

Yes, that's the only reason. If you think the model will be be initialized only by a file then it's not necessary to add an __init__.

[jemorrison]

@braingram I am hoping you can help me. I have made a new datamodel MIRILrsModel (it is a specwcs model).
The model name is up for debate so I can change it to make it more standard, but my question is I set up a init and
populate_meta method for the model. This is first time I have done it this way. I don't know how to call it and test it.
I thought if I did this:
model2 = datamodels.MIRILrsModel()
The init method would call populate_meta at least.
But that does not seem to be the case.
Do you know how to test that init or populate_meta is work. I am just looking for a simple test I can run and then one where I set the value in the initi step.

[braingram]

@braingram I am hoping you can help me. I have made a new datamodel MIRILrsModel (it is a specwcs model). The model name is up for debate so I can change it to make it more standard, but my question is I set up a init and populate_meta method for the model. This is first time I have done it this way. I don't know how to call it and test it. I thought if I did this: model2 = datamodels.MIRILrsModel() The init method would call populate_meta at least. But that does not seem to be the case. Do you know how to test that init or populate_meta is work. I am just looking for a simple test I can run and then one where I set the value in the initi step.

The __init__ and populate_meta look similar to the other models in wcs_ref_models.py. When I call MIRILrsModel() it does call populate_meta. Is this not the case for you?

[jemorrison]

@braingram
I think I solved my problem with the class and populate_meta. Now I want to try and pass a wavetable to the init method and see if it gets filled in correctly. Maybe this is over-kill, but since wavetable is a 2-D array and in the schema it is read in from the FITS extension WAVETABLE - how could I test passing this table in on initialization. Would I need to set up a class wavetable and fill in the values and then pass wavetable in when I define MIRILrsModel or something else ?

[braingram]

@braingram I think I solved my problem with the class and populate_meta. Now I want to try and pass a wavetable to the init method and see if it gets filled in correctly. Maybe this is over-kill, but since wavetable is a 2-D array and in the schema it is read in from the FITS extension WAVETABLE - how could I test passing this table in on initialization. Would I need to set up a class wavetable and fill in the values and then pass wavetable in when I define MIRILrsModel or something else ?

Thanks for the update.

Having the test (or tests) closely match the usage is jwst seems most useful. Is there a jwst PR using this new model? If it's opening a reference file from CRDS one option would be to construct a test that opens a "faked" reference file with the new datamodel. Here's a test that does something similar:

stdatamodels/src/stdatamodels/jwst/datamodels/tests/test_open.py

Line 87 in f7587b7

def test_open_hdulist(tmp_path):

I assume one other use will be for defining a reference file. In that case providing an array to the constructor, then saving the model, then verifying the contents seems like a good test to consider.

Finally if this is a reference model mapped to a CRDS reference file the test_schema_against_crds file could use an update (to use the new reference model):

stdatamodels/src/stdatamodels/jwst/datamodels/tests/test_schema_against_crds.py

Line 170 in f7587b7

"specwcs": dm.SpecwcsModel,

[jemorrison]

@braingram I just pushed up changes.
This is what I am trying to do - maybe it is not needed. I wrote this python script to just see if I could set the MIRI LRS specwcs model up rather than using a reference file in CRDS.
from stdatamodels.jwst import datamodels
import jwst

class wavetable_test:
x_center = np.array([1,2,3])
y_center= np.array([4,5,6])
wavelength = np.array([12.4,13.4, 14.5])
x0 = np.array([1,2,3])
x1 = np.array([1,2,3])
x2 = np.array([1,2,3])
x3 = np.array([1,2,3])
y0 = np.array([1,2,3])
y1 = np.array([1,2,3])
y2 = np.array([1,2,3])
y3 = np.array([1,2,3])
print(wavetable_test.x_center)
model = datamodels.MiriLRSSpecwcsModel(x_ref=430, y_ref=400, wavetable=wavetable_test)

I get an error when I run this:
File "/Users/morrison/Pipeline/testing/MIRI_LRS_3848/test_specwcs_datamodel.py", line 20, in
model = datamodels.MiriLRSSpecwcsModel(x_ref=430, y_ref=400, wavetable=wavetable_test)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/morrison/git/stdatamodels/src/stdatamodels/jwst/datamodels/wcs_ref_models.py", line 404, in init
self.wavetable = wavetable
^^^^^^^^^^^^^^
File "/Users/morrison/git/stdatamodels/src/stdatamodels/model_base.py", line 692, in setattr
properties.ObjectNode.setattr(self, attr, value)
File "/Users/morrison/git/stdatamodels/src/stdatamodels/properties.py", line 331, in setattr
val = _cast(val, schema)
^^^^^^^^^^^^^^^^^^
File "/Users/morrison/git/stdatamodels/src/stdatamodels/properties.py", line 47, in _cast
if (_is_struct_array_schema(schema) and len(val) and
^^^^^^^^
TypeError: object of type 'type' has no len()

[jemorrison]

@braingram
I have such a mess now with the main getting the style checks updated. I can not figure out the conflicts. I think I will close this PR - and open another clean on and then add my changes to the new PR

[braingram]

Thanks for sharing the example. That error is caused by the schema expecting a structured array for the wavetable. Let me know if more information would be helpful.

@braingram
I have such a mess now with the main getting the style checks updated. I can not figure out the conflicts. I think I will close this PR - and open another clean on and then add my changes to the new PR

Sounds good. This repository should hopefully be done with major style changes (and now aligns with the in-progress changes in jwst). These can certainly be disruptive to ongoing work.