Produce rateints and calints files for AMI observations

[stscijgbot-jp]

Issue JP-2904 was created on JIRA by Stephanie La Massa:

We request that the JWST pipeline produce rateints and calints files for AMI observations.

[stscijgbot-jp]

Comment by Katie Kaleida on JIRA:

Moving ticket from the JWSTDMS project to the JP project for CAL to work this in priority order with the rest of their backlog.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Will need input on this from the AMI working group as to how to process and store the results of the per-integration data in the Level 3 processing. Right now, when using a single 2D image per exposure as input, fringe parameters are computed per exposure, stored as a table in the "ami" products created by the "ami_analyze" step in calwebb_ami3, and then potentially averaged together across multiple exposures by the "ami_average" step. If 3D calints products are fed into calwebb_ami3, should the fringe parameters be computed per-integration and, if so, what's the preferred way of storing the multiple results in a single "ami" product for each exposure? When averaging across multiple exposures, should the averaging first be done across the multiple integrations for each exposure and then averaged over multiple exposures, or should all of the per-integration results across all exposures be averaged in a single go? Details like this need to be specified.

[stscijgbot-jp]

Comment by Stephanie La Massa on JIRA:

I'll have a conversation with Anand Sivaramakrishnan , Rachel Cooper , and Deepashri Thatte about this and will get back to you. I think it makes sense for this work to be folded in with integrating the new version of ImPlaneIA into the ami_analyze step of calwebb_ami3, which I believe is work that the AMI team can start dedicating resources to sometime soon.

[stscijgbot-jp]

Comment by Stephanie La Massa on JIRA:

I've talked with the AMI team and here's the plan: Rachel Cooper will lead the effort to integrate the latest version of the stand-alone AMI analysis algorithm into the pipeline. As part of this process, Rachel will work with SCSB to update the pipeline to produce calints and rateints files for AMI data.

Howard Bushouse: should I follow up with you off-line to work out the details for getting this process started?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Stephanie La Massa Rachel Cooper Yes, let's discuss this off-line to outline a plan of attack.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

rateints files are already produced for all exposures, so that one's covered. In order to produce calints products, we'll need to change the association generator rules so that level-2b processing uses rateints files as input (instead of rate files), which will then automatically create calints products coming out of level-2b processing. We'd just need to verify that all the steps that get applied handle multi-integration data properly.

I assume that the capability to use calints products as input to level-3 AMI processing would then be implemented as part of the larger set of updates covered in JP-3155. Rachel Cooper do you agree?

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

That sounds good. Updates to make ami_analyze handle multi-integration data are already underway. The current plan is to keep it flexible so that users can run AMI3 on a 2D cal file if they wish (after being read in it gets expanded to 3D anyway), but the default pipeline "flow" should be to take the calints files as input to AMI3.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

OK, we can change the ASN rules so that the default flow is to produce calints files from level-2b processing and use them as input to level-3 processing. Unfortunately we can't do both at the same time, so we have to pick one or the other.

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

Just so I'm clear – you're saying we have to do either 1) update ASN rules to level-2b produces calints files, or 2) update ASN rules so calints files are the expected input to level-3. (Eventually we want both, but one has to happen first?) If that's the case, I guess we might as well do the first one (change level 2b outputs) first since the rest of the level 2 pipeline for AMI isn't changing at the moment. 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The level-2a pipeline (calwebb_detector1) is setup to always produce both rate and rateints products for a given exposure. It's easy to do this, because the "uncal" input to level-2a processing can be used to create both types of output. But level-2b (calwebb_image2) processing can only take in and put out one kind of product: either a rate as input and a cal as output, or a rateints as input and a calints as output. You can't give it both rate and rateints and have it create both cal and calints.

Right now AMI processing is setup to use rate as the input to level-2b processing and create a cal as output. We can change the ASN rules to use rateints as input, which will give you calints as output, but it can't do both (just one or the other). So I don't see how level-3 can be setup to use both cal and calints as input at the same time, because we can't produce both cal and calints at the same time. This might be doable in off-line processing by a user.

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

Understood, thanks! When we (AMI team) process files we typically run a script that runs the Image2 pipeline on the rate and then the rateints files one after the other, so I had forgotten that that can't happen in a single image2 call. Let's go ahead with changing the default input to image2 to be rateints instead of rate for ami, and users can process the rate files on their own if they want. Could you point me to roughly the part in the code where this is set, or any available documentation about association rules? I had a read through the associations code but didn't find anything that stood out as what I need to change. 

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

Hello, I wanted to check in on this because this ticket and https://jira.stsci.edu/browse/JP-1714 are prerequisites for the updates to the AMI3 stage that I have been working on and am hoping to have ready by September 1. Is anyone in SCSB available help me to make the necessary association generator changes to have the image2 step take rateints and produce calints files by default for AMI exposures, and make the default input to ami_analyze be the 3d calints.fits files (with the option to still accept 2d cal.fits files)?

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Rachel Cooper The good news is that updates to the ASN generator rules were included (as stubs) in a recent update for other things and hence only need to be uncommented to activate them. Specifically, the change for image2 ASN's to switch from rate to rateints as input, is here: https://github.com/spacetelescope/jwst/blob/master/jwst/associations/lib/rules_level2_base.py#L216

and the equivalent change to make ami3 ASN's switch from cal to calints as input is here:

https://github.com/spacetelescope/jwst/blob/master/jwst/associations/lib/rules_level3_base.py#L271

As to changing the ami_analyze step itself, to accept either cal or calints, I guess could be done by simply inserting some kind of logic near the top level that would detect whether the input contains multiple integrations and then go into a loop over integrations.

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

Hi Howard, That's great to see! Should I uncomment those lines as part of the PR I'll be opening, or leave that for you or whoever made the changes? I have already made the necessary changes to the ami_analyze step itself to operate on 3D files, but I wasn't sure if there was something outside of the step code in the more general pipeline structure that needed to be changed (edit: other than the association generator; my reading comprehension finally caught up)

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Rachel Cooper I think it would be fine for you to include the 2 changes to ASN rules in your PR. That way it's all packaged together logically and in one place. The ASN rule change won't work correctly without your ami_analyze updates and vice versa.

Keep in mind, however, that once those ASN rules go into use, only rateints and calints products will flow through the pipelines for AMI observations, at least within the operations environment. While you may be setting up the ami_analyze step to be flexible enough to allow a user to use either cal or calints as input, you can only have one type of product going through the standard flow, not both.

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

That sounds good, I'll include the ASN rule changes then. Understood; we intend for the rateints and calints products to be used in the typical pipeline flow but leave the possibility for pipeline-savvy users to run the ami3 steps on 2D files independently if they want.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

I see that the updates to the association rules to switch AMI processing from rate and cal files to rateints and calints files are included in the main PR for all the AMI pipeline updates at https://github.com/spacetelescope/jwst/pull/7862/files so this ticket can be considered In Progress right now (along with JP-3153) and will be resolved when that PR 7862 gets merged. At which point both JP-3153 and this ticket will be declared resolved.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #7862

[stscijgbot-jp]

Comment by David Law on JIRA:

Rachel Cooper Rachel Plesha Can you please confirm that this is now working as intended?

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

I can confirm that the Detector1Pipeline produces rateints files and the Image2Pipeline saves calints files for AMI exposures when a rateints file is provided as input, but this has always been true. I am not sure how to test the default behavior of Image2 during automated data processing without explicitly calling Image2 on a rateints file. The associations generator code appears to do the right thing, but again when I create level3 associations I am manually providing the correct (calints) product names and don't know how to confirm that the automated way will perform correctly.

[stscijgbot-jp]

Comment by David Law on JIRA:

Rachel Cooper If it's really a question of whether or not this is happening automatically, we can also check in with DMS.  Also potentially worth discussing is what to do with any stale files that would result if we've swapped processing rate/cal files for rateint/calint files.  I'll add to agenda for the May 8 JP meeting.

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

David Law Thanks! Unless there's a way to run detector1 --> image2 --> ami3 without having to provide input filenames/associations at each stage that I'm unaware of (definitely a possibility), I think it is a question of how it will be handled automatically. I'll plan to attend that meeting to provide more info if necessary.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

From Howard:

Regarding testing for the use of rateints/calints products when processing NIRISS AMI observations with DMS B10.2, the directory “████████████████████████████████████████████████████████████ contains results for processing of program 1093, which has a lot of AMI observations. All of the jw01093ami3_asn.json files show that they are requesting calints products as input (for the science and psf members), and the corresponding jw01093image2_asn.json files show that they are requesting rateints products as input. So I think that proves that the desired changes to the ASN generator rules for AMI mode are working correctly.

I do not have permissions to this directory so cannot confirm. Rachel Cooper would you like to see these datasets, too?

[stscijgbot-jp]

Comment by Rachel Cooper on JIRA:

Based on email thread with Howard Bushouse and others, the image2 and ami3 associations are now correctly specifying rateints and calints input files, respectively.