JP-3153: post-commissioning AMI3 updates (#7862)

Co-authored-by: Brett <[email protected]>
Co-authored-by: Ned Molter <[email protected]>

CHANGES.rst

@@ -9,9 +9,22 @@
 ami
 ---
 
-- Replaced use of deprecated ``scipy.integrate.simps`` 
+- Replaced use of deprecated ``scipy.integrate.simps``
   with ``scipy.integrate.simpson``. [#8320]
 
+- Overhaul of AMI processing. See documentation for full details. [#7862]
+
+- Additional optional input arguments for greater user processing flexibility.
+  See documentation for details. [#7862]
+
+- Bad pixel correction applied to data using new NRM reference file to calculate 
+  complex visibility support (M. Ireland method implemented by J. Kammerer). [#7862]
+
+- Make ``AmiAnalyze`` and ``AmiNormalize`` output conform to the OIFITS standard. [#7862]
+
+- Disable ``AmiAverage`` step. [#7862]
+
+
 associations
 ------------
 

docs/jwst/ami_analyze/description.rst

@@ -20,7 +20,9 @@ the SUB80 subarray, in order to reduce execution time.
 
 Arguments
 ---------
-The ``ami_analyze`` step has four optional arguments:
+The ``ami_analyze`` step has several optional arguments. In most cases the 
+default arguments will be suitable but more advanced users may wish to test
+other options:
 
 :--oversample: The oversampling factor to be used in the model fit (default=3).
 
@@ -33,48 +35,89 @@ The ``ami_analyze`` step has four optional arguments:
 :--rotation_search: List of start, stop, and step values that define the list of
                     rotation search values. The default setting of '-3 3 1'
                     results in search values of [-3, -2, -1, 0, 1, 2, 3].
-
+
+:--bandpass: Synphot spectrum or suitable array to override filter/source 
+             (default=None)
+
+:--usebp: If True, exclude pixels marked DO_NOT_USE from fringe fitting 
+          (default=True)
+
+:--firstfew: If not None, process only the first few integrations (default=None)
+
+:--chooseholes: If not None, fit only certain fringes e.g. ['B4','B5','B6','C2']
+                (default=None)
+
+:--affine2d: User-defined Affine2d object (default=None)
+
+:--run_bpfix: Run Fourier bad pixel fix on cropped data (default=True)
+
 
 Inputs
 ------
 
-2D calibrated image
+3D calibrated image
 ^^^^^^^^^^^^^^^^^^^
-:Data model: `~jwst.datamodels.ImageModel`
-:File suffix: _cal
+:Data model: `~jwst.datamodels.DataModel`
+:File suffix: _calints
 
-The ``ami_analyze`` step takes a single calibrated image as input, which should be
-the "_cal" product resulting from :ref:`calwebb_image2 <calwebb_image2>` processing.
+The ``ami_analyze`` step takes a single calibrated image cube as input, which should be
+the "_calints" product resulting from :ref:`calwebb_image2 <calwebb_image2>` processing.
 Multiple exposures can be processed via use of an ASN file that is used as input
-to the :ref:`calwebb_ami3 <calwebb_ami3>` pipeline. The ``ami_analyze`` step itself does
-not accept an ASN as input.
+to the :ref:`calwebb_ami3 <calwebb_ami3>` pipeline. **Note:** The ``ami_analyze`` step will also
+accept a 2D "_cal" product but errors will not be computed in the output.
+The ``ami_analyze`` step itself does not accept an ASN as input.
 
 Outputs
 -------
 
+The ``ami_analyze`` step produces three output files. The first two (``_ami-oi.fits`` and ``_amimulti-oi.fits``) contain the interferometric observables, and the third (``_amilg.fits``) contains the data, LG model, and residuals.  These are described in more detail below.
+
+The output file name syntax is exposure-based, using the input file name as the root, with
+the addition of the association candidate ID and the "_ami-oi", "_amimulti-oi", or "amilg" product type suffix, e.g.
+"jw87600027001_02101_00002_nis_a3001_ami-oi.fits."
+
+Interferometric observables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+:Data model: `~jwst.datamodels.AmiOIModel`
+:File suffix: _ami-oi.fits, _amimulti-oi.fits
+
+. The inteferometric observables are saved as OIFITS files, a registered FITS format
+for optical interferometry, containing the following list of extensions:
+
+1)  ``OI_ARRAY``: AMI subaperture information  
+2)  ``OI_TARGET``: target properties  
+3)  ``OI_T3``: extracted closure amplitudes, phases 
+4)  ``OI_VIS``: extracted visibility (fringe) amplitudes, phases
+5)  ``OI_VIS2``: squared visibility (fringe) amplitudes
+6)  ``OI_WAVELENGTH``: filter information
+
+For more information on the format and contents of OIFITS files, see the `OIFITS2 standard <https://doi.org/10.1051/0004-6361/201526405>`_.
+
+The _ami-oi.fits file contains tables of median observables over all integrations of the input file.  Errors 
+are computed as the sigma-clipped standard deviation over integrations.
+The _amimulti-oi.fits file contains observables for each integration, and does not contain error estimates. The
+structure is the same as the _ami-oi.fits file, but the following data columns are 2D, with the second dimension being 
+the number of integrations: "PISTONS", "PIST_ERR", "VISAMP", "VISAMPERR", "VISPHI", "VISPHIERR", "VIS2DATA", "VIS2ERR", "T3AMP", "T3AMPERR", "T3PHI", "T3PHIERR".
+
 LG model parameters
 ^^^^^^^^^^^^^^^^^^^
-:Data model: `~jwst.datamodels.AmiLgModel`
-:File suffix: _ami
+:Data model: `~jwst.datamodels.AmiLgFitModel`
+:File suffix: _amilg.fits
 
-The ``ami_analyze`` step produces a single output file, containing the
-following list of extensions:
+The _amilg.fits output file contains the cropped and cleaned data, model, and residuals (data - model) as well as 
+the parameters of the best-fit LG model. It contains the following extensions:
 
-1) ``FIT``: a 2D image of the fitted model
-2) ``RESID``: a 2D image of the fit residuals
-3) ``CLOSURE_AMP``: table of closure amplitudes
-4) ``CLOSURE_PHA``: table of closure phases
-5) ``FRINGE_AMP``: table of fringe amplitudes
-6) ``FRINGE_PHA``: table of fringe phases
-7) ``PUPIL_PHA``: table of pupil phases
-8) ``SOLNS``: table of fringe coefficients
-
-The output file name syntax is exposure-based, using the input file name as the root, with
-the addition of the association candidate ID and the "_ami" product type suffix, e.g.
-"jw87600027001_02101_00002_nis_a3001_ami.fits."
+1) ``CTRD``: a 3D image of the centered, cropped data
+2) ``N_CTRD``: a 3D image CTRD normalized by data peak
+3) ``FIT``: a 3D image of the best-fit model
+4) ``N_FIT``: a 3D image of FIT normalized by data peak
+5) ``RESID``: a 3D image of the fit residuals
+6) ``N_RESID``: a 3D image of RESID normalized by data peak
+7) ``SOLNS``: table of fringe coefficients
 
 Reference Files
 ---------------
-The ``ami_analyze`` step uses a THROUGHPUT reference file.
+The ``ami_analyze`` step uses a THROUGHPUT reference file and NRM reference file.
 
 .. include:: ../references_general/throughput_reffile.inc
+.. include:: ../references_general/nrm_reffile.inc

docs/jwst/ami_average/description.rst

@@ -3,6 +3,15 @@ Description
 
 :Class: `jwst.ami.AmiAverageStep`
 :Alias: ami_average
+
+.. Attention:: 
+	The ``ami_average`` step has been removed from the default ami3 pipeline
+	until the effects of averaging multiple exposures has been more thoroughly examined.
+	It may be updated in the future; until then the legacy code is left in place and skipped by default. It
+	does not use the OIFITS-format (`~jwst.datamodels.AmiOIModel`) input that the current 
+	``ami_analyze`` step produces. It uses the deprecated `~jwst.datamodels.AmiLgModel`
+	for both input and output.
+
 
 The ``ami_average`` step is one of the AMI-specific steps in the ``ami``
 sub-package and is part of Stage 3 :ref:`calwebb_ami3 <calwebb_ami3>` processing.

docs/jwst/ami_normalize/description.rst

@@ -19,31 +19,33 @@ The ``ami_normalize`` step does not have any step-specific arguments.
 Inputs
 ------
 
-LG model parameters
-^^^^^^^^^^^^^^^^^^^
-:Data model: `~jwst.datamodels.AmiLgModel`
-:File suffix: _amiavg and _psf-amiavg
-
-The ``ami_normalize`` step takes two inputs: the first is the LG results for
-a science target and the second is the LG results for the PSF target. These should
-be the "_amiavg" and "_psf-amiavg" products resulting from the
-:ref:`ami_average <ami_average_step>` step. The inputs can be in the form of file
-names or `~jwst.datamodels.AmiLgModel` data models.
+Interferometric Observables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+:Data model: `~jwst.datamodels.AmiOIModel`
+:File suffix: _ami-oi.fits and/or _psf-ami-oi.fits
+
+The ``ami_normalize`` step takes two inputs: the first is the 
+interferometric observables for a science target and the second 
+is the interferometric observables for the PSF target. These should
+be the _ami-oi.fits and _psf-ami-oi.fits products resulting from the
+:ref:`ami_analyze <ami_analyze_step>` step, or two _ami-oi.fits files if the steps 
+are run independently. The inputs can be in the form of filenames or 
+`~jwst.datamodels.AmiOIModel` data models.
 
 Outputs
 -------
 
-Normalized LG model parameters
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-:Data model: `~jwst.datamodels.AmiLgModel`
-:File suffix: _aminorm
+Normalized Interferometric Observables
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+:Data model: `~jwst.datamodels.AmiOIModel`
+:File suffix: _aminorm-oi.fits
 
-The output is a new LG product for the science target in which the closure
-phases and fringe amplitudes have been normalized using the PSF target
+The output is a new set of interferometric observables for the science target
+in which the closure phases and fringe amplitudes have been normalized using the PSF target
 closure phases and fringe amplitudes. The remaining components of the science
 target data model are left unchanged. The output file name syntax is source-based,
 using the product name specified in the input ASN file and having a product type
-of "_aminorm", e.g. "jw87600-a3001_t001_niriss_f480m-nrm_aminorm.fits."
+of "_aminorm-oi.fits", e.g. "jw87600-a3001_t001_niriss_f480m-nrm_aminorm-oi.fits."
 
 Reference Files
 ---------------