TweakReg: doc updates.

docs/roman/package_index.rst

@@ -14,3 +14,4 @@
    stpipe/index.rst
    refpix/index.rst
    source_detection/index.rst
+   tweakreg/index.rst

docs/roman/pipeline_steps.rst

@@ -12,3 +12,5 @@ Pipeline Steps
    assign_wcs/index.rst
    flatfield/index.rst
    photom/index.rst
+   source_detection/index.rst
+   tweakreg/index.rst

docs/roman/tweakreg/README.rst

@@ -0,0 +1,231 @@
+Description
+===========
+
+:Class: `roman.tweakreg.TweakRegStep`
+:Alias: tweakreg
+
+Overview
+--------
+This step uses the coordinates of point-like sources from an input catalog
+(i.e. the result from `SourceDetectionStep` saved in the
+`meta.tweakreg_catalog` attribute) and compares them with the
+coordinates from a Gaia catalog to compute corrections to
+the WCS of the input images such that sky catalogs obtained from the image catalogs
+using the corrected WCS will align on the sky.
+
+Custom Source Catalogs
+----------------------
+The default catalog used by ``tweakreg`` step can be disabled by
+providing a file name to a custom source catalog in the
+``meta.tweakreg_catalog`` attribute of input data models.
+The catalog must be in a format automatically recognized by
+:py:meth:`~astropy.table.Table.read`. The catalog must contain
+either ``'x'`` and ``'y'`` or ``'xcentroid'`` and ``'ycentroid'`` columns which
+indicate source *image* coordinates (in pixels). Pixel coordinates are
+0-indexed.
+
+For the ``tweakreg`` step to use user-provided input source catalogs,
+``use_custom_catalogs`` parameter of the ``tweakreg`` step must be set to
+`True`.
+
+In addition to setting the ``meta.tweakreg_catalog`` attribute of input data
+models to the custom catalog file name, the ``tweakreg_step`` also supports two
+other ways of supplying custom source catalogs to the step:
+
+1. Adding ``tweakreg_catalog`` attribute to the ``members`` of the input ASN
+   table - see `~roman.datamodels.ModelContainer` for more details.
+   Catalog file names are relative to ASN file path.
+
+2. Providing a simple two-column text file, specified via step's parameter
+   ``catfile``, that contains input data models' file names in the first column
+   and the file names of the corresponding catalogs in the second column.
+   Catalog file names are relative to ``catfile`` file path.
+
+Specifying custom source catalogs via either the input ASN table or
+``catfile``, will update input data models' ``meta.tweakreg_catalog``
+attributes to the catalog file names provided in either in the ASN table or
+``catfile``.
+
+.. note::
+    When custom source catalogs are provided via both ``catfile`` and
+    ASN table members' attributes, the ``catfile`` takes precedence and
+    catalogs specified via ASN table are ignored altogether.
+
+.. note::
+    1. Providing a data model file name in the ``catfile`` and leaving
+       the corresponding source catalog file name empty -- same as setting
+       ``'tweakreg_catalog'`` in the ASN table to an empty string ``""`` --
+       would set corresponding input data model's ``meta.tweakreg_catalog``
+       attribute to `None`. In this case, ``tweakreg_step`` will automatically
+       generate a source catalog for that data model.
+
+    2. If an input data model is not listed in the ``catfile`` or does not
+       have ``'tweakreg_catalog'`` attribute provided in the ASN table,
+       then the catalog file name in that model's ``meta.tweakreg_catalog``
+       attribute will be used. If ``model.meta.tweakreg_catalog`` is `None`,
+       ``tweakreg_step`` will automatically generate a source catalog for
+       that data model.
+
+Alignment
+---------
+The source catalog (either created by `SourceDetectionStep` or provided by the user)
+gets cross-matched and fit to an astrometric reference catalog
+(set by ``TweakRegStep.abs_refcat``). The pipeline initially supports fitting to
+any Gaia Data Release (defaults to `GAIADR3`).
+
+WCS Correction
+--------------
+The linear coordinate transformation computed in the previous step
+is used to define tangent-plane corrections that need to be applied
+to the GWCS pipeline in order to correct input image WCS.
+This correction is implemented by inserting a ``v2v3corr`` frame with
+tangent plane corrections into the GWCS pipeline of the image's WCS.
+
+Step Arguments
+--------------
+``TweakRegStep`` has the following arguments:
+
+**Catalog parameters:**
+
+* ``use_custom_catalogs``: A boolean that indicates whether
+  to ignore source catalog in the input data model's ``meta.tweakreg_catalog``
+  attribute (Default=`False`).
+
+  .. note::
+    If `True`, the user must provide a valid custom catalog that will be assigned to
+    `meta.tweakreg_catalog` and used throughout the step.
+
+* ``catalog_format``: A `str` indicating one of the catalog output file format
+  supported by :py:class:`astropy.table.Table` (Default='ascii.ecsv').
+
+  .. note::
+    - This option must be provided whenever `use_custom_catalogs = True`.
+
+    - The full list of supported formats can be found on
+      the `astropy`'s `Built-In Table Readers/Writers`_ webpage.
+
+.. _`Built-In Table Readers/Writers`: https://docs.astropy.org/en/stable/io/unified.html#built-in-table-readers-writers
+
+* ``catfile``: Name of the file with a list of custom user-provided catalogs
+  (Default='').
+
+  .. note::
+    - This option must be provided whenever `use_custom_catalogs = True`.
+
+* ``catalog_path``: A `str` indicating the catalogs output file path (Default='').
+
+  .. note::
+      All catalogs will be saved to this path.
+      The default value is the current working directory.
+
+**Reference Catalog parameters:**
+
+* ``expand_refcat``: A boolean indicating whether or not to expand reference
+  catalog with new sources from other input images that have been already
+  aligned to the reference image (Default=False).
+
+**Object matching parameters:**
+
+* ``minobj``: A positive `int` indicating minimum number of objects acceptable
+  for matching (Default=15).
+
+* ``searchrad``: A `float` indicating the search radius in arcsec for a match
+  (Default=2.0).
+
+* ``use2dhist``: A boolean indicating whether to use 2D histogram to find
+  initial offset (Default=True).
+
+* ``separation``: Minimum object separation in arcsec (Default=1.0).
+
+* ``tolerance``: Matching tolerance for ``xyxymatch`` in arcsec (Default=0.7).
+
+**Catalog fitting parameters:**
+
+* ``fitgeometry``: A `str` value indicating the type of affine transformation
+  to be considered when fitting catalogs. Allowed values:
+
+  - ``'shift'``: x/y shifts only
+  - ``'rshift'``: rotation and shifts
+  - ``'rscale'``: rotation and scale
+  - ``'general'``: shift, rotation, and scale
+
+  The default value is "rshift".
+
+  .. note::
+      Mathematically, alignment of images observed in different tangent planes
+      requires ``fitgeometry='general'`` in order to fit source catalogs
+      in the different images even if mis-alignment is caused only by a shift
+      or rotation in the tangent plane of one of the images.
+
+      However, under certain circumstances, such as small alignment errors or
+      minimal dithering during observations that keep tangent planes of the
+      images to be aligned almost parallel, then it may be more robust to
+      use a ``fitgeometry`` setting with fewer degrees of freedom such as
+      ``'rshift'``, especially for "ill-conditioned" source catalogs such as
+      catalogs with very few sources, or large errors in source positions, or
+      sources placed along a line or bunched in a corner of the image (not
+      spread across/covering the entire image).
+
+* ``nclip``: A non-negative integer number of clipping iterations
+  to use in the fit (Default = 3).
+
+* ``sigma``: A positive `float` indicating the clipping limit, in sigma units,
+  used when performing fit (Default=3.0).
+
+**Absolute Astrometric fitting parameters:**
+
+Parameters used for absolute astrometry to a reference catalog.
+
+* ``abs_refcat``: String indicating what astrometric catalog should be used.
+  Currently supported options are (Default='GAIADR3'): ``'GAIADR1'``, ``'GAIADR2'``,
+  or ``'GAIADR3'``.
+
+  .. note::
+    If `None` or an empty string is passed in, `TweakRegStep` will
+    use the default catalog as set by `tweakreg_step.DEFAULT_ABS_REFCAT`.
+
+* ``abs_minobj``: A positive `int` indicating minimum number of objects
+  acceptable for matching. (Default=15).
+
+* ``abs_searchrad``: A `float` indicating the search radius in arcsec for
+  a match. It is recommended that a value larger than ``searchrad`` be used for
+  this parameter (e.g. 3 times larger) (Default=6.0).
+
+* ``abs_use2dhist``: A boolean indicating whether to use 2D histogram to find
+  initial offset. It is strongly recommended setting this parameter to `True`.
+  Otherwise the initial guess for the offsets will be set to zero
+  (Default=True).
+
+* ``abs_separation``: Minimum object separation in arcsec. It is recommended
+  that a value smaller than ``separation`` be used for this parameter
+  (e.g. 10 times smaller) (Default=0.1).
+
+* ``abs_tolerance``: Matching tolerance for ``xyxymatch`` in arcsec (Default=0.7).
+
+* ``abs_fitgeometry``: A `str` value indicating the type of affine
+  transformation to be considered when fitting catalogs. Allowed values:
+
+  - ``'shift'``: x/y shifts only
+  - ``'rshift'``: rotation and shifts
+  - ``'rscale'``: rotation and scale
+  - ``'general'``: shift, rotation, and scale
+
+  The default value is "rshift". Note that the same conditions/restrictions
+  that apply to ``fitgeometry`` also apply to ``abs_fitgeometry``.
+
+* ``abs_nclip``: A non-negative integer number of clipping iterations
+  to use in the fit (Default = 3).
+
+* ``abs_sigma``: A positive `float` indicating the clipping limit, in sigma
+  units, used when performing fit (Default=3.0).
+
+* ``save_abs_catalog``: A boolean specifying whether or not to write out the
+  astrometric catalog used for the fit as a separate product (Default=False).
+
+
+Further Documentation
+---------------------
+The underlying algorithms as well as formats of source catalogs are described
+in more detail at
+
+https://tweakwcs.readthedocs.io/en/latest/

docs/roman/tweakreg/astrometric_utils.rst

@@ -0,0 +1,13 @@
+=================
+astrometric_utils
+=================
+
+The ``astrometric_utils`` module provides functions for generating
+astrometric catalogs of sources for the field-of-view covered by a
+set of images.
+
+.. currentmodule:: romancal.tweakreg.astrometric_utils
+
+.. automodule:: romancal.tweakreg.astrometric_utils
+   :members:
+   :undoc-members:

docs/roman/tweakreg/index.rst

@@ -0,0 +1,23 @@
+.. _tweakreg_step:
+
+========
+TweakReg
+========
+
+.. moduleauthor:: Mihai Cara <[email protected]>
+
+.. toctree::
+   :maxdepth: 2
+
+   README.rst
+   tweakreg_examples.rst
+
+**Also See:**
+
+.. toctree::
+   :maxdepth: 2
+
+   tweakreg_step
+   astrometric_utils
+
+.. automodapi:: romancal.tweakreg

docs/roman/tweakreg/tweakreg_examples.rst

@@ -0,0 +1,84 @@
+Examples
+========
+In the examples below, `img` is either a string with the filename of a Roman `ASDF` file
+or a Roman datamodel `ImageModel`.
+
+1. To run TweakReg on a python session one image file
+   (i.e. one Roman's SCA) with the default parameters:
+
+.. doctest-skip::
+
+        >>> from romancal.tweakreg.tweakreg_step import TweakRegStep
+        >>> step = TweakRegStep()
+        >>> step.process([img])
+
+2. To run TweakReg on a Roman's exposure with default astrometric parameters and save
+   the absolute catalog data:
+
+.. doctest-skip::
+
+        >>> from romancal.tweakreg.tweakreg_step import TweakRegStep
+        >>> step = TweakRegStep()
+        >>> step.save_abs_catalog = True # save the catalog data used for absolute astrometry
+        >>> step.abs_refcat = 'GAIADR3' # use Gaia DR3 for absolute astrometry
+        >>> step.catalog_path = '/path/for/the/abs/catalog' # save the Gaia catalog to this path
+        >>> step.process([img])
+
+3. To run TweakReg using a custom source catalog with the default parameters:
+
+   - make sure the source catalog is in one of the supported formats. The file content
+     below is an example of a valid catalog (`ascii.csv` format):
+
+.. doctest-skip::
+
+        $ cat ref_catalog_1
+
+        x,y
+        846.1321662446178,945.839358133909
+        664.7073537074112,1028.4613139252003
+        1036.160742774408,642.3379043578552
+        935.8827367579428,594.1745467413945
+        1491.9672737821606,1037.4723609624757
+        735.1256651803337,1410.2791591559157
+        1358.2876707625007,651.7112260833995
+        526.4715950130742,751.745104066621
+        1545.082698426152,703.601696337681
+        374.9609365496525,972.6561578187437
+        1110.3498547121228,1644.2214966576498
+        341.18333252240654,891.4733849441861
+        820.0520846885105,312.0088351823117
+        567.7054174813052,386.8883078361564
+        1447.356249085851,1620.3390168916592
+        1400.4271386280673,1674.3765672924937
+        1681.9744852889235,571.6748779060324
+        959.7317254404431,197.8757865066898
+        1806.3360866990297,769.0603031839573
+        487.1560001146406,257.30706691141086
+        1048.7910126076483,85.36675265982751
+        1075.508595999755,29.085099663125334
+
+   - create `catfile` containing the filename of the input Roman datamodel and
+     its corresponding catalog, one per line, as below
+
+.. doctest-skip::
+
+        $ cat /path/to/catfile/catfilename
+
+        img1 ref_catalog_1
+        img2 ref_catalog_2
+        img3 ref_catalog_3
+
+    The content of `catfile` will allow TweakReg to assign the custom catalog to the
+    correct input Roman datamodel. In the example above, source catalog
+    `ref_catalog_1` will be assign to `img1`, and so on.
+
+    Now we can execute the following:
+
+.. doctest-skip::
+
+        >>> from romancal.tweakreg.tweakreg_step import TweakRegStep
+        >>> step = TweakRegStep()
+        >>> step.use_custom_catalogs = True # use custom catalogs
+        >>> step.catalog_format = "ascii.ecsv" # custom catalogs format
+        >>> step.catfile = '/path/to/catfile/catfilename' # path to datamodel:catalog mapping
+        >>> step.process([img])

docs/roman/tweakreg/tweakreg_step.rst

@@ -0,0 +1,13 @@
+=============
+tweakreg_step
+=============
+
+The ``tweakreg_step`` function (class name ``TweakRegStep``)
+is the top-level function used to call the "tweakreg" operation
+from the Roman Calibration pipeline.
+
+.. currentmodule:: romancal.tweakreg.tweakreg_step
+
+.. automodule:: romancal.tweakreg.tweakreg_step
+   :members:
+   :undoc-members:

romancal/tweakreg/astrometric_utils.py

@@ -11,6 +11,9 @@
 ASTROMETRIC_CAT_ENVVAR = "ASTROMETRIC_CATALOG_URL"
 DEF_CAT_URL = "http://gsss.stsci.edu/webservices"
 
+# VO request timeout (in seconds)
+TIMEOUT = 30.0
+
 if ASTROMETRIC_CAT_ENVVAR in os.environ:
     SERVICELOCATION = os.environ[ASTROMETRIC_CAT_ENVVAR]
 else:
@@ -184,7 +187,7 @@ def get_catalog(ra, dec, sr=0.1, catalog="GAIADR3"):
 
     spec = spec_str.format(ra, dec, sr, fmt, catalog)
     service_url = f"{SERVICELOCATION}/{service_type}?{spec}"
-    rawcat = requests.get(service_url, headers=headers)
+    rawcat = requests.get(service_url, headers=headers, timeout=TIMEOUT)
     r_contents = rawcat.content.decode()  # convert from bytes to a String
     rstr = r_contents.split("\r\n")
     # remove initial line describing the number of sources returned