Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace ModelContainer with ModelLibrary #1241

Merged
merged 61 commits into from
Jul 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
f0ee6f1
Revert "Remove unnecessary global variable and unit tests."
braingram Jul 18, 2024
91a374a
update flux to use ModelLibrary
braingram May 16, 2024
a100cd2
add WIP library
braingram May 16, 2024
cb3b882
remove ModelContainer from elp
braingram May 16, 2024
d372e3e
remove ModelContainer from highlevel_pipeline
braingram May 16, 2024
2ed7e92
update skymatch to use ModelLibrary
braingram May 16, 2024
ea149d1
WIP update to outlier_detection
braingram May 17, 2024
087a447
update resample to use ModelLibrary
braingram May 17, 2024
5f1ee5a
TMP: set stpipe to fork
braingram May 17, 2024
f418bce
add table_name try to fix hlp test
braingram May 17, 2024
7b3fae3
fix outlier median calc to use weights
braingram May 17, 2024
5e20b8a
allow ModelLibrary to run on_disk for hlp, reorganize outlier detection
braingram May 20, 2024
9337996
update tweakreg to use ModelLibrary
braingram May 22, 2024
db1ff48
remove ModelContainer
braingram May 22, 2024
88cf87d
skip questionable outlier detection test
braingram May 22, 2024
f9afd21
undo outlier detection resample single usage
braingram May 22, 2024
6453b7f
fix single model filename usage for tweakreg
braingram May 22, 2024
ba7592c
support overwrite=True in save
braingram May 23, 2024
098f57b
copy save from model container
braingram May 23, 2024
5fbf49f
replace container with library in docs
braingram May 23, 2024
06e91d9
start of index/map_function
braingram May 31, 2024
9b7a06c
port over jwst library unit tests
braingram May 31, 2024
a3e0ede
switch ModelStore to use path instead of directory name
braingram May 31, 2024
8130ac0
allow ModelLibrary(asn_dict)
braingram May 31, 2024
2bae2bc
cleanup ModelLibrary attributes
braingram Jun 10, 2024
bf4dcd4
use shelve, change __exit__ error handling
braingram Jun 10, 2024
9dd0224
update outlier_detection to use shelve
braingram Jun 10, 2024
6935a04
update flux to use shelve
braingram Jun 10, 2024
b2be289
update resample to use shelve
braingram Jun 10, 2024
4bb051a
update tweakreg to use shelve
braingram Jun 10, 2024
22c7135
update skymatch to use shelve
braingram Jun 10, 2024
e195e20
fix library exception test
braingram Jun 10, 2024
f3549f5
remove ModelLibrary.__getitem__ usage
braingram Jun 10, 2024
154de59
remove _ModelStore
braingram Jun 11, 2024
2450ac0
minor docs
braingram Jun 11, 2024
df502cf
remove _is_asn in skymatch
braingram Jun 11, 2024
ffa61d0
remove ModelLibrary.asn_table_name
braingram Jun 12, 2024
ff01a6e
remove meta.asn.exptype usage
braingram Jun 12, 2024
994eae6
flush out get_crds_parameters
braingram Jun 12, 2024
4392df7
use library from stpipe
braingram Jun 17, 2024
af15e74
fix outlier_detection tests
braingram Jun 17, 2024
9c20805
fix flux tests
braingram Jun 17, 2024
0c467f8
fix tweakreg tests
braingram Jun 17, 2024
64acdae
rebase
braingram Jul 11, 2024
9026f89
use library._save
braingram Jul 11, 2024
3b99769
remove use of private _models
braingram Jul 11, 2024
e662456
return non-shelved model in tweakreg
braingram Jul 11, 2024
a22bf23
fixing another borrow
braingram Jul 11, 2024
883d6a3
add changelog
braingram Jul 11, 2024
137eefe
fix bug introduced in rebase
braingram Jul 12, 2024
16532dc
clean up comments
braingram Jul 12, 2024
8221e7a
modify asn metadata setting in models
braingram Jul 12, 2024
acd1b3d
fix asn_pool->pool_name
braingram Jul 12, 2024
c9d162a
clean up comments
braingram Jul 14, 2024
42dba28
switch to stpipe main
braingram Jul 16, 2024
00e2eaf
fix skipped test
braingram Jul 16, 2024
f7fb783
fix docstring
braingram Jul 17, 2024
38996e4
use filenames instead of meta.filename for resample members
braingram Jul 17, 2024
d896b4c
fix ResampleData docstring
braingram Jul 18, 2024
e66656c
fix error message
braingram Jul 18, 2024
b5349d9
redo #1314
braingram Jul 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGES.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ general

- replace usages of ``copy_arrays`` with ``memmap`` [#1316]

- Replace ModelContainer with ModelLibrary [#1241]

source_catalog
--------------
- Add PSF photometry capability. [#1243]
Expand Down
8 changes: 4 additions & 4 deletions docs/roman/data_products/product_types.rst
Original file line number Diff line number Diff line change
Expand Up @@ -69,13 +69,13 @@ the user is running the pipeline. The input for each optional step is the output
+===================================================+=================+==============================+==================+=====================+=======================================+
| | | asn | | | |
+---------------------------------------------------+-----------------+------------------------------+------------------+---------------------+---------------------------------------+
| :ref:`flux <flux_step>` | asn | flux (opt) | ModelContainer | MJy/sr | A list of _cal files |
| :ref:`flux <flux_step>` | asn | flux (opt) | ModelLibrary | MJy/sr | A list of _cal files |
+---------------------------------------------------+-----------------+------------------------------+------------------+---------------------+---------------------------------------+
| :ref:`sky_match <skymatch_step>` | asn | skymatch (opt) | ModelContainer | MJy/sr | A list of _cal files |
| :ref:`sky_match <skymatch_step>` | asn | skymatch (opt) | ModelLibrary | MJy/sr | A list of _cal files |
+---------------------------------------------------+-----------------+------------------------------+------------------+---------------------+---------------------------------------+
| :ref:`outlier_detection <outlier_detection_step>` | | outlier_detection_step (opt) | ModelContainer | MJy/sr | A list of _cal files |
| :ref:`outlier_detection <outlier_detection_step>` | | outlier_detection_step (opt) | ModelLibrary | MJy/sr | A list of _cal files |
+---------------------------------------------------+-----------------+------------------------------+------------------+---------------------+---------------------------------------+
| :ref:`resample <resample_step>` | | resamplestep (opt) | ModelContainer | MJy/sr | A list of _cal files |
| :ref:`resample <resample_step>` | | resamplestep (opt) | ModelLibrary | MJy/sr | A list of _cal files |
+---------------------------------------------------+-----------------+------------------------------+------------------+---------------------+---------------------------------------+
| | | i2d | MosaicModel | MJy/sr | A 2D resampled image |
+---------------------------------------------------+-----------------+------------------------------+------------------+---------------------+---------------------------------------+
9 changes: 0 additions & 9 deletions docs/roman/datamodels/container.rst

This file was deleted.

2 changes: 1 addition & 1 deletion docs/roman/datamodels/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,4 @@
models.rst
metadata.rst
datamodels_asdf.rst
container.rst
library.rst
9 changes: 9 additions & 0 deletions docs/roman/datamodels/library.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
.. _library:

==============
ModelLibrary
==============

.. automodule:: romancal.datamodels.library
:members:
:undoc-members:
2 changes: 1 addition & 1 deletion docs/roman/flux/main.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The ``flux`` step can take:

* a single 2D input image (in the format of either a string with the full
path and filename of an ASDF file or a Roman
Datamodel/:py:class:`~romancal.datamodels.container.ModelContainer`);
Datamodel/:py:class:`~romancal.datamodels.library.ModelLibrary`);
* an association table (in JSON format).


Expand Down
8 changes: 4 additions & 4 deletions docs/roman/outlier_detection/outlier_detection.rst
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ Specifically, this routine performs the following operations:
resampling.
* Resampled images will be written out to disk with suffix `_outlier_i2d` by default.
* **If resampling is turned off** through the use of the ``resample_data`` parameter,
a copy of the unrectified input images (as a ModelContainer)
a copy of the unrectified input images (as a ModelLibrary)
will be used for subsequent processing.

#. Create a median image from all grouped observation mosaics.
Expand Down Expand Up @@ -115,7 +115,7 @@ The outlier detection algorithm can end up using massive amounts of memory
depending on the number of inputs, the size of each input, and the size of the
final output product. Specifically,

#. The input :py:class:`~romancal.datamodels.ModelContainer` all input exposures would
#. The input :py:class:`~romancal.datamodels.ModelLibrary` all input exposures would
have been kept open in memory to make processing more efficient.

#. The initial resample step creates an output product for EACH input that is the
Expand All @@ -137,9 +137,9 @@ with the use of the ``in_memory`` parameter. The full impact of this parameter
during processing includes:

#. The ``save_open`` parameter gets set to `False`
when opening the input :py:class:`~romancal.datamodels.container.ModelContainer`
when opening the input :py:class:`~romancal.datamodels.library.ModelLibrary`
object. This forces all input models in the input
:py:class:`~romancal.datamodels.container.ModelContainer` to get written out to disk.
:py:class:`~romancal.datamodels.library.ModelLibrary` to get written out to disk.
It then uses the filename of the input model during subsequent processing.

#. The ``in_memory`` parameter gets passed to the :py:class:`~romancal.resample.ResampleStep`
Expand Down
2 changes: 1 addition & 1 deletion docs/roman/outlier_detection/outlier_detection_step.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ and described in :ref:`outlier-detection-imaging`.
.. note::
Whether the data are being provided in an `association file`_ or as a list of ASDF filenames,
they must always be wrapped with a
:py:class:`~romancal.datamodels.container.ModelContainer`, which will handle and
:py:class:`~romancal.datamodels.library.ModelLibrary`, which will handle and
read in the input properly.

.. _association file: https://jwst-pipeline.readthedocs.io/en/latest/jwst/associations/asn_from_list.html
Expand Down
14 changes: 7 additions & 7 deletions docs/roman/outlier_detection/outlier_examples.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Examples
========
Whether the data are contained in a list of ASDF files or provided as an ASN file, the
`ModelContainer` class must be used to properly handle the data that will be used in
`ModelLibrary` class must be used to properly handle the data that will be used in
the outlier detection step.

1. To run the outlier detection step (with the default parameters) on a list of 2 ASDF
Expand All @@ -10,9 +10,9 @@ the outlier detection step.
.. code-block:: python

from romancal.outlier_detection import OutlierDetectionStep
from romancal.datamodels import ModelContainer
# read the file list into a ModelContainer object
mc = ModelContainer(["img_1.asdf", "img_2.asdf"])
from romancal.datamodels import ModelLibrary
# read the file list into a ModelLibrary object
mc = ModelLibrary(["img_1.asdf", "img_2.asdf"])
step = OutlierDetectionStep()
step.process(mc)

Expand Down Expand Up @@ -52,9 +52,9 @@ the outlier detection step.
.. code-block:: python

from romancal.outlier_detection import OutlierDetectionStep
from romancal.datamodels import ModelContainer
# read the file list into a ModelContainer object
mc = ModelContainer("asn_sample.json")
from romancal.datamodels import ModelLibrary
# read the file list into a ModelLibrary object
mc = ModelLibrary("asn_sample.json")
step = OutlierDetectionStep()
step.process(mc)

Expand Down
2 changes: 1 addition & 1 deletion docs/roman/resample/main.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ The ``resample`` step can take:

* a single 2D input image (in the format of either a string with the full
path and filename of an ASDF file or a Roman
Datamodel/:py:class:`~romancal.datamodels.container.ModelContainer`);
Datamodel/:py:class:`~romancal.datamodels.library.ModelLibrary`);
* an association table (in JSON format).

The parameters for the drizzle operation itself are set by
Expand Down
2 changes: 1 addition & 1 deletion docs/roman/tweakreg/README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ models to the custom catalog file name, the ``tweakreg_step`` also supports two
other ways of supplying custom source catalogs to the step:

1. Adding ``tweakreg_catalog`` attribute to the ``members`` of the input ASN
table - see `~roman.datamodels.ModelContainer` for more details.
table - see `~roman.datamodels.ModelLibrary` for more details.
Catalog file names are relative to ASN file path.

2. Providing a simple two-column text file, specified via step's parameter
Expand Down
2 changes: 1 addition & 1 deletion docs/roman/tweakreg/tweakreg_examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ or a Roman datamodel `ImageModel`.
.. note::
If the input is a single Roman ``DataModel``,
either ``step.call([img])`` or ``step.call(img)`` will work. For multiple elements as input,
they must be passed in as either a list or a ModelContainer.
they must be passed in as either a list or a ModelLibrary.

#. To run TweakReg in a Python session on an association file with the default parameters:

Expand Down
3 changes: 2 additions & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ dependencies = [
"scipy >=1.11",
"stcal>=1.7.0",
# "stcal @ git+https://github.com/spacetelescope/stcal.git@main",
"stpipe >=0.5.0",
#"stpipe >=0.5.0",
"stpipe @ git+https://github.com/spacetelescope/stpipe.git@main",
"tweakwcs >=0.8.6",
"spherical-geometry >= 1.2.22",
"stsci.imagestats >= 1.6.3",
Expand Down
4 changes: 2 additions & 2 deletions romancal/datamodels/__init__.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
from .container import ModelContainer
from .library import ModelLibrary

__all__ = ["ModelContainer"]
__all__ = ["ModelLibrary"]
Loading
Loading