Add apt parser sub_package

README

@@ -28,6 +28,10 @@ miri.datamodels
    .sim    - Data models supporting MIRI simulators
 See http://miri.ster.kuleuven.be/pub/Public/MIRISim_Public/miri_datamodels.pdf
 
+miri.apt_parser
+             - A tool to read APT XML files.
+    .mirisim - Process them via the MIRI simulator
+
 See also the following Jupyter notebooks:
 
 https://github.com/JWST-MIRI/MiriTE/blob/master/notebooks/How_to_use_the_data_models_to_obtain_calibration_reference_files.ipynb

apt_parser/README.adoc

@@ -0,0 +1,244 @@
+= APT_parser Documentation
+:author: Christophe Cossou
+:revnumber: 1.0
+:revdate: {docdate}
+:doctype: article
+:encoding: utf-8
+:lang: en
+:numbered:
+:toc: left
+:toclevels: 4
+:source-language: python
+
+== Purpose
+Parse .aptx or XML relative to APT files. Will return a list of simulations that these files contains.
+
+Only works for MIRI observations.
+
+Version 0.2.0 supports APT 27.1
+Version 0.3.0 add support for MIRISim simulation generation (IMA/MRS)
+Version 0.4.0 add support for LRS MIRISim simulation generation
+
+NOTE: There is no installation procedure as it is now a sub-package of MiriTE.
+
+== Tests
+To launch the unit test suite, simply run:
+[source, bash]
+----
+pytest
+----
+
+However, in the case you installed a *develop* version of the package, run:
+[source, bash]
+----
+python -m pytest
+----
+This is due to the *mirisim* submodule that cannot be imported in tests without forcing the local package. Or, if that doesn't work, install the package before running the tests.
+
+== How to parse an APT file
+[source,python]
+----
+import miri.apt_parser
+import logging
+
+# Only if using as a standalone code
+miri.apt_parser.init_log()
+LOG = logging.getLogger('parse_apt')
+
+simulations = miri.apt_parser.parse_apt("1027.aptx")
+----
+
+== Constants
+
+* *MIRI_CAR* : Dictionnary where each key is the STScI proposal number and the value is the corresponding MIRI CAR number
+* *MIRI_STSCI_ID* The opposite of *MIRI_CAR* dictionnary
+
+== Functions
+=== analyse_apt_list
+
+Given a list of APT files, will run parse_apt on each of them, then count_datavolume on each simulation list.
+
+.Will return a tuple of 3 dict. For each dict, a key represent a different APT file. The values are in that order:
+. list of simulation parameter for each APT
+. list of memory prediction for each APT (in GB)
+. list of computation time prediction for each APT (in hours)
+
+.Parameters:
+[cols="1, 1, 4",options="header",]
+|====
+|Parameter |Type or Values |Description
+| files | list(str) |List of APT filenames (relative or absolute path also works)
+|====
+
+[source,python]
+----
+import os
+import glob
+import miri.apt_parser
+
+folder = "apt"
+files = glob.glob(os.path.join(folder, "*.aptx"))
+(APT_sim, mem_volume, time_volume) = miri.apt_parser.analyse_apt_list(files)
+----
+
+=== count_datavolume
+
+Will extract from the given input the amount of time (in hours)
+and the memory (in GB) you need to process each simulation through the JWST pipeline
+
+.Parameters:
+[cols="1, 1, 4",options="header",]
+|====
+|Parameter |Type or Values |Description
+| sim_dict | dict | key represent a different APT file. Value are output of *parse_apt* function
+|====
+
+[source,python]
+----
+import miri.apt_parser
+(mem_volume, time_volume) = miri.apt_parser.count_datavolume(APT_sim)
+----
+
+=== init_log
+
+Init logging configuration file. Must be used before any other
+package you could think of and that might use logging as well.
+If it doesn't work the way you expect, try inverting the imports
+to see if it changes.
+
+Logs will be displayed on screen, but also stored in a file. This function
+has as a parameter the filename, and loglevel for the file, or screen output.
+
+You can also set a dictionnary of extra parameters you want to add in your logging configDict.
+
+
+.Parameters:
+[cols="1, 1, 4",options="header",]
+|====
+|Parameter |Type or Values |Description
+| log | str | filename where to store logs. By default "pipeline.log"
+| stdout_loglevel | ERROR, WARNING, INFO, DEBUG |log level for standard output (default: INFO)
+| file_loglevel | ERROR, WARNING, INFO, DEBUG |log level for log file (default: DEBUG)
+|extra_config | dict | [optional] Set of extra properties to be added to the dict_config for logging
+|====
+
+[source,python]
+----
+import miri.apt_parser
+miri.apt_parser.init_log(log="parse_apt.log", stdout_loglevel="INFO"
+                    , file_loglevel="DEBUG", extra_config=None)
+----
+
+
+Example of extra config dict:
+[source,python]
+----
+extra_config = {"loggers":
+            {
+
+                "paramiko":
+                {
+                        "level": "WARNING",
+                    },
+
+                "matplotlib":
+                {
+                        "level": "WARNING",
+                    },
+
+                "astropy":
+                {
+                        "level": "WARNING",
+                    },
+            },}
+----
+
+NOTE: The architecture of the *extra_config* directory should match the original configDict.
+
+=== parse_apt
+Will parse an APT file given its filename (can be either .aptx or .apt/XML)
+
+The output will be a list of individual simulations for that APT file.
+
+.Parameters:
+[cols="1, 1, 4",options="header",]
+|====
+|Parameter |Type or Values |Description
+|filename | str | Filename of the input APT file (can be .aptx or XML)
+|====
+
+[source,python]
+----
+import miri.apt_parser
+
+simulations = miri.apt_parser.parse_apt(filename)
+----
+
+== MIRISim support
+Since v0.3.0, apt_parser can generate MIRISim parameters from the APT file.
+
+=== Only retrieve simulation parameters
+If you just want the list of simulation parameters without running anything, do:
+[source]
+----
+import miri.apt_parser
+import mirisim.apt
+miri.apt_parser.init_log()
+
+from mirisim import config_parser as parser  # SimConfig, SimulatorConfig, SceneConfig
+
+observations = miri.apt_parser.parse_apt("1232.aptx")
+
+simulations = mirisim.apt.get_simulations(observations)
+----
+
+=== One scene to sim them all
+
+Here is an example on how to run simulation directly from the APT:
+[source]
+----
+import miri.apt_parser
+import mirisim.apt
+miri.apt_parser.init_log()
+
+from mirisim import config_parser as parser  # SimConfig, SimulatorConfig, SceneConfig
+
+# See MIRISim documentation for details to create a scene
+scene_config = parser.SceneConfig.from_default()
+
+observations = miri.apt_parser.parse_apt("1232.aptx")
+
+mirisim.apt.run(observations, scene_config)
+----
+
+.Optional parameters:
+* *dryrun*: If `True`, will create folders with simulation `.ini` files without running the simulation
+* *simulator*: Provide a custom *SimulatorConfig* object in case you don't want default values
+
+=== One scene per target
+In the APT file, you have a list of targets, for instance:
+
+* `1 NGC-188-FTS107-1`
+* `2 NGC-188-FTS107-2`
+
+You then have to provide one SceneConfig object per target:
+[source]
+----
+import miri.apt_parser
+import mirisim.apt
+apt_parser.init_log()
+
+from mirisim import config_parser as parser  # SimConfig, SimulatorConfig, SceneConfig
+
+# See MIRISim documentation for details to create a scene
+scene_config = parser.SceneConfig.from_default()
+
+scene = {
+'1 NGC-188-FTS107-1':scene_config,
+'2 NGC-188-FTS107-2':scene_config
+}
+
+observations = miri.apt_parser.parse_apt("1232.aptx")
+
+mirisim.apt.run(observations, scene)
+----

apt_parser/__init__.py

@@ -0,0 +1,6 @@
+__version__ = "0.6.0"
+
+from . import constants
+from .library import count_datavolume, init_log, parse_apt, analyse_apt_list
+from . import templates
+from . import utils

apt_parser/constants.py

@@ -0,0 +1,115 @@
+# We store the URI to avoid specifying it everywhere.
+ns = {'apt': 'http://www.stsci.edu/JWST/APT',
+      "mmrscge": "http://www.stsci.edu/JWST/APT/Template/MiriMRSCrossGratingEngineering",
+      "nsmsasd": "http://www.stsci.edu/JWST/APT/Template/NirspecMSAShortDetect",
+      "nid": "http://www.stsci.edu/JWST/APT/Template/NirissDark",
+      "ncipri": "http://www.stsci.edu/JWST/APT/Template/NircamIprImaging",
+      "nif": "http://www.stsci.edu/JWST/APT/Template/NirissFocus",
+      "wfscfp": "http://www.stsci.edu/JWST/APT/Template/WfscFinePhasing",
+      "nii": "http://www.stsci.edu/JWST/APT/Template/NirissImaging",
+      "mmimf": "http://www.stsci.edu/JWST/APT/Template/MiriMimf",
+      "mlrs": "http://www.stsci.edu/JWST/APT/Template/MiriLRS",
+      "mc": "http://www.stsci.edu/JWST/APT/Template/MiriCoron",
+      "md": "http://www.stsci.edu/JWST/APT/Template/MiriDark",
+      "nsfgwt": "http://www.stsci.edu/JWST/APT/Template/NirspecFilterGratingWheelTest",
+      "wfscga": "http://www.stsci.edu/JWST/APT/Template/WfscGlobalAlignment",
+      "nsil": "http://www.stsci.edu/JWST/APT/Template/NirspecInternalLamp",
+      "idfu": "http://www.stsci.edu/JWST/APT/Template/IsimDictionaryFileUpdate",
+      "mi": "http://www.stsci.edu/JWST/APT/Template/MiriImaging",
+      "nsimg": "http://www.stsci.edu/JWST/APT/Template/NirspecImaging",
+      "niami": "http://www.stsci.edu/JWST/APT/Template/NirissAmi",
+      "niwfss": "http://www.stsci.edu/JWST/APT/Template/NirissWfss",
+      "ncif": "http://www.stsci.edu/JWST/APT/Template/NircamInternalFlat",
+      "nsfss": "http://www.stsci.edu/JWST/APT/Template/NirspecFixedSlitSpectroscopy",
+      "fgsif": "http://www.stsci.edu/JWST/APT/Template/FgsInternalFlat",
+      "ncef": "http://www.stsci.edu/JWST/APT/Template/NircamExternalFlat",
+      "ncei": "http://www.stsci.edu/JWST/APT/Template/NircamEngineeringImaging",
+      "niec": "http://www.stsci.edu/JWST/APT/Template/NirissExternalCalibration",
+      "nsmsam": "http://www.stsci.edu/JWST/APT/Template/NirspecMSAMasking",
+      "niif": "http://www.stsci.edu/JWST/APT/Template/NirissInternalFlat",
+      "ncpili": "http://www.stsci.edu/JWST/APT/Template/NircamPilImaging",
+      "nsmsaa": "http://www.stsci.edu/JWST/APT/Template/NirspecMSAAnneal",
+      "mcpc": "http://www.stsci.edu/JWST/APT/Template/MiriCpc",
+      "fgsec": "http://www.stsci.edu/JWST/APT/Template/FgsExternalCalibration",
+      "nsd": "http://www.stsci.edu/JWST/APT/Template/NirspecDark",
+      "nsf": "http://www.stsci.edu/JWST/APT/Template/NirspecFocus",
+      "ns": "http://www.stsci.edu/JWST/APT/Instrument/Nirspec",
+      "nisoss": "http://www.stsci.edu/JWST/APT/Template/NirissSoss",
+      "nsmimf": "http://www.stsci.edu/JWST/APT/Template/NirspecMimf",
+      "ncts": "http://www.stsci.edu/JWST/APT/Template/NircamTimeSeries",
+      "ncd": "http://www.stsci.edu/JWST/APT/Template/NircamDark",
+      "mef": "http://www.stsci.edu/JWST/APT/Template/MiriExternalFlat",
+      "ncc": "http://www.stsci.edu/JWST/APT/Template/NircamCoron",
+      "ncf": "http://www.stsci.edu/JWST/APT/Template/NircamFocus",
+      "mmrs": "http://www.stsci.edu/JWST/APT/Template/MiriMRS",
+      "nci": "http://www.stsci.edu/JWST/APT/Template/NircamImaging",
+      "sk": "http://www.stsci.edu/JWST/APT/Template/StationKeeping",
+      "wfscc": "http://www.stsci.edu/JWST/APT/Template/WfscCommissioning",
+      "iat": "http://www.stsci.edu/JWST/APT/Template/IsimAsicTuning",
+      "sr": "http://www.stsci.edu/JWST/APT/Template/SafeModeRecovery",
+      "rtc": "http://www.stsci.edu/JWST/APT/Template/RealtimeCommanding",
+      "nsfr": "http://www.stsci.edu/JWST/APT/Template/NirspecFocusReference",
+      "ncw": "http://www.stsci.edu/JWST/APT/Template/NircamWheelExercise",
+      "nsbots": "http://www.stsci.edu/JWST/APT/Template/NirspecBrightObjectTimeSeries",
+      "mann": "http://www.stsci.edu/JWST/APT/Template/MiriAnneal",
+      "nsmos": "http://www.stsci.edu/JWST/APT/Template/NirspecMOS",
+      "ncgts": "http://www.stsci.edu/JWST/APT/Template/NircamGrismTimeSeries",
+      "wfsccp": "http://www.stsci.edu/JWST/APT/Template/WfscCoarsePhasing",
+      "nsifus": "http://www.stsci.edu/JWST/APT/Template/NirspecIFUSpectroscopy",
+      "msa": "http://www.stsci.edu/JWST/APT/Template/NirspecMSA",
+      "ncwfss": "http://www.stsci.edu/JWST/APT/Template/NircamWfss",
+      "nifs": "http://www.stsci.edu/JWST/APT/Template/NirissFlatSuite",
+      "fgsf": "http://www.stsci.edu/JWST/APT/Template/FgsFocus",
+      "po": "http://www.stsci.edu/JWST/APT/Template/PointingOnly"}
+
+# Given the STScI proposal number, return the corresponding MIRI CAR number
+MIRI_CAR = {"1024": "MIRI-007", "1027": "MIRI-011", "1028": "MIRI-012",
+            "1029": "MIRI-013", "1030": "MIRI-014", "1031": "MIRI-015",
+            "1032": "MIRI-016", "1033": "MIRI-017", "1034": "MIRI-018",
+            "1037": "MIRI-050", "1038": "MIRI-051", "1039": "MIRI-052",
+            "1040": "MIRI-053", "1042": "MIRI-055", "1043": "MIRI-056",
+            "1045": "MIRI-058", "1046": "MIRI-060", "1047": "MIRI-061",
+            "1048": "MIRI-062", "1049": "MIRI-063", "1050": "MIRI-064",
+            "1051": "MIRI-072", "1052": "MIRI-073", "1053": "MIRI-074",
+            "1054": "MIRI-075", "1171": "OTE-34", "1172": "OTE-35",
+            "1173": "MIRI-001", "1259": "MIRI-076", "1261": "MIRI-077",
+            "1406": "MIRI-078", "1011": "MIRI-002", "1012": "MIRI-004",
+            "1023": "MIRI-005"}
+
+# Invert previous dict
+MIRI_STSCI_ID = {v: k for k, v in MIRI_CAR.items()}
+
+# For some dither patterns, number of points is not necessarily
+# specified in the APT file. We then encode for these dither
+# pattern how many points there is
+DITHER_POINTS = {"4-Point-Sets": 4, "2-Point": 2, "REULEAUX": 12,  # Imager
+                 "ALONG_SLIT_NOD": 2, "1-PIXEL_SLIT_SCAN": 45, "2-PIXEL_SLIT_SCAN": 23, "7-PIXEL_SLIT_SCAN": 7,  # LRS
+                 "7x3_PIXEL_MAP_CENTER": 21, "7x3_PIXEL_MAP_NOD1": 21, "7x3_PIXEL_MAP_NOD2": 21,  # LRS
+                 "SHORT_CROSS_SCAN_NOD_1": 9, "SHORT_CROSS_SCAN_NOD_2": 9, "INTRAPIXEL_SLIT_SCAN_NOD_1": 11,
+                 "INTRAPIXEL_SLIT_SCAN_NOD_2": 11,  # LRS
+                 "INTRAPIXEL_SLIT_SCAN_CENTER": 11,
+                 # LRS. A lot are missing but in the .xsd spec, only slit nod and mapping is referenced
+                 "4-POINT-MIRI-F770W-WITH-NIRCam": 4, "4-POINT-MIRI-F1000W-WITH-NIRCam": 4,  # Dual mode with NIRCam
+                 "4-POINT-MIRI-F1800W-WITH-NIRCam": 4, "4-POINT-MIRI-F2550W-WITH-NIRCam": 4,  # Dual mode with NIRCam
+                 "4-Point": 4,  # DUAL mode Imager/MRS
+                 "MRS 2-Point": 2, "MRS 4-Point": 4,  # MRS
+                 "5-POINT-SMALL-GRID": 5, "9-POINT-SMALL-GRID": 9,  # Coronograph
+                 }
+
+SUBARRAY_PIX = {
+    "BOTH": 1032 * 1024,  # Artificial subarray for MRS mem and time prediction
+    "FULL": 1032 * 1024,
+    "BRIGHTSKY": 512 * 512,
+    "SUB256": 256 * 256,
+    "SUB128": 136 * 128,
+    "SUB64": 72 * 64,
+    "SLITLESSPRISM": 72 * 416,
+    "MASK1065": 288 * 224,
+    "MASK1140": 288 * 224,
+    "MASK1550": 288 * 224,
+    "FOUR_QPM": 288 * 224,
+    "MASKLYOT": 320 * 304,
+    "LYOT": 320 * 304,
+}
+
+DISPERSER = {"SHORT(A)": "SHORT", "MEDIUM(B)": "MEDIUM", "LONG(C)": "LONG"}