Merge pull request #21 from KeckObservatory/dev

v1.4.3

docs/status.md

@@ -4,19 +4,21 @@ KPF has been commissioned on sky, and is available for use. Many aspects of the
 
 A detailed summary of the instrument status was presented at the September 2023 Keck Science Meeting. The slides from that presentation are available in [PDF format](Keck Science Meeting 2023 Breakout Session.pdf).
 
-<font color="red">**Important Notice**</font>: KPF will have a lengthy “servicing mission” during 24B to perform several upgrades.  This will involve warm up of the detectors.  The work is expected to commence around Oct 27 and be complete before Nov 24.
+<font color="red">**Important Notice**</font>: KPF will have a lengthy “servicing mission” during 25A to perform several upgrades.  This will involve warm up of the detectors.  The work is expected to be completed during the Keck I pier repair shutdown in March and April, so no additional off sky time will be needed.
 
 
 ### Subsystem Status
 
 This is an attempt to summarize the status of various sub-systems of the instrument.  Each sub-system name is color coded to indicate the status at a glance: <font color="green">green</font> means functioning normally, <font color="orange">orange</font> means mostly normal, but with some caveats or minor issues, and <font color="red">red</font> means the sub-system is compromised in some way.
 
+- **<font color="red">Detector Noise</font>**: After servicing mission 2 in November of 2024, additional pattern noise has been present on the detectors.  This severaly increases the effective read noise of the system and damages sensitivity.  The noise is 4-5 times the nominal value. This is expected to be addressed during servicing mission 3 in March of 2025.
+- **<font color="red">Detector Cooling Systems</font>**: The CCR currently cooling the green detector has been unstable resulting in the occasional temperature transient. These transients vary in magnitude from a few mK to 10s of K. This is expected to be addressed during servicing mission 3 in March of 2025. The red side is unaffected.
+- **<font color="green">Detector Errors</font>**: The red and green detectors suffer from occasional “start state errors” in which the affected detector remains in the start phase and does not produce a useful exposure. The observing scripts now detect this occurrence, abort the current exposure (with read out) and start a fresh exposure on both cameras. No action is necessary on the part of the observer.  This costs about a minute of time for this to happen, but the resulting data should be normal (unless another error occurs).  The occurrence rate for these problems is 0.34% on the green detector and 0.22% on the red, so around one in every 180 exposures is affected by one of the two detectors experiencing this error.
 - **<font color="orange">Tip Tilt Corrections</font>**: The tip tilt stage X-axis has degraded once again (as of 2024 Aug 22) and we are not making fast tip tilt corrections in X.  Troubleshooting is underway.
 - **<font color="green">Ca H&K Detector</font>**: The CA H&K detector is operational.
 - **<font color="green">Double Star Observations</font>**: Operational.
 - **<font color="green">Etalon</font>**: Operational and providing the expected flux.
 - **<font color="green">LFC</font>**: Is operating normally. 
-- **<font color="green">Detector Systems</font>**: The red and green detectors suffer from occasional “start state errors” in which the affected detector remains in the start phase and does not produce a useful exposure. The observing scripts now detect this occurrence, abort the current exposure (with read out) and start a fresh exposure on both cameras. No action is necessary on the part of the observer.  This costs about a minute of time for this to happen, but the resulting data should be normal (unless another error occurs).  The occurrence rate for these problems is 0.34% on the green detector and 0.22% on the red, so around one in every 180 exposures is affected by one of the two detectors experiencing this error.
 - **<font color="green">Simultaneous Calibration (SimulCal)</font>**: Simultaneous calibrations are supported.  Observers have the option of manually specifying the ND filters to balance the calibration flux or using the `AutoNDFilters` option in the OB to have an algorithm set the filters based on the KPF ETC, the target parameters in the OB, and a reference calibration brightness value.  See the [Nighttime Calibrations](nighttimecals.md) page for more info.
 - **<font color="orange">Exposure Meter Terminated Exposures</font>**: The control system supports exposure meter terminated exposures (`ExpMeterMode: control` in the OB), however we are still documenting performance on sky.
 - **<font color="orange">Nod to Sky Observations</font>**: For observations which need a sky measurement other than the built in sky fibers, nodding away to a sky position can be accomplished manually by running separate OBs for the target and sky and asking the OA to offset the telescope as appropriate.  We plan to build a separate Nod To Sky observing mode which will accomplish this within a single OB, but that is not yet ready.

docs/tiptiltinstructions.md

@@ -8,9 +8,9 @@
     - [GUI Layout](#gui-layout)
     - [Camera Controls Tab](#camera-controls-tab)
     - [Object Detection Tab](#object-detection-tab)
+    - [Sky Subtraction Tab](#sky-subtraction-tab)
     - [Offset Guiding Tab](#offset-guiding-tab)
     - [Settings Tab](#settings-tab)
-    - [Plot Time Spans Tab](#plot-time-spans-tab)
     - [Tip Tilt Control and Telemetry](#tip-tilt-control-and-telemetry)
     - [Image Display](#image-display)
     - [Instrument Status](#instrument-status)
@@ -104,6 +104,15 @@ This is where you can set the object detection and deblending parameters discuss
 
 You can set these values on the command line by modifying the relevant keywords: `kpgduide.OBJECT_INTENSITY`, `kpfguide.OBJECT_AREA`, and `kpfguide.OBJECT_DBCONT`.
 
+#### Sky Subtraction Tab
+
+![A screenshot of the Tip Tilt GUI's Sky Subtraction tab.](figures/TipTiltGUI_SkySubtraction.png)
+>  A screenshot of the Tip Tilt GUI's Sky Subtraction tab.
+
+Detection of faint targets when the gain is high and the FPS is low (of order 2 FPS or less) might be enhanced by subtracting the sky background. To do this, once the target has been acquired to the KPF PO, click the "Obtain Sky Frame" button.  The telescope will offset by the amounts specified in the Sky Offset East and North fields, take a set of images there, and use that for sky subtraction.  This will **not** necessarily increase the flux value, but may improve object detection because the background will be smoother and less noisy.
+
+The "Reset Sky Frame" button will reset the subtraction file to default (a bias frame).  Any time the camera gain or FPS are changed, the sky frame will be reset to default.
+
 #### Offset Guiding Tab
 
 Not currently implemented.
@@ -113,18 +122,13 @@ Not currently implemented.
 ![A screenshot of the Tip Tilt GUI's Settings tab.](figures/TipTiltGUI_Settings.png)
 >  A screenshot of the Tip Tilt GUI's Settings tab.
 
-The three options here control various settings for the `kpfguide` algorithms.
+The options here control various settings for the `kpfguide` algorithms.
 
 The "X Axis" and "Y Axis" settings allow you to bypass the tip tilt mirror and control the star position only using offloads to the telescope (conceptually similar to normal Magiq guiding). These values should be "Mirror" unless there is a problem with the tip tilt system.
 
 The "DAR" setting determines whether the target pixel for the star is modified to account for differential atmospheric refraction (DAR) between the guide wavelengths (950-1200nm) and the science wavelengths (centered on 550nm).  This should be set to "Yes" under normal observing conditions.
 
-#### Plot Time Spans Tab
-
-![A screenshot of the Tip Tilt GUI's Plot Time Spans tab.](figures/TipTiltGUI_PlotTimeSpans.png)
->  A screenshot of the Tip Tilt GUI's Plot Time Spans tab.
-
-These two pulldowns allow you to change the time span of the two plots in the [Tip Tilt Control and Telemetry](#tip-tilt-control-and-telemetry) region of the GUI.  They have no effect on the tip tilt performance, only the plots.
+The "Tip Tilt ROI Size" can be adjusted using the pulldown.  The default value of 128 pix is recommended, but in bad seeing conditions, going to larget frames may help, but processing the extra pixels might slow down the tip tilt calculations, so keep an eye on the "Tip Tilt FPS" value and make sure it does not stay below the camera FPS value.
 
 #### Tip Tilt Control and Telemetry
 

kpf/OB_GUI/KPF_OB_GUI.py

@@ -9,7 +9,6 @@
 import subprocess
 import yaml
 import datetime
-from copy import deepcopy
 from astropy.coordinates import SkyCoord
 
 import ktl                      # provided by kroot/ktl/keyword/python
@@ -966,6 +965,7 @@ def render_OB(self):
         self.TakeSimulCal.setText(f"{self.OB.SEQ_Observations1.get('TakeSimulCal')}")
         take_simulcal = self.OB.SEQ_Observations1.get('TakeSimulCal')
         auto_nd = self.OB.SEQ_Observations1.get('AutoNDFilters')
+        self.AutoNDFilters.setChecked(take_simulcal and auto_nd)
         self.CalND1Label.setEnabled(take_simulcal and not auto_nd)
         self.CalND2Label.setEnabled(take_simulcal and not auto_nd)
         self.CalND1.setEnabled(take_simulcal and not auto_nd)
@@ -1651,21 +1651,23 @@ def run_load_slewcalOB(self):
                 self.calOB = CalibrationOB({})
                 self.log.debug(f"  Opening: {fname}")
                 with open(fname, 'r') as f:
-                    contents = yaml.safe_load(f)
+                    OB = yaml.safe_load(f)
                 self.log.debug('  Read in YAML')
+                contents = OB.get('SEQ_Calibrations', [{}])[0]
                 self.log.debug(contents)
                 self.enable_dark_seq1_state_change(0)
                 self.enable_dark_seq2_state_change(0)
                 self.enable_cal_seq1_state_change(2)
-                self.update_calOB('cal1_CalSource', self.kpfconfig['SIMULCALSOURCE'].read())
+                self.update_calOB('cal1_CalSource', contents.get('CalSource', ''))
                 self.update_calOB('cal1_Object', contents.get('Object', 'slewcal'))
                 self.update_calOB('cal1_CalND1', contents.get('CalND1', 'OD 0.1'))
-                self.update_calOB('cal1_CalND2', contents.get('CalND1', 'OD 0.1'))
+                self.update_calOB('cal1_CalND2', contents.get('CalND2', 'OD 0.1'))
                 self.update_calOB('cal1_nExp', contents.get('nExp', 1))
                 self.update_calOB('cal1_ExpTime', contents.get('ExpTime', 0))
                 self.update_calOB('cal1_SSS_Science', contents.get('SSS_Science', True))
                 self.update_calOB('cal1_SSS_Sky', contents.get('SSS_Sky', True))
-                self.update_calOB('cal1_TakeSimulCal', contents.get('TimedShutter_SimulCal', True))
+                self.update_calOB('cal1_TakeSimulCal', contents.get('TakeSimulCal', True))
+                self.update_calOB('cal1_ExpMeterMode', 'off')
             except Exception as e:
                 self.log.warning('Unable to load slew cal data')
                 self.log.debug(e)

kpf/TT_GUI/TipTiltGUI.py

@@ -161,6 +161,8 @@ def __init__(self, log, dark=False, monitor=False, *args, **kwargs):
         self.TIPTILT_ROIDIM = kPyQt.kFactory(kpfguide['TIPTILT_ROIDIM'])
         self.PIX_TARGET = kPyQt.kFactory(kpfguide['PIX_TARGET'])
         self.SUB_HIGH = kPyQt.kFactory(kpfguide['SUB_HIGH'])
+        self.SENSORSETP = kPyQt.kFactory(kpfguide['SENSORSETP'])
+        self.SENSORTEMP = kPyQt.kFactory(kpfguide['SENSORTEMP'])
 
         kpffiu = ktl.cache('kpffiu')
         self.TTXSRV = kPyQt.kFactory(kpffiu['TTXSRV'])
@@ -232,8 +234,8 @@ def __init__(self, log, dark=False, monitor=False, *args, **kwargs):
         self.GuiderParameters = {}
         # Sky Subtraction
         self.default_sub_file = '/kroot/rel/default/data/kpfguide/kpfguide_gain_high.fits'
-        self.SkyOffsetEastValue = 10
-        self.SkyOffsetNorthValue = 10
+        self.SkyOffsetEastValue = 8
+        self.SkyOffsetNorthValue = 8
 
     def setupUi(self):
         self.log.debug('setupUi')
@@ -364,7 +366,7 @@ def setupUi(self):
         self.ResetSkyFrameBtn = self.findChild(QPushButton, 'ResetSkyFrameBtn')
         self.ResetSkyFrameBtn.clicked.connect(self.reset_sky_frame)
 
-        sky_subtraction_enabled = self.enable_control and False
+        sky_subtraction_enabled = self.enable_control
         self.SkyOffsetEastLabel.setEnabled(sky_subtraction_enabled)
         self.SkyOffsetNorthLabel.setEnabled(sky_subtraction_enabled)
         self.SkyOffsetEast.setEnabled(sky_subtraction_enabled)
@@ -651,6 +653,8 @@ def setupUi(self):
         self.StatusBar.addPermanentWidget(self.kpfguide2StatusLabel)
         self.kpfguide3StatusLabel = QLabel('')
         self.StatusBar.addPermanentWidget(self.kpfguide3StatusLabel)
+        self.CRED2TemperatureLabel = QLabel('')
+        self.StatusBar.addPermanentWidget(self.CRED2TemperatureLabel)
 
         # kpfmon Statuses
         self.HB_GUIDE1STA.stringCallback.connect(self.update_kpfguide1status)
@@ -666,11 +670,15 @@ def setupUi(self):
         self.ST_GUIDE3STA.stringCallback.connect(self.update_kpfguide3status)
         self.ST_GUIDE3STA.primeCallback()
 
-        # CONTINUOUS and SAVE
+        # kpfguide keyword callbacks
         self.CONTINUOUS.stringCallback.connect(self.update_CONTINUOUS)
         self.CONTINUOUS.primeCallback()
         self.SAVE.stringCallback.connect(self.update_SAVE)
         self.SAVE.primeCallback()
+        self.SENSORSETP.stringCallback.connect(self.update_SENSORTEMP)
+        self.SENSORSETP.primeCallback()
+        self.SENSORTEMP.stringCallback.connect(self.update_SENSORTEMP)
+        self.SENSORTEMP.primeCallback()
 
 
     ##----------------------------------------------------------
@@ -728,6 +736,17 @@ def update_SAVE(self, value):
             self.SAVEStatusLabel.setStyleSheet('background-color: transparent;')
 
 
+    ##----------------------------------------------------------
+    ## update SENSORTEMP
+    def update_SENSORTEMP(self, value):
+        deltaT = self.SENSORTEMP.ktl_keyword.binary - self.SENSORSETP.ktl_keyword.binary
+        self.CRED2TemperatureLabel.setText(f'CRED2 deltaT={deltaT:+3.1f} C')
+        if abs(deltaT) > 1:
+            self.CRED2TemperatureLabel.setStyleSheet('background-color: red;')
+        else:
+            self.CRED2TemperatureLabel.setStyleSheet('background-color: transparent;')
+
+
     ##----------------------------------------------------------
     ## Enable/Disable Camera Control and Telemetry
     def enable_control_and_telemetry(self, enabled):
@@ -762,6 +781,7 @@ def update_CameraGain(self, value):
         self.log.debug(f'update_CameraGain: {value}')
         self.CameraGainValue.setText(f'{value}')
         self.CameraGain.setCurrentText('')
+#         self.reset_sky_frame()
         self.colorize_recommended_values()
 
     def set_CameraGain(self, value):
@@ -830,6 +850,7 @@ def update_CameraFPS(self, value):
         self.CameraFPSValue.setText(f"{float(value):.1f}")
         self.CameraFPSSelector.setCurrentText('')
         self.colorize_recommended_values()
+#         self.reset_sky_frame()
 
     def set_CameraFPS(self, value):
         if value != '':
@@ -842,14 +863,14 @@ def set_CameraFPS(self, value):
     def update_SkyOffsetEastValue(self, value):
         try:
             self.SkyOffsetEastValue = float(value)
-        except TypeError:
+        except ValueError:
             pass
 #             self.self.SkyOffsetEast.setText(str(self.SkyOffsetEastValue))
 
     def update_SkyOffsetNorthValue(self, value):
         try:
             self.SkyOffsetNorthValue = float(value)
-        except TypeError:
+        except ValueError:
             pass
 #             self.self.SkyOffsetNorth.setText(str(self.SkyOffsetNorthValue))
 
@@ -859,27 +880,46 @@ def reset_sky_frame(self):
 
     def en(self, e, n):
         dcs = ktl.cache('dcs1')
+        self.log.info(f'RAOFF={float(e):.3f}')
         dcs['RAOFF'].write(float(e))
+        self.log.info(f'DECOFF={float(n):.3f}')
         dcs['DECOFF'].write(float(n))
+        self.log.info('REL2CURR=t')
         dcs['REL2CURR'].write('t')
+        self.log.info('Sleep 2')
+        time.sleep(2)
+        self.log.info('WaitFor AXESTAT==tracking')
+        dcs['AXESTAT'].waitFor('==tracking')
+        self.log.info('Sleep 2')
+        time.sleep(2)
 
     def obtain_sky_frame(self):
+        # Turn loops off while we go get a sky frame
+        self.ALL_LOOPS.write('Inactive')
         # Offset to sky position
         self.log.info(f'Offsetting: en {self.SkyOffsetEastValue:.1f} {self.SkyOffsetNorthValue:.1f}')
         self.en(self.SkyOffsetEastValue, self.SkyOffsetNorthValue)
         time.sleep(0.5)
+        # Set SUB_HIGH to nothing to make sure bias remains in the resulting subtracted frame
+        self.SUB_HIGH.ktl_keyword.write('')
         # Take Image Cube to get sky
-        sky_multiplier = 4
+        sky_multiplier = 9
         duration = sky_multiplier*1/self.FPS.ktl_keyword.read(binary=True)
-        self.log.info(f'Taking sky frame: TakeGuiderImageCube {duration:.0f} seconds')
-        sky_file = TakeGuiderCube.execute({'duration': duration,
+        self.log.info(f'Taking sky frame: TakeGuiderImageCube {duration:.1f} seconds')
+        trigger_file = TakeGuiderCube.execute({'duration': duration,
                                            'ImageCube': False})
-        sky_file = Path(sky_file)
-        if sky_file.exists() == False:
-            self.log.error(f'Could not find {sky_file} on disk')
+        trigger_file = Path(trigger_file)
+        if trigger_file.exists() == False:
+            self.log.error(f'Could not find {trigger_file} on disk')
         else:
-            self.log.info(f'Updating SUB_HIGH with LASTCUBEFILE')
+            trigger_file_hdul = fits.open(trigger_file)
+            sky_file_hdu = fits.PrimaryHDU(data=trigger_file_hdul[1].data)
+            sky_file_hdul = fits.HDUList([sky_file_hdu])
+            sky_file = trigger_file.parent / 'sky.fits'
+            sky_file_hdul.writeto(sky_file, overwrite=True)
+            self.log.info(f'Updating SUB_HIGH')
             self.SUB_HIGH.ktl_keyword.write(f'{sky_file}')
+
         # Offset back to target
         self.log.info(f'Offsetting: en {-self.SkyOffsetEastValue:.1f} {-self.SkyOffsetNorthValue:.1f}')
         self.en(-self.SkyOffsetEastValue, -self.SkyOffsetNorthValue)

kpf/TT_GUI/TipTiltGUI.ui

@@ -41,7 +41,7 @@
       </font>
      </property>
      <property name="toolTip">
-      <string>Enable/disable all componets of the tip tilt loop</string>
+      <string/>
      </property>
      <property name="text">
       <string>Tip Tilt On/Off</string>

kpf/cahk/PowerCycleCaHK.py

@@ -29,9 +29,9 @@ class PowerCycleCaHK(KPFTranslatorFunction):
     @classmethod
     def pre_condition(cls, args, logger, cfg):
         kpfpower = ktl.cache('kpfpower')
-        outlets = [('J1', 'kpfexpose2 Galil RIO controller'),
-                   ('J2', 'kpfexpose2 Galil output bank'),
-                   ('J5', 'Andor Newton PS'),
+        outlets = [('J1', 'Galil RIO (expose2)'),
+                   ('J2', 'Galil Output Bank (expose2)'),
+                   ('J5', 'Andor Newton (kpf_hk)'),
                    ]
         for outlet_id, outlet_name in outlets:
             name = kpfpower[f'OUTLET_{outlet_id}_NAME'].read()

kpf/scripts/EndOfNight.py

@@ -28,6 +28,7 @@ class EndOfNight(KPFTranslatorFunction):
     '''Send KPF in to an end of night configuration.
 
     - kpffiu.MODE = Stowed
+    - reset guider bias/sky subtraction file to default
     - Power off FVCs
     - Power off LED back illuminators
     - close AO hatch
@@ -53,6 +54,10 @@ def perform(cls, args, logger, cfg):
         log.info('Setting FIU mode to Stowed')
         ConfigureFIU.execute({'mode': 'Stowed', 'wait': False})
 
+        # Reset CRED2 subtraction file to default
+        kpfguide = ktl.cache('kpfguide')
+        kpfguide[f'SUB_HIGH'].write(f'/kroot/rel/default/data/kpfguide/kpfguide_gain_high.fits')
+
         # ---------------------------------
         # User Verification for AO Shutdown
         # ---------------------------------