THEMIS and MRO SSD Kernels in ISIS

[jlaura]

ISIS version(s) affected: 3.8.0

Description
The smithed THEMIS kernels and MRO SSD kernels are currently not included in ISIS. These kernels need to be shipped to ISIS users.

How to reproduce
Download smithed kernels, use them manually with spiceinit and be amazed by how much better good pointing really is.

Possible Solution
copy and update kernels.db files

Additional context

[krlberry]

@jlaura Where are the smithed THEMIS kernels located?

[jlaura]

@rfergason @lwellerastro Can you all aim us at the kernels to be included? I have seen some data/kernels under /work/projects, but I want to make sure we get the correct kernels.

[jessemapel]

For the MRO SPKs, they can be found here. Anything with _ssd_ in its name is from the JPL SSD group. We should prefer the mro110c kernels, but if they are not available, then we can use the mro95a kernels.

[rfergason]

The final smithed CK and SPK kernels that Lynn produced from the THEMIS control network solution are located in: /archive/projects/themis_control/Kernels

There are separate CK and SPK kernels for the dayIR images and nightIR images, as daytime and nighttime IR images were not bundled in the same solution.

[krlberry]

The kernels are now in and updated. Leaving this issue open until all related ISIS tests are passing.

[jlaura]

Is this safe to close? I didn't check / see the CRON jobs - looks like the working Jenkins stuff is still good to go.

[krlberry]

Kernels were updated and tests are passing

[lwellerastro]

Hi - how do I access the updated Themis kernels? I am working with a test image under isis3.10.2 (and under isis4.2.0) and it doesn't look like I'm picking up smithed kernels. This is an image that was controlled and is included in the updated kernels.

thm2isis from=/pds_san/PDS_Archive/Mars_Odyssey/THEMIS/USA_NASA_PDS_ODTSDP_100XX/ODTSDP_10014/data/odtir1_0014/i147xxrdr/I14705003RDR.QUB to=I14705003RDR_smithed.cub

spiceinit from=I14705003RDR_smithed.cub cksmithed=true spksmithed=true

From the label after spiceinit:

Group = Kernels
NaifFrameCode = -53031
LeapSecond = $base/kernels/lsk/naif0012.tls
TargetAttitudeShape = $base/kernels/pck/pck00009.tpc
TargetPosition = (Table, $base/kernels/spk/de430.bsp,
$base/kernels/spk/mar097.bsp)
InstrumentPointing = (Table, $odyssey/kernels/ck/m01_sc_ext3.bc,
$odyssey/kernels/fk/m01_v29.tf)
Instrument = Null
SpacecraftClock = $odyssey/kernels/sclk/ORB1_SCLKSCET.00258.tsc
InstrumentPosition = (Table, $odyssey/kernels/spk/m01_ext3.bsp)
InstrumentAddendum = $odyssey/kernels/iak/themisAddendum003.ti
ShapeModel = $base/dems/molaMarsPlanetaryRadius0005.cub
InstrumentPositionQuality = Reconstructed
InstrumentPointingQuality = Reconstructed
CameraVersion = 2
End_Group

I've compared it to the default spiced version and to my jigsaw updated image and it is definitely picking up the default spice even when asking for smithed. Am I doing something wrong?

Sorry if re-opening this post wasn't the thing to do.

[jessemapel]

@lwellerastro What data area are you using on our network? I don't know if these were checked into the old ISIS3 area.

[lwellerastro]

I was working under isis3.10.2. Do I have to ask for a data area? Since this was closed almost a year ago I would assume the updates are under any version of isis released after that. Is that not the case?

Edit - I just noticed in my original post I mention this doesn't work under isis4.2.0 either so I added that data area information as well.

After activating conda:

(isis3.10.2) ast{114}> printenv |egrep ISIS
ISISROOT=/usgs/cpkgs/anaconda3_linux/envs/isis3.10.2
ISIS3DATA=/usgs/cpkgs/isis3/data
ISIS3TESTDATA=/usgs/cpkgs/isis3/testData

(isis4.2.0) ast{104}> printenv |egrep ISIS
ISISROOT=/usgs/cpkgs/anaconda3_linux/envs/isis4.2.0
ISISDATA=/usgs/cpkgs/isis3/isis_data/
ISISTESTDATA=/usgs/cpkgs/isis3/isis_testData/

Is this telling you anything?

[jessemapel]

Is this telling you anything?

It's very much broken in both places :(

[lwellerastro]

I'm sorry - sad emoji's x 3 (responding to you w/ a confused one (the only close to sad choice) did not seem right).

Should this be a new post then?

Also, CTX folks should get a heads up to check any themis files they may have used for ground recently to see if this impacted them.

[jessemapel]

No, I think this is the right place to post.

[krlberry]

No fix yet, but an update: I can see why the smithed kernels aren't getting picked up. The kernel's coverage is a fraction of a second smaller than the start/stop time on the image, so the kernel is being identified as "not covering the entire image" and as such is not used.

For example, looking at: I14705003RDR.QUB
themis_dayir_merged_2018Jul13_ck.bc has coverage from:
2005-APR-08 02:45:46.059 to 2005-APR-08 02:47:53.322 in UTC, but the example image's start and stop times (UTC) are as follows:

START_TIME = 2005-04-08T02:45:45.797
STOP_TIME = 2005-04-08T02:47:45.797

[lwellerastro]

@kberryUSGS, I used these kernels extensively in building the final Themis products by pointing to them specifically in my spiceinit command, so I know they work. I think we ran across something similar with my smithed Enceladus kernels. There is likely a post out there. I'll link it here when I find it.

Here it is: #3669
There's a lot in this post...it might be easier just to ask @jessemapel

[krlberry]

@lwellerastro The Enceladus issue does look like the same problem, thanks!

[jessemapel]

Yep, this sounds like the same problem. For that case I just wrote a little notebook/script to go through the kerneldb and add some amount of time padding to each image segment.

[lwellerastro]

@kberryUSGS, my example command points to the pds archive in its entirety and I did not control all of the Themis IR, hence why you had so many failures via your test pool.

If you already have a test script that ingests then spiceinits images please pull data from the following lists. On is for daytime IR and the other is for nighttime IR data both of which should be in smithed kernels, so you might pull some files from both. When you have something you think is working ok, I can run all the data through as a test. There are over 70k files combined, so I don't expect you to that sort of testing.

daytime IR pds source location: /work/projects/themis_control/Completed_Tiles/MergedNetworks/DayIR/pds_san_rdr.lis
nighttime IR pds source location: /work/projects/themis_control/Completed_Tiles/MergedNetworks/NightIR/pds_san_rdr.lis

[krlberry]

Thank you very much for the test data! I'll pull ~50 from each list to test as a first-pass.

[lwellerastro]

@kberryUSGS would you mind pulling from the start and end of the lists? They are likely in order of acquisition time so by grabbing some from the end you can sort cover the full time frame of what's in the kernel.

[krlberry]

Good idea, thanks!

[krlberry]

Here is what is going on:

Summary: spiceinit uses the StartTime in the label to decide which kernels to use, but the StartTime in the label is incorrect for THEMIS images. A band-dependent offset to the start time is applied inside the THEMIS camera model to correct for this. When ckwriter/spkwriter generated the smithed kernels, they use the corrected times with the offsets applied, which is the correct behavior. This means that when the smithed kernels are used with the original images, their labels still have the incorrect StartTimes, so spiceinit complains that it can't find a kernel that covers the image time as determined by the incorrect StartTime on the image. But, the information about the time-offset isn't calculated until after the kernels are selected by spiceinit.

More details:
THEMIS has a band-dependent offset to the start time of the image that is applied within the camera model. Some THEMIS images start with Band 1. These have a very small offset applied and a small amount of kernel padding (0.3 seconds) added to the kernel databases works well. Other THEMIS images start with Band 4 (ex: I00818001RDR.QUB.) These images are offset by 2.6 seconds, so a larger kernel padding is needed for these. The maximum possible start-time offset is 7.7 seconds, if an image were to start with Band 10 (I have not checked to see if any of these exist yet.)

The band-dependent time offset is applied in the camera model, after spiceinit selects which kernels to use. For the smithed kernels, spkwriter and ckwriter have correctly written out the position/pointing information vs. the actual timing information with the offsets applied, but then when the smithed kernel is used with the image again, spiceinit still uses the (incorrect) start time in the image label to determine which kernels to use, and since the start time in the label doesn't overlap with the start time in the actual kernel, the kernel isn't selected, unless padding to the kernel database is added to correct for the offset.

Here is an example:

StartTime on the Label : 2002-02-19T22:08:28.604 (outside of kernel time)
StartTime of associated Kernel segment: 2002-02-19T22:08:31.131
Actual start time with offset applied for first line of the image: 2002-02-19T22:08:31.134 (within kernel time)

There isn't really a pre-existing mechanism for different kernel database padding depending on the image, so I'm not sure what the best solution is here, but here are some options:
(1) Pad by the maximum time offset in the dataset, which will ensure that the smithed kernels get picked up, but has the possibility of picking them up when they shouldn't be.
(2) Write a script to look at the image labels of all the images used to generate the smithed ck/spk, and pad individual segments of the kernels in the kernel databases, depending on which bands and summing mode are used in the image corresponding to that time range. (The offset is a function of band number and summing mode.)
(3) Does anyone have any better ideas? I'm not sure what the right solution is here.

The only strategy that I considered is "compensating for the offset to the image label StartTime by padding the kernel database to compensate", so I'm particularly interested if anyone else thinks of a different angle on this.

[lwellerastro]

This is concerning, though I'll admit I don't completely understand what is going on.

Somewhat early in the Themis IR project (which covered about 6 years), I worked with developers to determine how to best work with the data to ensure I was building kernels properly. After running spiceinit on the ingested multi-band Themis image, filter 9 was extracted (which could resign in any one of several band locations based on the number of bands in the image), and used for controlling the data set and building kernels. It was recommended to do it this way (spice, then extract as opposed to extract then spice) to ensure the extracted band would been spiced using the full time range of the multi-band image and the kernels would encompass that.

I did a lot of testing to determine this was indeed the proper procedure and in fact had to go back to the first years products and redo some of them since they were done in the in the wrong order (extract then spice).

So I''m confused why the multi-band image label and kernel segment+padding don't completely cover the time span properly. Is this the kernels database doing this? In order to make the final distributed products every image in the kernel was re-ingested, spiced using the new kernels (directly accessed via spiceinit) then mosaicked. There were no failures. Of course all of this was done via isis3production a long time ago (I created the kernels during 2018).

Your example indicates the kernel wouldn't work for the image since its start time is outside the start time of the label. Did spiceinit actually fail for the image you were working with while pointing directly to the new kernels? I'm confused.

[krlberry]

@lwellerastro As far as I can tell, the smithed kernels are correct and do actually cover the entire time range of the images. The only problem is with the kernel databases and spiceinit not realizing that the kernels do cover the entire time range of the image because spiceinit doesn't have "all the information," it just has the time range of the kernel specified in the kernel database, and the time range of the image according to the label (which is off.)

So I''m confused why the multi-band image label and kernel segment+padding don't completely cover the time span properly. Is this the kernels database doing this? In order to make the final distributed products every image in the kernel was re-ingested, spiced using the new kernels (directly accessed via spiceinit) then mosaicked. There were no failures. Of course all of this was done via isis3production a long time ago (I created the kernels during 2018).

Yeah, it's just the kernel database. Did spiceinit used to pick up the smithed kernels using spksmithed=true cksmithed=true in an earlier version of ISIS, or did you need to specify the smithed ck and spk, using ck= and spk=?

Your example indicates the kernel wouldn't work for the image since its start time is outside the start time of the label. Did spiceinit actually fail for the image you were working with while pointing directly to the new kernels? I'm confused.

spiceinit with spksmithed=true and cksmithed=true will not pick up this kernel (unless I add 2.6 seconds of padding to the kernel database entries.)

spiceinit with spk=themis_nightir_merged_2018Mar02_spk.bsp ck=themis_nightir_merged_2018Mar02_ck.bc will work and do the right thing because it bypasses the kernel databases.

[lwellerastro]

Thanks @kberryUSGS - it's a database problem. I thought so but was confused.

The maximum possible start-time offset is 7.7 seconds, if an image were to start with Band 10 (I have not checked to see if any of these exist yet.)

I think for this to be true, it would have to be a single band themis image, which might not happen. I have a data base of all my smithed images and logs of processing. I'll have a look and see if I can determine the common starting bands for my images if that helps any.

[krlberry]

Thank you for asking the question! I knew my explanation was kind of long and confusing, but couldn't figure out how to improve it any more, so this is probably a lot more clear now for anyone else reading the ticket, too.

It would be definitely helpful to know which starting bands were used in your images!

[lwellerastro]

Well I was wrong, there are single band themis IR images that at least start with band 9 (this band was a requirement, so if there are single band images that only contain band 10, I wouldn't have them).

I went through all daytime and nighttime IR Themis images contained within the kernels and the following are starting bands encompassing all of the images: 1, 3, 4 and 9.

And just to break it down another way, here is the various combos of band bin filter information for all processed/controlled/kernel-based images:
BAND_BIN_FILTER_NUMBER = (1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
BAND_BIN_FILTER_NUMBER = (3, 4, 5, 6, 7, 8, 9, 10)
BAND_BIN_FILTER_NUMBER = (4, 7, 8, 9, 10)
BAND_BIN_FILTER_NUMBER = (4, 9, 10)
BAND_BIN_FILTER_NUMBER = (9)

[krlberry]

@lwellerastro Thanks for the information! For images starting with Band 9, there would be a maximum start-time offset (between label start time and 'actual' start time) of 6.84 seconds, assuming a summing mode of 1. Would you also be willing to check what the summing modes used in the images in your set are? The associated ISIS keyword is SpatialSumming.

[lwellerastro]

@kberryUSGS - we only worked with summing modes 1 and 2.

[krlberry]

Since the maximum offset is around 7 seconds, which is over 400 exposure durations, I decided that it didn't make sense to pad all the kernel database entries by the maximum possible offset, as this could lead to spiceinit selecting these kernels when they don't actually cover the image time.

I proceeded with "write a script to update the kernel database with segment-specific padding." The script worked on my test case of 25 images and I'm now running it on the full sets of themis day IR and night IR images. It's going to take a little while for the updated kernel databases to be generated and be available for testing, as the script needs to parse each image label to calculate the amount of offset, which is over 70k files, as Lynn mentioned earlier. (As of right now, it's 15% done, and I started running it about an hour ago.)

[KrisBecker]

Apologies for being late here.

I don't see padding the times in the kernel database as good solution here. For one, it adds a (time coverage) falsehood to kernels/infrastructure that is otherwise precise and unforgiving. Second, typically other spacecraft payload instruments use these databases for the same purpose and it could break spiceinit on those devices.

The specific/real issue is spiceinit's time coverage algorithm. It uses/relies on StartTime (and EndTime if it exists) label keyword values to find kernel coverage. Nearly all ISIS camera models use SCLK values in the SpacecraftClockStartTime (and ExposureDuration) and generally require very intimate details about spacecraft->instrument timing handshaking. This is especially true with line scan instruments, variable scan rates and time dependent filters. Using just StartTime adds a potentially large amount of uncertainty in kernel lookups as you are experiencing here.

Although spiceinit provides STARTPAD/ENDPAD parameters, usage will cause expansion (or shrinkage) of the ephemeris caches, which will compromise coverage in the case of negative time padding.

A few options come to mind: 1) add additional parameters (for both CK and SPK?) to spiceinit specifically to expand the time coverage resolution aspect of spiceinit's algorithm that does not affect the cached time ranges (this is also not w/o risk), and 2) for a more radical, but robust approach, add additional methods and constructors to Camera and Camera Models that provide the real coverage ranges that require loading only applicable SCLK and LSK kernels (for most cases, however, some more complex instruments may require additional kernels).

A word about image acquisition timing: Currently, there exists no explicit, reliable method to get both the real start time and/or the exposure duration in standard units from the perspective of the camera model regarding time coverage for the active observation. There are methods that do provide timing at a particular pixel, but not for all (e.g., framing) instruments. And you can get start/end cache times, but these times are affected by STARTPAD/ENDPAD and framers only return the time at the center of the exposure time.

A word about StartTime: These values are typically generated very shortly after image data is received on the ground by instrument teams using currently available resources (i.e., kernels). The StartTime, stored as a UTC string, are typically computed using the SpacecraftClockStartTime and SCLK and LSK kernels available at the time. However, many if not all SCLK kernels account for spacecraft clock drift that is determined at a later date. For example OREX SCLKs don't stabilize until about the second released SCLK kernel with a release cadence of about a month per new SCLK. In summary, StartTimes are not a reliable source for image observation start times.

Now a word about ckwriter/spkwriter: The documentation of these application and the comments in the output kernels state that all the kernels specified in the image labels initially applied by spiceinit must be used when using the generated CK/SPK kernels, otherwise you may not get reliable results.

[krlberry]

Hi Kris, Thanks for responding to this.

I don't see padding the times in the kernel database as good solution here. For one, it adds a (time coverage) falsehood to kernels/infrastructure that is otherwise precise and unforgiving. Second, typically other spacecraft payload instruments use these databases for the same purpose and it could break spiceinit on those devices.

I updated my approach and am now shifting the times in the kernel database according to the offset that is applied in the THEMIS camera model to the label StartTime, rather than padding the database entries to expand their coverage. As such, the updated kernel database will connect the StartTime in the labels of the images to the correct kernel.

It's an extremely good point about the other payload instruments. I updated the database so these kernels with the shifted time ranges will only be used for ThemisIR images.

The specific/real issue is spiceinit's time coverage algorithm. It uses/relies on StartTime (and EndTime if it exists) label keyword values to find kernel coverage. Nearly all ISIS camera models use SCLK values in the SpacecraftClockStartTime (and ExposureDuration) and generally require very intimate details about spacecraft->instrument timing handshaking. This is especially true with line scan instruments, variable scan rates and time dependent filters. Using just StartTime adds a potentially large amount of uncertainty in kernel lookups as you are experiencing here.

I completely agree! I think that a more robust way to deal with this problem in the future is a great idea, but I think it's beyond the scope of this specific ticket (getting smithed kernels to be picked up by spiceinit for THEMIS.)

[krlberry]

spiceinit with spksmithed=true and cksmithed=true now successfully picks up the smithed ck and spk kernels for all the ~70k THEMIS IR images I tested it on, except for 11 images. These images were not included in the creation of the smithed kernels and genuinely do not have pointing/position data in the smithed kernels, so this is the correct, expected result.

The smithed parts of the THEMIS kernel databases were updated to use start/stop times which were shifted by the offset between the original image label start/stop times and the actual start/stop times in the image (corresponding to the kernel start/stop times.) This is because spiceinit uses the image label times to select which kernels to use. Since there is a large offset applied to the time from the image label in the THEMIS camera model, these no longer matched up, and spiceinit was unable to find kernels produced from the original images. In some cases, this shift still wasn't enough, and a per-image padding to the times on the order of 0.01 seconds was added as needed.

This was accomplished using a script, and now, when automated kernel updates occur, everything will be updated except the smithed section of the kernel database, which is kept constant through the updates.

[jlaura]

🎉 Well done @kberryUSGS and the rest of the team that supported your efforts! I am going to close as this appears to be working now. Anyone should feel empowered to reopen if they see any issues when using the smithed kernels with THEMIS IR data.

[lwellerastro]

except for about 4 images. These images were not included in the ISIS solution

Could you email me the filenames of the failed images? I want to understand why they were included in the list I supplied to you. Sorry for including some bad apples in the list. I hope they didn't cause you too much grief!