Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JP-2297: Properly handle DO_NOT_USE DQ flag for multi-integration processing #60

Merged
merged 12 commits into from
Oct 22, 2021

Conversation

kmacdonald-stsci
Copy link
Collaborator

@kmacdonald-stsci kmacdonald-stsci commented Oct 21, 2021

It is possible during ramp fitting for a pixel to be marked as DO_NOT_USE due to saturation in one integration, but not another. For pixels good data in one or more integrations having unusable date in other integrations, the final DQ flag should not be marked as DO_NOT_USE.

Fixes spacetelescope/jwst#6379

Resolves JP-2297

@codecov
Copy link

codecov bot commented Oct 21, 2021

Codecov Report

Merging #60 (8713b83) into main (5ab6cf5) will increase coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main      #60      +/-   ##
==========================================
+ Coverage   94.48%   94.51%   +0.03%     
==========================================
  Files          14       14              
  Lines        1614     1624      +10     
==========================================
+ Hits         1525     1535      +10     
  Misses         89       89              
Impacted Files Coverage Δ
src/stcal/ramp_fitting/ols_fit.py 93.53% <100.00%> (+0.02%) ⬆️
src/stcal/ramp_fitting/utils.py 98.30% <100.00%> (+0.04%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5ab6cf5...8713b83. Read the comment docs.

@hbushouse hbushouse changed the title JO-2297: Properly hand DO_NOT_USE DQ flag for multi-integration processing JP-2297: Properly handle DO_NOT_USE DQ flag for multi-integration processing Oct 21, 2021
Copy link
Collaborator

@hbushouse hbushouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one minor comment/question

src/stcal/ramp_fitting/utils.py Outdated Show resolved Hide resolved
Copy link
Collaborator

@hbushouse hbushouse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a change log entry. Other than that, it looks good.

# -----------------------------------------------------------------------------
# Set up functions

# Need test for multi-ints near zero with positive and negative slopes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Near zero" ? I don't quite follow.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was an old comment I didn't see was still there. I removed this because it doesn't apply here and is confusing.


def test_utils_dq_compress_final():
"""
Set up a multi-integration 3 pixel data array each ramp as the following:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't follow what is being tested here - can you add some clarification ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am expanding the comment for this test. Let me know if I need to expand it further.

Copy link
Contributor

@dmggh dmggh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Looks good.

)

* Initial changes, with tests, to zero out Poisson variances for pixels with negative median rates.

* Modifying test file structure.

* Adding comment.

* Making style changes.

* Moving ramp fitting unit tests to the top level 'tests' directory.

* Changed test comments and change log based on code review feedback.

* Making changes to the change log.
@nden
Copy link
Collaborator

nden commented Oct 22, 2021

@kmacdonald-stsci Please rebase to resolve the conflict.

@nden
Copy link
Collaborator

nden commented Oct 22, 2021

Also there's a flake 8 issue

…tils.dq_comopress_final function.

Adding a test for the new dq_compress_final function.

Changing a line to make it clearer a uint32 is required for the computation.

Changing style failures in ramp fitting test file.

Adding to the change log.

Removing a confusing comment and expanding the comment explaining the test to make it more understandable.

Correcting style fails.
CHANGES.rst Outdated
@@ -4,6 +4,7 @@
ramp_fitting
------------

- For slopes with negative median rates, the Poisson variance is zero. [#33]
- Changed the way the final DQ array gets computed when handling the DO_NOT_USE
flag for multi-integration data.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the parser need to have the continuation line intended to the same level as the start of the text in the preceding line? (i.e. this may need to be intended 2 spaces)

CHANGES.rst Outdated

- For slopes with negative median rates, the Poisson variance is zero. [#33]
- Changed the way the final DQ array gets computed when handling the DO_NOT_USE
flag for multi-integration data.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is missing the PR number.

Copy link
Collaborator

@nden nden Oct 22, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also the previous PR number is incorrect. It is #59

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah. I put issue number, not PR number. I'm changing this now.

@nden
Copy link
Collaborator

nden commented Oct 22, 2021

Thanks @kmacdonald-stsci
Roman tests are failing for unrelated reasons.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Error in calculating the final DQ for MIRI multiple integration exposures
4 participants