Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

copy reference pixels during dqinit #1121

Merged
merged 2 commits into from
Mar 1, 2024

Conversation

braingram
Copy link
Collaborator

@braingram braingram commented Feb 29, 2024

dqinit adds data views to the resulting datamodel which when saved triggers asdf to save the base array resulting in large file sizes. Similar to #1098 this PR adds copies of those arrays to the output datamodel to avoid larger-than-needed file sizes.

Without this PR running the elp benchmarks (each step in `elp) with a 330Mb input file produces files with sizes as follows:

step output file size
dqinit 1.5 G
saturation 1.5 G
refpix 2.2 G
linearity 2.2 G
darkcurrent 2.2 G
rampfit 332 M
assignwcs 332 M
flatfield 395 M
photom 395 M
sourcedetection 395 M

Note the jump to 2.2 G for the refpix output.

With this PR the file sizes are:

step output file size
dqinit 1.5 G
saturation 1.5 G
refpix 1.5 G
linearity 1.5 G
darkcurrent 1.5 G
rampfit 332 M
assignwcs 332 M
flatfield 395 M
photom 395 M
sourcedetection 395 M

Note that the jump in file size is not seen until later in the pipeline because the base array saved for the views in dqinit is shared until later when the base array is replaced.

Regtests run with no errors: https://plwishmaster.stsci.edu:8081/job/RT/job/Roman-Developers-Pull-Requests/623/

Checklist

  • added entry in CHANGES.rst under the corresponding subsection
  • updated relevant tests
  • updated relevant documentation
  • updated relevant milestone(s)
  • added relevant label(s)
  • ran regression tests, post a link to the Jenkins job below. How to run regression tests on a PR

@braingram braingram changed the title copy arrays added to datamodel during refpix copy arrays added to datamodel during refpix and dqinit Feb 29, 2024
Copy link

codecov bot commented Feb 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 76.73%. Comparing base (1cacc96) to head (046faae).
Report is 1 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1121   +/-   ##
=======================================
  Coverage   76.73%   76.73%           
=======================================
  Files         105      105           
  Lines        7054     7054           
=======================================
  Hits         5413     5413           
  Misses       1641     1641           
Flag Coverage Δ *Carryforward flag
nightly 62.73% <ø> (-0.09%) ⬇️ Carriedforward from 1cacc96

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@braingram braingram changed the title copy arrays added to datamodel during refpix and dqinit copy reference pixels during dqinit Feb 29, 2024
@braingram braingram marked this pull request as ready for review February 29, 2024 22:32
@braingram braingram requested a review from a team as a code owner February 29, 2024 22:32
Copy link
Collaborator

@schlafly schlafly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@braingram braingram merged commit 5f7cb8b into spacetelescope:main Mar 1, 2024
30 checks passed
@braingram braingram deleted the refpix_copy branch March 1, 2024 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants