Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Execution Block ID uid://A001/X15a0/X118 Sgr_A_st_u_03_TM1 #241

Open
6 of 15 tasks
keflavich opened this issue Sep 3, 2022 · 22 comments
Open
6 of 15 tasks

Execution Block ID uid://A001/X15a0/X118 Sgr_A_st_u_03_TM1 #241

keflavich opened this issue Sep 3, 2022 · 22 comments
Assignees
@xinglunju
Labels
Delivered Done! EB Execution Block Needs Reimaging: Spectral Resolution size mitig Needs to be reimaged with full spectral resolution Reclean Needs to be or has been recleaned TM1

Comments

@keflavich
Copy link
Contributor

keflavich commented Sep 3, 2022 •
edited
Loading

Sgr_A_st_u_03_TM1
uid://A001/X15a0/X118

Product Links:

Reprocessed Product Links:
@keflavich keflavich added EB Execution Block TM1 labels Sep 3, 2022
@xinglunju xinglunju self-assigned this Sep 19, 2022
@xinglunju xinglunju mentioned this issue Sep 19, 2022
Open
@xinglunju
Copy link
Contributor

Calibration

  • The phase calibrator J1744 was not properly flagged in spw29. Significant HCO+ absorption feature is seen (see plot below). This does not affect bandpass, but would affect phase calibration.
    image

Continuum imaging

Looks good. Although this field has a weird shape like #126 , the psf and the imaging in general turn out to be ok.
image

Line imaging

  • All spws size-mitigated and binned by factor 2. Need reclean.
  • Otherwise, the image cubes look great.
  • The beams are not quite good: the largest beam is found for spw31, which is 2.16 x 1.28 arcsec, with a mean of 1.66 arcsec. The beam ratio is high (2.16/1.28=1.69), while the weblog says 'Maximum expected beam axial ratio (from OT): 1.5'.

@xinglunju xinglunju added Reclean Needs to be or has been recleaned Needs Reimaging: Spectral Resolution size mitig Needs to be reimaged with full spectral resolution labels Sep 19, 2022
@ashleythomasbarnes ashleythomasbarnes mentioned this issue Sep 21, 2022
Open
@xinglunju
Copy link
Contributor

xinglunju commented Nov 8, 2022 •
edited
Loading

There is already a re-imaging argument for this SB in the json file, probably added by other folks. But it is only for SPW33. Should I just overwrite that one, or add a new argument for the other SPWs? @keflavich
By the way, the hipergator pipeline run already includes all the necessary tclean arguments to solve the size-mitigation issues, however, the weblog shows many errors in the imaging step and there are no image products in globus. Is there anyway to re-run the hipergator pipeline?

@keflavich
Copy link
Contributor Author

If the parameters need to be modified, just overwrite them. SPW33 was auto-generated, so it makes sense that it needs new arguments.

There are image products, though:

/orange/adamginsburg/ACES/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw25.cube.I.iter1.image
/orange/adamginsburg/ACES/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image
/orange/adamginsburg/ACES/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw29.cube.I.iter1.image
/orange/adamginsburg/ACES/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image
/orange/adamginsburg/ACES/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw33.cube.I.iter1.image
/orange/adamginsburg/ACES/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw35.cube.I.iter1.image

@xinglunju
Copy link
Contributor

But these image products are from the original pipeline with the size-mitigation issues, right? We need images from the hipergator pipeline that uses the correct parameters (no binning, no dropping spw33...), which do not exist...

@keflavich
Copy link
Contributor Author

spw33 is in that list, so no, these are not from the pipeline that dropped it. The modification dates are October, so I believe these all come from the post-rebinning pipeline. However, to make this easier to look up, I'm running our cube_stats_grid.py script so we can check easily.

@xinglunju
Copy link
Contributor

I don't think so... the files with modification dates in October are mom0 fits, but the image products themselves are still from Sep. This SB does have spw33, even in the original pipeline--in the last comment I meant to refer to the other SBs with the same issue (#238 #152 ) that have dropped spw33

@keflavich
Copy link
Contributor Author

Ah... yes, this one has modification dates of September. #238 and #152 have more recent modification dates. Still waiting on stats to finish to see if these have the right sizes
https://data.rc.ufl.edu/secure/adamginsburg/ACES/tables/cube_stats.js.html

@xinglunju
Copy link
Contributor

@keflavich I confirm that all the six SPWs are still 2x binned in Globus. We need to re-image with new parameters (or re-run the hipergator version?)

@keflavich
Copy link
Contributor Author

are the override_json commands correct?

@xinglunju
Copy link
Contributor

There is an overwrite_json command for SPW33, but not for other SPWs.
This is something that confused me: if you check the tclean commands in hipergator pipeline, the nchan keywords have the correct values without binning, so can we just re-run the hipergator pipeline and get unbinned images of all the SPWs? Or do I need to add all the overwrite commands to the json file?

@keflavich
Copy link
Contributor Author

you need to add the override commands to the json file. While the hipergator pipeline runs happen, they have a failure rate of 100% because CASA MPI doesn't work and there is a time limit on our jobs. The override_json files specify the final imaging parameters that will be run.

Basically, I had to abandon the CASA/ALMA pipeline because it wasn't flexible enough to work with our computing setup here.

@xinglunju
Copy link
Contributor

I see, then I will add the override commands for this SB as well as the other two 12m SBs.

@xinglunju
Copy link
Contributor

override_json commands updated for this SB.

@keflavich
Copy link
Contributor Author

override_tclean commands were updated in #295 (@xinglunju - it's good to reference where you made the change)

@keflavich keflavich mentioned this issue Dec 7, 2022
Merged
@keflavich
Copy link
Contributor Author

Files have been moved into unmitigated/ subdirectory; recleaning will proceed soon.

@xinglunju xinglunju mentioned this issue Dec 8, 2022
Open
@xinglunju
Copy link
Contributor

xinglunju commented Mar 1, 2023 •
edited
Loading

Mar 1 2023 update: cubes of all SPWs seem to be 2x binned on Globus, according to the cube_stats.js list, although the file sizes are big enough. I will download and double check.

@keflavich
Copy link
Contributor Author

These cubes appear to be the right size:

login4.ufhpc /orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working$ gethead NAXIS3 *fits
uid___A001_X15a0_X118.s36_0.Sgr_A_star_sci.spw25_27_29_31_33_35.cont.I.iter1.image.tt0.pbcor.fits                 1
uid___A001_X15a0_X118.s36_0.Sgr_A_star_sci.spw33_35.cont.I.iter1.image.tt0.pbcor.fits                             1
uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw25.cube.I.iter1.image.pbcor.fits                                    1914
uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw27.cube.I.iter1.image.pbcor.fits                                    1914
uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw29.cube.I.iter1.image.pbcor.fits                                    1914
uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw31.cube.I.iter1.image.pbcor.fits                                    1914
uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw33.cube.I.iter1.image.pbcor.fits                                    3834
uid___A001_X15a0_X118.s38_0.Sgr_A_star_sci.spw35.cube.I.iter1.image.pbcor.fits                                    3834

I'm not sure why this appears wrong in cube stats though.

@keflavich
Copy link
Contributor Author

...likely because i introduced a bug in imstats 3 months ago and it hasn't run since then

@eacmills eacmills mentioned this issue Mar 23, 2023
Closed
@d-l-walker d-l-walker mentioned this issue Jun 7, 2023
Open
@xinglunju
Copy link
Contributor

All cubes are available on Globus and are not size-mitigated. Setting this one to 'done'.

@keflavich
Copy link
Contributor Author

keflavich commented Nov 27, 2023 •
edited
Loading

The s36 files for the continuum were broken (the images were made from uncalibrated data). They have been removed, as s10 data were present and healthy.

rm -r /orange/adamginsburg/ACES//data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X118/calibrated/working/uid___A001_X15a0_X118.s36*

@pyhsiehATalma pyhsiehATalma mentioned this issue Jan 31, 2024
Open
@betacygni
Copy link

QA - Line contamination in continuum images from high/low frequencies

Summary: both files look reasonably good

Files checked:

  • New: spw25_27 and spw33_35
  • Old: spw25_27_29_31_33_35 and oldhigh_spw33_35

Results:
Both spw33_35 and spw25_27 look reasonably good (no obvious contamination)

Attached image:

  • Top-left: spw33_35
  • Top-right: oldhigh_spw33_35
  • Bottom-left: spw25_27
  • Bottom-right: spw25_27_29_31_33_35
QA_field_u_cont

@betacygni
Copy link

betacygni commented May 8, 2024 •
edited
Loading

QA - Line contamination in continuum images from high/low frequencies

Summary: New images look good, simple field at first glance

Files checked:
New: spw25_27 and spw33_35
Old: v1.1_20240314_low_spw25_27 and v1.1_20240314_high_spw33_35

Attached image:
Top-left: spw25_27
Top-right: old_spw25_27
Bottom-left: spw33_35
Bottom-right: old_spw33_35

Screenshot 2024-05-08 at 14 38 14

@github-project-automation github-project-automation bot moved this to Completed execution blocks (imaging, QA, calibration all done) in Data Reduction Aug 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xinglunju xinglunju

Labels
Delivered Done! EB Execution Block Needs Reimaging: Spectral Resolution size mitig Needs to be reimaged with full spectral resolution Reclean Needs to be or has been recleaned TM1
Projects
Data Reduction
Status: Completed execution blocks (imaging, QA, calibration all done)
Milestone
No milestone
Development

No branches or pull requests
3 participants
@keflavich @xinglunju @betacygni