Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Execution Block ID uid://A001/X15a0/X19c Sgr_A_st_aq_03_TM1 #140

Open
5 of 15 tasks
Tracked by #179
keflavich opened this issue May 18, 2022 · 29 comments
Open
5 of 15 tasks
Tracked by #179

Execution Block ID uid://A001/X15a0/X19c Sgr_A_st_aq_03_TM1 #140

keflavich opened this issue May 18, 2022 · 29 comments
Assignees
@d-l-walker
Labels
Delivered EB Execution Block Needs Reimaging: Divergence Needs reimaging b/c of divergence Needs Reimaging: Spectral Resolution size mitig Needs to be reimaged with full spectral resolution TM1

Comments

@keflavich
Copy link
Contributor

keflavich commented May 18, 2022 •
edited
Loading

Sgr_A_st_aq_03_TM1
uid://A001/X15a0/X19c

Product Links:

Reprocessed Product Links:
@keflavich keflavich added EB Execution Block TM1 labels May 18, 2022
@d-l-walker d-l-walker self-assigned this Aug 24, 2022
@d-l-walker d-l-walker added Needs Reimaging: Divergence Needs reimaging b/c of divergence Needs Reimaging: Spectral Resolution size mitig Needs to be reimaged with full spectral resolution and removed Delivered labels Aug 24, 2022
@d-l-walker
Copy link
Contributor

Calibration looks fine.

Imaging issues:

  • SPWs 33 & 35 both diverged
  • All SPWs are binned x2
  • Slight slope in spectrum for SPW 27 due to only a few continuum channels on one cube edge
  • SPW 29 has a very broad emission. Looks like the two continuum bits on either edge were added manually to anchor the fitting.
    uid___A001_X15a0_X19c s38_0 Sgr_A_star_sci spw29 cube I iter1 image spectrum
  • Continuum identification in SPWs 33 & 35 looks poor. I think we can add a lot more here to increase the continuum sensitivity.
    uid___A001_X15a0_X19c s38_0 Sgr_A_star_sci spw35 cube I iter1 image spectrum

@keflavich I can see that re-imaging is already happening on HPG, but I think this is with size mitigation. So, I will re-run the imaging pipeline locally with:

  • Increased cyclefactor to avoid diverged in SPWs 33 & 35
  • Native spectral resolution to avoid size mitigation
  • Refined continuum selection in several SPWs as necessary

@keflavich
Copy link
Contributor Author

Before you run the pipeline locally, could you please upload the revised imaging parameters that are not size-mitigated?

@d-l-walker d-l-walker mentioned this issue Aug 24, 2022
Open
17 tasks
@d-l-walker
Copy link
Contributor

Just a minor update on this -- the PL re-imaging without size mitigation is currently underway at Manchester.
I didn't update the imaging parameters yet, as I'm using the PL tasks to automatically set the parameters, not manually specifying them. I also made a few changes to the PL re-run (revised continuum selection, higher cyclefactor).

Once all the cleaning has finished, I'll place the files in the /upload/ directory, and update override_tclean_commands.json as necessary.

@d-l-walker
Copy link
Contributor

Progress update: I set this running ages ago, but it appears to have failed part-way through. I executed the full imaging PL again late last week and it hasn't died so far, so hopefully this will finish soon. I will upload the the products and updated imaging parameters once finished.

@eacmills eacmills mentioned this issue Mar 23, 2023
Closed
@d-l-walker
Copy link
Contributor

Update: The full PL imaging re-run is still ongoing for this. It's taking a while because I'm doing the full imaging re-run (contsub, per-SPW mfs, continuum, and cubes), plus with a higher cyclefactor to avoid the divergence in SPWs 33 & 35.

SPW 33 is currently being cleaned, and everything else before this (contiuum, and SPWs 25, 27, 29, 31) all look good. Hopefully shouldn't take too much longer now.

@d-l-walker
Copy link
Contributor

Full re-imaging has finished, but SPWs 33 & 35 both diverged using cyclefactor=2.0. I'll re-run those with a higher value, and upload all of the finished products for the other SPWs/continuum.

@d-l-walker
Copy link
Contributor

SPWs 33 & 35 are still ongoing with the higher cyclefactor.
I've started adding all of the other products to /upload/Sgr_A_st_aq_03_TM1_reclean/. Once the remaining SPWs have finished, I'll add those and then the whole folder can be merged with the target destination.

@d-l-walker
Copy link
Contributor

This field is Sgr A*.

SPW 25 looks neat, lots of SiO masers:

Screenshot 2023-05-31 at 14 17 44

Sgr A* is more obvious in the SPW 35 peak intensity map:

Screenshot 2023-05-31 at 14 21 54

TO DO: At some point, we should look into removing Sgr A* from the data.

@pyhsiehATalma
Copy link
Contributor

It is hard to subtract Sgr A* indeed, but I could take a look of the data. Here are things I would like to check and try some non-pipeline approaches.

  1. is line-free region sufficient for continuum subtract?
  2. split Sgr A* fields and subtract continuum seperately from other fields
  3. using "mstrasfrom" to subtract continuum, it is faster because the continuum subtraction is done in visibility.

I guess a self-calibration may help to improve the subtraction and to minimize the residual of continuum.

@pyhsiehATalma
Copy link
Contributor

pyhsiehATalma commented Jun 20, 2023 •
edited
Loading

I am looking into the continuum subtraction of Sgr A* (SPW 35 here). We have two EB, I started from X57d2.
ms1='uid___A002_Xf9226e_X57d2_target.ms'
ms2='uid___A002_Xf9226e_X4fe0_target.ms'

  1. First I split Sgr A* fields.
    listobs(vis=ms1,outfile='12m.log')
  2. writing ds9 region file of fields.
    cat 12m.log | grep "none" | awk '{print $1,$4,$5}' | sed 's/([0-9]):([0-9]):([0-9.]*) /\1:\2:\3 /' | sed 's/([0-9][0-9]).([0-9][0-9]).([0-9][0-9])./\1:\2:\3./' | awk '{printf("circle(%s,%s,30\42) # text={%s}\n",$2,$3,$1)}' > sgra.reg

field id covering Sgr A*: 8, 13, 14, 21
splitting Sgr A* fields to work on continuum usb

split(vis=ms1,
outputvis=ms1+'_field_sgra',
datacolumn='data',
field= '8,13,14,21')

ds9

  1. checking the "line-free" region for continuum subtraction in pipeline log, which is,

35:99.8629877786-99.9953119974GHz;100.2082026224-100.3087885599GHz

  1. marking these regions with rectangles on the plotms figure

plotms

  1. I guess the regions are too narrow for continuum subtraction. The solution is likely an extraplation. We should try wider range for subtraction. I will work on it and make an image to see it we could improve the continuum subtraction.

@keflavich
Copy link
Contributor Author

I moved the files from the upload directory to /orange/adamginsburg/ACES/rawdata/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X19c/reclean

@d-l-walker
Copy link
Contributor

Thanks @keflavich. The only outstanding item now is SPW 35, which @pyhsiehATalma is working on.

@keflavich keflavich mentioned this issue Jun 28, 2023
Open
@keflavich keflavich mentioned this issue Nov 14, 2023
Open
@keflavich
Copy link
Contributor Author

spw33 is still broken in several channels:
image
but I wonder if that's because I failed to delete the old, bad data?

@keflavich
Copy link
Contributor Author

Yep, that's it. Dan's reclean version is good, the version in the main image directory is bad.

@d-l-walker
Copy link
Contributor

This SB still needs to be updated to undo size mitigation (see #179). Note: the latest products are NOT size mitigated. I just need to update the JSON file, since I did the re-imaging locally.

@pyhsiehATalma pyhsiehATalma mentioned this issue Jan 24, 2024
Open
@ashleythomasbarnes
Copy link
Collaborator

Divergence at channel 638 @d-l-walker - could you check?

@keflavich
Copy link
Contributor Author

Confirm divergence remains, but it's much lighter than before
image

(recall this is Sgr A*'s window, so there's also muck around Sgr A*)

@d-l-walker
Copy link
Contributor

Thanks @keflavich. This was already done with cyclefactor=3.0. I can bump it up again, but it'll take a while (:

@pyhsiehATalma
Copy link
Contributor

thanks @d-l-walker @keflavich. I was also testing to subtract the residual continuum with the narrow band CS cube (-300 km/s to 300 km/s). Left images are after subtraction and middle images are before subtraction. The fitting order is 1, and it works well. I am also surprised to see that some artifacts (which look like sidelobes from Sgr A*) were removed after the continuum subtraction. I thought 'imcontsub' just scales the image with the continuum level in the image base. I don't understand why these artifacts from Sgr A* could be removed in the image base.

截圖 2024-01-25 13 39 34
截圖 2024-01-25 13 43 45
截圖 2024-01-25 13 44 14

@ashleythomasbarnes
Copy link
Collaborator

@pyhsiehATalma - could you confirm which cube you're using here?

@keflavich
Copy link
Contributor Author

cyclefactor to 3.5 in 0051bbc#diff-f2ff6354caa768635d4300c5a481a90d1c95f24abf94f9a7724ba94d1cc770e4R2759

@pyhsiehATalma
Copy link
Contributor

@pyhsiehATalma - could you confirm which cube you're using here?

Hi @ashleythomasbarnes for the time being, i am using cubes under,

/data/2021.1.00172.L/science_goal.uid___A001_X1590_X30a8/group.uid___A001_X1590_X30a9/member.uid___A001_X15a0_X19c/reclean/

@keflavich
Copy link
Contributor Author

cyclefactor 3.5 fixed the divergence problem, apparently. There are now other problems with the spw33 cube....

@keflavich
Copy link
Contributor Author

image

@betacygni
Copy link

QA - Line contamination in continuum images from high/low frequencies

New image for continuum of spw33_35 is missing in Globus

@d-l-walker
Copy link
Contributor

@keflavich the statcont cubes for this field are all (except SPW 33) size mitigated.

The un-mitigated products do exist in /member.uid___A001_X15a0_X19c/reclean/. I uploaded them a while ago, but I guess they weren't transferred to the main working directory.

I think we just need to remove the cubes for SPWs 25,27,29,31,35, convert all of the relevant recleaned cube files to .fits (I uploaded the CASA images only apparently), and run statcont on the .pbcor cubes. SPW 33 should be ignored, as my cube still had divergence, which you have since fixed.

d-l-walker added a commit to d-l-walker/reduction_ACES that referenced this issue Apr 15, 2024
@d-l-walker
undo size mitigation for aq (ACES-CMZ#140)
2f2e452
@d-l-walker d-l-walker mentioned this issue Apr 15, 2024
Merged
keflavich pushed a commit that referenced this issue Apr 15, 2024
@d-l-walker
undo size mitigation for aq (#140) (#420)
9921a3d
@keflavich
Copy link
Contributor Author

Moved files: mv *spw{25,27,29,31,35}* sizemitigated/. Then, linked reclean files: ln -vs ../../reclean/* .

Statcont run is started, files should show up imminently.

@keflavich
Copy link
Contributor Author

We have redundant cubes now. reclean files are s3. Need to check which is better, if either.

@keflavich
Copy link
Contributor Author

I moved *s38_0*spw33* to a subdirectory redundant/

keflavich pushed a commit to keflavich/reduction_ACES that referenced this issue Jun 1, 2024
@d-l-walker @keflavich
undo size mitigation for aq (ACES-CMZ#140) (ACES-CMZ#420)
7cacad1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@d-l-walker d-l-walker

Labels
Delivered EB Execution Block Needs Reimaging: Divergence Needs reimaging b/c of divergence Needs Reimaging: Spectral Resolution size mitig Needs to be reimaged with full spectral resolution TM1
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@keflavich @d-l-walker @betacygni @ashleythomasbarnes @pyhsiehATalma