Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

issue with apero_reject in 0.7.291 #797

Open
larnoldgithub opened this issue Dec 14, 2024 · 8 comments
Open

issue with apero_reject in 0.7.291 #797

larnoldgithub opened this issue Dec 14, 2024 · 8 comments

Comments

@larnoldgithub
Copy link
Collaborator

larnoldgithub commented Dec 14, 2024 •
edited
Loading

apero_reject.py --identifier=3094104,3094105 --test=True --autofill="1,1,1,24BQ13-Nov14_bad_pm_calib"

gives an error

apero_reject.py --identifier=3094104,3094105 --test=True --autofill="1,1,1,24BQ13-Nov14_bad_pm_calib"
21:12:52.445- |REJECT| ***************************************************************************
21:12:52.451- |REJECT| *
21:12:52.451- |REJECT| * SPIROU @PID-00017342107718117170-0W41 (V0.7.291)
21:12:52.452- |REJECT| * py3.9.16 git:v0.7.291-stable-test
21:12:52.452- |REJECT| *
21:12:52.458- |REJECT| ***************************************************************************
21:12:52.463- |REJECT| █████╗ ██████╗ ███████╗██████╗ ██████╗
21:12:52.469- |REJECT| ██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔═══██╗
21:12:52.474- |REJECT| ███████║██████╔╝█████╗ ██████╔╝██║ ██║
21:12:52.480- |REJECT| ██╔══██║██╔═══╝ ██╔══╝ ██╔══██╗██║ ██║
21:12:52.486- |REJECT| ██║ ██║██║ ███████╗██║ ██║╚██████╔╝
21:12:52.491- |REJECT| ╚═╝ ╚═╝╚═╝ ╚══════╝╚═╝ ╚═╝ ╚═════╝
21:12:52.497- |REJECT| ***************************************************************************
21:12:52.506-|REJECT| DRS Setup:
21:12:52.511-|REJECT|
21:12:52.512-|REJECT| DRS_ROOT: /apero/apero-drs-v0.7.290/apero
21:12:52.512-|REJECT| DRS_DATA_RAW: /data/spirou4/arlinks
21:12:52.512-|REJECT| DRS_DATA_REDUC: /data/spirou4/apero-data/offline290/red
21:12:52.512-|REJECT| DRS_DATA_WORKING: /data/spirou4/apero-data/offline290/tmp
21:12:52.512-|REJECT| DRS_CALIB_DB: /data/spirou4/apero-data/offline290/calib
21:12:52.512-|REJECT| DRS_TELLU_DB: /data/spirou4/apero-data/offline290/tellu
21:12:52.513-|REJECT| DRS_DATA_ASSETS: /data/spirou4/apero-data/offline290/asset
21:12:52.513-|REJECT| DRS_DATA_OUT: /data/spirou4/apero-data/offline290/out
21:12:52.513-|REJECT| DRS_DATA_MSG: /data/spirou4/apero-data/offline290/msg
21:12:52.513-|REJECT| DRS_DATA_RUN: /data/spirou4/apero-data/offline290/run
21:12:52.513-|REJECT| DRS_DATA_PLOT: /data/spirou4/apero-data/offline290/plot
21:12:52.513-|REJECT| DRS_DATA_OTHER: /data/spirou4/apero-data/offline290/other
21:12:52.513-|REJECT| LBL_PATH: /data/spirou4/apero-data/offline290/lbl
21:12:52.514-|REJECT| DRS_CONFIG: /apero/config/offline290/user_config.ini
21:12:52.514-|REJECT| DRS_CONFIG: /apero/config/offline290/user_constants.ini
21:12:52.514-|REJECT| DRS_CONFIG: apero.constants.default.default_config
21:12:52.514-|REJECT| DRS_CONFIG: apero.constants.default.default_keywords
21:12:52.514-|REJECT| DRS_CONFIG: config.instruments.spirou.default_config.py
21:12:52.514-|REJECT| DRS_CONFIG: config.instruments.spirou.default_keywords.py
21:12:52.515-|REJECT| DRS_CONFIG: core.instruments.default.default_constants.py
21:12:52.515-|REJECT| DRS_CONFIG: core.instruments.spirou.default_constants.py
21:12:52.515-|REJECT| DATABASE: MYSQL
21:12:52.515-|REJECT| DATABASE-calib: aperooffline290@localhost:calib_offline290_db
21:12:52.515-|REJECT| DATABASE-tellu: aperooffline290@localhost:tellu_offline290_db
21:12:52.515-|REJECT| DATABASE-findex: aperooffline290@localhost:findex_offline290_db
21:12:52.515-|REJECT| DATABASE-log: aperooffline290@localhost:log_offline290_db
21:12:52.516-|REJECT| DATABASE-astrom: aperooffline290@localhost:astrom_offline290_db
21:12:52.516-|REJECT| DATABASE-lang: aperooffline290@localhost:lang_offline290_db
21:12:52.516-|REJECT| DATABASE-reject: aperooffline290@localhost:reject_offline290_db
21:12:52.516-|REJECT| PRINT_LEVEL: all
21:12:52.516-|REJECT| LOG_LEVEL: all
21:12:52.516-|REJECT| DRS_PLOT: 0
21:12:52.522- |REJECT| ***************************************************************************
21:12:52.528-|REJECT| Arguments used:
21:12:52.533-|REJECT|
21:12:52.533-|REJECT| --IDENTIFIER: 3094104,3094105
21:12:52.533-|REJECT| --OBJNAME: None
21:12:52.534-|REJECT| --AUTOFILL: 1,1,1,24BQ13-Nov14_bad_pm_calib
21:12:52.534-|REJECT| --TEST: True
21:12:52.539- |REJECT| ***************************************************************************
21:12:54.163-!!|REJECT| E[01-010-00001]: Unhandled error has occurred: Error <class 'AttributeError'>
21:12:54.163-!!|REJECT|
21:12:54.164-!!|REJECT| Traceback (most recent call last):
21:12:54.164-!!|REJECT| File "/apero/apero-drs-v0.7.290/apero/core/utils/drs_startup.py", line 439, in run
21:12:54.164-!!|REJECT| llmain = func(recipe, params)
21:12:54.164-!!|REJECT| File "/apero/apero-drs-v0.7.290/tools/bin/apero_reject.py", line 114, in main
21:12:54.164-!!|REJECT| drs_reject.add_file_reject(params, recipe, identifier)
21:12:54.165-!!|REJECT| File "/apero/apero-drs-v0.7.290/apero/tools/module/listing/drs_reject.py", line 79, in add_file_reject
21:12:54.165-!!|REJECT| google_sheet = gspd.spread.Spread(sheet_id)
21:12:54.165-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/spread.py", line 126, in init
21:12:54.165-!!|REJECT| self.client = Client(user, config, scope, creds)
21:12:54.165-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/client.py", line 96, in init
21:12:54.165-!!|REJECT| self._root = self._drive_request(file_id="root", params={"fields": "name,id"})
21:12:54.166-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/client.py", line 197, in _drive_request
21:12:54.166-!!|REJECT| res = self.request(method, url, params=params, json=data)
21:12:54.166-!!|REJECT| File "/conda/miniconda3/envs/apero-env-v0.7.290/lib/python3.9/site-packages/gspread_pandas/util.py", line 292, in request
21:12:54.166-!!|REJECT| return ClientV4.request(client, *args, kwargs)
21:12:54.166-!!|REJECT| AttributeError: type object 'Client' has no attribute 'request'
21:12:54.167-!!|REJECT|
21:12:54.255-|REJECT| ***************************************************************************
21:12:54.265-@!|REJECT| W[40-003-00005]: Recipe apero_reject has NOT been successfully completed (2.440
21:12:54.265-@!|REJECT| seconds)
21:12:54.275-**|REJECT| ***************************************************************************

By the way, @neil, the example you gave me earlier this year was using identifier like
identifier=3094104d , so with a letter after the odometer, and this is the way I entered the last bad odomters for SPIRou.
Is this correct ?

Please let me know

I hope the apero_reject tool will work again soon.

apero_astrometrics.py seems to work.

Thanks
Luc
@njcuk9999
Copy link
Owner

This is the same error as you've had before, its with gspread and gspread-pandas

21:12:54.166-!!|REJECT| return ClientV4.request(client, *args, kwargs)
21:12:54.166-!!|REJECT| AttributeError: type object 'Client' has no attribute 'request'

You should note down this error as you seem to get it a lot!

Can you please:

  1. Check you are in the correct environment
  2. Check the versions of gspread and gspread-pandas

They are both defined in the requirements file now so they should be exactly the same as that:

gspread-pandas==3.2.2
gspread==5.12.4

If they are not it means you:

  1. are not in the correct environment that did the pip install -r requirements
  2. have installed something with pip after the requirements file (which has updated these modules)

Otherwise I've no idea whats going on.

As far as I can tell now that we have both these requirements this isn't an apero issue.

The reject script (for SPIROU) uses the FILENAME header key so the identifier should be exactly the same as that key to be rejected.

i.e.

FILENAME= '2503707d'           / filename anticipated by fitspipe server

This actually means that most of the bad list is incorrect for SPIRou and I never realised...
I'm not sure how to fix this but here is the list and the "SPIROU" tab (note for NIRPS we use the filename itself)
https://docs.google.com/spreadsheets/d/1gvMp1nHmEcKCUpxsTxkx-5m115mLuQIGHhxJCyVoZCM/edit?usp=sharing

@larnoldgithub
Copy link
Collaborator Author

I had seen it was again with gspread but thought it has been corrected with the 291 upgrade of the tools.

a 'pip freeze' on my installation says
gspread==6.1.3
gspread-pandas==3.2.2

so I need to downgrade gspread.

I have another installation I'm using for tests running gspread==5.12.4

@larnoldgithub
Copy link
Collaborator Author

regarding the issue with FILENAME in the reject list, at least all odometers with 'OBJECT' string in the comment G column are o.fits files.
All other are calibrations (CALIB or calib) but unfortunately I don't know when it's d,f,a or c.fits.
I could re-run the script generating the full list, but it may take some time.

@larnoldgithub
Copy link
Collaborator Author

worth noting that the amount of bad calibration is not so large I believe.

@larnoldgithub
Copy link
Collaborator Author

ok @njcuk9999
Chris only did a:
pip install gspread==5.12.4
after going into the correct env.
I tested with adding a new target with apero_astrometrics, and adding a new bad odometer (actuall FILENAME) and it worked.

@njcuk9999
Copy link
Owner

Do we have any idea why gspread was the wrong version?

Re: the reject list - I'll have a think about what I can do maybe there is a way to use it as a prefix instead of an exact match but it will be a lot slower!

@larnoldgithub
Copy link
Collaborator Author

gspread was the wrong version on my quicklook and offline version, because they were both installed from the .290 stable test in sep or oct. I had the gspread corrected on a third setup which I'm using for tests and other things (like astrometrics and reject); I thought when upgrading quicklook and offline to new .291 source last week with updated tools it would had solved the issue with gspread (it did at least when I verified Gl699 was correctly identified as an existng obj with apero_astrometrics) but actually Chris and I only realised yesterday that the py packages had to be updated too. Chris just downgraded gspread.

@larnoldgithub
Copy link
Collaborator Author

For the reject list, I guess you would have to query the db for each odometer to get the DPRTYPE and thus build the corresponding filename ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@njcuk9999 @larnoldgithub