BUG: SDSS KeyNotFoundException for query_region

[fmannucci]

Hi, I am having the same problem with:
astroquery 0.4.7.d
python 3.10.9
numpy 1.23.5.

Using (for example) the example in the doc, as any other call:

from astropy.coordinates import SkyCoord
from astroquery.sdss import SDSS

pos = SkyCoord('0h8m05.63s +14d50m23.3s', frame='icrs')
xid = SDSS.query_region(pos, spectro=True)


Traceback (most recent call last):

  File "/var/folders/m2/32fmbv0s3hg46rj5ccx2r5g40000gn/T/ipykernel_49747/284473386.py", line 2, in <module>
    xid = SDSS.query_region(pos, spectro=True)

  File "/Users/filippo/opt/anaconda3/envs/Py310/lib/python3.10/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
    return self.fn(obj, *args, **kwds)

  File "/Users/filippo/opt/anaconda3/envs/Py310/lib/python3.10/site-packages/astroquery/utils/process_asyncs.py", line 29, in newmethod
    result = self._parse_result(response, verbose=verbose)

  File "/Users/filippo/opt/anaconda3/envs/Py310/lib/python3.10/site-packages/astroquery/sdss/core.py", line 905, in _parse_result
    arr = np.atleast_1d(np.genfromtxt(io.BytesIO(response.content),

  File "/Users/filippo/opt/anaconda3/envs/Py310/lib/python3.10/site-packages/numpy/lib/npyio.py", line 2291, in genfromtxt
    raise ValueError(errmsg)

ValueError: Some errors were detected !
    Line #3 (got 3 columns instead of 1)
    Line #5 (got 3 columns instead of 1)
    Line #6 (got 4 columns instead of 1)
    Line #8 (got 4 columns instead of 1)
    Line #12 (got 4 columns instead of 1)
    Line #15 (got 6 columns instead of 1)
    Line #17 (got 4 columns instead of 1)
    Line #21 (got 3 columns instead of 1)
    Line #31 (got 3 columns instead of 1)

It works fine on another machine running python 3.8 and astroquery 0.4.3

[bsipocz]

cc @weaverba137

With a quick look I see this in the response, but haven't yet got time to dive deeper:

ipdb> response.text
"System.Collections.Generic.KeyNotFoundException: The given key 'searchtool' was not present in the dictionary.\r\n   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)\r\n   at SkyserverCore.Common.ResponseREST.ProcessRequestREST(HttpRequest Request, HttpResponse Response, Boolean isSyntax) in D:\\ManuWork\\skyserver-core\\SkyserverCore\\Common\\ResponseREST.cs:line 86\r\n   at SkyserverCore.Controllers.SearchToolsController.X_Results() in D:\\ManuWork\\skyserver-core\\SkyserverCore\\Controllers\\SearchToolsController.cs:line 49\r\n   at lambda_method(Closure , Object , Object[] )\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeInnerFilterAsync()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResourceFilter>g__Awaited|24_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResourceExecutedContextSealed context)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeFilterPipelineAsync()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)\r\n   at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)\r\n   at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Session.SessionMiddleware.Invoke(HttpContext context)\r\n   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)\r\n\r\nHEADERS\r\n=======\r\nConnection: Keep-Alive\r\nAccept: */*\r\nAccept-Encoding: gzip, deflate, br\r\nHost: skyserver.sdss.org\r\nUser-Agent: astroquery/0.4.7.dev8454 python-requests/2.28.1\r\nX-Forwarded-Proto: https\r\nX-Forwarded-Protocol: https\r\nX-Forwarded-Port: 443\r\nX-Forwarded-For: 73.109.45.152\r\nMS-ASPNETCORE-TOKEN: 001b9d6a-c083-4859-9037-b2b64008b21c\r\nX-Original-For: 127.0.0.1:46435\r\nX-Original-Proto: http\r\n"

[bsipocz]

This maybe related to (or even a duplicate of) #2212

[astro-fermi]

I am getting the exactly same error. All commands were working fine last week.

from astroquery.sdss import SDSS
from astropy import coordinates

pos = coordinates.SkyCoord(15.03034u.deg,-0.538486u.deg,frame='fk5')
xid = SDSS.query_region(pos, spectro=True)

Traceback (most recent call last):
File "", line 1, in 
File "/opt/miniconda3/lib/python3.8/site-packages/astroquery/utils/class_or_instance.py", line 25, in f
return self.fn(obj, *args, **kwds)
File "/opt/miniconda3/lib/python3.8/site-packages/astroquery/utils/process_asyncs.py", line 29, in newmethod
result = self._parse_result(response, verbose=verbose)
File "/opt/miniconda3/lib/python3.8/site-packages/astroquery/sdss/core.py", line 905, in _parse_result
arr = np.atleast_1d(np.genfromtxt(io.BytesIO(response.content),
File "/opt/miniconda3/lib/python3.8/site-packages/numpy/lib/npyio.py", line 2291, in genfromtxt
raise ValueError(errmsg)
ValueError: Some errors were detected !
Line #3 (got 3 columns instead of 1)
Line #5 (got 3 columns instead of 1)
Line #6 (got 4 columns instead of 1)
Line #8 (got 4 columns instead of 1)
Line #12 (got 4 columns instead of 1)
Line #15 (got 6 columns instead of 1)
Line #17 (got 4 columns instead of 1)
Line #21 (got 3 columns instead of 1)
Line #31 (got 2 columns instead of 1)

[weaverba137]

This sounds like something that should be reported to the server maintainers.

[fmannucci]

thanks. Actually, it works fine if I make the same request from another machine running python 3.8 and astroquery 0.4.3, I guess it does not depend on the server.

[bsipocz]

Yes, the query payload has changed since that older version, and the complaint above is about a missing searchtool field, which seems to be present. So not exactly clear yet whether it's on the server or astroquery side. Nevertheless, we started to see failing tests, too which points to the direction that something has changed upstream.

in 0.4.3.:

{'cmd': 'SELECT DISTINCT p.ra, p.dec, p.objid, p.run, p.rerun, p.camcol, p.field, s.z, s.plate, s.mjd, s.fiberID, s.specobjid, s.run2d, s.instrument FROM PhotoObjAll AS p  JOIN SpecObjAll s ON p.objID = s.bestObjID  WHERE ((p.ra between 15.0298 and 15.0309) and (p.dec between -0.539042 and -0.53793))',
 'format': 'csv',
 'searchtool': 'SQL'}

in current dev

{'format': 'csv',
'uquery': 'SELECT\r\np.ra, p.dec, p.objid, p.run, p.rerun, p.camcol, p.field, s.z, s.plate, s.mjd, s.fiberID, s.specobjid, s.run2d FROM #upload u JOIN #x x ON x.up_id = u.up_id JOIN PhotoObjAll AS p ON p.objID = x.objID JOIN SpecObjAll AS s ON p.objID = s.bestObjID ORDER BY x.up_id',
'searchtool': 'CrossID',
'radius': 0.03333333333333333,
'firstcol': 0,
'photoScope': 'allObj',
'photoUpType': 'ra-dec',
'searchType': 'photo'}

[weaverba137]

To help narrow this down, could someone test the dev version with release=16?

[bsipocz]

The payload is the same, the response is a bit different though:

...
 <div class="content">
     <h2>An error has occurred in Skyserver</h2>
    
    <div style="color: yellow; font-weight: bold">
        Exception of type &#39;System.Web.HttpUnhandledException&#39; was thrown.: Select proper tool
    </div>
...

Also, I have noticed that the response is in fact a 500 with dr17 (it's a 200 with dr16), so we probably want to do a raise on status as well, at least before sending it to parsing (after all, all these table parsing issues are red herrings and misleading to users, they'll see the same misleading error for all the unrelated cases where the response is not the expected table but some html explanation.

[bsipocz]

As to narrow it down, I would start checking the test suite, those used to work with dr17, too, but now running some issues.

[pllim]

Looks like payload changes are from #2477 . Maybe the cone search way of things broke. Did it ever work?

[bsipocz]

Of course it worked!

As said above, investigation is ongoing about what has changed upstream.

[weaverba137]

I don't have time to test this today, but I would recommend trying again with the exact same code, but explicitly setting astroquery.sdss.conf.skyserver_baseurl = 'https://skyserver.sdss.org'. Note that the code currently has http.

This may have been a temporary issue on the SDSS side with their load-balancer and https redirect, but it really should be changed to https anyway.

[bsipocz]

Hmm, changing the baseurl indeed helps with the remote tests, I'll open a quick PR for it momentarily and see whether there are still any outstanding issues after that.

[weaverba137]

Caution: https might not work for every data release, especially prior to dr12. Testing should reveal that though.

[bsipocz]

Yeap, testing is all clean, so if this was all, it was more painless than I expected. I'll have a more careful look at the coverage reports, too:

astroquery/sdss/tests/test_sdss.py ............................................................................. [ 16%]
................................................................................................................ [ 39%]
................................................................................................................ [ 62%]
................................................................................................................ [ 86%]
......................                                                                                           [ 91%]
astroquery/sdss/tests/test_sdss_remote.py ....................xx....................                             [ 99%]
docs/sdss/sdss.rst .