Asking a syntax of paging

[tuocvu]

Dear All,
I've recently make a query as:
optimade-get https://nomad-lab.eu/prod/rae/optimade/ --use-async --filter 'nelements = 2 AND elements HAS ALL "Sc","H"' --max-results-per-provider 1000 --http-timeout 3000 --output-file bi-H-Sc.json
╭─────────────────────────────────────────────────────────────────────────────────╮
│ Performing query structures/?filter=nelements = 2 AND elements HAS ALL "Sc","H" │
╰─────────────────────────────────────────────────────────────────────────────────╯
Error: Provider 'https://nomad-lab.eu/prod/rae/optimade/' returned: ['RuntimeError: 500 -
https://nomad-lab.eu/prod/rae/optimade/v1/structures?filter=nelements+%3D+2+AND+elements+H
AS+ALL+%22Sc%22%2C%22H%22&page_offset=460: ExtraData: unpack(b) received extra data.']
✓ nomad-lab.eu/prod/rae/optimade/v1/structures ━━━━━━━━━━━━━━━━━━━━━━━━━━━ 460/460 0:00:32

I have encountered 940 structures in the results. However, due to the default page_offset being set to 460, I am unable to retrieve all the data. Your suggestions on how to address this through paging would be greatly appreciated.

Thank you for your assistance.
Tuoc Vu
Hanoi Univ. of Science and Technology

[ml-evs]

Hi @tuocvu, thanks for the question. The optimade-get client will do paging for you, until it runs out of results. What you're seeing in the error message is actually an error on the server side: optimade-get tried to pull the page with offset 460 and the server returned "500 ExtraData: unpack received extra data", probably due to a bad entry somewhere in NOMAD.

You can see the same results in your browser at https://nomad-lab.eu/prod/rae/optimade/v1/structures?filter=nelements+%3D+2+AND+elements+HAS+ALL+%22Sc%22%2C%22H%22&page_offset=460.

In fact if I reduce the page limit to 1 we can see that it is only this structure that is failing:

https://nomad-lab.eu/prod/rae/optimade/v1/structures?filter=nelements+%3D+2+AND+elements+HAS+ALL+%22Sc%22%2C%22H%22&page_offset=468&page_limit=1

The next step would be to contact NOMAD and ask them about the issue, providing them specifics on the failing query and entry. I recommend using the matsci.org forums where they maintain a presence at https://matsci.org/c/nomad/32 (you can link them to this GitHub issue too).