Update filter examples and validate optional cases

[ml-evs]

This PR completes the automation of scraping the spec for examples, since they are now all parseable as of Materials-Consortia/OPTIMADE#263. It adds the optional filters as tests in the validator that are printed with less emphasis, and do not cause the validator to return a non-zero exit code.

[codecov]

Codecov Report

Merging #227 into master will decrease coverage by 0.23%.
The diff coverage is 69.81%.

@@            Coverage Diff             @@
##           master     #227      +/-   ##
==========================================
- Coverage   87.57%   87.33%   -0.24%     
==========================================
  Files          43       43              
  Lines        1916     1943      +27     
==========================================
+ Hits         1678     1697      +19     
- Misses        238      246       +8     

Flag	Coverage Δ
#unittests	87.33% <69.81%> (-0.24%)	⬇️

Impacted Files	Coverage Δ
optimade/validator/validator.py	67.81% <60.97%> (+0.43%)	⬆️
optimade/validator/data/__init__.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16917b2...6715bfc. Read the comment docs.

[CasperWA]

Good work, thanks @ml-evs !

I have some minor suggested changes.
It is generally a very practical code, which is fine by me. But the amount of if-else statements is increasing rapidly, which tells me there may be a better design for the whole thing. However, this is definitely for a later point, when we're all bored and want to do something "fun" 😅

[ml-evs]

Good work, thanks @ml-evs !

I have some minor suggested changes.
It is generally a very practical code, which is fine by me. But the amount of if-else statements is increasing rapidly, which tells me there may be a better design for the whole thing. However, this is definitely for a later point, when we're all bored and want to do something "fun" sweat_smile

Yes, the important thing now to consider is validating the results properly (beyond checking that the response is in the right format)... this has been useful to catch some problems with the filterparser and transformer, but it's pretty meaningless if all we're doing his querying non-existent fields!

I'm not entirely sure how we should go about doing this. We could potentially just find one structure in the API and use that to construct a load of queries that should return at least that structure, and kind of trawl and validate that way. Hopefully this can be guided more by other implementations!

[CasperWA]

Yes, the important thing now to consider is validating the results properly (beyond checking that the response is in the right format)... this has been useful to catch some problems with the filterparser and transformer, but it's pretty meaningless if all we're doing his querying non-existent fields!

True. We need more real-world implementations as testbeds.
I think we can and should use providers.optimade.org as a testbed for the index meta-database, and then I am drawing a blank for the regular server. The best approach is probably still our own setup? Either running a server locally or querying heroku.

I'm not entirely sure how we should go about doing this. We could potentially just find one structure in the API and use that to construct a load of queries that should return at least that structure, and kind of trawl and validate that way. Hopefully this can be guided more by other implementations!

I don't think scraping the spec further will end in a fruitful result. Rather, we should either re-use some queries/model structures from the other OPTIMADE repository tests and/or use real-world implementations and/or have the consortium come up with a list of queries/structures that can be used. The latter would also solidify this validator as the "official" OPTIMADE implementation validator to be used.

I don't know if I misunderstood your comment in answering here, but I think my comments are still valid in their own right 😅

[ml-evs]

I think we can and should use providers.optimade.org as a testbed for the index meta-database, and then I am drawing a blank for the regular server. The best approach is probably still our own setup? Either running a server locally or querying heroku.

100% agree

I don't think scraping the spec further will end in a fruitful result. Rather, we should either re-use some queries/model structures from the other OPTIMADE repository tests and/or use real-world implementations and/or have the consortium come up with a list of queries/structures that can be used. The latter would also solidify this validator as the "official" OPTIMADE implementation validator to be used.

I don't know if I misunderstood your comment in answering here, but I think my comments are still valid in their own right sweat_smile

It's not the point I was making, but I do agree! In order to avoid getting people to put random data into their databases so that it can be queried, my suggestion was more to find any structure in the database (i.e. /structures?page_limit=1), then perform queries that should definitely return that structure, i.e. query for its ID, query for formula, query on its relationships etc. in order to check that the underlying mechanisms are working.

The spec scraping stuff is only really useful to check what the database says it has implemented, and what causes backend errors.

[ml-evs]

We should leave this to an issue discussing the future of the validator though :)

[CasperWA]

(...) In order to avoid getting people to put random data into their databases so that it can be queried, my suggestion was more to find any structure in the database (i.e. /structures?page_limit=1), then perform queries that should definitely return that structure, i.e. query for its ID, query for formula, query on its relationships etc. in order to check that the underlying mechanisms are working.

Ah. It would be a good check, I guess, of the filter working. Especially if using NOT to check a structure is not returned. E.g., if an implementation just disregards the filter and returns all regardless of input.
However, it also seems incomplete and not completely rigorous in its testing method. I think we'll have to come up with a robust algorithm/logic for testing this.

I would even go so far as to say that it's the implementation's own responsibility to ensure their system works properly... However, I definitely see the benefit of having a central "official" tester to make sure it works generally as expected.

[CasperWA]

Very good.
All of the txt files are already added to the MANIFEST in some way, right?

[ml-evs]

All of the txt files are already added to the MANIFEST in some way, right?

Yep!