The GA4GH API needs a compliance test suite

[cassiedoll]

Main goal: Given an API endpoint, output which of the API methods are passing, failing, or unimplemented.

Bonus:

• add failure reasons/suggested fixes for the non-passing methods
• output an overall score
• keep track of passing implementations somewhere so that new users can easily find implementors
• periodically re-run the tests to make sure regressions don't occur

This would be lovely as another repository alongside schemas.

[pgrosu]

+1 Agreed, though it would be easier start with a smaller scope in order to make it easier to manage, by using "standardized" test data with the expected outputs. These would be provided by us and later augmented by the community at large. Definitely, having it comprehensive to the required data-sets and edge-cases as posted in @114 will make it easier to communicate as to the state of the API. Scripts to push these unit tests results into a comprehensive summary matrix will be something nice to have later on.

[cassiedoll]

I'm working on a possible impl for this - will send demo link soon!

[pgrosu]

Cool! :)

[cassiedoll]

I have a first demo in my github staging area:
http://cassiedoll.github.io/compliance-staging.html

The default endpoint is Google, but this should work against any other API provider which supports CORS.

The code itself is a small amount of js + html. I'm cleaning things up to try and make writing new tests as easy as possible. (right now it's testing the v0.1 APIs and the v0.5 ones are all todos - because there isn't a v0.5 impl to test against)

I'll send out a proper PR against a new repo (maybe ga4gh/compliance ?) in the next day or so.

Comments on the demo are very welcome!
And if anybody out there has a wip v0.5 impl, I'd love to test your API :)

[pgrosu]

Hi Cassie,

This looks really nice and I can see that a lot of work went into it. It's definitely much more fun to work with than the QA tests I used to run and had to generate reports for :)

Look forward to the new details in the PR.

Thanks,
Paul

[skeenan]

Awesome work Cassie!!

EBI and NCBI score 29/57

On 19 August 2014 20:29, cassiedoll [email protected] wrote:

I have a first demo in my github staging area:
http://cassiedoll.github.io/compliance-staging.html

The default endpoint is Google, but this should work against any other API
provider which supports CORS http://enable-cors.org/.

The code itself is a small amount of js + html. I'm cleaning things up to
try and make writing new tests as easy as possible. (right now it's testing
the v0.1 APIs and the v0.5 ones are all todos - because there isn't a v0.5
impl to test against)

I'll send out a proper PR against a new repo (maybe ga4gh/compliance ?) in
the next day or so.

Comments on the demo are very welcome!
And if anybody out there has a wip v0.5 impl, I'd love to test your API :)

—
Reply to this email directly or view it on GitHub
#113 (comment).

[cassiedoll]

Thanks guys :) PR is coming in just a second.

@skeenan - NCBI and EBI actually both score a 0 in my latest code, because they don't support CORS atm. They are both just missing a simple header in their responses: http://enable-cors.org/server.html

I'm hoping both backends can add that header - cause it makes it so much easier to write example integrations (and of course tests :) If they can't though, we can come up with workarounds.

[cassiedoll]

Please take a look: ga4gh/compliance#1

[skeenan]

@cassiedoll in my naive test using the EBI and NCBI endpoints
'something' seemed to run.

EBI endpoint = http://193.62.52.16

this API scores
29 out of 57 points

NCBI endpoint = http://trace.ncbi.nlm.nih.gov/Traces/gg

this API scores
29 out of 57 points

On 20 August 2014 01:15, cassiedoll [email protected] wrote:

Thanks guys :) PR is coming in just a second.

@skeenan - NCBI and EBI actually both score a 0 in my latest code, because
they don't support CORS atm. They are both just missing a simple header in
their responses: http://enable-cors.org/server.html

I'm hoping both backends can add that header - cause it makes it so much
easier to write example integrations (and of course tests :) If they can't
though, we can come up with workarounds.

—
Reply to this email directly or view it on GitHub.

[cassiedoll]

@skeenan - that's the old version of the code, which wasn't doing the right status checking. the new tests are our of 68 points and have been refreshed here: http://cassiedoll.github.io/compliance-staging.html

[skeenan]

Great thanks. All behaving as expected now.

On 20 August 2014 16:34, cassiedoll [email protected] wrote:

@skeenan https://github.com/skeenan - that's the old version of the
code, which wasn't doing the right status checking. the new tests are our
of 68 points and have been refreshed here:
http://cassiedoll.github.io/compliance-staging.html

—
Reply to this email directly or view it on GitHub
#113 (comment).

[pgrosu]

Hi Cassie,

This is awesome! I was wondering if we could have a pull-down menu for the API endpoint and Dataset ID, which would appropriately populate the fields. People could still type in their own preference for both, but this would provide users validity against some common endpoints and IDs to test.

Also would it be possible to have a link for each test to see the detail on what it fetched, the results and where it succeeded/failed.

Later we can have a link at the bottom that provides a report for all the tests performed and where it succeeded/failed.

It would be nice to also have an About link (or Readme file) describing some of the details which get performed. This could include things such as the scoring algorithm and how each component of the API is tested among other things. It would help with providing some transparency to its implementation, in case others might want to add more tests or pieces of information.

Otherwise this is super!

Many thanks :)
Paul

[cassiedoll]

I think many of your issues are already addressed:

1. There are 3 links in gray (ebi, google, ncbi) - which will directly populate those 3 backends and run the tests
2. If you click on one of the tan boxes, it will expand and tell you exactly what passed and failed. Click the 'debug json' link to see the actual json response from the server.

Hope that helps!

[pgrosu]

Oh, wow! That's just too cool!

Many thanks and superb work!
Paul

[cassiedoll]

Marking this closed as the first commit is in.
All other compliance discussion should now happen in that repo: https://github.com/ga4gh/compliance