[ENH] Notebook testing performance

[jdkent]

Closes #None

Changes proposed in this pull request:

• tests the false positive rate for ale, kda, and mkda
• correlate p-values from ale, kda, and mkda to see which methods are more correlated with each other

[codecov]

Codecov Report

Merging #383 (5751b49) into master (8dc6c55) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #383   +/-   ##
=======================================
  Coverage   82.32%   82.32%           
=======================================
  Files          40       40           
  Lines        3848     3848           
=======================================
  Hits         3168     3168           
  Misses        680      680           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8dc6c55...5751b49. Read the comment docs.

[tsalo]

I think the current differences are probably due to the null method used. The default for ALE and MKDA should be analytical, while the default for KDA is still the empirical null.

Regarding the example itself, can you make the notebook render in the example gallery?

[jdkent]

I messed up some commits, I'll get it fixed up:
some updates on the notebook

observations:

• empirical and analytic null methods correlate strongly with each other (~0.99)
• the ale analytic null method results in slightly more false positives than expected.
• both mkda null methods result in a conservative false positive rate
• the ale analytic null method is not evenly distributed in p-values from [0-1] (there appears to be bias towards certain p-values).

if these two lines

step_size = 0.0001
hist_bins = np.round(np.arange(0, max_poss_ale + 0.001, step_size), 4)

are changed to:

step_size = 0.00001
hist_bins = np.round(np.arange(0, max_poss_ale + 0.001, step_size), 5)

the bias disappears in my simulations (suggesting a certain granularity of the null distribution is necessary)

[tsalo]

It seems like the MKDA analytic and empirical results are exactly the same... which is rather impressive if there are no bugs.

[jdkent]

they appear really close, ALE does better when:

step_size = 0.00001
hist_bins = np.round(np.arange(0, max_poss_ale + 0.001, step_size), 5)

the ale analytic & empirical correlation is: 0.9999446144830378

Here's the false positive rates:

the p-value distribution:

p-value correlation:

So increasing the granularity of the hist_bins by 10x for ALE appears to bring the analytic and empirical methods really close to each other (but this may bite into the performance gains that Tal has been working on)

[tsalo]

We can profile the difference, but I don't think it'll be that big a deal. I think your newest results are a pretty compelling argument for 100000 bins.

[tsalo]

Regarding the example gallery, I think that we will want to use nbexamples.

Regarding merging, I think we should get #393 merged first, then an adapted version of #378, and then re-run your notebook with all three estimators and the new structure.

[tsalo]

Can you also measure and plot elapsed time in this notebook?

[tsalo]

I think the remaining elements that need to be added before we merge this are:

• Test KDA analytic/empirical nulls
• Track and plot elapsed time for each of the methods Dropped for now.
• Render the notebook in our examples gallery using nbexamples Dropped for now.

@jdkent is there anything else you think this PR needs?

[tsalo]

I spent some time trying to figure out how to use nbexamples and if there was a way to just render pre-run notebooks with sphinx-gallery, but I struck out on both counts. I don't think it's worth holding this up for that, so I think we should move forward with the KDA kernel and updating w.r.t. master.

[tsalo]

I wonder if we should place full-on analyses, like this notebook, in a separate repository. It's useful for NiMARE users, so we should definitely link to it in the documentation, but it's not exactly an "example" and it's also very computationally intensive to run.

[jdkent]

I was recently thinking the same thing, I'll create a separate repo

[tsalo]

Let me know when you're ready to work on the new repo. I'm happy to contribute if you want any help.