Convert example notebooks to Markdown files #243

bkktimber · 2022-06-30T14:13:09Z

Convert all notebooks in examples to Markdown files (#128). Could you review this PR? @rlouf

A few important guidelines and requirements before we can merge your PR:

We should be able to understand what the PR does from its title only;
There is a high-level description of the changes;
There are links to all the relevant issues, discussions and PRs;
The branch is rebased on the latest main commit;
Commit messages follow these guidelines;
pre-commit is installed and configured on your machine, and you ran it before opening the PR;
There are tests covering the changes;
The doc is up-to-date;
If I add a sampler I added/updated related examples

Consider opening a Draft PR if your work is still in progress but you would like some feedback from other contributors.

codecov · 2022-06-30T14:30:39Z

Codecov Report

Merging #243 (9d69352) into main (1581424) will not change coverage.
The diff coverage is n/a.

❗ Current head 9d69352 differs from pull request most recent head 7e5738b. Consider uploading reports for the commit 7e5738b to get more accurate results

@@           Coverage Diff           @@
##             main     #243   +/-   ##
=======================================
  Coverage   98.63%   98.63%           
=======================================
  Files          43       43           
  Lines        1757     1757           
=======================================
  Hits         1733     1733           
  Misses         24       24

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

rlouf · 2022-06-30T14:49:17Z

examples/GP_EllipticalSliceSampler.md

+```python
+
+```


rlouf · 2022-06-30T14:50:00Z

examples/Introduction.md

+states.position["loc"].block_until_ready()
+```
+
+In this example the result is a dictionnary and each entry has shape `(num_samples, num_chains)`. Here's how to access the samples of the second chains for `loc`:


It looks like there are some lines missing here?

rlouf · 2022-06-30T14:50:48Z

examples/Pathfinder.md

+
+
+* L-BFGS algorithm struggles with float32s and log-likelihood functions; it's suggested to use double precision numbers. In order to do that in `jax` a configuration variable needs to be set up at initialization time (see [here](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#double-64bit-precision))
+


The linebreaks are not necessary

rlouf · 2022-06-30T14:51:36Z

examples/TemperedSMC.md

+    name: python3
+---
+
+<!-- #region pycharm={"name": "#%% md\n"} -->


Please remove all the  stuff added by PyCharm

many things added by PyCharm in this file, please remove all of them

rlouf · 2022-06-30T14:51:52Z

examples/TemperedSMC.md

+
+We plot the resulting tempered density for 5 different values of $\lambda_k$: from $\lambda_k =1$ which correponds to the original density to $\lambda_k=0$. The lower the value of $\lambda_k$ the easier it is to sampler from the posterior log-density.
+
+```python pycharm={"name": "#%%\n"}


rlouf · 2022-06-30T14:52:01Z

examples/TemperedSMC.md

+
+We first try to sample from the posterior density using an HMC kernel.
+
+```python pycharm={"name": "#%%\n"}


rlouf · 2022-06-30T14:52:07Z

examples/TemperedSMC.md

+ax.legend(list(lambdas))
+```
+
+```python pycharm={"name": "#%%\n"}


rlouf · 2022-06-30T14:53:23Z

examples/use_with_pymc.md

+    name: blackjax
+---
+
+<!-- #region id="397995ab" -->


Please clean the file as the previous one.

rlouf · 2022-06-30T14:53:35Z

examples/use_with_tfp.md

+
+# Use BlackJAX with TFP
+
+<!-- #region -->


rlouf · 2022-06-30T14:54:39Z

This looks great, well done. It looks like some text editors added some unnecessary comments/ids, could you remove them manually please? Next we can work on running the notebooks in CI to include them in the documentation :)

bkktimber · 2022-06-30T15:18:39Z

Sure 😃

bkktimber · 2022-07-02T02:54:05Z

I removed most of the metadata from the markdown files and kept only notebooks' metadata at the top of each file.

I used jupytext.toml file to set jupytext config on my local. Should I include the config file in PR too?

rlouf · 2022-07-02T10:38:29Z

I removed most of the metadata from the markdown files and kept only notebooks' metadata at the top of each file.

Looks much better!

I used jupytext.toml file to set jupytext config on my local. Should I include the config file in PR too?

If it is important for people who clone the repo to have the same experience running / keeping the notebooks in sync with the md files I would say yes.

bkktimber · 2022-07-02T14:33:51Z

I added markdown conversion instruction and updated documentation in docs/examples.rst

rlouf · 2022-07-02T19:25:52Z

Great, Now we need to have the CI run the notebooks and display them with images in the documentation so people can visualize their content browsing the doc.

bkktimber · 2022-07-03T02:46:20Z

I follow this link to add rendering configuration in docs/conf.py. Not sure is there another place I need to change?

rlouf · 2022-07-03T10:36:00Z

I don't know. Does it work when you build the doc locally?

bkktimber · 2022-07-05T15:06:05Z

Just to drop an update. Rendering matplotlib's plots from markdown don't seem to work. I check how Matplotlib and Scipy deal with their docs. they use .py files with matplotlib.sphinxext.plot_directive.

I'll try to follow those repos. Hope I can finish this issue by this weekend.

rlouf · 2022-07-05T19:51:22Z

Did you manage to build the docs without the images at least?

bkktimber · 2022-07-05T23:48:29Z

yes, I did. the docs without images look fine.

rlouf · 2022-07-06T05:00:36Z

Nice! I'll take a look around too for the figures

rlouf · 2022-07-06T09:36:32Z

I am wondering if you can't just convert the .md files to .ipynb in CI and then use nbsphinx to execute and add to the documentation?

bkktimber · 2022-07-06T13:44:54Z

Oh, that works too. I thought we are going to remove all .ipynb for good.

That means we keep examples in Markdown format on git and convert them into notebooks for documentation. is that correct?

rlouf · 2022-07-06T14:15:33Z

That means we keep examples in Markdown format on git and convert them into notebooks for documentation. is that correct?

That sounds a like a good strategy to me. Notebooks are not under version control, results are displayed in the docs, everyone happy!

rlouf · 2022-07-26T14:58:19Z

Any progress on this @bkktimber ?

rlouf · 2022-08-17T22:49:53Z

This looks good! Two things:

Is it possible to not have use_with_pymc, use_with_numpyro and use_with_tfp in the built documentation? This way we can remove a few large dependencies in requirements-docs.py
I don't see anything related to building the documentation in CI, is that normal?

bkktimber · 2022-08-18T15:15:53Z

Is it possible to not have use_with_pymc, use_with_numpyro and use_with_tfp in the built documentation? This way we can remove a few large dependencies in requirements-docs.py

I removed them in the latest commit.

I don't see anything related to building the documentation in CI, is that normal?

Assuming all examples are in Markdown format before committing to the repo, all markdown - jupyter conversions are done in examples/conf.py. I also write steps to prepare example files to build documentation in examples/README

With that assumption, I use make build-docs to build the documentation. I find workflows/build_docs.yml also uses the same commands used in the Makefile. So my understanding is that if building documentation with Makefile works, building documentation in CI with build_docs.yml should work without adding extra steps. Please correct me if I'm wrong.

rlouf · 2022-08-18T16:46:53Z

Looks good. Before merging could you rebase your commits? There should be one commit for converting to .md and another one to build them in CI I believe.

bkktimber · 2022-08-19T00:06:23Z

Got it. I rebased and pushed.

rlouf · 2022-08-24T21:44:29Z

All right this looks fantastic. @junpenglao wdyt?

examples/README.md

rlouf · 2022-08-24T21:47:02Z

examples/README.md

+
+## Converting Notebook Files to Text Files
+
+Please convert your `.ipynb` to `markdown` before commit your example to this repo. You can use the command below:


Suggested change

Please convert your `.ipynb` to `markdown` before commit your example to this repo. You can use the command below:

If you implemented your example in a notebook you can convert your `.ipynb` file to Markdown using the command below:

examples/README.md

rlouf · 2022-08-24T21:48:31Z

examples/README.md

+
+3. Verify the generated documentation in `docs/builds`
+
+### Execution Times


Just saw that, very useful!

bkktimber · 2022-08-25T01:01:49Z

Thank you 😄 Should I commit your suggestions or leave them to you when the PR is merged?

rlouf · 2022-08-25T03:58:13Z

I will apply my suggestions and rebase once @junpenglao has reviewed the changes.

junpenglao

some minor nit, great job!

docs/examples.rst

examples/README.md

rlouf · 2022-09-01T20:59:07Z

@bkktimber It looks like at least one notebook wasn't fully converted and the markdown version is shorter than the notebook version. Could you double-check that every notebook has been fully converted?

bkktimber · 2022-09-02T16:27:55Z

@rlouf not sure what was missing. I double-checked with the document on main and could not find any missing. I uploaded documentation built with this commit here for reference.

Could you suggest the missing part, please?

rlouf · 2022-09-02T19:38:28Z

I was thinking about Introduction.md but it is identical to the original notebook. My bad @bkktimber! At least this PR will make it easy to go over the notebooks and correct the text. I'll rebase the commits if needed and merge, and then we'll just need to look at the docs to see if we can admire your work.

rlouf · 2022-09-02T20:38:04Z

Merging. Great job!

bkktimber · 2022-09-03T03:26:56Z

No worry @rlouf! Let's see how it's doing 😁

rlouf · 2022-09-03T05:52:49Z

A notebook fails in CI, I need to investigate.

rlouf · 2022-09-22T05:53:38Z

Just a quick note to say that I've worked with the documentation recently and it's fantastic, so thank you again!

bkktimber · 2022-09-27T14:24:10Z

It's my pleasure. Glad to know my contribution is helpful.

rlouf reviewed Jun 30, 2022

View reviewed changes

examples/GP_EllipticalSliceSampler.md Outdated

Comment on lines 178 to 180

```python

```

Copy link

Member

rlouf Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Empty cell

rlouf reviewed Jun 30, 2022

View reviewed changes

examples/TemperedSMC.md Outdated

ax.legend(list(lambdas))

```

```python pycharm={"name": "#%%\n"}

Copy link

Member

rlouf Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here too

rlouf reviewed Jun 30, 2022

View reviewed changes

examples/use_with_pymc.md Outdated

name: blackjax

---



Copy link

Member

rlouf Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please clean the file as the previous one.

rlouf reviewed Jun 30, 2022

View reviewed changes

examples/use_with_tfp.md Outdated

# Use BlackJAX with TFP



Copy link

Member

rlouf Jun 30, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same

This was linked to issues Jul 3, 2022

Make sure notebooks run in test suite #68

Closed

Use jupytext-compatible markdown files instead of notebooks? #128

Closed

rlouf linked an issue Jul 3, 2022 that may be closed by this pull request

Included images are not properly displayed in the documentation #160

Closed

bkktimber force-pushed the issues/128 branch from 57f5af6 to 7f26350 Compare July 28, 2022 16:33

bkktimber force-pushed the issues/128 branch from 9d69352 to 15cc1fd Compare August 19, 2022 00:04

rlouf reviewed Aug 24, 2022

View reviewed changes

examples/README.md Outdated Show resolved Hide resolved

rlouf reviewed Aug 24, 2022

View reviewed changes

examples/README.md Outdated Show resolved Hide resolved

rlouf reviewed Aug 24, 2022

View reviewed changes

junpenglao previously approved these changes Aug 26, 2022

View reviewed changes

docs/examples.rst Show resolved Hide resolved

examples/README.md Outdated Show resolved Hide resolved

elanmart mentioned this pull request Aug 28, 2022

pmap seems to drastically improve performance in the example notebook #251

Closed

bkktimber dismissed junpenglao’s stale review via 2ca47bd August 31, 2022 00:23

junpenglao previously approved these changes Aug 31, 2022

View reviewed changes

rlouf dismissed junpenglao’s stale review via 072f7d0 August 31, 2022 06:38

bkktimber and others added 3 commits September 2, 2022 22:03

Convert example notebooks to markdown files

e9b33b4

Render markdown example files in documentation

cfd10c8

Explain how to interact with the examples

7e5738b

rlouf force-pushed the issues/128 branch from dedcd5b to 7e5738b Compare September 2, 2022 20:17

rlouf merged commit bebfb0f into blackjax-devs:main Sep 2, 2022



		* L-BFGS algorithm struggles with float32s and log-likelihood functions; it's suggested to use double precision numbers. In order to do that in `jax` a configuration variable needs to be set up at initialization time (see [here](https://jax.readthedocs.io/en/latest/notebooks/Common_Gotchas_in_JAX.html#double-64bit-precision))


		We plot the resulting tempered density for 5 different values of $\lambda_k$: from $\lambda_k =1$ which correponds to the original density to $\lambda_k=0$. The lower the value of $\lambda_k$ the easier it is to sampler from the posterior log-density.

		```python pycharm={"name": "#%%\n"}


		We first try to sample from the posterior density using an HMC kernel.

		```python pycharm={"name": "#%%\n"}


		## Converting Notebook Files to Text Files

		Please convert your `.ipynb` to `markdown` before commit your example to this repo. You can use the command below:

	Please convert your `.ipynb` to `markdown` before commit your example to this repo. You can use the command below:
	If you implemented your example in a notebook you can convert your `.ipynb` file to Markdown using the command below:


		3. Verify the generated documentation in `docs/builds`

		### Execution Times

Convert example notebooks to Markdown files #243

Convert example notebooks to Markdown files #243

Conversation

bkktimber commented Jun 30, 2022 • edited Loading

codecov bot commented Jun 30, 2022 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rlouf commented Jun 30, 2022

bkktimber commented Jun 30, 2022

bkktimber commented Jul 2, 2022

rlouf commented Jul 2, 2022

bkktimber commented Jul 2, 2022 • edited Loading

rlouf commented Jul 2, 2022

bkktimber commented Jul 3, 2022

rlouf commented Jul 3, 2022

bkktimber commented Jul 5, 2022

rlouf commented Jul 5, 2022

bkktimber commented Jul 5, 2022

rlouf commented Jul 6, 2022

rlouf commented Jul 6, 2022

bkktimber commented Jul 6, 2022

rlouf commented Jul 6, 2022

rlouf commented Jul 26, 2022

rlouf commented Aug 17, 2022

bkktimber commented Aug 18, 2022 • edited Loading

rlouf commented Aug 18, 2022

bkktimber commented Aug 19, 2022

rlouf commented Aug 24, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkktimber commented Aug 25, 2022

rlouf commented Aug 25, 2022

junpenglao left a comment

Choose a reason for hiding this comment

rlouf commented Sep 1, 2022

bkktimber commented Sep 2, 2022

rlouf commented Sep 2, 2022

rlouf commented Sep 2, 2022

bkktimber commented Sep 3, 2022

rlouf commented Sep 3, 2022

rlouf commented Sep 22, 2022

bkktimber commented Sep 27, 2022

bkktimber commented Jun 30, 2022 •

edited

Loading

codecov bot commented Jun 30, 2022 •

edited

Loading

bkktimber commented Jul 2, 2022 •

edited

Loading

bkktimber commented Aug 18, 2022 •

edited

Loading