To avoid pointless conflicts while working with jupyter notebooks (with different execution counts or cell metadata), it is recommended to clean the notebooks before committing anything (done automatically if you install the git hooks with nbdev_install_git_hooks). The following functions are used to do that.
tst = {'cell_type': 'code',
+ 'execution_count': 26,
+ 'metadata': {'hide_input': True, 'meta': 23},
+ 'outputs': [{'execution_count': 2,
+ 'data': {
+ 'application/vnd.google.colaboratory.intrinsic+json': {
+ 'type': 'string'},
+ 'plain/text': ['sample output',]
+ },
+ 'output': 'super'}],
+
+ 'source': 'awesome_code'}
+tst1 = tst.copy()
+
+clean_cell(tst)
+test_eq(tst, {'cell_type': 'code',
+ 'execution_count': None,
+ 'metadata': {'hide_input': True},
+ 'outputs': [{'execution_count': None,
+ 'data': {'plain/text': ['sample output',]},
+ 'output': 'super'}],
+ 'source': 'awesome_code'})
+
+clean_cell(tst1, clear_all=True)
+test_eq(tst1, {'cell_type': 'code',
+ 'execution_count': None,
+ 'metadata': {},
+ 'outputs': [],
+ 'source': 'awesome_code'})
+tst2 = {
+ 'metadata': {'tags':[]},
+ 'outputs': [{
+ 'metadata': {
+ 'tags':[]
+ }}],
+
+ "source": [
+ ""
+ ]}
+clean_cell(tst2, clear_all=False)
+test_eq(tst2, {
+ 'metadata': {},
+ 'outputs': [{
+ 'metadata':{}}],
+ 'source': []})
+tst = {'cell_type': 'code',
+ 'execution_count': 26,
+ 'metadata': {'hide_input': True, 'meta': 23},
+ 'outputs': [{'execution_count': 2,
+ 'data': {
+ 'application/vnd.google.colaboratory.intrinsic+json': {
+ 'type': 'string'},
+ 'plain/text': ['sample output',]
+ },
+ 'output': 'super'}],
+ 'source': 'awesome_code'}
+nb = {'metadata': {'kernelspec': 'some_spec', 'jekyll': 'some_meta', 'meta': 37},
+ 'cells': [tst]}
+
+clean_nb(nb)
+test_eq(nb['cells'][0], {'cell_type': 'code',
+ 'execution_count': None,
+ 'metadata': {'hide_input': True},
+ 'outputs': [{'execution_count': None,
+ 'data': { 'plain/text': ['sample output',]},
+ 'output': 'super'}],
+ 'source': 'awesome_code'})
+test_eq(nb['metadata'], {'kernelspec': 'some_spec', 'jekyll': 'some_meta'})
+assert clean_cr(fr'a{BSLASH}r\nb{BSLASH}rc\n') == fr'a\nb\nc\n'
+By default (fname left to None), the all the notebooks in lib_folder are cleaned. You can opt in to fully clean the notebook by removing every bit of metadata and the cell outputs by passing clear_all=True. disp is only used for internal use with git hooks and will print the clean notebook instead of saving it. Same for read_input_stream that will read the notebook from the input stream instead of the file names.
nbdev comes with the following commands. To use any of them, you must be in one of the subfolders of your project: they will search for the settings.ini recursively in the parent directory but need to access it to be able to work. Their names all begin with nbdev so you can easily get a list with tab completion.
chisel_nbdev_build_docs builds the documentation from the notebookschisel_nbdev_build_lib builds the library from the notebookschisel_nbdev_bump_version increments version in settings.py by onechisel_nbdev_clean_nbs removes all superfluous metadata form the notebooks, to avoid merge conflictschisel_nbdev_detach exports cell attachments to dest and updates referenceschisel_nbdev_diff_nbs gives you the diff between the notebooks and the exported librarychisel_nbdev_fix_merge will fix merge conflicts in a notebook filechisel_nbdev_install_git_hooks installs the git hooks that use the last two command automatically on each commit/mergechisel_nbdev_nb2md converts a notebook to a markdown filechisel_nbdev_new creates a new nbdev projectchisel_nbdev_read_nbs reads all notebooks to make sure none are brokenchisel_nbdev_test_nbs runs tests in notebookschisel_nbdev_trust_nbs trusts all notebooks (so that the HTML content is shown)chisel_nbdev_update_lib propagates any change in the library back to the notebooksBy default (fname left to None), the whole library is built from the notebooks in the lib_folder set in your settings.ini.
By default (fname left to None), the whole library is treated. Note that this tool is only designed for small changes such as typo or small bug fixes. You can't add new cells in notebook from the library.
By default (fname left to None), the whole library is tested from the notebooks in the lib_folder set in your settings.ini.
By default (fname left to None), the whole documentation is build from the notebooks in the lib_folder set in your settings.ini, only converting the ones that have been modified since the their corresponding html was last touched unless you pass force_all=True. The index is also converted to make the README file, unless you pass along mk_readme=False.
By default (fname left to None), the all the notebooks in lib_folder are checked.
By default (fname left to None), the all the notebooks in lib_folder are trusted. To speed things up, only the ones touched since the last time this command was run are trusted unless you pass along force_all=True.
When you have merge conflicts after a git pull, the notebook file will be broken and won't open in jupyter notebook anymore. This command fixes this by changing the notebook to a proper json file again and add markdown cells to signal the conflict, you just have to open that notebook again and look for >>>>>>> to see those conflicts and manually fix them. The old broken file is copied with a .ipynb.bak extension, so is still accessible in case the merge wasn't successful.
Moreover, if fast=True, conflicts in outputs and metadata will automatically be fixed by using the local version if trust_us=True, the remote one if trust_us=False. With this option, it's very likely you won't have anything to do, unless there is a real conflict.
test_eq(bump_version('0.1.1' ), '0.1.2')
+test_eq(bump_version('0.1.1', 1), '0.2.0')
+This command installs git hooks to make sure notebooks are cleaned before you commit them to GitHub and automatically trusted at each merge. To be more specific, this creates:
+nbdev_trust_nbs.gitconfig file that uses nbev_clean_nbs has a filter/diff on all notebook files inside nbs_folder and a .gitattributes file generated in this folder (copy this file in other folders where you might have notebooks you want cleaned as well)nbdev_new is a command line tool that creates a new nbdev project from the current directory, which must be a cloned git repo.
After you run nbdev_new, please check the contents of settings.ini look good, and then run nbdev_build_lib.
import nbdev.showdoc
+The most important function defined in this module is notebooks2script, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to library. The main things to remember are:
# export on each cell you want exported# exports on each cell you want exported with the source code shown in the docs # exporti on each cell you want exported without it being added to __all__, and without it showing up in the docs.# default_exp followed by the name of the module (with points for submodules and without the py extension) everything should be exported in (if one specific cell needs to be exported in a different module, just indicate it after #export: #export special.module)__all__ automatically__all__ if it's not picked automatically, write an exported cell with something like #add2all "my_name"For bootstrapping nbdev we have a few basic foundations defined in imports, which we test a show here. First, a simple config file class, Config that read the content of your settings.ini file and make it accessible:
cfg = Config(cfg_name='settings.ini')
+test_eq(cfg.lib_name, 'chisel_nbdev')
+test_eq(cfg.git_url, "https://github.com/ucsc-vama/chisel_nbdev")
+test_eq(cfg.path("lib_path"), Path.cwd().parent/'chisel_nbdev')
+test_eq(cfg.path("nbs_path"), Path.cwd())
+# test_eq(cfg.path("doc_path"), Path.cwd().parent/'docs')
+# test_eq(cfg.custom_sidebar, 'False')
+A jupyter notebook is a json file behind the scenes. We can just read it with the json module, which will return a nested dictionary of dictionaries/lists of dictionaries, but there are some small differences between reading the json and using the tools from nbformat so we'll use this one.
fname can be a string or a pathlib object.
test_nb = read_nb('test.ipynb')
+The root has four keys: cells contains the cells of the notebook, metadata some stuff around the version of python used to execute the notebook, nbformat and nbformat_minor the version of nbformat.
test_nb.keys()
+test_nb['metadata']
+f"{test_nb['nbformat']}.{test_nb['nbformat_minor']}"
+The cells key then contains a list of cells. Each one is a new dictionary that contains entries like the type (code or markdown), the source (what is written in the cell) and the output (for code cells).
+ +test_nb['cells'][0]
+assert(is_scala_nb('test.ipynb'))
+assert(not is_scala_nb('00_export_scala.ipynb'))
+The following functions are used to catch the flags used in the code cells.
+ +pat can be a string or a compiled regex. If code_only=True, this function ignores non-code cells, such as markdown.
cell = test_nb['cells'][2].copy()
+# print(cell)
+assert check_re(cell, '//export') is not None
+assert check_re(cell, re.compile('//export')) is not None
+assert check_re(cell, '# bla') is None
+cell['cell_type'] = 'markdown'
+assert check_re(cell, '//export') is None # don't export markdown
+assert check_re(cell, '//export', code_only=False) is not None # unless specified
+cell = test_nb['cells'][2].copy()
+cell['source'] = "a b c"
+# print(cell)
+assert check_re(cell, 'a') is not None
+assert check_re(cell, 'd') is None
+# show that searching with patterns ['d','b','a'] will match 'b'
+# i.e. 'd' is not found and we don't search for 'a'
+assert check_re_multi(cell, ['d','b','a']).span() == (2,3)
+This function returns a regex object that can be used to find nbdev flags in multiline text
+body regex fragment to match one or more flags,n_params number of flag parameters to match and catch (-1 for any number of params; (0,1) for 0 for 1 params),comment explains what the compiled regex should do.is_export returns;
False for an internal export)) if cell is to be exported or None if cell will not be exported.The cells to export are marked with //export///exporti///exports, potentially with a module name where we want it exported. The default module is given in a cell of the form //default_exp bla inside the notebook (usually at the top), though in this function, it needs the be passed (the final script will read the whole notebook to find it).
//export///exporti///exports will be exported to the default modulespecial.module appended will be exported in special.module (located in lib_name/special/module.py)//export will have its signature added to the documentation//exports will additionally have its source code added to the documentation//exporti will not show up in the documentation, and will also not be added to __all__.cell = test_nb['cells'][2].copy()
+cell['source'] = "// export ModA"
+test_eq(is_non_def_export(cell), ('ModA', True))
+cell = test_nb['cells'][2].copy()
+test_eq(is_export(cell, 'export'), ('export', True))
+cell['source'] = "// exports"
+test_eq(is_export(cell, 'export'), ('export', True))
+cell['source'] = "// exporti"
+test_eq(is_export(cell, 'export'), ('export', False))
+cell['source'] = "// export mod"
+test_eq(is_export(cell, 'export'), ('mod', True))
+test_eq(find_non_default_exports(test_nb['cells']), ['NewScript', 'NewScript2'])
+Stops at the first cell containing // default_exp (if there are several) and returns the value behind. Returns None if there are no cell with that code.
test_eq(find_default_export(test_nb['cells']), 'test')
+assert find_default_export(test_nb['cells'][3:]) is None
+Until now the above code has been verbatim to the 00_export.ipynb from nbdev minus changes to regexes swapping '#' to '//'.
+Now Scala syntax parsing is required as well as a target build infrastructure (sbt, mill, or Ammonite scripts).
For documentation we need to extract out the names of our classes, objects, methods, etc including:
+This function finds all of the function/class/object names.
+ +test_eq(export_names("def my_func(x: Int): Unit = {\n\tprint(x)\n}\n class MyClass(){}"), ["my_func", "MyClass"])
+To be able to build back a correspondence between functions and the notebooks they are defined in, we need to store an index. It's done in the private module _nbdev inside your library, and the following function are used to define it.
return_type tells us if the tuple returned will contain lists of lines or strings with line breaks.
We treat the first comment line as a flag +
+ +def _test_split_flags_and_code(expected_flags, expected_code):
+ cell = nbformat.v4.new_code_cell('\n'.join(expected_flags + expected_code))
+ test_eq((expected_flags, expected_code), split_flags_and_code(cell))
+ expected=('\n'.join(expected_flags), '\n'.join(expected_code))
+ test_eq(expected, split_flags_and_code(cell, str))
+
+_test_split_flags_and_code([
+ '//export'],
+ ['// TODO: write this function',
+ 'def func(x) = ???'])
+A new module filename is created each time a notebook has a cell marked with #default_exp. In your collection of notebooks, you should only have one notebook that creates a given module since they are re-created each time you do a library build (to ensure the library is clean). Note that any file you create manually will never be overwritten (unless it has the same name as one of the modules defined in a #default_exp cell) so you are responsible to clean up those yourself.
fname is the notebook that contained the #default_exp cell.
Create module files for all #default_export flags found in files and return a list containing the names of modules created.
Note: The number if modules returned will be less that the number of files passed in if files do not #default_export.
By creating all module files before calling _notebook2script, the order of execution no longer matters - so you can now export to a notebook that is run "later".
You might still have problems when
+#default_export yetin which case _notebook2script will print warnings like;
Warning: Exporting to "core.py" but this module is not part of this buildIf you see a warning like this
+FileNotFoundErrorassert not nbglob(recursive=True).filter(lambda x: '.ipynb_checkpoints' in str(x))
+Finds cells starting with #export and puts them into the appropriate module. If fname is not specified, this will convert all notebook not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.
silent makes the command not print any statement and to_dict is used internally to convert the library to a dictionary.
The most important function defined in this module is notebook2html, so you may want to jump to it before scrolling though the rest, which explain the details behind the scenes of the conversion from notebooks to the html documentation. The main things to remember are:
#hide at the top of any cell you want to completely hide in the docsshow_doc cells have that marker added)get_metadata<code> and </code> when you have homonyms and don't want those links#default_cls_lvl followed by a number (default is 2)add_jekyll_notes)#hide_input at the top of a cell if you don't want code to be shown in the docs#export or show_doc have their code hidden automatically#hide_output at the top of a cell if you don't want output to be shown in the docs#collapse_input or #collapse_output to include code or output in the docs under a collapsable elementh = HTMLParseAttrs()
+t = h('<img src="src" alt="alt" width="700" caption="cap" />')
+test_eq(t['width'], '700')
+test_eq(t['src' ], 'src')
+t['width'] = '600'
+test_eq(h.show(), '<img src="src" alt="alt" width="600" caption="cap" />')
+The following functions are applied on individual cells as a preprocessing step before the conversion to html.
+ +Those outputs usually can't be rendered properly in html.
+ +This concerns all the cells with #export or #hide_input flags and all the cells containing a show_doc for a function or class.
for source in ['show_doc(read_nb)', '// export\nfrom local.core import *', '// hide_input\n2+2',
+ 'line1\n show_doc (read_nb) \nline3', '// export with.mod\nfrom local.core import *']:
+ cell = {'cell_type': 'code', 'source': source}
+ cell1 = hide_cells(cell.copy())
+ assert 'metadata' in cell1
+ assert 'hide_input' in cell1['metadata']
+ assert cell1['metadata']['hide_input']
+
+flag = '// exports'
+cell = {'cell_type': 'code', 'source': f'{flag}\nfrom local.core2 import *'}
+test_eq(hide_cells(cell.copy()), cell)
+This concerns all the cells with #hide_output.
for source in ['// hide-output\nfrom local.core import *', '// hide_output\n2+2']:
+ cell = {'cell_type': 'code', 'source': source}
+ cell1 = hide_cells(cell.copy())
+ assert 'metadata' in cell1
+ assert 'hide_output' in cell1['metadata']
+ assert cell1['metadata']['hide_output']
+
+cell = {'cell_type': 'code', 'source': '// hide-outputs\nfrom local.core import *'}
+test_eq(hide_cells(cell.copy()), cell)
+The rest of the cell is displayed without any modification.
+ +flag = '// exports'
+cell = {'cell_type': 'code', 'source': f'{flag}\nfrom local.core import *'}
+test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'from local.core import *'})
+cell['cell_type'] = 'markdown'
+test_eq(clean_exports(cell.copy()), cell)
+cell = {'cell_type': 'code', 'source': f'{flag} core\nfrom local.core import *'}
+test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': 'from local.core import *'})
+
+cell = {'cell_type': 'code', 'source': f'// comment \n// exports\nprint("something")'}
+test_eq(clean_exports(cell.copy()), {'cell_type': 'code', 'source': '// exports\nprint("something")'})
+cell = {'cell_type': 'markdown', 'source': 'This is a `DocsTestClass`'}
+# test_eq(treat_backticks(cell), {'cell_type': 'markdown',
+# 'source': 'This is a [`DocsTestClass`](/export.html#DocsTestClass)'})
+Supported styles are Warning, Note Tip and Important:
Typing > Warning: There will be no second warning! will render in the docs:
+{% include warning.html content='There will be no second warning!' %}
+Typing > Important: Pay attention! It's important. will render in the docs:
+{% include important.html content='Pay attention! It’s important.' %}
+Typing > Tip: This is my tip. will render in the docs:
+{% include tip.html content='This is my tip.' %}
+Typing > Note: Take note of this. will render in the docs:
+{% include note.html content='Take note of this.' %}
+Typing > Note: A doc link to [`add_jekyll_notes`](/export_scala2html.html#add_jekyll_notes) should also work fine. will render in the docs:
+{% include note.html content='A doc link to add_jekyll_notes should also work fine.' %}
m=_re_image.search('')
+test_eq(m.groups(), ('', None))
+# using ) or whitespace to close the group means we don't need a special case for captions
+m=_re_image.search('')
+test_eq(m.groups(), (')
+This is to ensure that all images defined in nbs_folder/images and used in notebooks are copied over to doc_folder/images.
dest_img = Config().path("doc_path")/'images'/'logo.jpg'
+cell = {'cell_type': 'markdown', 'source':'Text\n'}
+try:
+ copy_images(cell, Path('ToImport.ipynb'), Config().path("doc_path"))
+ test_eq(cell["source"], 'Text\n')
+ #Image has been copied
+ assert dest_img.exists()
+ cell = {'cell_type': 'markdown', 'source':'Text\n'}
+ copy_images(cell, Path('ToImport.ipynb'), Config().path("doc_path"))
+ test_eq(cell["source"], 'Text\n')
+finally: dest_img.unlink()
+This function is slightly different as it ensures that a notebook convert to a file that will be placed in dest will have the images location updated. It is used for the README.md file (generated automatically from the index) since the images are copied inside the github repo, but in general, you should make sure your images are going to be accessible from the location your file ends up being.
cell = {'cell_type': 'markdown', 'source': 'Text\n'}
+cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
+test_eq(cell1['source'], 'Text\n')
+
+cell = {'cell_type': 'markdown', 'source': 'Text\n'}
+cell1 = adapt_img_path(cell, Path('01_export.ipynb'), Path('.').absolute().parent)
+test_eq(cell1['source'], 'Text\n')
+Escape Latex in liquid
+ +cell = {'cell_type': 'markdown',
+ 'source': 'lala\n$$equation$$\nlala'}
+cell = escape_latex(cell)
+test_eq(cell['source'], 'lala\n{% raw %}\n$$equation$$\n{% endraw %}\nlala')
+#collapse_input open in a code cell will include your code under a collapsable element that is open by default.print('This code cell is not collapsed by default but you can collapse it to hide it from view!')
+print("Note that the output always shows with `%collapse_input`.")
+#collapse_input in a code cell will include your code in a collapsable element that is closed by default. For example:print('The code cell that produced this output is collapsed by default but you can expand it!')
+print('The input of this cell is visible as usual.\nHowever, the OUTPUT of this cell is collapsed by default but you can expand it!')
+The following functions are applied to the entire list of cells of the notebook as a preprocessing step before the conversion to html.
+ +cells = [{'cell_type': 'code', 'source': source, 'hide': hide} for hide, source in [
+ (False, '// export\nfrom local.core import *'),
+ (False, '// exporti mod file'), # Note: this used to get removed but we're more strict now
+ (True, '// hide\nfrom local.core import *'),
+ (False, '// hide_input\nfrom local.core import *'),
+ (False, '//exports\nsuper code'),
+ (True, '//default_exp notebook.export'),
+ (False, 'show_doc(read_nb)'),
+ (False, '//hide (last test of to_concat)'),
+ (True, '// exporti\n1 + 1')]] + [
+ {'cell_type': 'markdown', 'source': source, 'hide': hide} for hide, source in [
+ (False, '//hide_input\nnice'),
+ (True, '//hide\n\nto hide')]]
+
+for a,b in zip([cell for cell in cells if not cell['hide']], remove_hidden(cells)):
+ test_eq(a,b)
+tst_nb = read_nb('ToImport.ipynb')
+test_eq(find_default_level(tst_nb['cells']), 3)
+A scala notebook will not have any python imports in scope. This _import_str must also be included when a new show_docs cell is added. This way when we execute just the doc cells in a scala notebook, we will be able to generate the markdown from show_doc.
for i,cell in enumerate(tst_nb['cells']):
+ if cell['source'].startswith('//export\nclass Add'): break
+tst_cells = [c.copy() for c in tst_nb['cells'][i-1:i+1]]
+added_cells = add_show_docs(tst_cells, cls_lvl=3)
+test_eq(len(added_cells), 3)
+test_eq(added_cells[0], tst_nb['cells'][i-1])
+test_eq(added_cells[2], tst_nb['cells'][i])
+test_eq(added_cells[1], _show_doc_cell('Add', cls_lvl=3))
+test_eq(added_cells[1]['source'], f"{_import_str}show_doc('Add', default_cls_level=3)")
+
+for flag in ['//export', '//exports']:
+ for show_doc_source in [
+ ('show_doc(my_func)', 'show_doc(my_func, title_level=3)')]:
+ #Check show_doc isn't added if it was already there.
+ tst_cells1 = [{'cell_type':'code', 'source': f'{flag}\ndef my_func(x):\n return x'},
+ {'cell_type':'code', 'source': show_doc_source[0]}]
+ test_eq(add_show_docs(tst_cells1), tst_cells1)
+ #Check show_doc is added
+ test_eq(len(add_show_docs(tst_cells1[:-1])), len(tst_cells1))
+ tst_cells1 = [{'cell_type':'code', 'source': f'{flag} with.mod\ndef my_func(x):\n return x'},
+ {'cell_type':'markdown', 'source': 'Some text'},
+ {'cell_type':'code', 'source': show_doc_source[1]}]
+ test_eq(add_show_docs(tst_cells1), tst_cells1)
+ #Check show_doc is added when using mod export
+ test_eq(len(add_show_docs(tst_cells1[:-1])), len(tst_cells1))
+You can fake headers in your notebook to navigate them more easily with collapsible headers, just make them finish with a dash and they will be removed. One typical use case is to have a header of level 2 with the name of a class, since the show_doc cell of that class will create the same anchor, you need to have the one you created manually disappear to avoid any duplicate.
cells = [{'cell_type': 'markdown',
+ 'metadata': {},
+ 'source': '### Fake-'}] + tst_nb['cells'][:10]
+cells1 = remove_fake_headers(cells)
+test_eq(len(cells1), len(cells)-1)
+test_eq(cells1[0], cells[1])
+In the markdown cell with the title, you can add the summary as a block quote (just put an empty block quote for an empty summary) and a list with any additional metadata you would like to add, for instance:
+ +# Title
+> Awesome summary
+- toc:FalseThe toc: False metadata will prevent the table of contents from showing on the page.
+ +tst_nb = read_nb('ToImport.ipynb')
+test_eq(get_metadata(tst_nb['cells']), {
+ 'keywords': 'chisel',
+ 'summary': 'This file imports the necessary Chisel dependencies and defines an Add module that future notebooks will import.',
+ 'title': 'Creating a basic Chisel module to be imported by other files.'})
+
+#The cell with the metada is popped out, so if we do it a second time we get the default.
+test_eq(get_metadata(tst_nb['cells']), {'keywords': 'chisel', 'title' : 'Title'})
+Cells containing:
+notebook2scriptare not run while building docs. This avoids failures caused by importing empty or partially built modules.
+Cells containing:
+show_doc (which could be indented) orfrom LIB_NAME.core import *are executed and must run without error. If running these cells raises an exception, the build will stop.
+Cells containing zero indented imports. e.g.
+from module import * orimport moduleare executed but errors will not stop the build.
+If you need to show_doc something, please make sure it's imported via a cell that does not depend on previous cells being run. The easiest way to do this is to use a cell that contains nothing but imports.
jupyter_latex_envs is a jupyter extension https://github.com/jfbercher/jupyter_latex_envs.
+You can find relevant section here
+Note, that nbdev now only supports [<a class="latex_cit" id="call-" href="#cit-"></a>] conversion and not the rest, e.g., \figure{} and so on.
It's important to execute all show_doc cells before exporting the notebook to html because some of them have just been added automatically or others could have outdated links.
fake_nb = {k:v for k,v in tst_nb.items() if k != 'cells'}
+fake_nb['cells'] = [tst_nb['cells'][0].copy()] + added_cells
+fake_nb = execute_nb(fake_nb, mod='export')
+assert len(fake_nb['cells'][-2]['outputs']) > 0
+The following functions automatically adds jekyll templates if they are missing.
+ +store_true()
+__file__ = str(Config().path("lib_path")/'export_scala2html.py')
+Hide cells starting with #export and only leaves the prose and the tests. If fname is not specified, this will convert all notebooks not beginning with an underscore in the nb_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.
By default, only the notebooks that are more recent than their html counterparts are modified, pass force_all=True to change that behavior.
This is used to convert the index into the README.md.
t = ''
+test_eq(_re_att_ref.match(t).groups(), ('screenshot', None))
+
+t = ''
+test_eq(_re_att_ref.match(t).groups(), ('screenshot', "Deploying to Binder"))
+The default sidebar lists all html pages with their respective title, except the index that is named "Overview". To build a custom sidebar, set the flag custom_sidebar in your settings.ini to True then change the sidebar.json file in the doc_folder to your liking. Otherwise, the sidebar is updated at each doc build.
nbdev_build_docs(force_all=True)
+When working with jupyter notebooks (which are json files behind the scenes) and GitHub, it is very common that a merge conflict (that will add new lines in the notebook source file) will break some notebooks you are working on. This module defines the function fix_conflicts to fix those notebooks for you, and attempt to automatically merge standard conflicts. The remaining ones will be delimited by markdown cells like this:
{% include image.html alt="Fixed notebook" width="700" caption="A notebook fixed after a merged conflict. The file couldn't be opened before the command was run, but after it the conflict is highlighted by markdown cells." max-width="700" file="/images/merge.PNG" %}
+ +This is an example of broken notebook we defined in tst_nb. The json format is broken by the lines automatically added by git. Such a file can't be opened again in jupyter notebook, leaving the user with no other choice than to fix the text file manually.
print(tst_nb)
+Note that in this example, the second conflict is easily solved: it just concerns the execution count of the second cell and can be solved by choosing either option without really impacting your notebook. This is the kind of conflicts fix_conflicts will (by default) fix automatically. The first conflict is more complicated as it spans across two cells and there is a cell present in one version, not the other. Such a conflict (and generally the ones where the inputs of the cells change form one version to the other) aren't automatically fixed, but fix_conflicts will return a proper json file where the annotations introduced by git will be placed in markdown cells.
The first step to do this is to walk the raw text file to extract the cells. We can't read it as a JSON since it's broken, so we have to parse the text.
+ +This function returns the beginning of the text (before the cells are defined), the list of cells and the end of the text (after the cells are defined).
+ +start,cells,end = extract_cells(tst_nb)
+test_eq(len(cells), 3)
+test_eq(cells[0], """ {
+ "cell_type": "code",
+<<<<<<< HEAD
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "3"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "z=3\n",
+ "z"
+ ]
+ },""")
+When walking the broken cells, we will add conflicts marker before and after the cells with conflicts as markdown cells. To do that we use this function.
+ +tst = ''' {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "A bit of markdown"
+ ]
+ },'''
+assert get_md_cell("A bit of markdown") == tst
+ts = [''' {
+ "cell_type": "code",
+ "source": [
+ "'''+code+'''"
+ ]
+ },''' for code in ["a=1", "b=1", "a=1"]]
+assert same_inputs(ts[0],ts[2])
+assert not same_inputs(ts[0], ts[1])
+This is the main function used to walk through the cells of a notebook. cell is the cell we're at, cf the conflict state: 0 if we're not in any conflict, 1 if we are inside the first part of a conflict (between <<<<<<< and =======) and 2 for the second part of a conflict. names contains the names of the branches (they start at [None,None] and get updated as we pass along conflicts). prev contains a copy of what should be included at the start of the second version (if cf=1 or cf=2). added starts at False and keeps track of whether we added any markdown cells (this flag allows us to know if a fast merge didn't leave any conflicts at the end). fast and trust_us are passed along by fix_conflicts: if fast is True, we don't point out conflict between cells if the inputs in the two versions are the same. Instead we merge using the local or remote branch, depending on trust_us.
The function then returns the updated text (with one or several cells, depending on the conflicts to solve), the updated cf, names, prev and added.
tst = '\n'.join(['a', f'{conflicts[0]} HEAD', 'b', conflicts[1], 'c'])
+c,cf,names,prev,added = analyze_cell(tst, 0, [None,None], None, False,fast=False)
+test_eq(c, get_md_cell('`<<<<<<< HEAD`')+'\na\nb')
+test_eq(cf, 2)
+test_eq(names, ['HEAD', None])
+test_eq(prev, ['a\nc'])
+test_eq(added, True)
+Here in this example, we were entering cell tst with no conflict state. At the end of the cells, we are still in the second part of the conflict, hence cf=2. The result returns a marker for the branch head, then the whole cell in version 1 (a + b). We save a (prior to the conflict hence common to the two versions) and c (only in version 2) for the next cell in prev (that should contain the resolution of this conflict).
This begins by backing the notebook fname to fname.bak in case something goes wrong. Then it parses the broken json, solving conflicts in cells. If fast=True, every conflict that only involves metadata or outputs of cells will be solved automatically by using the local (trust_us=True) or the remote (trust_us=False) branch. Otherwise, or for conflicts involving the inputs of cells, the json will be repaired by including the two version of the conflicted cell(s) with markdown cells indicating the conflicts. You will be able to open the notebook again and search for the conflicts (look for <<<<<<<) then fix them as you wish.
If fast=True, the function will print a message indicating whether the notebook was fully merged or if conflicts remain.
The library is primarily developed in notebooks so any big changes should be made there. But sometimes, it's easier to fix small bugs or typos in the modules directly. nbdev_update_lib is the function that will propagate those changes back to the corresponding notebooks. Note that you can't create new cells with that functionality, so your corrections should remain limited.
We need to get the name of the object we are looking for, and then we'll try to find it in our index file.
+ +from nbdev.export import DocsTestClass
+test_eq(get_name(in_ipython), 'in_ipython')
+test_eq(get_name(DocsTestClass.test), 'test')
+test_eq(qual_name(DocsTestClass.test), 'DocsTestClass.test')
+You can either pass an object or its name (by default is_name will look if func is a string or not to determine if it should be True or False, but you can override if there is some inconsistent behavior).
If passed a method of a class, the function will return the notebook in which the largest part of the function name was defined in case there is a monkey-matching that defines class.method in a different notebook than class. If return_all=True, the function will return a tuple with the name by which the function was found and the notebook.
For properties defined using property or our own add_props helper, we approximate the name by looking at their getter functions, since we don't seem to have access to the property name itself. If everything fails (a getter cannot be found), we return the name of the object that contains the property. This suffices for source_nb to work.
test_eq(source_nb("test.Add"), 'test.ipynb')
+test_eq(source_nb("Add"), 'ToImport.ipynb')
+test_eq(source_nb("Operator"), 'test.ipynb')
+test_eq(source_nb("MulDiv"), 'test.ipynb')
+assert source_nb(int) is None
+If someone decides to change a module instead of the notebooks, the following functions help update the notebooks accordingly.
+ +If fname is not specified, this will convert all modules and submodules in the lib_folder defined in setting.ini. Otherwise fname can be a single filename or a glob expression.
silent makes the command not print any statement.
Before making a commit, you may want to check there is no diff between the exported library and the notebooks. You may also want to make this part of your CI, so that you don't accidentally merge a PR that introduces some changes between the two. This function is there to print this diff.
+ +If you receive an output, you'll need to either run notebook2script() or nbdev_update_lib() to fix the difference.
nbdev_diff_nbs()
+nbdev_trust_nbs()
+Until can build an actual package with chisel-nbdev python code, import my 00_export_scala.ipynb -> nbdev/export.py using this hacky method:
Everything that is not an exported cell is considered a test, so you should make sure your notebooks can all run smoothly (and fast) if you want to use this functionality as the CLI. You can mark some cells with special flags (like slow) to make sure they are only executed when you authorize it. Those flags should be configured in your settings.ini (separated by a | if you have several of them). You can also apply flags to one entire notebook by using the all option, e.g. #all_slow, in code cells.
If tst_flags=slow|fastai in settings.ini, you can:
#slow flag#fastai flag.nb = read_nb("04_test_scala.ipynb")
+assert get_all_flags(nb['cells']) == set()
+test_eq(get_cell_flags({'cell_type': 'code', 'source': "//hide\n"}), [])
+If there is a mix of different notebook kernels (i.e Scala and Python notebooks), we only want to run the Scala tests.
+ +# files
+nbdev_test_nbs()
+