Documentation

[ffevotte]

The documentation effort made by @Vincent-Sivadon is still under way, but I'm creating this WIP PR so that we have a space to discuss issues related to it.

[ffevotte]

One issue that was brought up in earlier discussions comes from the use of a halo or zeros in the Gauss-Seidel example. IIUC, this is currently implemented by using a modified version of PseudoTiledMatrix that is aware of the existence of a halo, but in which only the "interior" of the matrix is tiled. This makes it possible to make the algorithm loops completely unaware of the existence of a halo. In particular, all tiles can assume that they have neighboring data in all directions.

OTOH, an other approach would consist in making the matrix itself unaware of the halo. The algorithm itself should therefore take care of iterating only on valid indices. In terms of documentation, this would have the added benefit that boundary tiles have less than 4 neighbors, so that the number of data dependencies is not always the same. It would be interesting to see whether this can be handled in a friendly way by DataFlowTasks and, if so, to use Gauss Seidel as an example to show how to.

A first idea that comes to mind to declare dependencies that don't always exist would be something along those lines:

@R (i>2 ? A[i-1,:] : nothing)

But I'm not sure whether this already works out of the box from a syntax viewpoint. Maybe it also requires implementing new methods like

memory_overlap(::Nothing, ::Any) = false

[codecov]

Codecov Report

Merging #22 (4a51faa) into main (097ce4d) will decrease coverage by 1.76%.
The diff coverage is 46.42%.

@@            Coverage Diff             @@
##             main      #22      +/-   ##
==========================================
- Coverage   83.33%   81.56%   -1.77%     
==========================================
  Files          10       10              
  Lines         660      678      +18     
==========================================
+ Hits          550      553       +3     
- Misses        110      125      +15     

Impacted Files	Coverage Δ
src/dag.jl	75.00% <0.00%> (-5.29%)	⬇️
src/plotdag.jl	81.81% <0.00%> (-3.90%)	⬇️
src/scheduler.jl	76.13% <ø> (ø)
src/plotgeneral.jl	89.92% <76.47%> (-2.71%)	⬇️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[maltezfaria]

One issue that was brought up in earlier discussions comes from the use of a halo or zeros in the Gauss-Seidel example. IIUC, this is currently implemented by using a modified version of PseudoTiledMatrix that is aware of the existence of a halo, but in which only the "interior" of the matrix is tiled. This makes it possible to make the algorithm loops completely unaware of the existence of a halo. In particular, all tiles can assume that they have neighboring data in all directions.

OTOH, an other approach would consist in making the matrix itself unaware of the halo. The algorithm itself should therefore take care of iterating only on valid indices. In terms of documentation, this would have the added benefit that boundary tiles have less than 4 neighbors, so that the number of data dependencies is not always the same. It would be interesting to see whether this can be handled in a friendly way by DataFlowTasks and, if so, to use Gauss Seidel as an example to show how to.

A first idea that comes to mind to declare dependencies that don't always exist would be something along those lines:

@R (i>2 ? A[i-1,:] : nothing)

But I'm not sure whether this already works out of the box from a syntax viewpoint. Maybe it also requires implementing new methods like

memory_overlap(::Nothing, ::Any) = false

I am lost here, can you guys give me a reference to what you are calling the Gauss-Seidel algorithm? I can't reconcile what I have in mind, which is the iterative method, to this five-point stencil description that I found on the docs 😕

[ffevotte]

I am lost here, can you guys give me a reference to what you are calling the Gauss-Seidel algorithm? I can't reconcile what I have in mind, which is the iterative method, to this five-point stencil description that I found on the docs confused

Yes, I couldn't agree more. I should have mentioned that this is also a question I asked Vincent yesterday. @Vincent-Sivadon seems to have found this terminology in a paper on dataflow parallelism that I haven't seen yet...

[ffevotte]

Thanks for this write-up!

Here are my comments on the brand-new README.

[ffevotte]

Yep! If/when #26 (or something like it) gets merged, we'll have to update the doc here accordingly.

[ffevotte]

A global comment on this branch is: is there a reason why we need to have generated documentation files (such as build/*.html) checked in the git repo ?

[ffevotte]

Commit 9085534 introduces a way to automatically generate the README.md file from a Literate.jl julia script (located in docs/readme/README.jl). A collateral benefit is that the same contents are now also generated in the form of a Jupyter notebook (which can be seen directly from github or from nbviewer).

The new version of README.md has been generated this way, and we can now be sure that all code in it is runnable (which incidentally increases the number of test cases, although these are run during documentation generation and are therefore not accounted for in the coverage analysis).

I'll now go ahead and try to resolve all remaining issues with README.md, which should get us close to the final contents. Maybe a last review from y'all would be good though, in order to make sure we're on the same page in terms of how we want to present DataFlowTasks to new users.

More work is still needed to finalize the full Documenter docs, especially if we want to include all examples. But maybe we can reach pretty quickly a "presentable" state, where all features are documented. And from there we can add more tutorials/examples (maybe using DemoCards.jl or Literate.jl to more easily ensure they are runnable).

[ffevotte]

I think the README is now in a state where it can be reviewed by anyone willing to.

I've commented on the few issues that still need to be discussed collectively IMO.

[ffevotte]

Unresolved questions regarding reset:

• do we want to keep it? (cf. DAG environnement not beeing cleaned #24)
• even if so, do we want to advertize its use?

OTOH: if we don't want to have/advertize it, make sure removing it doesn't affect the outcome of the examples shown here.

[ffevotte]

Oh, and if we want to keep reset(), we'll have to add tests for it.

[ffevotte]

Are these lines needed? Would resetlogger! be enough?

[ffevotte]

Naive question: why not something simpler, like

err = norm(F.L*F.U - A) / norm(A)

Also: I arbitrarily put the tolerance to machine precision, but this might not be the right thing to do.