Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transition from Requires.jl to v1.9-style weak dependencies #42

Merged
merged 5 commits into from
Jul 12, 2023
Merged

Transition from Requires.jl to v1.9-style weak dependencies #42

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f0664f2
[WIP] Transition from Requires.jl to v1.9-style weak dependencies
ffevotte Jul 11, 2023
73ad895
Update documentation
ffevotte Jul 11, 2023
dbaad0e
Document extended functions
ffevotte Jul 11, 2023
ebdab4a
fix some typos
maltezfaria Jul 12, 2023
e7d5561
update docs
maltezfaria Jul 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,24 @@ Compat = "34da2185-b29b-5c13-b0c7-acf172513d20"
DataStructures = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
Pkg = "44cfe95a-1eb2-52ea-b672-e2afdf69b78f"
Requires = "ae029012-a4dd-5104-9daa-d747884805df"
StaticArrays = "90137ffa-7385-5640-81b9-e52037218182"

[compat]
DataStructures = "0.18"
Pkg = "1.6"
Requires = "1.3"
StaticArrays = "1.5"
julia = "1.6"

[extensions]
DataFlowTasks_GraphViz_Ext = "GraphViz"
DataFlowTasks_Makie_Ext = "Makie"

[extras]
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

[targets]
test = ["Test"]

[weakdeps]
GraphViz = "f526b714-d49f-11e8-06ff-31ed36ee7ee0"
Makie = "ee78f7c6-11fb-53f2-987a-cfe4a2b5a57a"
32 changes: 16 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,22 +68,22 @@ represented as a Directed Acyclic Graph. Reconstructing the `DAG` (as well as
the parallalel traces) can be done using the `@log` macro:

````julia
using GraphViz # triggers additional code loading, powered by Requires.jl
using GraphViz # triggers additional code loading, powered by weak dependencies (julia >= 1.9)
log_info = DataFlowTasks.@log let
@dspawn fill!(@W(A), 0) label="write whole"
@dspawn @RW(view(A, 1:2)) .+= 2 label="write 1:2"
@dspawn @RW(view(A, 3:4)) .+= 3 label="write 3:4"
res = @dspawn @R(A) label="read whole"
fetch(res)
end
dag = DataFlowTasks.plot_dag(log_info)
dag = GraphViz.Graph(log_info)
````

![](docs/readme/example_dag.svg)

In the example above, the tasks *write 1:2* and *write 3:4* access
different parts of the array `A` and are
therefore independant, as shown in the DAG.
therefore independent, as shown in the DAG.

## Example : Parallel Cholesky Factorization

Expand All @@ -100,12 +100,12 @@ version of this algorithm decomposes the matrix A into tiles (of even sizes,
in this simplified version). At each step of the algorithm, we do a Cholesky
factorization on the diagonal tile, use a triangular solve to update all of
the tiles at the right of the diagonal tile, and finally update all the tiles
of the submatrix with a schur complement.
of the submatrix with a Schur complement.

If we have a matrix A decomposed in `n x n` tiles, then the algorithm will
have `n` steps. The `i`-th step (with `i ∈ [1:n]`) will perform

- `1` cholesky factorization of the (i,i) block,
- `1` Cholesky factorization of the (i,i) block,
- `(i-1)` triangular solves (one for each block in the `i`-th row),
- `i*(i-1)/2` matrix multiplications to update the submatrix.

Expand All @@ -128,7 +128,7 @@ function cholesky_tiled!(A, ts)
T = [view(A, tilerange(i, ts), tilerange(j, ts)) for i in 1:n, j in 1:n]

for i in 1:n
# Diagonal cholesky serial factorization
# Diagonal Cholesky serial factorization
cholesky!(T[i,i])

# Left blocks update
Expand Down Expand Up @@ -163,7 +163,7 @@ function cholesky_dft!(A, ts)
T = [view(A, tilerange(i, ts), tilerange(j, ts)) for i in 1:n, j in 1:n]

for i in 1:n
# Diagonal cholesky serial factorization
# Diagonal Cholesky serial factorization
@dspawn cholesky!(@RW(T[i,i])) label="chol ($i,$i)"

# Left blocks update
Expand Down Expand Up @@ -194,7 +194,7 @@ The code below shows how to use this `cholesky_tiled!` function, as well as
how to profile the program and get information about how tasks were scheduled:

````julia
# DataFlowTasks environnement setup
# DataFlowTasks environment setup

# Context
n = 2048
Expand All @@ -216,7 +216,7 @@ err = norm(F.L*F.U-A,Inf)/max(norm(A),norm(F.L*F.U))

## Debugging and Profiling

DataFlowTasks comes with debugging and profiling tools which help
DataFlowTasks comes with debugging and profiling tools that help
understanding how task dependencies were inferred, and how tasks were
scheduled during execution.

Expand All @@ -242,19 +242,19 @@ In this more complex example, we can see how quickly the DAG complexity
increases (even though the test case only has 4x4 blocks here):

````julia
dag = DataFlowTasks.plot_dag(log_info)
dag = GraphViz.Graph(log_info)
````

![](docs/readme/cholesky_dag.svg)

The parallel trace plot shows a timeline of the tasks execution on available
threads. It helps understanding how tasks were scheduled. The same window also
The parallel trace plot shows a timeline of the tasks' execution on available
threads. It helps in understanding how tasks were scheduled. The same window also
carries other general information allowing to better understand the
performance limiting factors:

````julia
using CairoMakie # or GLMakie in order to have more interactivity
trace = DataFlowTasks.plot_traces(log_info; categories=["chol", "ldiv", "schur"])
trace = plot(log_info; categories=["chol", "ldiv", "schur"])
````

![](docs/readme/cholesky_trace.svg)
Expand All @@ -263,13 +263,13 @@ We see here that the execution time is bounded by the length of the critical
path: with this block size and matrix size, the algorithm does not expose
enough parallelism to occupy all threads without waiting periods.

We'll cover in details the usage and possibilities of the visualization in the
We'll cover in detail the usage and possibilities of the visualization in the
documentation.

Note that the debugging & profiling tools need additional dependencies such as
`Makie` and `GraphViz`, which are only meant to be used interactively during
the development process. These packages are therefore only considered as
optional depdendencies; assuming they are available in your work environment,
optional dependencies; assuming they are available in your work environment,
calling e.g. `using GraphViz` will load some additional code from
`DataFlowTasks` (see also the documentation of `DataFlowTasks.@using_opt` if
you prefer an alternative way of handling these extra dependencies).
Expand All @@ -285,7 +285,7 @@ implementations for the sequential building blocks operating on tiles:

This approach is pursued in
[`TiledFactorization.jl`](https://github.com/maltezfaria/TiledFactorization),
where all the above mentioned building blocks are combined with the
where all the above-mentioned building blocks are combined with the
parallelization strategy presented here to create a *pure Julia*
implementation of the matrix factorizations. The performances of this
implementation is assessed in the following plot, by comparison to MKL on a
Expand Down
1 change: 1 addition & 0 deletions docs/Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[deps]
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
CairoMakie = "13f3f980-e62b-5c42-98c6-ff1f3baf88f0"
DataFlowTasks = "d1549cb6-e9f4-42f8-98cc-ffc8d067ff5b"
Copy link
Owner

@maltezfaria maltezfaria Jul 12, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I never add the project itself as a dependency to docs, as I was under the impression this was not necessary (Documenter looks into its parent folder for a project right?).

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think Documenter looks into its parent folder to find the documented project, so at the time when the documentation is built, the parent package has to be present in the documentation environment in some way. However, one of the steps in the GitHub CI action takes care of Pkg.deving the parent project in the documentation subproject. So both ways work: if the committed docs/Project.toml does not mention DataFlowTasks, things will get set up correctly just ahead of time.

I've recently seen packages where even the docs/Manifest.toml was committed so as to make sure that the parent project is deved from the parent directory. Graphs.jl is an example of this.

Personally, I'd be in favor of keeping DataFlowTasks as a documentation dependency in the committed docs/Project.toml, because this simplifies the process to locally build the documentation. But I'm open to compromise 😄

Documenter = "e30172f5-a6a5-5a46-863b-614d45cd2de4"
GraphViz = "f526b714-d49f-11e8-06ff-31ed36ee7ee0"
Images = "916415d5-f1e6-5110-898d-aaa5f9f070e0"
Expand Down
10 changes: 9 additions & 1 deletion docs/make.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,16 @@ end

DocMeta.setdocmeta!(DataFlowTasks, :DocTestSetup, :(using CairoMakie, GraphViz, DataFlowTasks); recursive=true)

modules = [DataFlowTasks]
if isdefined(Base, :get_extension)
using GraphViz, CairoMakie
const GraphViz_Ext = Base.get_extension(DataFlowTasks, :DataFlowTasks_GraphViz_Ext)
const Makie_Ext = Base.get_extension(DataFlowTasks, :DataFlowTasks_Makie_Ext)
append!(modules, (GraphViz_Ext, Makie_Ext))
end

makedocs(;
modules=[DataFlowTasks],
modules = modules,
repo="https://github.com/maltezfaria/DataFlowTasks.jl/blob/{commit}{path}#{line}",
sitename="DataFlowTasks.jl",
format=Documenter.HTML(;
Expand Down
Loading