[WIP] workflow toml specification and initial implementation #1
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1 +/- ##
======================================
Coverage 0 0.00%
======================================
Files 0 4 +4
Lines 0 71 +71
======================================
- Misses 0 71 +71
Continue to review full report at Codecov.
|
johnnychen94
force-pushed
the
jc/workflow
branch
from
7b21a19 to
962c9f7
Compare
| [stages.run] | ||
| metrics = ["time"] | ||
| out = ["results.csv"] | ||
| driver = "csv" |
There was a problem hiding this comment.
Currently, it outputs
id,time
dilate_juliaimages_morphology,0.623235
erode_juliaimages_morphology,0.627855
dilate_skimage_morphology,1.5773831250000003
and if I change to metrics = ["time", "memory"], then it becomes
id,time,memory
dilate_juliaimages_morphology,0.636088,1280096
erode_juliaimages_morphology,0.641488,1280096
dilate_morphology_skimage,1.8287903700000008,
cc: @ashwani-rathee
There was a problem hiding this comment.
I should make metrics an optional field and defaults to all results. Or just remove this field.
| [stages.run] | ||
| metrics = ["time"] | ||
| out = ["results.csv"] | ||
| driver = "csv" |
There was a problem hiding this comment.
I should make metrics an optional field and defaults to all results. Or just remove this field.
| count, time = timeit.Timer('dilation(img, square(3))', globals=globals()).autorange() | ||
|
|
||
| # export | ||
| print(json.dumps({"time": 1e3*time/count})) # ms |
There was a problem hiding this comment.
shell runner pass data via stdout=IOBuffer()
| Dict( | ||
| "memory" => rst.memory, # byte | ||
| "time" => median(rst.times)/1e6, # ms | ||
| ) |> JSON3.write |
There was a problem hiding this comment.
Julia runner could choose to directly pass a Dict, but I want to make it language-agnostic so for consistency, I choose to let it pass a json String.
|
closed in favor of #8 |
Key features I have in mind:
python,julia,shell, and others)My design:
The idea comes from mainly two designs: dvc yaml and datasets toml
Example: benchmark framework
The initial purpose of this is to support a generic and wide-range benchmark framework, there are some key challenges to achieve this:
fin framework A might not have its corresponding function in other frameworksA natural idea is to separate the data producing stage and data analysis stage. We can produce any many benchmark results as we want, as long as we properly tag them. Then in the data visualization stage, we use filters to collect results that we're interested.
BenchmarkTools and PkgBenchmark use a tree design(nested groups) to organize the benchmark tasks. I reckon it a bad design because 1) it introduces a overly compact form and makes it very hard to adjust benchmark targets, 2) it makes future result filtering much harder. Thus I choose to use "name"+"tags" design, in the visualization and analysis stage:
fhas multiple implementations.A unique ID is generated by join "name" and "tags" (e.g.,
join([name, sort(tags)...])cc: @ashwani-rathee