Macros and DSLs

[zenon]

I try out Soss.jl, a tool for probabilistic programming.

I can't attach .jl files, so I changed them to txt.

This (non Pluto) works fine:
sosstest01.txt

This (Pluto) doesn't:
sosstest02.txt

Error message:
Cyclic references among m, y and forwardModel:
Combine all definitions into a single reactive cell using a begin ... end block.

Am I doing sth wrong, or has Pluto an issue with macros?

Kind greetings, z.

[fonsp]

Pluto doesn't understand macros, it just looks at the code as if the macro wasn't there.

For example, if a macro turns

@hello dog

into

dog = 123

then Pluto things that this cell references dog, and that it does not assign to it. This leads to all sorts of problems - one being that the scope can't be managed properly, and you get an error like rafaqz/UnitfulMoles.jl#12

---

This is a difficult problem: the macro is not known to Pluto - it is only defined inside the notebook process. To understand (i.e. expand) macros, there needs to be an analysis step after normal analysis, but before runtime (because the analysis determines which cells will run).

So Domain Specific Languages like Soss.jl might not work well.

[zenon]

Hello Fons, thank you for the fast answer.

Hm. I was afraid it would come out this way. :-)

Kind greetings, z.

[fonsp]

About DSLs: I am thinking that maybe Pluto can include a "mock version" for a bunch of popular packages with DSLs.

So not the original macro, but a macro that transforms the code into something that has the intended references and definitions. For example, Replace=with~.jl could transform

@replaceit x ~ y + 1

(Pluto thinks that x, y and + are the references, with no assignments)
to

x = y + 1

Another example

Another example is @bind inside Pluto:

@bind x Slider(1:3)

looks like a reference to x, but it is actually an assignment to x. This one is solved by expanding the macro (because @bind is available inside the Pluto package).

[fonsp]

Although, looking at this issue again - the problem is not the DSL, it's a bug in Pluto's syntax analysis! #225

[fonsp]

Your original issue should be fixed now! But the DSL problem is still there

[zenon]

Yes! It now works (even if I understand that there are no guarantees :-) )

Thank you for the hint.

(Sorry for the late answer. I took an offline time ....)

[aplavin]

Another important case when this issue happens is with the Query.jl package: df |> @map({a = _.b}) gets treated as an assigment to a by Pluto.

[fonsp]

See a workaround for some macros here: #371

[fonsp]

Query.jl and VegaLite.jl both use {a = 1} to mean something similar to a named tuple, (a = 1,) not as an assignment to a. {} is not part of base Julia syntax, so we can change Pluto to always interpret {} that way.

This will fix those two packages, but it might break others. Which other packages use {} inside its custom macro sytnax?

[mcabbott]

I was wondering how to make things like @einsum a[i] := b[i,j] * c[j] work; here := means this assigns to a new array. Like {a=1} I don't think this is allowed outside of macros, might this be another candidate case?

[fonsp]

Yep! But you need to figure out what other packages are affected

[mcabbott]

Excellent. I know half a dozen who use this notation (some of which I'm guilty of writing) and can't immediately think of other macros using :=, but will dig a bit.

[fonsp]

Although note that a[1] = 2 counts as a reference to a in Pluto, not an assignment to a, so in the specific example that you gave, there is no differnce.

[mcabbott]

Yes, only with := is it assignment. Something like @einsum a[i] = b[i,j] * c[j] ought to count as a reference to a,b,c, although ideally not to i,j.

[fonsp]

Here are three options for solving this:

Option 1

Pluto will macroexpand everything. This is though to implement, but possible - read more here: #177 (comment)

This introduces a new problem: not all macros are just code rewrites, some macros have side effects, as part of the expansion process.

Something that has a side effect but is not an example of what I mean: @show x has the side effect of writing to stdout, but this side effect is executed by the resulting expression - the macro expansion itself has no side effect:

julia> macroexpand(Main, :(@show x))
quote
    Base.println("x = ", Base.repr(begin
                #= show.jl:613 =#
                var"#2#value" = x
            end))
    var"#2#value"
end

(note that nothing was printed to stdout except the expanded expression)

But macros are not guaranteed to have this nice property. I can't think of an example of a macro that runs a side effect during the expansion, but please comment if you know of one.

Option 2

Pluto will check for popular macros and run special AST analysis. I don't like this, because the AST analysis scripts are often spaghetti code, and having lots of them will become a burden to work with.

Option 3

Pluto will contain mock versions of popular macros. This is a stripped down version of the real macro that converts the expression into something that does the correct assignments and references. So not the original macro, just something that works enough for practical use.

Example

I will use Base.@gensym in this example:

help?> @gensym
  @gensym

  Generates a gensym symbol for a variable. For example, @gensym x y is transformed into x = gensym("x"); y =
  gensym("y").

And our example will be:

@gensym x y

Without special treatment, Pluto will see @gensym x y and detect:

Symbol references	Symbol assignments	Function calls	Function definitions
:x, :y	none	@gensym	none

Option 1

Note: because this macro is in Base, we can just expand the macro without the problems of #177 (comment) . (And this is what Pluto actually does already but shhh that messes with my example.)

julia> macroexpand(Main, :(@gensym x y))
quote
    x = Base.gensym("x")
    y = Base.gensym("y")
    Base.nothing
end

And we see:

Symbol references	Symbol assignments	Function calls	Function definitions
Base	:x, :y	@gensym (we add it), Base.gensym	none

Perfect! But hard to implement

Option 2

We write an extra check in https://github.com/fonsp/Pluto.jl/blob/v0.11.14/src/analysis/ExpressionExplorer.jl like:

elseif ex.head === :macrocall && ex.args[1] === Symbol("@gensym")
	symbols_to_be_assigned = ex.args[2:end]
	return SymbolsState(Set{Symbol}(), Set{Symbol}(symbols_to_be_assigned))
...

fairly simple for this macro, but it can get very complicated.

Option 3

We would write a mock version, for example

function mock_gensym(s::Symbol, ss...)
	:($(s) = $(mock_gensym(ss...)))
end
mock_gensym() = 1234

(It doesn't need to be a macro, it can just be a function if that is easier to write.)

mock_gensym(:x, :y)
:(x = (y = 1234))

Which, as far as Pluto's analysis is concerned, is equivalent to the expanded macro.

---

I think that Option 3 might be best, with Option 1 as something to revisit in a couple of months/years

[mcabbott]

So under option 3, nothing has to be re-written if you change how SymbolsState works. That's certainly better than 2.

Where should these mock macros / mocking functions live? Just some MacroZoo folder inside Pluto?

And should they be a little bit namespaced? Presumably Pluto knows what packages have been loaded, so it might make sense to restrict each rule to notebooks which have loaded its owner, to avoid surprises because of some package you've never heard of.

[Pangoraw]

Since macros are non-deterministic, we can't know what code will be executed before a macro call. For this reason, the static analyzers have little to no support for macros. One common solution is to do what has been done with Pluto for now, which is to write package+macro specific static handlers.

However, in the case of Pluto, the goals are different than those of static analysis tools because we actually need to know which variables/functions will referenced/assigned after a macro call. So instead of doing static analysis which would require a lot of work to deal with package version and to support every package, the idea would be to do the analysis in several steps.

First we macro expand the macro call, which will give us the expression to be executed. After analyzing this expression, we have the dependencies/children for this cell and can run the cell but by replacing the call with the expanded expression so that there is no surprise with regard to non-deterministic macro expansion. The whole process would look something like this:

1. Gather all macro calls using the traditional ExpressionExplorer
2. Run all imports/using cells on the notebook process
3. Macro expand all the macro calls on the process notebook
4. Re-explore the corresponding cells with the expanded expressions instead of the macro calls.
5. Compute the cell graph as usual
6. Run the cells, no-need to re-run imports/usings

[schneiderfelipe]

What about macros like @md_str that interpret a string and build a (deep) Expr including interpolated Julia code? I just wrote one (to work with HTML like JSX), but I can't make @bind work with it. I'm not sure if that's a typical macro pattern, but it's definitely useful.

[fonsp]

That's not yet possible inside Pluto, but it will be an exciting possibility after it is fixed! HypertextLiteral.jl has "JSX-like" macro too, and it uses a regular macro instead of a _str macro (and also to support nested interpolation):

@htl(""""
<hello>$(world)</world>
""")

@md_str works in Pluto because it gets special treatment.

Also, @schneiderfelipe you should come to PlutoCon next week to discuss JSX! Or send me an email: [email protected].

[schneiderfelipe]

That's not yet possible inside Pluto, but it will be an exciting possibility after it is fixed!

Nice! I developed something similar to HypertextLiteral.jl, but it is a standard @htm_str macro that behaves very much like @md_str in terms of interpolation (JSX.jl repo). The idea was to support a "component" syntax and to accept arbitrary objects in the structure.

screenshot

I was hoping to make it work in Pluto.jl just like @md_str.

Also, @schneiderfelipe you should come to PlutoCon next week to discuss JSX! Or send me an email: [email protected].

Definitely! 🥳 👍🏽

[Pangoraw]

I just wrote one (to work with HTML like JSX), but I can't make @Bind work with it

@schneiderfelipe string macros appear as regular macro calls in the expression tree so this should be fixed once #1032 is merged:

[zahachtah]

adding an issue with ModelingToolkit.jl that seems to be related. discussed here.

in short:

using ModelingToolkit

@parameters t σ ρ β

@variables x(t) y(t) z(t)

gives

"cannot assign a value to variable workspace3.t from module workspace6"

[SyxP]

I believe the following problem is related. Deleting a @memoize function (using Memoize or some similar packages.)

The example is as follows:

### A Pluto.jl notebook ###
# v0.14.8

using Markdown
using InteractiveUtils

# ╔═╡ dd790891-acb2-459f-b8b6-e1af2e29bf98
using Memoize

# ╔═╡ 3958d75c-38bc-4ea0-ad85-8a11fdd299c2
@memoize function f()
	1
end

# ╔═╡ 8bee9b3b-a664-4293-b762-c90e89530bfa
f()

# ╔═╡ Cell order:
# ╠═dd790891-acb2-459f-b8b6-e1af2e29bf98
# ╠═3958d75c-38bc-4ea0-ad85-8a11fdd299c2
# ╠═8bee9b3b-a664-4293-b762-c90e89530bfa

Deleting the second cell still keeps f().