rebar3 compile bare from mix fails 'unable to run 'pre hooks'

[ghost]

Example repo :
https://github.com/bryanhuntesl/poolboy_error

Docker image you can run to quickly reproduce the issue :
bryanhuntesl/reproduce-poolboy-mix-rebar3-compilation-bug

When mix calls to a rebar3 project in it's typical fashion - rebar3 compile bare --paths ... the rebar3 project fails to compile.

When rebar3 is executed within the project directory, compilation succeeds.

Invoked via mix (failure):

[root@47d39b1e6774 poolboy_error]# mix compile
===> Compiling gen_fsm_compat
===> Unable to run pre hooks for 'compile', command 'erl_vsn' not found.
** (Mix) Could not compile dependency :gen_fsm_compat, "/root/.asdf/installs/elixir/1.6.5/.mix/rebar3 bare compile --paths "/root/poolboy_error/_build/dev/lib/*/ebin"" command failed. You can recompile this dependency with "mix deps.compile gen_fsm_compat", update it with "mix deps.update gen_fsm_compat" or clean it with "mix deps.clean gen_fsm_compat"

Invoked within the dependency directory (success):

[root@47d39b1e6774 poolboy_error]# cd deps/gen_fsm_compat/
[root@47d39b1e6774 gen_fsm_compat]# /root/.asdf/installs/elixir/1.6.5/.mix/rebar3 compile bare
===> Verifying dependencies...
===> Compiling gen_fsm_compat

Invoked within the dependency directory with --path argument (failure)

[root@47d39b1e6774 gen_fsm_compat]# /root/.asdf/installs/elixir/1.6.5/.mix/rebar3 bare compile --paths "/root/poolboy_error/_build/dev/lib/*/ebin"
===> Compiling gen_fsm_compat
===> Unable to run pre hooks for 'compile', command 'erl_vsn' not found.

Environment

• Add the result of rebar3 report to your message:

$ rebar3 report "my failing command"
...

rebar3 debug output and rebar3 report are attached

Looking into the debug output - it seems unable to run 'pre hooks' - I've seen this behavior in other projects too.

Unable to run pre hooks for 'compile', command 'erl_vsn' not found.

run.rebar3.debug.txt

rebar3.report.txt

[ghost]

@ericmj ^ Elixir guys are mentioning a lot of similar issues.

[ericmj]

I show how to reproduce with only rebar3 in the issue linked above: rebar3 bare compile --paths ""

[ferd]

So I'm not too familiar with the bare compiler; my understanding is that it allows hex to handle all the dependencies, but will only run the compilation for its own stuff. This seems to be relying on a rebar_erl_vsn plugin, which is declared as a provider hook. When was the last version where this worked?

So there's a few things I can think of that go wrong:

• has the erl_vsn plugin been downloaded? Is it expected that rebar3 installs the plugins for the run on a bare compile, or that the caller provides it?
• is the path for the plugin passed in?
• around december 2017 we released a bunch of fixes to plugin paths that would cause failures due to loading/unloading around custom resources (see https://github.com/erlang/rebar3/releases/tag/3.5.0); this ended up in a kind of fixup of how plenty of path management was done around the compiler and plugin. This leads to some unloading and reloading of plugin code within the compiler to avoid polluting build artifacts themselves (I'm still majorly annoyed the erlang compiler requires the deps to be loaded in memory to do its thing, rather than using paths) and I could imagine this having an impact assuming the two previous points are covered

I'm thinking the third one is a likely candidate, but only if all of the Elixir folks with problems went from pre-3.5.0 versions to post-3.5.0 versions around the same time

[ghost]

@Licenser - I'm sure the issue isn't sepecific to your erl_vsn plugin but perhaps you'd care to comment.

@ferd I pushed a docker image with the bug environment to docker - bryanhuntesl/reproduce-poolboy-mix-rebar3-compilation-bug

[ericmj]

Elixir still uses rebar 3.3.6 by default, I will check if the issue exists in both versions.

Elixir handles the deps for rebar projects but since plugins are handled separately we can not do anything with them. The bare compiler is supposed to fetch and compile them.

[tsloughter]

Found the issue. For some reason bare compile runs in the bare namespace. So it is looking for {bare, erl_vsn} which doesn't exist.

Not sure of the fix yet...

[ferd]

Would it make sense to force the namespace to default in State when passing the value to the compiler, and then revert it once we get the new state back?

[tsloughter]

That is what I did to fix it. No need to revert since state is not returned.

Line 49 of rebar_hooks:

case rebar_core:do(HookProviders, rebar_state:namespace(State1, default)) of

I don't think this is a feature that is used... so it should be safe. Only something like alpaca or efene would be using it I believe, and I'm not seeing it used in their providers currently.

[ferd]

I'd have forced the default profile at https://github.com/erlang/rebar3/blob/master/src/rebar_prv_bare_compile.erl#L49 and then the rest would have worked as usual -- maybe that's what you did.

As for rebar_hooks, I guess the question is whether hooks should always use an explicit namespace when not using default? I think I'd tend to err on the side of caution and leave things as they are, since the requirements are to do the same when specifying the hooks in the provider declaration, along with deps and everything. Better to operate with a full blown namespace switch in the usual case than doing a half-and-half ?

[tsloughter]

Oh true, I guess we could set the namespace (not profile) in bare compile since it is a special provider.

[tsloughter]

Right, right now it either uses the explicit name space, like {bare, compile}, {namespace(), provider()}, or the current namespace, so running rebar3 bare compile will look for the providers {bare, provider()}. But what is confusing is the fact that the hooked provider is still defaulting to default, it is essentially translating:

{compile, erl_vsn}

to

{{default, compile}, {bare, erl_vsn}}

Edit: I was wrong about namespace and do, hehe, removed that part.

[tsloughter]

Oh right, it is that namespaces act as do. So:

rebar3 plugins list, upgrade

Is what works and upgrade "inherits" the namespace from the first command and is why in core:do we use the state's namespace.