Including in a release Erlang and Elixir modules

[saleyn]

I have an Erlang project that includes one Elixir *.ex file in the src directory alongside with *.erl files. I added this line to compile it:

{pre_hooks, [{erlc_compile, "elixirc -o $REBAR_BUILD_DIR/lib/my_app/ebin src/*.ex"}]}.

I can verify that it gets compiled fine with rebar3 compile. However when this project is used by another project as a dependency by being referenced via a link under _checkouts, while the compilation of that project compiles it, making a release doesn't result in the compiled _build/default/my_app/ebin/Elixir*.ex being copied into the release tree. What might cause it to ignore this file and how can I configure rebar3 (or mix if the dependent project is Elixir one) to include this file?

[ferd]

You may want to use https://github.com/Supersonido/rebar_mix as a plugin, which also provides hooks to consolidate protocols to get them included in a release.

[tsloughter]

Can you provide an example project to reproduce? Sounds like it should work fine so I'd need to play around with it.

[saleyn]

@ferd, as far as I understand rebar_mix is designed to handle compilation of complete Elixir applications, whereas in my case I only need to include a single Elixir file in the Erlang project. Is that doable with that plugin? In my case the application is a dependency of a parent Elixir app, which takes care of all consolidations, so I don't need the consolidated modules in this one.

[saleyn]

I found something interesting that may lead to a clue. If I have application one, which has Erlang and Elixir source files under src dir, and application two, that has one listed as a dependency:

{deps, [
  {one, "1.0", {git, "[email protected]:...", {branch, "master"}}},
]}.

and one is symlinked in _checkouts, then the above said behavior is observed - Elixir compiled modules don't get listed in the one.app's modules tuple, and those files are not included in the release when running rebar3 release.

However, if instead I use the rebar3_path_deps plugin, and reference one like this, then the Elixir's compiled beam files are included in the release:

{plugins, [rebar3_path_deps]}.
{deps, [
  {one, {path, "../one"}}
]}.

So, I think the issue is that in the phase of producing one.app those Elixir modules are skipped for some reason, and hence, not included in the release in that case.

[tsloughter]

Maybe it is an issue with REBAR_BUILD_DIR then.

[saleyn]

I put together this example to illustrate the issue. There are two branches there working and not-working.

$ tree                                                                       
├── one                                                                                             
│   ├── rebar.config                                                                                
│   └── src                                                                                         
│       ├── one.app.src                                                                             
│       ├── one.erl                                                                                 
│       └── three.ex                                                                                
└── two                                                                                             
    ├── rebar.config                                                                                
    ├── rebar.lock                                                                                  
    └── src                                                                                         
        ├── two.app.src                                                                             
        └── two.erl 

1. Checkout the working branch, cd to directory two and run rebar3 release. Then check:

tree _build/default/rel/two/lib/one-1.0/
_build/default/rel/two/lib/one-1.0/                                                                 
└── ebin                                                                                            
    ├── Elixir.DecodeError.beam                                                                     
    ├── Elixir.EncodeError.beam                                                                     
    ├── one.app                                                                                     
    └── one.beam

As you see the Elixir beams are generated. This two version of the project is using the rebar3_path_deps plugin.

2. Checkout the not-working branch, remove _build, and run:

$ rebar3 release                                                    
===> Verifying dependencies...                                                                      
===> App one is a checkout dependency and cannot be locked.
===> Analyzing applications...     
===> Compiling one                             
===> Missing artifact ebin/Elixir.DecodeError.beam

$ tree _build
_build
└── default
    ├── checkouts
    │   └── one
    │       ├── ebin
    │       │   ├── one.app
    │       │   └── one.beam
    │       ├── include -> ../../../../_checkouts/one/include
    │       ├── priv -> ../../../../_checkouts/one/priv
    │       └── src -> ../../../../_checkouts/one/src
    └── lib
        └── one
            └── ebin
                ├── Elixir.DecodeError.beam
                └── Elixir.EncodeError.beam

So, looks like the files get generated in the wrong place?

If we comment out the {artifacts, [...]} in project one and rerun rebar3 release, it completes but what we get is this:

$ tree _build/default/rel/two/lib/one-1.0 _build/default/lib/one
_build/default/rel/two/lib/one-1.0
└── ebin
    ├── one.app
    └── one.beam
_build/default/lib/one
└── ebin
    ├── Elixir.DecodeError.beam
    └── Elixir.EncodeError.beam

and the Elixir modules are not listed in the one.app file:

$ cat _build/default/rel/two/lib/one-1.0/ebin/one.app 
{application,one,
             [{description,"Sample application one"},
              {vsn,"1.0"},
              {registered,[]},
              {applications,[kernel,stdlib]},
              {modules,[one]},
              {env,[{config,[]}]}]}.

[saleyn]

@tsloughter, possibly you are right, the issue is with my use of $REBAR_BUILD_DIR as the target location of the Elixir's beam files. If that's the case, how would you suggest to compile the Elixir's files so that they are placed correctly irrespective of use of _checkouts, path or being a normal git or hex dependency?

[tsloughter]

@saleyn that would still be the right way to do it, it would just mean there is a bug. I was just suggesting a way you might be able to check by seeing if that value is wrong (like print it out in the shell command that runs the elixir compile) or searching for the beam file to see where it ended up.

[tsloughter]

Thanks for the example, I'll try it in the morning.

[saleyn]

Somewhat unrelated, but is there an option in rebar to specify that a dependency is compile-only? E.g. if project one in this case is dependent on parse_trans, which is only needed in the compilation phase of one but not needed by two, is there a way to make it so, similarly how mix does it with the runtime: false option for dependencies?

[ferd]

IIRC the issue is probably that we moved _checkout directories' build directory to ?DEFAULT_CHECKOUTS_OUT_DIR avoid the symlinking issue of prior times:

rebar3/apps/rebar/src/rebar_app_utils.erl

Lines 256 to 261 in 048412e

	AppInfo = case rebar_app_info:discover(CheckoutsDir, State) of
	{ok, App} ->
	OutDir = filename:join(rebar_dir:checkouts_out_dir(State), Name),
	rebar_app_info:out_dir(
	rebar_app_info:source(
	rebar_app_info:is_checkout(App, true), checkout), OutDir);

This was done as part of #2276 and moves checkout deps' artifacts to _build/<profile>/checkouts/ which unfortunately means $REBAR_BUILD_DIR/lib/my_app/ebin must be $REBAR_BUILD_DIR/checkouts/my_app/ebin or more succintly $REBAR_CHECKOUTS_OUT_DIR/my_app/ebin for the time being.

This isn't great for portability, maybe there's a need to add a value to rebar_env that adds a sort of REBAR_<appname>_OUTDIR set of dynamic env vars. This environment is however created before/after hooks and so they may not return a consistent set of values across all runs and hook points.

[saleyn]

Indeed, the current approach doesn't seem to be portable, as the project shouldn't care about the method a dependency is defined (i.e. _checkouts, path, git, etc). Sounds like an oversight.

[tsloughter]

@saleyn does it only not work when used as a checkout? I noticed in your example the one dependency from git is wrong, it should use git_subdir because one is a subdirectory of rebar-test.

[tsloughter]

Oooh, dammit, I forgot about that checkouts change.

Isn't there an env var for the path to the lib... instead of needing to append lib/my_app/ebin or checkouts/my_app/ebin? I'm going to go check now. If there isn't then it should be added.

[tsloughter]

Looks like there isn't.

I think the change would be to use rebar_state:current_app in rebar_env to add a variable with a path to the apps build dir. $REBAR_CURRENT_APP_BUILD_DIR or something like that.

[saleyn]

I have one more test case in question that doesn't involve _checkouts, where there is an Elixir project three referencing an Erlang project one which has a rebar clause to compile its Elixir elixir.ex file. See branch three:

$ git checkout three
$ cd three
$ mix do deps.get, compile
$ tree .
.
├── _build
│   └── dev
│       └── lib
│           ├── one
│           │   ├── ebin
│           │   │   ├── one.app
│           │   │   └── one.beam
│           │   └── mix.rebar.config
│           └── three
│               ├── consolidated
│               │   ...
│               └── ebin
│                   ├── Elixir.TestModule.beam
│                   └── three.app
├── deps
│   └── one
│       └── one
│           ├── _build
│           │   └── prod
│           │       └── lib
│           │           └── one
│           │               └── ebin
│           │                   ├── Elixir.DecodeError.beam
│           │                   └── Elixir.EncodeError.beam
│           ├── rebar.config
│           └── src
│               ├── one.app.src
│               ├── one.erl
│               └── elixir.ex
├── lib
│   └── three.ex
├── mix.exs
└── mix.lock

As can be seen, the beam files Elixir.{De,Ec}coderError.beam are generated in the wrong place of the tree under the deps node, and if we uncomment the artifacts tuple in the project one's config file, its compilation will fail:

$ vi deps/one/one/rebar.config 
$ grep artifacts -A 3 deps/one/one/rebar.config 
{artifacts, [
  "ebin/Elixir.DecodeError.beam",
  "ebin/Elixir.DecodeError.beam"
]}.
$ mix deps.compile
===> Analyzing applications...
===> Compiling one
===> Missing artifact ebin/Elixir.DecodeError.beam
** (Mix) Could not compile dependency :one, "/home/serge/.mix/rebar3 bare compile --paths /home/serge/tmp/rebar-test/three/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile one", update it with "mix deps.update one" or clean it with "mix deps.clean one"

[saleyn]

Is there a workaround for the test case above that I can use with the current version of the rebar3 to be able to compile a mixed Erlang/Elixir project by specifying the proper location for the compiled Elixir files?

For the sake of completeness I added the build example using the mix tool for a project with a dependency where both have a mix of Erlang/Elixir files, and the mix tool produces correct result:

$ cd ../two
$ rm -fr deps _build
$ git checkout -b mix
$ git pull origin mix
$ (cd .. && tree one two)
one
├── mix.exs
└── src
    ├── elixir.ex
    └── one.erl
two
├── mix.exs
├── mix.lock
└── src
    ├── two.erl
    └── two.ex

$ mix do deps.get, compile
$ tree _build/dev/lib/{one,two}/ebin
_build/dev/lib/one/ebin
├── Elixir.DecodeError.beam
├── Elixir.EncodeError.beam
├── one.app
└── one.beam
_build/dev/lib/two/ebin
├── Elixir.TestModule.beam
├── two.app
└── two.beam

[ferd]

Well for your example with one/three, I don't quite know. The environment setting does specify using absolute paths:

rebar3/apps/rebar/src/rebar_env.erl

Line 43 in dc3eca5

{"REBAR_BUILD_DIR", filename:absname(rebar_dir:base_dir(State))},

but it sets the path based on the current working directory initially (

rebar3/apps/rebar/src/rebar_dir.erl

Lines 39 to 43 in dc3eca5

	%% @doc returns the directory root for build artifacts
	%% for the current profile, such as `_build/default/'.
	-spec base_dir(rebar_state:t()) -> file:filename_all().
	base_dir(State) ->
	profile_dir(rebar_state:opts(State), rebar_state:current_profiles(State)).

) and then expands it because rebar3 sort of always expects to be working from the project root.

But mix is doing its build by switching the directory, and so the hooks are run without knowing of the appropriate project hook. This ends up being in a bad situation where for Elixir's build model, the hook would need to be {compile, "elixirc -o ebin src/*.ex"} but that won't work for Rebar3 itself because Rebar3 would expect the hook to be specific about where it sends the data.

This is handled properly in the bare erlang compiler (which is used as an interface for other apps by changing the output directories in

rebar3/apps/rebar/src/rebar_prv_bare_compile.erl

Line 59 in dc3eca5

DefaultOutDir = os:getenv("REBAR_BARE_COMPILER_OUTPUT_DIR", rebar_dir:get_cwd()),

)

If you output the whole environment when running the hook under mix, you'll also see that it generates a few extra values and others exist:

# this one tells you where the project root is and why paths get relative to this
REBAR_ROOT_DIR=/tmp/rebar-test/three/deps/one/one/.
...
# these are added by Mix
REBAR_BARE_COMPILER_OUTPUT_DIR=/tmp/rebar-test/three/_build/dev/lib/one
REBAR_PROFILE=prod
REBAR_CONFIG=/tmp/rebar-test/three/_build/dev/lib/one/mix.rebar.config

The sort of issue then is that when REBAR_BARE_COMPILER_OUTPUT_DIR is present, the output should be redirected there instead of $REBAR_BUILD_DIR/lib/one/ebin.

This can be done:

diff --git a/one/rebar.config b/one/rebar.config
index c5ea44a..6a49ed8 100644
--- a/one/rebar.config
+++ b/one/rebar.config
@@ -1,5 +1,5 @@
 {pre_hooks,
-  [{compile, "elixirc -o $REBAR_BUILD_DIR/lib/one/ebin src/*.ex"}]}.
+  [{compile, "elixirc -o ${REBAR_BARE_COMPILER_OUTPUT_DIR:-$REBAR_BUILD_DIR/lib/one/ebin} src/*.ex"}]}.

 {deps, []}.

The project will now be able to compile itself fine when called from within rebar3, but also to have the hook locate the code properly when a third-party build tool calls the shots and has a different file structure.

[saleyn]

Thanks @ferd. Maybe it would be better for rebar to set REBAR_BARE_COMPILER_OUTPUT_DIR env to $REBAR_BUILD_DIR/lib/one/ebin, if not already set? This way the projects would not have to deal with this conditional logic.

I followed your suggestion, and that fixed the example project two in branch three, but the project three in branch three where a mix-based project three had one as a dependency is still failing:

[.../three]$ mix do deps.get, compile
===> Analyzing applications...
===> Compiling one
===> Missing artifact ebin/Elixir.DecodeError.beam
** (Mix) Could not compile dependency :one, "/home/serge/.mix/rebar3 bare compile --paths /home/serge/tmp/rebar-test/three/_build/dev/lib/*/ebin" command failed. Errors may have been logged above. You can recompile this dependency with "mix deps.compile one", update it with "mix deps.update one" or clean it with "mix deps.clean one"
[.../three]$ tree _build/dev/lib/
_build/dev/lib/
├── one
│   ├── ebin
│   │   ├── one.app
│   │   └── one.beam
│   ├── Elixir.DecodeError.beam
│   ├── Elixir.EncodeError.beam
│   └── mix.rebar.config
└── three

So, the Elixir output files are not placed under ebin.

[ferd]

Well that value is not set by rebar3 ever, it is set for rebar3. It could for example be set by default but not used with the rebar3 bare compiler and be wrong to use in a hook then.

I'm just assuming it won't be set, but that may be wrong. If someone had the value set, then hooks would start dumping files in the wrong environment.

It's a hack I'd be hard pressed to support, and just a mess because we're trying to make two tools with different project structures agree when calling each other somewhat recursively.

I don't know of a good long term solution, just shitty hacks. Maybe that patch would be acceptable. I guess chances are low enough?

[saleyn]

I guess the alternative is to introduce another variable (e.g. "REBAR_PROJECT_OUTPUT_DIR"?) that would be set to the target location of the current project's output directory. It would check for the REBAR_BARE_COMPILER_OUTPUT_DIR override and make necessary adjustments. So the config would be:
{pre_hooks, [{compile, "elixirc -o $REBAR_PROJECT_OUTPUT_DIR/ebin src/*.ex"}]}.

[ferd]

Even if we changed the hook mechanism to specifically know which commands are run so that the value is overriden in case of a bare compile call so it runs the compile hooks and adds a REBAR_PROJECT_OUTPUT_DIR as needed, we would still have problems.

Rebar3 just has the basic directories and not necessarily app-specific directories as variables in hooks (it would just link to _build/lib rather than _build/lib/libname) whereas the Elixir path points out to a direct appname/ebin dir.

The reason for that is that the hook variables are uniform across all hooks, and some hooks are not app-specific, and therefore can't link to a specific app's path. However the rebar3 bare compile is special because it is always about just one app. The variable levels are just not compatible. We could maybe do it if we were to start introducing one variable per app, but not all apps are visible at the same time depending on the build phase...

[ferd]

Maybe we could set a value that we guarantee only exists when being called by the bare compiler (we clear it if not) which would indicate a way to do that workaround?

It still forces the logic onto the caller but I don't think there is such a thing as a transparent way to make it work.

[saleyn]

My objective is to be able to bundle a mix of Erlang/Elixir modules in one app, and make sure that if used as a dependency the app can be properly compiled whether the top-level build is done by rebar3 or by mix. Right now they are producing different results, and from my experiments, mix works pretty much "out of the box", but rebar3 does involve some pain. :(

[ferd]

Yeah. That's because they have entirely different compiling models. We're sort of stuck with that. Rebar3 gave full standalone modes to Mix and had to write custom compiler extensions to build Elixir properly, but it isn't perfect. I'm pretty sure we do a lot of things that are really painful for people using Bazel too.

Regarding hooks:

rebar3/apps/rebar/src/rebar_hooks.erl

Lines 90 to 101 in dc3eca5

	Hooks ->
	Env = rebar_env:create_env(State, Opts),
	CommandHooks = lists:filter(
	fun({_, C, _}) when C =:= Command -> true;
	({C, _}) when C =:= Command -> true;
	(_) -> false
	end,
	Hooks
	),
	?DEBUG("\t{~p, ~p}.",
	[Type, CommandHooks]),
	lists:foreach(fun(Hook) -> apply_hook(Dir, Env, Hook) end, CommandHooks)

is where this takes place. I'll try to think of a way things could be twisted to create a potential REBAR_APP_OUTDIR variable that could point to ebin/, but it wouldn't always be there if the hook is run on a multi-app level (eg. umbrella project top-level's compile hook).

I'm gonna have to think a bit about what would be a good way to make it work. I'm not sure it can be done without being a huge mess.