Add support for rebar3 #118

davisp · 2016-03-18T20:49:36Z

This should hopefully allow users to use rebar3 to compile jiffy while
also not breaking any project still using rebar2.

sescobb27 · 2016-03-21T04:59:24Z

Hi, I'm erlang newbie starting a cowboy project with rebar3, this work perfect with rebar3 shell with one addition to rebar.config file extracted from port_compiler

{overrides,
 [{override, jiffy, [
     {plugins, [pc]},
     {artifacts, ["priv/jiffy.so"]},
     {provider_hooks, [
       {post,
         [
           {compile, {pc, compile}},
           {clean, {pc, clean}}
         ]
        }
      ]}
  ]}
]}.

the problem is when running the release

$ reba3 release
$ _build/default/rel/ec_api/bin/ec_api console

then running jiffy:decode(<<"{\"foo\": \"bar\"}">>). throws ** exception error: undefined function jiffy:decode/1
so my question is, is that feature a work in progress? or can you hint me what i'm doing wrong?
thanks.

davisp · 2016-03-21T17:05:15Z

@sescobb27 Thanks for giving this a whirl. I'm still learning rebar3 so its definitely a bit in development in terms of wanting people to test it out and see if it breaks anything.

The first thing I'd try is to make sure that you've set the jiffy dependency branch to "feature-add-rebar3-support" and then remove the override from your application's rebar.config when using this branch. I've got a test repo at [1] that I've just pushed. Jiffy appears to work fine after a rebar3 release && ./_build/default/rel/goop/bin/goop console. Let me know if it works for you.

[1] https://github.com/davisp/goop

sescobb27 · 2016-03-29T02:45:40Z

hi, sorry for my late response i was on vacations.
i had the same issue trying your example, don't know if it's because i'm on linux

i had to add the following code to make it work

{overrides,
 [{override, jiffy, [
     {plugins, [pc]},
     {artifacts, ["priv/jiffy.so"]},
     {provider_hooks, [
       {post,
         [
           {compile, {pc, compile}},
           {clean, {pc, clean}}
         ]
        }
      ]}
  ]}
]}.

btw in my app it worked using rebar3 shell but didn't using the release bin console

davisp · 2016-03-29T11:29:52Z

@sescobb27 Can you try adding this line:

io:format("~p~n", [OptsToAdd]),

on line 60 in _build/default/lib/jiffy/rebar.config.script and then re-run rebar3 release?

sescobb27 · 2016-03-30T05:05:49Z

in goop i got the following

[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
===> Verifying dependencies...
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
===> Fetching pc ({git,"[email protected]:blt/port_compiler.git",
                              {branch,"master"}})
===> Plugin {pc,{git,"[email protected]:blt/port_compiler.git",
                            {branch,"master"}}} not available. It will not be used.
===> Compiling jiffy
===> Unable to run pre hooks for 'compile', command 'compile' in namespace 'pc' not found.

in mine i got the following

[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
===> Verifying dependencies...
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
[{deps,[]},
 {erl_opts,[]},
 {plugins,[{pc,{git,"[email protected]:blt/port_compiler.git",
                    {branch,"master"}}}]},
 {provider_hooks,[{pre,[{compile,{pc,compile}},{clean,{pc,clean}}]}]}]
===> Compiling ec_api
===> Starting relx build process ...
===> Resolving OTP Applications from directories:
          /home/kiro/Documents/Erlang/ec_api/_build/default/lib
          /usr/lib/erlang/lib
          /home/kiro/Documents/Erlang/ec_api/_build/default/rel
===> Missing beam file elf_format <<"/usr/lib/erlang/lib/hipe-3.12/ebin/elf_format.beam">>
===> Resolved ec_api-0.0.1
===> Dev mode enabled, release will be symlinked
===> release successfully created!

kradalby · 2016-04-02T23:43:07Z

This branch works for me on OS X 10.11 with Erlang 18.3.

I just used it as a dependency in my rebar3 project.

Edit:
The same project does not build on Debian Jessie 8.3 with Erlang 18.3.

The compile errors thrown is:
===> Plugin {pc,{git,"[email protected]:blt/port_compiler.git",
{branch,"master"}}} not available. It will not be used.

and

===> Unable to run pre hooks for 'compile', command 'compile' in namespace 'pc' not found.

I am not sure if this is a rebar3 error or something else...

sescobb27 · 2016-04-06T14:02:51Z

in https://github.com/davisp/jiffy/pull/118/files#diff-f310c7cc098862c9835f2ba9675a4060R30 i just commented it

% Plugins = case IsRebar3 of
%     true -> [{pc, {git, PortCompilerUrl, {branch, "master"}}}];
%     false -> [rebar_gdb_plugin]
% end,

% ProviderHooks = if not IsRebar3 -> []; true ->
%     [{pre, [
%         {compile, {pc, compile}},
%         {clean, {pc, clean}}
%     ]}]
% end,

OptsToAdd = [
    {deps, Deps},
    {erl_opts, ErlOpts}
    % {plugins, Plugins},
    % {provider_hooks, ProviderHooks}
],

and it worked, my project compiles without problems BUT it didn't compile NIF library

Erlang/OTP 18 [erts-7.3] [source] [64-bit] [smp:8:8] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V7.3  (abort with ^G)
1> jiffy:decode(<<"{\"foo\": \"bar\"}">>).

=WARNING REPORT==== 6-Apr-2016::08:54:22 ===
The on_load function for module jiffy returned {error,
                                                {load_failed,
                                                 "Failed to load NIF library: '/home/kiro/Documents/Erlang/ec_api/_build/default/lib/jiffy/priv/jiffy.so: cannot open shared object file: No such file or directory'"}}
** exception error: undefined function jiffy:decode/1

In rebar3 it is required to have a Makefile or other instructions for building your C/C++ code outside of rebar itself., I think that is what port_compiler does but it is giving a lot of troubles in linux

lpgauth · 2016-04-06T20:08:06Z

The proper way to do this for rebar2 / rebar3 compatibility is to have a Makefile that's called by pre_hooks / post_hooks. The port compiler aka pc is sketchy at best if the NIF is a dependancy of a dependancy.

http://www.rebar3.org/docs/building-cc

davisp · 2016-04-06T20:33:05Z

As far as I'm concerned this is a bug in rebar3/port_compiler then. How can the standard build chain not support building ports and NIFs? Anyway, I'm very much not going to be maintaining oodles of Makefiles across multiple NIF projects I maintain because they screwed things up. If someone has something better than pc, I'm all ears. But I won't be committing Makefiles that require me to tweak things everytime someone with a new os/arch/vm/whatever comes along and needs to use a NIF I wrote.

sescobb27 · 2016-04-07T04:27:01Z

sorry to hear that, i know it would be a pain in the ass for you to maintain that, i was looking around in rebar3 documentation and found this, https://www.rebar3.org/docs/custom-compiler-plugins does not know if it could work, if it does maybe, i can try to implement it (as i said before i'm new to erlang but i can give it a try). just let me know 😄

fp · 2016-05-12T22:04:23Z

The port compiler (pc) should be available via hex.pm without the need to add the github repo.

{plugins, [pc]}.
{provider_hooks,
 [
  {pre,
   [
    {compile, {pc, compile}},
    {clean, {pc, clean}}
   ]}
]}.

should work just fine with rebar3.

The question is: does that make rebar2 choke when that is in the rebar.config file? If no one answer, I'll fiddle with it and verify.

Update: had a chance to look into this and there was one other issue with the test cases that needed sorting out because of the PWD directory differences between rebar/rebar3 eunit execution. To make it simpler to examine the change, I put up a separate PR #120. Hope this helps!

davisp · 2016-09-22T22:39:24Z

Hey everyone,

I'm taking another crack at getting rebar3 support to work without terribleness for end users. Can anyone give this a try? It seems to work for me with both rebar2 and rebar3. If anyone has any issues let me know.

davisp · 2016-09-22T22:41:40Z

Also, FYI, this removes everything about the port compiler plugin, so if you have anything in your overrides to inject the port compiler this may or may not break that.

I've gone and replaced the port_compiler with a new tool I hacked together from rebar2 that can be found here: https://github.com/davisp/erlang-native-compiler

PropEr broke my support for R14. Turns out that EQC Mini is quite usable so I've just switched to that. If EQC Mini exists it will be used, if not the test is skipped gracefully.

Allow Jiffy to be used easily in projects using either rebar 2 or 3.

ElectronicRU · 2016-10-10T07:10:57Z

It works for me, but just saying, you shouldn't force push to branches you expect people to clone and test. rebar3 commit locking chokes on that.

vendethiel · 2016-10-10T17:37:54Z

Rejoicing! Thank you very much.

This was referenced Mar 18, 2016

Rebar3 support? #117

Closed

https://hex.pm/packages/jiffy/0.14.7 does not contain plugins/rebar_gdb_plugin #116

Closed

When included in a rebar3 project, jiffy will not compile on its own #96

Closed

Rebar3 #110

Closed

sescobb27 mentioned this pull request Apr 9, 2016

Release/Compilation issue in linux blt/port_compiler#23

Open

tsloughter mentioned this pull request May 11, 2016

rebar_gdb_plugin not available erlang/rebar3#1181

Closed

romul mentioned this pull request Jul 12, 2016

Can't get it working TheRealReal/new-relixir#7

Closed

davisp force-pushed the feature-add-rebar3-support branch from 6fb6b04 to daad18e Compare September 22, 2016 22:35

davisp added 3 commits September 29, 2016 12:54

Replace PropEr with EQC

68b6040

PropEr broke my support for R14. Turns out that EQC Mini is quite usable so I've just switched to that. If EQC Mini exists it will be used, if not the test is skipped gracefully.

Update the list of Erlang VMs to use on Travis-CI

0789c08

Add support for rebar3

c7756da

Allow Jiffy to be used easily in projects using either rebar 2 or 3.

davisp force-pushed the feature-add-rebar3-support branch from 4e48aab to c7756da Compare September 29, 2016 17:56

davisp merged commit a7db724 into master Oct 10, 2016

davisp mentioned this pull request Dec 15, 2016

Add rebar3 compabibility #120

Closed

davisp deleted the feature-add-rebar3-support branch November 6, 2017 16:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for rebar3 #118

Add support for rebar3 #118

davisp commented Mar 18, 2016

sescobb27 commented Mar 21, 2016

davisp commented Mar 21, 2016

sescobb27 commented Mar 29, 2016

davisp commented Mar 29, 2016

sescobb27 commented Mar 30, 2016

kradalby commented Apr 2, 2016

sescobb27 commented Apr 6, 2016

lpgauth commented Apr 6, 2016

davisp commented Apr 6, 2016

sescobb27 commented Apr 7, 2016

fp commented May 12, 2016 •

edited

Loading

davisp commented Sep 22, 2016

davisp commented Sep 22, 2016

ElectronicRU commented Oct 10, 2016

vendethiel commented Oct 10, 2016

Add support for rebar3 #118

Add support for rebar3 #118

Conversation

davisp commented Mar 18, 2016

sescobb27 commented Mar 21, 2016

davisp commented Mar 21, 2016

sescobb27 commented Mar 29, 2016

davisp commented Mar 29, 2016

sescobb27 commented Mar 30, 2016

kradalby commented Apr 2, 2016

sescobb27 commented Apr 6, 2016

lpgauth commented Apr 6, 2016

davisp commented Apr 6, 2016

sescobb27 commented Apr 7, 2016

fp commented May 12, 2016 • edited Loading

davisp commented Sep 22, 2016

davisp commented Sep 22, 2016

ElectronicRU commented Oct 10, 2016

vendethiel commented Oct 10, 2016

fp commented May 12, 2016 •

edited

Loading