buildMix: fix: initial try #112477

happysalada · 2021-02-09T07:36:57Z

Motivation for this change

buildMix is broken
More details on #105002

Things done

Here are instructions on how to test.

clone the following repo https://github.com/happysalada/nix_demo
run nix-build ./build.nix in it.
(I have added the instructions in the beam-section to the build.nix file).

(upon your first try, nix will complain that the sha256 is empty, I'm just leaving this empty so that everybody can verify by themselves.)
(please remember that in order to run the release you will need to set the env var RELEASE_TMP to a writable directory)

Here are the questions I have that are open

by including the building of the js dependencies, the build does not seem to be a Fixed Output Derivation. By trying to add the following three lines at the end of the buildMix file

  outputHashAlgo = "sha256";
  outputHashMode = "recursive";
  outputHash = sha256;

it can be verified, that whatever sha256 output is given, the resulting hash will always be different.
I have to say I'm a little puzzled as to why that is the case.
.

happysalada · 2021-02-09T23:21:54Z

@hauleth you might be interested in this. Absolutely no worries if you don't have the time.
Most of the stuff here is based off your stuff. Thanks again for a lot of gems!

happysalada · 2021-02-09T23:26:31Z

@cw789 you might be interested into this. I want to distill some of the greatness you put in nix_elixir_example in the doc ideally. No worries if you don't have time.

happysalada · 2021-02-09T23:48:29Z

Initial adding of documentation

cw789 · 2021-02-10T09:01:12Z

Hi Raphael.
I can have a look, but won't be able to test at the moment, as I'm currently sick.

First thing I notice, buildMix seems to be a different thing then.
Former it was to build (compile) Elixir dependencies (packages / libraries).
Now it will create a release (services).

mkdir: cannot create directory ‘/nix/store/n8wm8iy4w51na8b4bzizx6yxhwd693fl-union-0.0.1/tmp’: Permission denied

To this trouble - releases always need a writable tmp directory,
wich default is release_directory/tmp.
As your release is within the nix store which is never writable,
you need to change the tmp directory before starting the release.
This could be archived by setting env variable RELEASE_TMP - to e.g. /home/xxx/my_release_tmp

You can do this also within env.sh (documented within mix release).

happysalada · 2021-02-10T09:05:36Z

@cw789 Thanks Chris, your help is much appreciated. ❤️
No rush at all on the review, I don't think this will be merged that quick.
Thanks a lot for the tips. I will have a look at those tomorrow.
I agree with you that the buildMix seems to have been different. I removed large part of what was there before in the hope that somebody would say why they were needed. I think this will take a while to be merged, so hopefully someone can say if we forgot/missed something.

pkgs/development/beam-modules/build-mix.nix

minijackson

Thanks for the PR! Here are small overview comments, I haven't gotten to test it properly.

pkgs/development/beam-modules/build-mix.nix

doc/languages-frameworks/beam.section.md

minijackson · 2021-02-11T17:26:12Z

doc/languages-frameworks/beam.section.md

-#### Building in a Shell (for Mix Projects) {#building-in-a-shell}
+#### Elixir - Phoenix project
+
+Here is an example shell.nix


Is there any way of automating the creation of a nix shell for Elixir projects? Obviously this would be asking for another PR for however wishing to do so, but this seem quite useful, and I think some things can be automated, like setting mix-related variables. Can't this use passthru.env?

I'm thinking i should have a look at @cw789 's repo on nix_elixir_demo, he has a lot of stuff in there relating to shell and making the environment work.
https://github.com/cw789/elixir_nix_example
I agree with you, it would be way better to define a shell that can be passed through.

minijackson · 2021-02-11T17:36:22Z

pkgs/development/beam-modules/build-mix.nix

+        env = shell self;
+      };
+    });
+in pkg lib.fix


I'm new to the lib.fix function, but shouldn't it be lib.fix pkg? Am I missing something?

I just copied that from before. I have to say I'm not sure. I tested both ways and it doesn't seem to make a difference. We'll have to ask somebody more knowledgeable here.

From my understanding, this allows the use of self, which is used only in passthru.env. I can confirm that trying to build mobilizon.env with pkg lib.fix produces an evaluation error, while with lib.fix pkg it builds but fails anaway with variable $src or $srcs should point to the source. In my opinion, we should drop the passthru and lib.fix pkg for now and open another PR if the need arises.

Isn't the passthru stuff used for potentially passing the shell ? I'm not sure, just wondering if by removing those, we are loosing the ability to define ourselves the shell so that the user doesn't need do

I am no sure, as far as I know, nix-shell for a package has always worked for me, whether the package defines an env. But I am not proposing to remove them indefinitely, though. I think defining the shell there is nice, because we can easily extend it using all the attributes of the package.

minijackson · 2021-02-11T20:29:53Z

As an aside, I would never have thought that you could spin up postgresql like that!

For testing I personally would create a nixos VM, but your method is good to have in mind 😃

minijackson · 2021-02-11T20:41:51Z

Also, if we are changing the meaning of buildMix, like @cw789 said, shouldn't we be using mix releases? I think they weren't actually part of Mix when buildMix was created, so it might make more sense now.

Also, I have a question that is maybe out of scope for this PR, but might be nice to keep in mind. It seems to me that configuration of Mix projects are given at build time. Does that mean that for future NixOS services of Mix applications, rebuilding your system will mean you will have to rebuild every configured Mix applications? Is there a way around it?

happysalada · 2021-02-11T23:34:46Z

@minijackson On the mix release comment, unless I misunderstand something, this is using mix releases to pack your application.

After @cw789 's comment, here is what I am thinking about.

ask user to mix release.init and add the RELEASE_TMP="/private/tmp"
I feel a bit weird having to define the tmp directory before creating a service, but I don't see any way around it at the moment.

happysalada · 2021-02-12T00:17:02Z

I have added a build.nix file to make the process simpler and it includes a demonstration of the RELEASE_TMP variable. It's working great.
There are two more variables needed to run a basic release though. DATABASE_URL and SECRET_KEY
I still feel weird about encoding the DATABASE_URL on building.
Perhaps we should include something in the doc to ask users to put that variable in the dynamically configurable variables? like in config.prod.exs.
Maybe it's just me.

happysalada · 2021-02-12T00:21:03Z

All things considered, would it be a possibility to ask users to put their DATABASE_URL and SECRET KEY in the build.nix file?
We could ask them to supply RELEASE_TMP at the same time.
Does it make sense to everybody?

minijackson · 2021-02-12T13:40:22Z

unless I misunderstand something, this is using mix releases to pack your application.

oops, that's my bad, I forgot to read after the comma. I got the impression that it wasn't when I read that there was no install phase, I figured that it was up to the end-user to choose what to install. In retrospect, that is obviously not the case.

ask user to mix release.init and add the RELEASE_TMP="/private/tmp"

Isn't mix release.init for upstream to do? I am not sure what the RELEASE_TMP variable would mean, can you give more details?

I've had a bit of thinking and experimenting with your PR. First of all, I managed to compile the Elixir part of Mobilizon without any particular issues 🎉

I was thinking that it would be best if we could specify in the configuration that we want these variables to be resolved at runtime, and fetch them from the environment (instead of fetching the environment at build time). I found this nice feature while looking through the docs: you can specify a runtime.exs file, which is exactly what I wanted.

With this, I added my custom Mobilizon runtime.exs in preBuild:

{
  # ...
  preBuild = ''
    cp "${./runtime.exs}" config/runtime.exs
  '';
}

With this, here's what I'm proposing for the moment: we add a runtimeConfiguration argument to buildMix which "environmentalizes" the configuration, and we automatically put in $src/config/runtime.exs and mix release takes care of the rest. This allows not recompiling the whole application on each configuration change, and would prevent secrets from being world readable in the Erlang bytecode stored in the Nix store.

An even better solution in my opinion would be to generate the runtime.exs in the style of RFC42, instead of going through environment variables. This would be nicer since there can be quite a lot of configuration options, so providing a runtime.exs which allows the modification of every variable can become quite tedious and error prone. The drawback for now is that we probably have to find a way to provide the runtime.exs outside of the release, which may be hacky. I'll try to find if this is possible.

But the good news is that I managed to get Mobilizon local instance up and running 🎉, although I had to compile the yarn part manually...

minijackson · 2021-02-12T13:43:01Z

pkgs/development/beam-modules/build-mix.nix

+      postUnpack = ''
+        export HEX_HOME="$TMPDIR/hex"
+        export MIX_HOME="$TMPDIR/mix"
+        export MIX_DEPS_PATH="$TMPDIR/deps"


Is there any particular reason to put mix deps in the directory, and not directly in $src/deps? I personally find that this makes the environment a bit different than usual, which can surprise users. At least it bite me a little bit while I was fixing the Mobilizon build, although this was definitely not a major issue.

Can you detail a bit what you mean?
I have no problem with changing this, just want to understand where the issue lies.
I was actually thinking of setting the deps_path to where dependencies are already and not even do a copy.

So regarding this, doing a cp on $src will not work
cp: cannot create directory '/nix/store/ycqg5a39gzjcvim7cszk7q7260glw7lm-source/deps': Permission denied
I learned something today as well.

I also tried not copying the dependency folder, but it doesn't work. The compilation of the dependencies will require that the dependency folder is writable. I also learned that today. I added a comment to prevent future people from being tricked.

Can you detail a bit what you mean?

When packaging mobilizon, I had to provide some locale files in a dependency directory, because it was downloading them at build time. My first instinct was to add this in pre-build:

cp -R ${cldrLocales}/* deps/ex_cldr/priv/cldr/locales/

That was because in a "normal" elixir project, the dependencies are under the same directory as the source code. But with the current implementation, this had to be corrected to:

cp -R ${cldrLocales}/* $MIX_DEPS_PATH/ex_cldr/priv/cldr/locales/

So regarding this, doing a cp on $src will not work

What about if you do cp --no-preserve=mode -R "${mixDeps}" ./deps? I think the $src is the nix store path of the source, not the current build directory, my bad.

This really is just nitpicking, though, if it's a hassle, you can just forget about it ^^

happysalada · 2021-02-13T02:00:41Z

@minijackson I like your idea of using runtime variables.
What about letting the user define their runtime.exs file, and then just patching the env.sh.exs file in the end.
Since env.sh.exs is actually just a shell file, it would just be a matter of concatenating export VAR="my_value" at the end of it.

I personally only use runtime env vars for secret stuff, however I'm concerned about the experience for people starting a new phoenix project. I don't know why, but the phoenix default is to have DATABASE_URL as a compile time dependency. I was thinking to add compileEnvVars and just inject that in the preConfigure hook if people need.

Then we could define runtimeEnvVars inside a service definition (in a later PR perhaps?) and those would just patch the env.sh file inside the release.
That way we stay completely out of defining runtime env vars when we are just building the service.

PS: Here is the explanation around RELEASE_TMP and mix release.init.

mix release.init
When you mean mix release.init done by upstream, we mean the end user right? The reason I mentioned asking the user to do it, is that it's not mandatory, some people never do it. Maybe we can add a check for those files on build to make sure they are there. The reason it can be useful is that it provides a file env.sh.exs that is just a shell file to pass on extra runtime environment variables. In the official elixir doc, they use the runtime.exs by just calling System.fetch_env!/1 . It looks like the job of supplying environment variable is instead carried out by env.sh
RELEASE_TMP
Upon starting a release, some space on the disk will be used to write some configuration variables in a file. It does so in the RELEASE_TMP specified directory. Usually it's just the same directory as the release, but if we build the release in the nix store, that location won't be writable. So we need to supply a writable location that will be used to store the BEAM config variables. If you don't set this variable and try to run your release, it should throw an error regarding a not writable path. I'm not sure why you didn't get an error.

PPS: out of curiosity, did you use the yarn2nix tool? I'm wondering how well it works, if it works well, I was thinking we could add some documentation about it too.

happysalada · 2021-02-13T03:54:07Z

After some more tests, here is what I found out.

mix release.init is unecessary, no need to force people to use it.
In order to run the release, simply setting RELEASE_TMP to a writable dir will work. (of course you will also need to set your other envirenment variable for your release).
I kept the depsPreConfigure hook in there, just in case people actually have build time env variable dependency. Even though I can not personally think of a good reason why. I just added a recommendation in the docs.

Here are the tasks that could be improved.

Add docs about yarn support? Maybe with yarn2nix?
Add an exemple service file? I don't think anybody is going to run their release raw.
Here is an early draft, just putting it here for discussion
with an example service of $name

    users.groups.${name} = { };
    users.users.${name) = {
      description = "${name} service user";
      group = "${name}";
      isSystemUser = true;
    };
    systemd.services.${name} = {
      description = "My super service";
      wantedBy = [ "multi-user.target" ];
      after = [ "network-online.target" ];
      requires = [ "network-online.target" ];
      ExecStart = "${pkgs.${name}}/bin/${name} start";
      User = "${name}";
      Group = "${name}";
      Restart = "no";
      StateDirectory = "${name}";
      ExecReload = "${pkgs.coreutils}/bin/kill -HUP $MAINPID";
      Environment = [
          "RELEASE_TMP=/private/tmp"
          "MY_VAR="MY_VAL"
      ];
    };

I'm not really sure what to put exactly in the ExecStart command

minijackson · 2021-02-13T17:10:11Z

What about letting the user define their runtime.exs file, and then just patching the env.sh.exs file in the end.
Since env.sh.exs is actually just a shell file, it would just be a matter of concatenating export VAR="my_value" at the end of it.

What do you mean by user? Do you mean the package maintainer, or nixos module maintainer?

If we want to go through with the RFC42-style module, I think it would be best to patch runtime.exs directly, instead of going through environment variables, which was a possibility I hadn't considered, thanks!

If we want to go through with runtime variables, though, I would personally use systemd.services.<service>.environment or systemd.services.<service>.serviceConfig.EnvironmentFile for secrets.

I personally only use runtime env vars for secret stuff, however I'm concerned about the experience for people starting a new phoenix project. I don't know why, but the phoenix default is to have DATABASE_URL as a compile time dependency. I was thinking to add compileEnvVars and just inject that in the preConfigure hook if people need.

Do you mean that the DATABASE_URL is used at compile time by Phoenix, and cannot be changed at runtime? From my understanding, this is against the official library guidelines, but it is still possible. From my experience with Mobilizon, I managed to change the Ecto database hostname without issues. But if you want to change a variable at build time, a package manager can easily do:

{ ... }:

buildMix {
  # ...
  DATABASE_URL = "...";
}

When you mean mix release.init done by upstream, we mean the end user right?

I meant the software developers (as in the Mobilizon devs). But as you said, it is not necessary, one less thing to worry about ^^

Maybe we can add a check for those files on build to make sure they are there

I think there is no need for that, if they are not there, mix release will create them from a template, and we can specify environment variables through systemd, of other means (for example, direnv, or just plain export when we are not using a nixos module).

If you don't set this variable and try to run your release, it should throw an error regarding a not writable path. I'm not sure why you didn't get an error.

Thanks for the clarification, it helped.

It seems RELEASE_TMP is only used when we are starting the application using "daemon" or "daemon_iex", or using "eval" with RUNTIME_CONFIG=true is set in the sys.config file (whatever that means). For comparison, I have RUNTIME_CONFIG=false in my Mobilizon build. I don't think it is a blocker if we just want NixOS modules, but this can definitely be annoying for development. But I think this can easily be fixed by making a wrapper.

PPS: out of curiosity, did you use the yarn2nix tool? I'm wondering how well it works, if it works well, I was thinking we could add some documentation about it too.

I did try to use it, but the "resolutions" feature of package.json doesn't seem to be implemented, so it fails to build for Mobilizon.

I don't think anybody is going to run their release raw.

I just did, to test out things ^^

I can also see it being useful for development.

Here is an early draft, just putting it here for discussion
with an example service of $name

I think your systemd service looks good. Here is some small nitpicks:

{
  systemd.services.${name} = {
    description = "My super service";
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    requires = [ "network-online.target" ];
    # Raw systemd service options inside serviceConfig
    serviceConfig = {
      ExecStart = "${pkgs.${name}}/bin/${name} start";
      # Encouraging people of having a nice set of privacy defaults,
      # but might not be applicable everywhere.
      # Best if it used the software developers upstream.
      DynamicUser = true;
      StateDirectory = "${name}";
      # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
      PrivateTmp = true;
      EnvironmentFile = "/path/to/my/secrets.env";
      # I am not sure about the Restart and ExecReload options,
      # does sending a SIGHUP signal does something to the Erlang VM?
    };
    # Using the nixos declarative way
    environment = {
      # Set to "/tmp", made private by systemd, although not necessary here
      # Maybe this should be in buildMix, set in the wrapper
      RELEASE_TMP = "/tmp";
      # Maybe an example for the StateDirectory?
      MY_DATA_DIR = "/var/lib/${name}";
    };
  };
}

Sorry for the long message, and thanks again!

doc/languages-frameworks/beam.section.md

kalbasit · 2021-02-15T18:16:35Z

doc/languages-frameworks/beam.section.md

+  '';
+  preConfigure = ''
+    # if you have build time environment variable add them here
+    export MY_ENV_VAR="my_value";


Is this the same as the one in the section above? So build environment variables must be exported in both steps? You can make them available to all steps by sending them to the underlying mkDerivation as Nix variables such as MY_ENV_VAR = "my_value";. You could add an attribute set that you include in the args sent to mkDerivation to make it clearer.

You're right. Adding an attribute called build_env_vars = { MY_ENV_VAR = "my_value";}; then using a merge in the mkDerivation in both the build and the fetch steps to make them available.
Thanks!

I added a buildEnvVars attribute. Trying to merge this and the default attrs resulted in an error, so I have removed the merging of the attrs with the rest of the variables for mkDerivation. What changes is that you can't override as easily steps for the derivartion. Here in particular, the preConfigure hook is not overridable anymore from passed attrs. It looses a bit of flexibility. Not sure how to proceed otherwise.

happysalada · 2021-02-15T22:06:59Z

What about letting the user define their runtime.exs file, and then just patching the env.sh.exs file in the end.
Since env.sh.exs is actually just a shell file, it would just be a matter of concatenating export VAR="my_value" at the end of it.

What do you mean by user? Do you mean the package maintainer, or nixos module maintainer?

If we want to go through with the RFC42-style module, I think it would be best to patch runtime.exs directly, instead of going through environment variables, which was a possibility I hadn't considered, thanks!

If we want to go through with runtime variables, though, I would personally use systemd.services.<service>.environment or systemd.services.<service>.serviceConfig.EnvironmentFile for secrets.

I personally only use runtime env vars for secret stuff, however I'm concerned about the experience for people starting a new phoenix project. I don't know why, but the phoenix default is to have DATABASE_URL as a compile time dependency. I was thinking to add compileEnvVars and just inject that in the preConfigure hook if people need.

Do you mean that the DATABASE_URL is used at compile time by Phoenix, and cannot be changed at runtime? From my understanding, this is against the official library guidelines, but it is still possible. From my experience with Mobilizon, I managed to change the Ecto database hostname without issues. But if you want to change a variable at build time, a package manager can easily do:
{ ... }:

buildMix {
  # ...
  DATABASE_URL = "...";
}
When you mean mix release.init done by upstream, we mean the end user right?

I meant the software developers (as in the Mobilizon devs). But as you said, it is not necessary, one less thing to worry about ^^

Maybe we can add a check for those files on build to make sure they are there

I think there is no need for that, if they are not there, mix release will create them from a template, and we can specify environment variables through systemd, of other means (for example, direnv, or just plain export when we are not using a nixos module).

If you don't set this variable and try to run your release, it should throw an error regarding a not writable path. I'm not sure why you didn't get an error.

Thanks for the clarification, it helped.

It seems RELEASE_TMP is only used when we are starting the application using "daemon" or "daemon_iex", or using "eval" with RUNTIME_CONFIG=true is set in the sys.config file (whatever that means). For comparison, I have RUNTIME_CONFIG=false in my Mobilizon build. I don't think it is a blocker if we just want NixOS modules, but this can definitely be annoying for development. But I think this can easily be fixed by making a wrapper.

PPS: out of curiosity, did you use the yarn2nix tool? I'm wondering how well it works, if it works well, I was thinking we could add some documentation about it too.

I did try to use it, but the "resolutions" feature of package.json doesn't seem to be implemented, so it fails to build for Mobilizon.

I don't think anybody is going to run their release raw.

I just did, to test out things ^^

I can also see it being useful for development.

Here is an early draft, just putting it here for discussion
with an example service of $name

I think your systemd service looks good. Here is some small nitpicks:
{
  systemd.services.${name} = {
    description = "My super service";
    wantedBy = [ "multi-user.target" ];
    after = [ "network-online.target" ];
    requires = [ "network-online.target" ];
    # Raw systemd service options inside serviceConfig
    serviceConfig = {
      ExecStart = "${pkgs.${name}}/bin/${name} start";
      # Encouraging people of having a nice set of privacy defaults,
      # but might not be applicable everywhere.
      # Best if it used the software developers upstream.
      DynamicUser = true;
      StateDirectory = "${name}";
      # Implied by DynamicUser, but just to emphasize due to RELEASE_TMP
      PrivateTmp = true;
      EnvironmentFile = "/path/to/my/secrets.env";
      # I am not sure about the Restart and ExecReload options,
      # does sending a SIGHUP signal does something to the Erlang VM?
    };
    # Using the nixos declarative way
    environment = {
      # Set to "/tmp", made private by systemd, although not necessary here
      # Maybe this should be in buildMix, set in the wrapper
      RELEASE_TMP = "/tmp";
      # Maybe an example for the StateDirectory?
      MY_DATA_DIR = "/var/lib/${name}";
    };
  };
}
Sorry for the long message, and thanks again!

Thanks for all the suggestions. I need a couple of days before I can come back to this. I'll reply soon.

happysalada · 2021-04-02T02:49:37Z

/status awaiting_reviewer

SuperSandro2000

I don't know elixier, mix or elang but since we did already quite a long bikeshedding session here I think this is in a good place and if something is broken we (I mean you) can always fix it later when possible someone else used it.

doc/languages-frameworks/beam.section.md

pkgs/development/beam-modules/default.nix

pkgs/development/beam-modules/fetch-mix-deps.nix

pkgs/development/beam-modules/mix-release.nix

happysalada · 2021-04-03T00:44:07Z

@SuperSandro2000 Thanks a lot for spending time on this!
I've taken care about the comments that I could take care of and I've responded to the other comments.
Please take a look when you have a moment.
Thanks again!

happysalada · 2021-04-05T05:36:21Z

@SuperSandro2000 I've taken care of the last comment. (I should have thought about adding it before).
It should be ready now.
Let me know if anything.
Thanks again for your patience with this whole PR.

marvin-mk2 · 2021-04-08T09:04:13Z

Reminder: Please review!

This Pull Request is awaiting review. If you are the assigned reviewer, please have a look. Try to find another reviewer if necessary. If you can't, please say so. If the status is not accurate, please change it. If nothing happens, this PR ~~will be~~ should be put back in the needs_reviewer queue in one day.

Note: This feature is currently broken. The bot will not actually change the status. If you see this message multiple times, please request a status change manually.

NobbZ · 2021-04-08T10:22:43Z

I wonder how we can get this on https://github.com/orgs/NixOS/projects/6#card-56314431

happysalada · 2021-04-08T10:38:05Z

@NobbZ great idea! I think it's a matter of asking the release manager right?
Let me know if you want to have a go at it, otherwise I don't mind taking care of it.
if it doesn't make it by the time the release is made, I can probably backport it. I've never done it before, but I've been wanting to try for some time.

balsoft

I have managed to compile two (private) Elixir projects with this, with the only quirk being the .bat removal. fixupPhase = ":"; mitigates that for now, but it would be nice to see an additional check before removing the bat.

pkgs/development/beam-modules/mix-release.nix

afontaine · 2021-04-08T13:27:49Z

pkgs/development/beam-modules/fetch-mix-deps.nix

  '';

  dontBuild = true;

-  installPhase = ''
+  installPhase = attrs.installPhase or ''
+    runHook preInstall
    mix deps.get --only ${mixEnv}


Does it make sense to compile the dependencies here with mix deps.compile and potentially cache that build for larger (phoenix) projects? Or does elixir/nix not like that?

mix deps.compile is done in the configure phase of the mixRelease.
It could be possible to compile dependencies here.
It did not feel natural to have a compilation step into a step to fetch dependencies.
The other concern I had, was that what if you wanted to compile your dependencies in a specific way.

If you wanted to cache the compilation step, you could potentially add another intermediary step.
I have to say I don't know yet the consequences that it would have.
Perhaps it's something that can be tried for 6 months somewhere so we can make a PR about it later?
Curious if anybody has a more definite opinion.
It could be something to ask on an official elixir forum as well.

mix deps.compile doesn't produce deterministic output

Oh right, I forgot about the config part of things 🤔

bummer 😞

@schneebyte thanks for the answer here. I kept on thinking about this.
Do you mean to say that mix deps.compile will produce something different based on the configuration?
I'm just wondering what is producing the non-deterministic behavior, is it some input that we can pass in? Or is it that no matter what we pass in, we will never get deterministic output?

Well there is compile time configuration, https://hexdocs.pm/mime/MIME.html for example.
And also the .mix directories in _build/env/lib/xyz/.mix change, but those could be deleted if you pass --no-deps-check to next mix command.

I actually got my Elixir project to produce deterministic output, until i added phoenix_live_dashboard to deps, suddenly one .beam file was changing between builds.

I narrowed it down a little. If i compile this a few times the beam file changes.

defmodule Foo do
def bar() do
~r/(?((.+?/){3})).(?(/.){3})/
end
end

Thanks a lot for the precision, very interesting!

Thank you so much @schneebyte for providing a minimal example! I was actually investigating this a while ago, and it seemed like the built in pcre2 library in Erlang sometimes doesn't produce deterministic compiled regex data. I feel like with the minimal example, we should be able to report a bug to the Erlang project

That example was actually wrong, formatting messed it up.
I narrowed it down to the different length of the names of named captures?
But only with the sigil in elixir, not sure how to do it with :re.compile.

~r/(?<xxx>.)(?<yyy>.)/
Is fine because xxx and yyy are both length 3

But with
~r/(?<xxxxx>.)(?<yy>.)/
the beam file differs because xxxxx and yy have different lengths.

Weird stuff.

happysalada · 2021-04-09T06:28:55Z

@balsoft I believe this is ready to merge. No worries at all if you don't want to take the responsability or don't have the time to invest in this.

happysalada · 2021-04-09T11:37:17Z

@balsoft ❤️ thanks a lot ❤️

@SuperSandro2000 @minijackson @cw789 @NobbZ @schneebyte @afontaine @kalbasit @hauleth
thanks a lot for the help on this!

nixos-discourse · 2021-04-11T17:38:24Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/announcing-mixnix-build-elixir-projects-with-nix/2444/20

happysalada added 2.status: work-in-progress This PR isn't done 6.topic: developer experience 6.topic: erlang labels Feb 9, 2021

happysalada mentioned this pull request Feb 9, 2021

RFC fixing build-mix #105002

Closed

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Feb 9, 2021

happysalada requested review from DianaOlympos, yorickvP and minijackson February 9, 2021 08:13

ofborg bot added the 8.has: documentation This PR adds or changes documentation label Feb 9, 2021

hauleth reviewed Feb 10, 2021

View reviewed changes

pkgs/development/beam-modules/build-mix.nix Outdated Show resolved Hide resolved

minijackson reviewed Feb 11, 2021

View reviewed changes

minijackson reviewed Feb 12, 2021

View reviewed changes

happysalada mentioned this pull request Feb 13, 2021

Remove deprecated use of stdenv*.lib hauleth/nix-elixir#6

Merged

kalbasit reviewed Feb 15, 2021

View reviewed changes

marvin-mk2 bot added the awaiting_reviewer (old Marvin label, do not use) label Apr 2, 2021

SuperSandro2000 requested changes Apr 2, 2021

View reviewed changes

marvin-mk2 bot added awaiting_changes (old Marvin label, do not use) awaiting_reviewer (old Marvin label, do not use) and removed awaiting_reviewer (old Marvin label, do not use) awaiting_changes (old Marvin label, do not use) labels Apr 2, 2021

beam-modules: remove unused pc versioning

1b1af19

happysalada force-pushed the fix_build_mix branch from a0e8ba6 to 8fdf5fa Compare April 3, 2021 00:41

happysalada force-pushed the fix_build_mix branch from 8fdf5fa to 138d6d3 Compare April 5, 2021 05:35

happysalada requested a review from SuperSandro2000 April 5, 2021 05:45

happysalada force-pushed the fix_build_mix branch from 138d6d3 to 1a14eda Compare April 8, 2021 10:41

balsoft reviewed Apr 8, 2021

View reviewed changes

pkgs/development/beam-modules/mix-release.nix Show resolved Hide resolved

beam-modules: buildMix -> mixRelease

481832b

happysalada force-pushed the fix_build_mix branch from 1a14eda to 481832b Compare April 8, 2021 11:55

balsoft approved these changes Apr 8, 2021

View reviewed changes

happysalada requested a review from balsoft April 8, 2021 11:56

balsoft approved these changes Apr 8, 2021

View reviewed changes

afontaine reviewed Apr 8, 2021

View reviewed changes

balsoft merged commit 544664d into NixOS:master Apr 9, 2021

ydlr mentioned this pull request May 9, 2021

restore buildMix and its bootstrapper #122374

Merged

8 tasks

happysalada deleted the fix_build_mix branch April 28, 2023 20:01

buildMix: fix: initial try #112477

buildMix: fix: initial try #112477

Conversation

happysalada commented Feb 9, 2021 • edited by sternenseemann Loading

Motivation for this change

Things done

happysalada commented Feb 9, 2021

happysalada commented Feb 9, 2021

happysalada commented Feb 9, 2021

cw789 commented Feb 10, 2021 • edited Loading

happysalada commented Feb 10, 2021

minijackson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada Feb 11, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

minijackson commented Feb 11, 2021

minijackson commented Feb 11, 2021

happysalada commented Feb 11, 2021

happysalada commented Feb 12, 2021

happysalada commented Feb 12, 2021

minijackson commented Feb 12, 2021

Choose a reason for hiding this comment

happysalada Feb 13, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada commented Feb 13, 2021 • edited Loading

happysalada commented Feb 13, 2021 • edited Loading

minijackson commented Feb 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

happysalada commented Feb 15, 2021

happysalada commented Apr 2, 2021

SuperSandro2000 left a comment

Choose a reason for hiding this comment

happysalada commented Apr 3, 2021

happysalada commented Apr 5, 2021

marvin-mk2 bot commented Apr 8, 2021

NobbZ commented Apr 8, 2021

happysalada commented Apr 8, 2021

balsoft left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schneebyte Apr 9, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

schneebyte Apr 9, 2021 • edited Loading

Choose a reason for hiding this comment

happysalada commented Apr 9, 2021

happysalada commented Apr 9, 2021

nixos-discourse commented Apr 11, 2021

happysalada commented Feb 9, 2021 •

edited by sternenseemann

Loading

cw789 commented Feb 10, 2021 •

edited

Loading

happysalada Feb 11, 2021 •

edited

Loading

happysalada Feb 13, 2021 •

edited

Loading

happysalada commented Feb 13, 2021 •

edited

Loading

happysalada commented Feb 13, 2021 •

edited

Loading

schneebyte Apr 9, 2021 •

edited

Loading

schneebyte Apr 9, 2021 •

edited

Loading