Application hot upgrade failed

[atimid]

I used reber3 as prod tar to package my program, I thought it was a command ./bin/relup_test install 1.1.1 to upgrade my program, but it failed.

Environment

rebar 3.15.1+build.4997.ref12d77966 on Erlang/OTP 22 Erts 10.7.2.8

Current behaviour

rebar.config

{erl_opts, [debug_info]}.
{deps, []}.

{relx, [
    {release, {relup_test, "1.1.1"},
        [relup_test,
            sasl]},

    {mode, dev},

    %% automatically picked up if the files
    %% exist but can be set manually, which
    %% is required if the names aren't exactly
    %% sys.config and vm.args
    {sys_config, "./config/sys.config"},
    {vm_args, "./config/vm.args"}

    %% the .src form of the configuration files do
    %% not require setting RELX_REPLACE_OS_VARS
    %% {sys_config_src, "./config/sys.config.src"},
    %% {vm_args_src, "./config/vm.args.src"}
]}.

{profiles, [{prod, [{relx,
    [%% prod is the default mode when prod
        %% profile is used, so does not have
        %% to be explicitly included like this
        {mode, prod}
        %% use minimal mode to exclude ERTS
        %% {mode, minimal}
    ]
}]}]}.

$ ./bin/relup_test install 1.1.1
Release 1.1.1 is already unpacked, now installing.
ERROR: release_handler:check_install_release failed: {'EXIT',
                                                      {{badmatch,
                                                        {error,beam_lib,
                                                         {missing_chunk,
                                                          <<70,79,82,49,0,0,3,
                                                            208,66,69,65,77,76,
                                                            105,110,101,0,0,0,
                                                            110,0,0,0,0,0,0,0,
                                                            0,0,0,0,14,0,0,0,
                                                            10,0,0,0,1,18,9,23,
                                                            9,24,9,26,9,27,9,
                                                            30,9,33,9,36,9,37,
                                                            9,42,9,45,0,67,47,
                                                            85,115,101,114,115,
                                                            47,121,97,110,103,
                                                            103,97,111,47,100,
                                                            101,109,111,47,114,
                                                            101,108,117,112,95,
                                                            116,101,115,116,47,
                                                            97,112,112,115,47,
                                                            114,101,108,117,
                                                            112,95,116,101,115,
                                                            116,47,115,114,99,
                                                            47,104,101,108,108,
                                                            111,95,115,101,114,
                                                            118,101,114,46,101,
                                                            114,108,0,0,65,116,
                                                            85,56,0,0,0,205,0,
                                                            0,0,21,12,104,101,
                                                            108,108,111,95,115,
                                                            101,114,118,101,
                                                            114,10,115,116,97,
                                                            114,116,95,108,105,
                                                            110,107,10,103,101,
                                                            110,95,115,101,114,
                                                            118,101,114,4,105,
                                                            110,105,116,6,101,
                                                            114,108,97,110,103,
                                                            4,115,101,108,102,
                                                            5,104,101,108,108,
                                                            111,10,115,101,110,
                                                            100,95,97,102,116,
                                                            101,114,11,104,97,
                                                            110,100,108,101,95,
                                                            99,97,108,108,18,
                                                            104,101,108,108,
                                                            111,95,115,101,114,
                                                            118,101,114,95,115,
                                                            116,97,116,101,5,
                                                            114,101,112,108,
                                                            121,2,111,107,11,
                                                            104,97,110,100,108,
                                                            101,95,99,97,115,
                                                            116,7,110,111,114,
                                                            101,112,108,121,11,
                                                            104,97,110,100,108,
                                                            101,95,105,110,102,
                                                            111,2,105,111,6,
                                                            102,111,114,109,97,
                                                            116,9,116,101,114,
                                                            109,105,110,97,116,
                                                            101,11,99,111,100,
                                                            101,95,99,104,97,
                                                            110,103,101,11,109,
                                                            111,100,117,108,
                                                            101,95,105,110,102,
                                                            111,15,103,101,116,
                                                            95,109,111,100,117,
                                                            108,101,95,105,110,
                                                            102,111,0,0,0,67,
                                                            111,100,101,0,0,1,
                                                            56,0,0,0,16,0,0,0,
                                                            0,0,0,0,164,0,0,0,
                                                            20,0,0,0,9,1,16,
                                                            153,16,2,18,34,0,1,
                                                            32,64,2,35,64,18,
                                                            19,64,2,51,64,71,0,
                                                            3,153,32,78,64,0,1,
                                                            48,153,48,2,18,66,
                                                            16,1,64,52,53,3,12,
                                                            0,0,9,16,19,64,114,
                                                            35,64,25,39,16,3,
                                                            153,64,7,48,32,64,
                                                            71,16,3,18,0,19,1,
                                                            80,153,80,2,18,146,
                                                            48,1,96,159,85,35,
                                                            32,162,16,64,48,
                                                            164,3,23,48,178,
                                                            194,35,19,1,112,
                                                            153,96,2,18,210,32,
                                                            1,128,159,117,19,
                                                            32,162,16,48,32,
                                                            164,3,23,32,226,19,
                                                            19,1,144,153,112,2,
                                                            18,242,32,1,160,43,
                                                            181,3,114,159,149,
                                                            19,32,162,13,16,
                                                            112,32,64,19,4,66,
                                                            19,16,3,164,3,23,
                                                            64,18,9,37,71,32,3,
                                                            69,3,2,19,64,71,48,
                                                            3,153,128,7,32,48,
                                                            16,48,0,164,3,23,
                                                            32,226,4,18,16,19,
                                                            1,176,159,149,19,
                                                            32,162,16,48,32,
                                                            164,3,23,32,226,19,
                                                            19,1,192,153,144,2,
                                                            18,10,18,32,1,208,
                                                            159,197,19,32,162,
                                                            64,194,3,19,1,224,
                                                            153,160,2,18,10,19,
                                                            48,1,240,159,229,
                                                            19,32,162,16,48,32,
                                                            164,3,23,32,194,19,
                                                            19,1,8,16,153,0,2,
                                                            18,10,20,0,1,8,17,
                                                            64,18,3,153,0,78,
                                                            16,64,1,8,18,153,0,
                                                            2,18,10,20,16,1,8,
                                                            19,64,3,19,64,18,3,
                                                            153,0,78,32,80,3,
                                                            83,116,114,84,0,0,
                                                            0,0,73,109,112,84,
                                                            0,0,0,76,0,0,0,6,0,
                                                            0,0,3,0,0,0,2,0,0,
                                                            0,4,0,0,0,5,0,0,0,
                                                            6,0,0,0,0,0,0,0,5,
                                                            0,0,0,8,0,0,0,3,0,
                                                            0,0,16,0,0,0,17,0,
                                                            0,0,2,0,0,0,5,0,0,
                                                            0,21,0,0,0,1,0,0,0,
                                                            5,0,0,0,21,0,0,0,2,
                                                            69,120,112,84,0,0,
                                                            0,112,0,0,0,9,0,0,
                                                            0,20,0,0,0,1,0,0,0,
                                                            19,0,0,0,20,0,0,0,
                                                            0,0,0,0,17,0,0,0,
                                                            19,0,0,0,3,0,0,0,
                                                            15,0,0,0,18,0,0,0,
                                                            2,0,0,0,13,0,0,0,
                                                            15,0,0,0,2,0,0,0,
                                                            10,0,0,0,13,0,0,0,
                                                            2,0,0,0,8,0,0,0,9,
                                                            0,0,0,3,0,0,0,6,0,
                                                            0,0,4,0,0,0,1,0,0,
                                                            0,4,0,0,0,2,0,0,0,
                                                            0,0,0,0,2,76,105,
                                                            116,84,0,0,0,95,0,
                                                            0,0,109,120,156,99,
                                                            96,96,96,97,96,96,
                                                            144,106,206,96,74,
                                                            97,96,205,201,79,
                                                            78,204,73,97,224,
                                                            201,72,205,201,201,
                                                            143,47,78,45,42,75,
                                                            45,2,202,170,129,
                                                            101,153,242,179,65,
                                                            148,16,178,100,124,
                                                            113,73,98,73,106,
                                                            54,3,75,101,98,94,
                                                            58,80,37,103,115,
                                                            54,3,43,88,1,144,
                                                            35,0,228,240,120,
                                                            128,56,10,86,10,
                                                            117,5,117,121,0,30,
                                                            119,30,55,0>>,
                                                          "Attr"}}},
                                                       [{release_handler_1,
                                                         get_vsn,1,
                                                         [{file,
                                                           "release_handler_1.erl"},
                                                          {line,779}]},
                                                        {release_handler_1,
                                                         add_vsns,3,
                                                         [{file,
                                                           "release_handler_1.erl"},
                                                          {line,735}]},
                                                        {release_handler_1,
                                                         '-eval/2-fun-0-',4,
                                                         [{file,
                                                           "release_handler_1.erl"},
                                                          {line,319}]},
                                                        {lists,foldl,3,
                                                         [{file,"lists.erl"},
                                                          {line,1263}]},
                                                        {release_handler_1,
                                                         eval,2,
                                                         [{file,
                                                           "release_handler_1.erl"},
                                                          {line,314}]},
                                                        {lists,foldl,3,
                                                         [{file,"lists.erl"},
                                                          {line,1263}]},
                                                        {release_handler_1,
                                                         do_check_script,3,
                                                         [{file,
                                                           "release_handler_1.erl"},
                                                          {line,65}]},
                                                        {release_handler,
                                                         do_check_install_release,
                                                         5,
                                                         [{file,
                                                           "release_handler.erl"},
                                                          {line,945}]}]}}

I have added -vsn("1.1.1") to the module that needs to be updated, but the file in the package that rebar3 as prod tar is still missing vsn information.

Expected behaviour

I expect the upgrade to succeed.

[ferd]

We're gonna need a bit more information:

• which OS is this on?
• what did the code look like on the first version (and what version was it)
• how was the code modified for the second version (version 1.1.1)

Right now it just looks like Release 1.1.1 is already unpacked, now installing. mentions the version you tried to install was already unpacked, which means it was already present in the system. This points to either some issue around which releases are being applied or the versions generated, or you've done the right thing, but only re-played operations for debugging purposes.

But with neither the code or any extra information there's no good way to reproduce this and narrow down the possible issue.

[atimid]

OS: macOS 10.15.7
code

[jwheare]

@atimid is this fixed if you add {debug_info, keep} to your prod profile relx options?

[jwheare]

This is caused by beam_file:strip_release being called, which strips out (among others) the "Attr" chunk from beam files for a release, used to get the vsn for a beam during release_handler installation.

https://github.com/erlware/relx/blob/master/src/rlx_assemble.erl#L24-L54

This happens with {mode, prod} and {debug_info, strip} set (prod sets it by default).

This isn't what you want. Note that this ignores the {erl_opts, [debug_info]}

I think the fix for this in rebar3/relx is better documentation of these options, and perhaps taking erl_opts into account. Breaking relup releases in {mode, prod} is probably not a great default.

Stripping debug_info should perhaps not also strip the attr/vsn from beams as well. That seems to be longstanding behaviour of beam_lib:strip_release, but perhaps there's a more suitable call to use for just debug info stripping.

[tsloughter]

Argh, definitely need better docs, yes.

Are you saying the longstanding behaviour of strip_release was to not strip the attr/vsn from the beam files as well?

[jwheare]

Ah sorry, I was saying that from what I can tell of the beam_lib commit history, the long standing behaviour of strip_release is that it DOES strip attributes.

[atimid]

@atimid is this fixed if you add {debug_info, keep} to your prod profile relx options?

{profiles, [{prod, [{relx,
                     [%% prod is the default mode when prod
                      %% profile is used, so does not have
                      %% to be explicitly included like this
                      {mode, prod},
                      {debug_info, keep} 
                      %% use minimal mode to exclude ERTS
                      %% {mode, minimal}
                     ]
            }]}]}.

I added this option in both version 1.1.0 and 1.1.1, but the problem still exists, the vsn is still missing in the compiled beam file, and the process of repeating the hot update still fails.

[tsloughter]

@atimid sorry, haven't had the time to try your example project. Can you try with {mode, dev}? That should keep it from doing any stripping at all but still build your a tarball you can attempt an upgrade with.

[atimid]

@tsloughter thanks, using {mode, dev} can solve this problem.

[sg2342]

The fix for this is in 3.19.0.
The issue should be closed.