RabbitMQ does not auto-recover from pause_minority due to a feature flags issue

[lukebakken]

Describe the bug

See the README here:

https://github.com/lukebakken/rabbitmq-server-8113

Upon resolution of network connectivity, the paused node errors with {erpc,noconnection}

Reproduction steps

https://github.com/lukebakken/rabbitmq-server-8113#reproduction-steps

Expected behavior

The paused node should automatically come back up when network connectivity is restored.

Additional context

RabbitMQ 3.11.15, Erlang 25.3.1 (current docker image).

[michaelklishin]

Interesting that it runs into

Error during startup: {error,incompatible_feature_flags}

upon boot.

[dcorbacho]

Interesting that it runs into

Error during startup: {error,incompatible_feature_flags}

upon boot.

Syncing feature flags captures the error and returns it tagged like that. Not the most descriptive, probably error_syncing_feature_flags is more accurate nowadays - ff are becoming more complex and they can fail for many reasons. Here: https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/src/rabbit_mnesia.erl#L678

[dumbbell]

@lukebakken shared the entire log files in our internal Slack, I'm attaching them here.
[email protected]
[email protected]
[email protected]

I tried to reproduce as explained but couldn't so far. The logs above were from RabbitMQ 3.11.15, but RabbitMQ 3.11.16 was picked when I executed the script.

There is no evidence in the logs above that this is something related to feature flags: the rmq0 Erlang node failed to open a connection to both rmq1 and rmq2 when it wanted to restart:

2023-05-05 22:57:05.477001+00:00 [warning] <0.2275.0> 'global' at '[email protected]' failed to connect to '[email protected]'
2023-05-05 22:57:05.477001+00:00 [warning] <0.2275.0> 
2023-05-05 22:57:05.495009+00:00 [notice] <0.44.0> Application mnesia exited with reason: stopped
2023-05-05 22:57:05.496061+00:00 [notice] <0.2315.0> Feature flags: checking nodes `[email protected]` and `[email protected]` compatibility...
2023-05-05 22:57:19.522172+00:00 [error] <0.2363.0> Feature flags: error while running:
2023-05-05 22:57:19.522172+00:00 [error] <0.2363.0> Feature flags:   rabbit_ff_registry:inventory[]
2023-05-05 22:57:19.522172+00:00 [error] <0.2363.0> Feature flags: on node `[email protected]`:
2023-05-05 22:57:19.522172+00:00 [error] <0.2363.0> Feature flags:   exception error: {erpc,noconnection}

We can see that global couldn't contact rmq1 and rabbit_ff_controller couldn't contact rmq2.

I will do some "research" to see if I can use RabbitMQ 3.11.15 instead.

[dumbbell]

So far, I was never able to reproduce the error :-/

[michaelklishin]

@mkuratczyk this is the issue I've mentioned during the sync-up today

[lukebakken]

@michaelklishin @dumbbell @mkuratczyk this appears to be fixed in 3.11.16. I'm re-testing with 3.11.15

[lukebakken]

It took two tries to reproduce with 3.11.15, but it did. The other day I could repro several times in a row, which is why I opened this issue.

I'm going to keep re-trying with 3.11.16 for a bit and will re-open this issue if I can repro.

[lukebakken]

Yep, I just reproduced with 3.11.16. Same set of reproduction steps.

[lukebakken]

User error! When re-connecting, docker network connect knows nothing about aliases defined via docker compose. You must re-connect using this command:

docker network connect --alias=rmq0.local rabbitnet rabbitmq-server-8113-rmq0-1