Federation Requires local AND upstream Rabbit to have Federation Plugin enabled -> Breaking Change since 3.x

[BenniG82]

Describe the bug

Since we upgraded our cental RabbitMQ from Version 3.x to Version 4.0.4 we are no longer able to setup a federation from our developer RabbitMQ-Instances to the central (upstream) RabbitMQ.

All developer RabbitMQ-Instances are running with Federation Plugin enabled.
The central RabbitMQ does not have the Federation Plugin enabled.

With RabbitMQ 3.x as central RabbitMQ everything worked even with no Federation Plugin installed.

With RabbitMQ 4.0.4 we get the following error on the satellites in Federation Status:

{server_initiated_close,404,
                        <<"NOT_FOUND - no exchange 'federation: exchange.demo -> rabbit@7c1db11dd767 B' in vhost '/'">>}

As you can see in the screenshot:

• Federation works for the connection with RabbitMQ 3.x with no plugins enabled conn3-wo-plugins
• Federation works for the connection with RabbitMQ 4.x with Federation Plugin enabled conn4-with-plugins
• Federation does not work for the connection with RabbitMQ 4.x with no Federation Plugin enabled conn4-wo-plugins

Reproduction steps

Use the following Docker-Compose file to setup:

• a satellite rabbit with RabbitMQ 4 with Federation Plugin enabled
• a upstream upstream4WoPlugins RabbitMQ 4.x with no Plugins enabled
• a upstream upstream4WithPlugins RabbitMQ 4.x with Federation Plugin enabled
• a upstream upstream3WoPlugins RabbitMQ 3.x with no Plugins enabled

docker-compose.yaml

services:

  rabbit:
    image: rabbitmq:4.0.4-management
    restart: always
    ports:
      - "15673:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: guest
    volumes:
      - "./rabbitmq/enabled_plugins:/etc/rabbitmq/enabled_plugins"

  upstream4WoPlugins:
    image: rabbitmq:4.0.4-management
    restart: always
    ports:
      - "15674:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: guest

  upstream4WithPlugins:
    image: rabbitmq:4.0.4-management
    restart: always
    ports:
      - "15675:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: guest
    volumes:
      - "./rabbitmq/enabled_plugins:/etc/rabbitmq/enabled_plugins"

  upstream3WoPlugins:
    image: rabbitmq:3.13-management
    restart: always
    ports:
      - "15676:15672"
    environment:
      RABBITMQ_DEFAULT_USER: admin
      RABBITMQ_DEFAULT_PASS: guest

File: rabbitmq/enabled_plugins

[rabbitmq_federation, rabbitmq_federation_management].

Execute the following commands for a demo setup

Create Exchange exchange.demo in all RabbitMQ instances

docker compose exec rabbit rabbitmqadmin --base-uri=http://localhost:15672 --username=admin --password=guest --vhost=/ declare exchange name=exchange.demo type=fanout durable=true 
docker compose exec upstream4WoPlugins rabbitmqadmin --base-uri=http://localhost:15672 --username=admin --password=guest --vhost=/ declare exchange name=exchange.demo type=fanout durable=true
docker compose exec upstream4WithPlugins rabbitmqadmin --base-uri=http://localhost:15672 --username=admin --password=guest --vhost=/ declare exchange name=exchange.demo type=fanout durable=true
docker compose exec upstream3WoPlugins rabbitmqadmin --base-uri=http://localhost:15672 --username=admin --password=guest --vhost=/ declare exchange name=exchange.demo type=fanout durable=true

Create Federation from rabbit (satellite) to conn4-wo-plugins (upstream 4.x with no plugins enabled)

docker compose exec rabbit rabbitmqctl set_parameter federation-upstream conn4-wo-plugins '{"uri":"amqp://admin:guest@upstream4WoPlugins:5672", "exchange": "exchange.demo"}'
docker compose exec rabbit rabbitmqctl set_policy federate4-wo-demo "^exchange.demo$" '{"federation-upstream-set":"all"}' --apply-to exchanges

=> This federation does not work

Create Federation from rabbit (satellite) to conn4-with-plugins (upstream 4.x with Fed. Plugin)

docker compose exec rabbit rabbitmqctl set_parameter federation-upstream conn4-with-plugins '{"uri":"amqp://admin:guest@upstream4WithPlugins:5672", "exchange": "exchange.demo"}'
docker compose exec rabbit rabbitmqctl set_policy federate4-with-demo "^exchange.demo$" '{"federation-upstream-set":"all"}' --apply-to exchanges

=> This federation works

Create Federation from rabbit (satellite) to conn3-wo-plugins (upstream 3.x with no plugins enabled)

docker compose exec rabbit rabbitmqctl set_parameter federation-upstream conn3-wo-plugins '{"uri":"amqp://admin:guest@upstream3WoPlugins:5672", "exchange": "exchange.demo"}'
docker compose exec rabbit rabbitmqctl set_policy federate3-wo-demo "^exchange.demo$" '{"federation-upstream-set":"all"}' --apply-to exchanges

=> This federation works

Expected behavior

I expect Federation to work without additional plugins on the upsteam server.
This worked in 3.x

Additional context

No response

[michaelklishin]

@BenniG82 I'm afraid in order to use a feature provided by a plugin you must have it enabled. The very first step in the docs explains how to enable the plugin.

Queue federation could work (one way) without queue federation on the upstream side. This was by accident.

By your logic, management UI should be available without the management plugin. Same for the MQTT implementation? That's not how plugins are supposed to work.

[michaelklishin]

We can mention this in 4.0.1 release notes but the 3.x behavior you claim is just an accident. Federation may look like a "regular consumer" to upstream clusters in some cases, but that was never a design goal with either exchange or queue federation.

As RabbitMQ core engineering team we cannot develop inter-cluster features as plugins with the assumption that "the plugin might not be available on the other side". If it's not, then it should be enabled or you cannot use this feature. This is not just the realities of distributed system development, this is common sense.

If it did work by accident before, that was just that, an accident. And this is not a breaking change, this is an incorrect use of plugins that goes against the very first recommendation in the docs.

I will verify if the docs might imply that enabling the plugin is optional because that was never a design goal.

[lukebakken]

@BenniG82 it turns out you found a bug in how RabbitMQ 3.x operates. When you enable the rabbitmq_prometheus plugin in 3.13.x, it implicitly enables the rabbitmq_federation plugin. In RabbitMQ 4.x, enabling rabbitmq_prometheus does NOT implicitly enable rabbitmq_federation.

But, why are you seeing this? The Docker image ships with an enabled_plugins file that enables rabbitmq_management and rabbitmq_prometheus. So, you see the behavior I describe using Docker out-of-the-box.

Like @michaelklishin said, there's really nothing here to fix, because the documentation is clear - both sides of a Federation link need to have the plugin enabled. It's just that, in the case you found, the rabbitmq_federation plugin was enabled implicitly, by mistake.

[michaelklishin]

By the way, this is because #11940 was merged in the 4.0.0 development cycle.

[BenniG82]

@lukebakken @michaelklishin
Thanks for the clarification.

Just one thing to add:

there's really nothing here to fix, because the documentation is clear - both sides of a Federation link need to have the plugin enabled

To me this precondition was not clear. I implicitly assumed, that the Federation Plugin is just a special kind of consumer.

Maybe you could add a note in the Getting-Started Section at https://www.rabbitmq.com/docs/federation#getting-started?
E.g.

Please note that the Federation Plugin needs to be enabled on both upstream and downstream.

Thanks for your lightning fast responses 🥇

[michaelklishin]

@BenniG82 I will update the docs later today (North American time zones) and leave a link here so that you can suggest more edits, if necessary. We are always happy to tweak the docs if something is not clear.

[michaelklishin]

rabbitmq/rabbitmq-website@a035d73