Improve the role change reversal and properly revert them

[scholzj]

Type of change

• Bugfix

Description

The current way how the reversal of the role change when a broker role is removed from a node that is not empty is implemented as simply adding the broker role back. But that is not the reversal in case the role is changed from broker to controller. That leads the operator into a strange in the middle state where the controller role is applied but the broker role is not removed. This PR fixes it by reverting the roles to the original state before this change and maintaining the current state.

Checklist

• Write tests
• Make sure all tests pass
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally

[scholzj]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[scholzj]

/azp run kraft-regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ppatierno]

This PR fixes it by reverting the roles to the original state before this change and maintaining the current state.

@scholzj maybe I haven't got the gist of this but I can't see where this fix is happening. I see the PR having some warning and log changes + test but not a fix somewhere in the code. I am sure there is something I am missing, could you elaborate a little bit more where I should look at, please?

[scholzj]

@ppatierno In KafkaClusterCreator, line 289 ... addToRoles -> withRoles

[ppatierno]

@ppatierno In KafkaClusterCreator, line 289 ... addToRoles -> withRoles

:-o ... How could I have missed that??

[ppatierno]

LGTM.