ENG-19808 mlxfwreset before reboot in SR-IOV operator

[punkerpunker]

Well, overall it seems like something wrong on the firmware level enabling the SR-IOV capability, from the kernel perspective things looks okay to me

I see in the sriov-network-operator there's an option to enable mstfwreset after mlxconfig change! I guess that's something that we can try leveraging:
https://github.com/openshift/sriov-network-operator/blob/79cb3c6ae721220754189300539a38c63e38e66c/pkg/plugins/mellanox/mellanox_plugin.go#L215

I think this is going to resolve the reboot loop we're getting when running config-daemon, and I'll try doing it tomorrow.
All paths so far goes to firmware, I don't think I'll find anything more in the kernel, tbh

Infra PR to enable featureGate - https://github.com/togethercomputer/infra/pull/4044

[github-actions]

Thanks for your PR,
To run vendors CIs, Maintainers can use one of:

• /test-all: To run all tests for all vendors.
• /test-e2e-all: To run all E2E tests for all vendors.
• /test-e2e-nvidia-all: To run all E2E tests for NVIDIA vendor.

To skip the vendors CIs, Maintainers can use one of:

• /skip-all: To skip all tests for all vendors.
• /skip-e2e-all: To skip all E2E tests for all vendors.
• /skip-e2e-nvidia-all: To skip all E2E tests for NVIDIA vendor.
  Best regards.