Backup-restore now restart the etcd member incase of etcd's advertise peerURL found to be updated.

[ishan16696]

What this PR does / why we need it:

1. Backup-restore will now call updateMemberPeerURL API call only when peerURL is found to be updated.
2. Backup-restpre will now restart it's corresponding etcd member after updating etcd's advertise peer URLs if found to be updated.
3. Introduced a CLI flag --use-etcd-wrapper(default: false) to enable/disable backup-restore to use etcd-wrapper related functionality.

Which issue(s) this PR fixes:
Fixes #787

Special notes for your reviewer:
This PR needs to be tested with PR of etcd-wrapper: gardener/etcd-wrapper#31 where /stop endpoint is introduced with --use-etcd-wrapper set to true for backup-restore.

Release note:

etcd-backup-restore now triggers a restart of the etcd member after updating etcd's advertise peer URLs if found updated.

Introduced a CLI flag `--use-etcd-wrapper` (default: false) to enable/disable the backup-restore to use etcd-wrapper related functionality. Note: enable this flag only if etcd-wrapper is deployed.

[unmarshall]

@ishan16696 What is the impact of calling etcd-wrapper endpoint from etcdbr on consumers who are only using etcdbr? Let us have this clarity first and then we can remove the reviewed/do-not-merge label.

[ishan16696]

What is the impact of calling etcd-wrapper endpoint from etcdbr on consumers who are only using etcdbr ?

yes, that's very valid point.
I would suggest to introduce a /version endpoint to wrapper just like etcd has version endpoint to detect whether corresponding backup-restore is running with wrapper or not.

curl http://127.0.0.1:2379/version
{"etcdserver":"3.5.9","etcdcluster":"3.5.0"}

backup-restore can send REST request to this URL of wrapper: http://127.0.0.1:9095/version
if response return this {"etcdwrapper":"v0.x."} then only backup-restore calls /stop endpoint in case of detection of peerURL change
else
backup-restore will try to call etcd endpoint http://127.0.0.1:2379/version then just print warning of unable to restart as etcd-wrapper is not found . wdyt ?

[unmarshall]

backup-restore will try to call etcd endpoint http://127.0.0.1:2379/version then just print warning of unable to restart as etcd-wrapper is not found . wdyt ?

I am not sure versioning of HTTP endpoints is the right approach. Instead one should start etcdbr with feature gate which should enable/disable wrapper handling.

[ishan16696]

Instead one should start etcdbr with feature gate which should enable/disable wrapper handling.

we already have a feature gate in druid as druid deploys etcd and backup-restore with either with etcd-custom-image or etcd-wrapper but IMO, backup-restore should not use feature gate for etcd

[ishan16696]

@unmarshall @shreyas-s-rao and I have decided to introduced the CLI flag: --use-etcd-wrapper to consume etcd-wrapper related functionality in backup-restore with default value of this flag set to false, hence causing no impact on our community users.
We can set --use-etcd-wrapper to true in etcd-druid as it always deploys etcd cluster with etcd-wrapper.

[unmarshall]

We have decided to make this backward-incompatible change. This has been added to the compatibility matrix in etcd-druid documentation.
/lgtm

[shreyas-s-rao]

@ishan16696 thanks for the PR. One small nit from my side, but overall LGTM. I have tested this with gardener/etcd-druid#883 and the upgrade cases work perfectly now.

[ishan16696]

/cla

[gardener-robot]

@ishan16696 I reached out successfully to the cla-assistant to recheck this pull request.

[shreyas-s-rao]

Thanks for addressing the comment.
/lgtm