Allow opting out of frequent etcd flushes to storage

[akalenyu]

What this PR does / why we need it:
etcd allows opting out of WAL [0] and data [1] fsyncs
(WAL records describe the commit changes and are being flushed to permanent storage very frequently)
which could significantly decrease the load on the worker pod filesystem.
This is (arguably) equivalent in terms of disasters to etcd in memory,
and has seemed to work well for other large projects [1].

As this is behind an env var, and our CI clusters are ephemeral,
there's no harm in evaluating the use of this, in hopes of squeezing
out the maximum of the resources available to us
(non-dedicated storage per worker)

[0] https://www.postgresql.org/docs/current/wal-intro.html
[1] etcd-io/etcd#18357 (comment)
[2] cert-manager/cert-manager@057ce50

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist

This checklist is not enforcing, but it's a reminder of items that could be relevant to every PR.
Approvers are expected to review this list.

• Design: A design document was considered and is present (link) or not required
• PR: The PR description is expressive enough and will help future contributors
• Code: Write code that humans can understand and Keep it simple
• Refactor: You have left the code cleaner than you found it (Boy Scout Rule)
• Upgrade: Impact of this change on upgrade flows was considered and addressed if required
• Testing: New code requires new unit tests. New features and bug fixes require at least on e2e test
• Documentation: A user-guide update was considered and is present (link) or not required. You want a user-guide update if it's a user facing feature / API change.
• Community: Announcement to kubevirt-dev was considered

Release note:

add a configurable for opting out of frequent etcd flushes

[akalenyu]

/retest

[mhenriks]

/lgtm

[akalenyu]

/lgtm

Just rebased

[kubevirt-bot]

@akalenyu: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
check-up-kind-1.27-vgpu	51b02e0	link	false	/test check-up-kind-1.27-vgpu

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[brianmcarey]

/approve

Thanks @akalenyu - this should be ok to add now as it is only enabled when the env variable is set.

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: brianmcarey

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [brianmcarey]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment