Prefer processes in coordinator selection for multi-region clusters

[johscheuer]

Description

Fixes: #2034

Type of change

Please select one of the options below.

• New feature (non-breaking change which adds functionality)

Discussion

Most of the code changes are adjustments for the testing code, to make sure we add some randomness.

Testing

Added additional unit tests and modified them.

Documentation

Follow-up

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 93cce33
• Duration 2:57:28
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[johscheuer]

The new benchmark will present a result like this:

------------------------------
• [0.035 seconds]
Localities chooseDistributedProcesses when a multi-region cluster is used with 4 dcs when measuring the performance for choose distributed processes [measurement]
/Users/jscheuermann/go/src/github.com/FoundationDB/fdb-kubernetes-operator/internal/locality/locality_test.go:630

  Report Entries >>
  Choose Distributed Processes - /Users/jscheuermann/go/src/github.com/FoundationDB/fdb-kubernetes-operator/internal/locality/locality_test.go:636 @ 06/04/24 11:52:09.548
    Choose Distributed Processes
    Name                                  | N  | Min   | Median | Mean  | StdDev | Max
    ==================================================================================
    ChooseDistributedProcesses [duration] | 50 | 300µs | 400µs  | 400µs | 100µs  | 1ms

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 93cce33
• Duration 2:25:22
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: a197e87
• Duration 2:30:20
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: 0b9b6aa
• Duration 2:25:00
• Result: ✅ SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of fdb-kubernetes-operator-pr on Linux CentOS 7

• Commit ID: b615ff2
• Duration 2:12:55
• Result: ❌ FAILED
• Error: Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[nicmorales9]

just a few things to double check but the core lgtm