New Selection Algorithm

[leszko]

Changes:

• New Selection Algorithm as discussed in Design Doc and Discord
• Periodically fetching Orchestrator Performance Scores
• Probability Selection Algorithm, which calculates probability taking into account weighted stake, price, and randomness
• Added flags:
  • -selectStakeWeight, -selectPriceWeight: weights of selection algorithms params
  • -selectPriceExpFactor: expresses how significant a small change of price is for the selection algorithm; default 100
  • -orchPerfStatsUrl: URL to the test stream service, Livepeer Inc service is available at: https://leaderboard-serverless.vercel.app/api/aggregated_stats
  • -minPerfScore: filter for the minimal O's performance score to be accepted for the selection
  • -region: region where B is deployed; used to get the right performance scores

Some notes about the approach

• In this version, the selection algorithm is not fully configurable, all you can configure right now are: 1) weights of price, stake, randomness, 2) filter for performance score 3) factor for exp price factor; in the future we may want to make the selection algorithm fully configurable using webhook service (aka plugin) or by allowing to specify an equation as a flag like selection=0.3*exp(price, 100)+0.4*rand+0.3*stake, but I think we should wait for some more use-cases / feature request to decide on the best approach
• For the performance scores, they are not part of the weighted probability calculation, but a filter, so only Os that have a regional performance score higher than -minPerfScore are taken into consideration for the selection. Note that the performance score is calculated as defined in https://github.com/livepeer/leaderboard-serverless, so perfScore = (RTT / segment_size) * success_rate. At first, I thought we should use the RTT expressed in ms, but that's not a good metric, because RTT depends on the segment length, so the performance score sounds like a better metric for the performance
• Note that if no orchestrator passes the performance score metric, then all orchestrators are allowed for the selection. This may result in using Orchestrator with lower performance than defined, however I chose this approach, because I think in practice we'll only see no Orchestrators passing this filter, when Test Stream service is down or gives wrong results (which happened a few times in the past)

[codecov]

Codecov Report

Merging #2872 (3355766) into master (ecc2839) will decrease coverage by 0.07833%.
The diff coverage is 58.72093%.

@@                 Coverage Diff                 @@
##              master       #2872         +/-   ##
===================================================
- Coverage   56.48581%   56.40748%   -0.07833%     
===================================================
  Files             88          89          +1     
  Lines          19242       19368        +126     
===================================================
+ Hits           10869       10925         +56     
- Misses          7774        7839         +65     
- Partials         599         604          +5     

Files	Coverage Δ
cmd/livepeer/livepeer.go	49.48980% <100.00000%> (+1.87075%)	⬆️
common/types.go	0.00000% <ø> (ø)
core/livepeernode.go	60.00000% <ø> (ø)
server/broadcast.go	77.55776% <100.00000%> (ø)
server/selection.go	93.38235% <100.00000%> (-5.92799%)	⬇️
server/mediaserver.go	67.35632% <0.00000%> (ø)
server/selection_algorithm.go	87.50000% <87.50000%> (ø)
cmd/livepeer/starter/starter.go	4.80530% <13.88889%> (+0.60582%)	⬆️

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecc2839...3355766. Read the comment docs.

Files	Coverage Δ
cmd/livepeer/livepeer.go	49.48980% <100.00000%> (+1.87075%)	⬆️
common/types.go	0.00000% <ø> (ø)
core/livepeernode.go	60.00000% <ø> (ø)
server/broadcast.go	77.55776% <100.00000%> (ø)
server/selection.go	93.38235% <100.00000%> (-5.92799%)	⬇️
server/mediaserver.go	67.35632% <0.00000%> (ø)
server/selection_algorithm.go	87.50000% <87.50000%> (ø)
cmd/livepeer/starter/starter.go	4.80530% <13.88889%> (+0.60582%)	⬆️

[thomshutt]

Really really neat change!