[RFE] Standalone measure subcommand

[rsevilla87]

Is your feature request related to a problem? Please describe.
It would be nice to have a measure subcommand to make kube-burner able to execute measurements (The current available are podLatency and pprof) without actually running a benchmark

Describe the solution you'd like
Adding a new measure subcommand to do something like:

kube-burner measure -c configFile --for 1h

This new feature will read the kube-burner configuration as usual and then execute the measurements defined there for the time defined by the flag "--for"

[rsevilla87]

cc: @chaitanyaenr @paigerube14

[smalleni]

@rsevilla87 @shashank-boyapally @vishnuchalla I thought the purpose of this command is to setup measurements for the next time interval and not grab measurements from the past. The way I am guessing this would be used is to just run kube-burner measurements on a customer's environment for the next 1 hour or so to identify the podready latency etc.

[vishnuchalla]

Currently, kube-burner listens to the pod creation event and sets a creation timestamp in our program memory using time.Now().UTC() (instead of us manually setting this we can also fetch the exact pod creationTimestamp using client-go library) and keeps on looking for the update events and captures each of these timestamps while the each stage gets updated until a pod gets to Ready state.

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-15T23:45:46Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T19:27:07Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T19:27:07Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-07-15T23:45:46Z"
    status: "True"
    type: PodScheduled

once it has all those timestamps then it calculates latencies for each step. This is while the workload is running. Now that we know how these times are being fetched, we can make tweaks in our current code to capture them for previously ran pods or currently running pods as they transition stages over time.

So instead of an option like -for 1h, which I believe is for the future runs, we can have options like --start_time and --end_time to get these metrics for a workload which was ran with in that time range, which will also help us to run measurements on previously ran workloads as well as long as their pods are present and also isolates this measurement calculation from a workload run.

[vishnuchalla]

@Sai, @chaitanyaenr Please feel free to share your thoughts on this. Thank you!

[rsevilla87]

Currently, kube-burner listens to the pod creation event and sets a creation timestamp in our program memory using time.Now().UTC() (instead of us manually setting this we can also fetch the exact pod creationTimestamp using client-go library) and keeps on looking for the update events and captures each of these timestamps while the each stage gets updated until a pod gets to Ready state.

status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-07-15T23:45:46Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T19:27:07Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-07-17T19:27:07Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-07-15T23:45:46Z"
    status: "True"
    type: PodScheduled

once it has all those timestamps then it calculates latencies for each step. This is while the workload is running. Now that we know how these times are being fetched, we can make tweaks in our current code to capture them for previously ran pods or currently running pods as they transition stages over time.

So instead of an option like -for 1h, which I believe is for the future runs, we can have options like --start_time and --end_time to get these metrics for a workload which was ran with in that time range, which will also help us to run measurements on previously ran workloads as well as long as their pods are present and also isolates this measurement calculation from a workload run.

It's technically possible to measure podLatency numbers from the past, but I personally don't see the point on it, it will require lot of coding and I think is more reasonable to take the measurement from upcoming events, this change shouldn't be very complex. In addition from pod latency measurements, the pprof measurement is also there, and this one cannot be obtained from the past.

[vishnuchalla]

cc @shashank-boyapally
Sounds good to me. Assuming that to calculate metrics for future, we can start working on the implementation part for collecting both podLatency and pporf in a standalone measure command. And we can think of extending to other use cases later only if required. Thank you!

[shashank-boyapally]

Yes @vishnuchalla, we should be able to implement metrics for future.

[smalleni]

cc @shashank-boyapally Sounds good to me. Assuming that to calculate metrics for future, we can start working on the implementation part for collecting both podLatency and pporf in a standalone measure command. And we can think of extending to other use cases later only if required. Thank you!

I agree on the direction.

[github-actions]

This issue has become stale and will be closed automatically within 7 days.