Tracing CI build performance

[kwonoj]

Status	Last Updated	Target version
In review		N/A

Summary

Observing CI performances with few metrics, such as SWC's bytesize / CI build time for each steps (check, test, ...) to track numbers over time to check regressions or apply optimizations.

Motivation

There are some continuously growing pain points. CI is one of the core around those who work on the codebase daily. We feel there is something going on, but we do not have any data points to answer some questions like

• How much it gets worse over ${N} time period?
• Which step of CI process is the biggest bottleneck currently?
• If we apply some optimizations / fixes, how much will it be improved?
• How large are swc binaries and how much does it grow over time?

Having a naive, simple tracing would able to cover some of those questions, and we can improve things over time.

Note this is not #3065 - this is meta-tracking to observe build performance of SWC itself, not tracking runtime performance of SWC building other source inputs. There is no concern of tracking PII at all as we'll run this over SWC's upstream CI only.

Detailed design

Proof of concept PR is available at kwonoj#98, it was able to collect traces like below.

Hope I could share dyanmic query itself, but there seems no publicly-shareable way.

The approach is

• Using honeycomb's free plan (https://www.honeycomb.io/pricing)
• Use honeycomb's gh action (https://github.com/marketplace/actions/honeycomb-buildevents), wraps existing CI commands with spans
• Done (😅 )

This is mostly non-invasive approach except one drawbacks we need to solve in #Drawbacks.

Drawbacks

One of the biggest drawback is honeycomb's gh action does not support windows yet. In result, POC PR have some hacky workaround to skip windows runner to install gh actions. Probably we'll need to create some stub, or more better have a fork to support windows collection preserves existing CI definitions nearly as-is.

Alternatives

We could do some other ways like

• manually collect timestampes only, render it in somewhere else
• use other providers
• do nothing, checking CI logs manually as same as today for the specific occasions

Choosing honeycomb was based on a simple decision, it provides an acceptable free-usage plan and has out of the box support for the github actions, which requires less work (except windows support part). If there are easier way, it'd be better.

Unresolved questions

[kwonoj]

https://github.com/inception-health/otel-export-trace-action could be a simpler way to enable this.