[CI Visibility] Add support for CI Visibility mode in ddtrace #2739

tonyredondo · 2024-06-12T09:37:50Z

What does this PR do?

This PR is a continuation of #2736 and adds support for CI Visibility mode inside ddtrace.

When the CI Visibility mode is enabled, the tracer is initialised with the CIVisibility Writer, Transport and the custom Payload format.

The PR tries to reduce the impact on the current tracer code, by only modifying the options file to check if the civisibility mode is enabled and change the configuration to the ci visibility defaults, including the new transport. It also changes the tracer file to set the civisibility writer when the civisibility mode is enabled.

Motivation

The CI Visibility agentless intake has a custom payload format based on events.
We want to reuse the current tracer implementation but making it compatible with the ci visibility format (aka CI Visibility Protocol) required to support Test Suite Level Visibility (tslv)

More information about the protocol and TSLV can be found here and here

Stacked PRs

Reviewer's Checklist

Changed code has unit tests for its functionality at or near 100% coverage.
System-Tests covering this feature have been added and enabled with the va.b.c-dev version tag.
There is a benchmark for any new code, or changes to existing code.
If this interacts with the agent in a new way, a system test has been added.
Add an appropriate team label so this PR gets put in the right place for the release notes.
Non-trivial go.mod changes, e.g. adding new modules, are reviewed by @DataDog/dd-trace-go-guild.

Unsure? Have a question? Request a review!

ddtrace/tracer/civisibility_transport_test.go

ddtrace/tracer/civisibility_tslv.go

ddtrace/tracer/civisibility_transport.go

pr-commenter · 2024-06-12T09:50:01Z

Benchmarks

Benchmark execution time: 2024-07-05 10:34:05

Comparing candidate commit 93cfbc8 in PR branch tony/civisibility-part2-ddtrace-modifications with baseline commit 1987b7b in branch tony/civisibility-part1-consts-and-utils.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 0 unstable metrics.

scenario:BenchmarkHttpServeTrace-24

🟩 execution_time [-747.244ns; -644.756ns] or [-4.317%; -3.725%]

ddtrace/tracer/civisibility_transport.go

ManuelPalenzuelaDD

LGTM!

darccio

Some comments of things I saw on a first cursory review. I'll be OOO, so maybe @dianashevchenko can take another look.

ddtrace/tracer/civisibility_payload.go

ddtrace/tracer/civisibility_transport.go

darccio

I had to review it at last hour before going OOO, so consider this approval just a way to unlock it :) I'd appreciate if more eyes can take a deeper look cc @dianashevchenko

ddtrace/tracer/option.go

ddtrace/tracer/civisibility_transport_test.go

ddtrace/tracer/civisibility_payload_test.go

ddtrace/tracer/civisibility_transport.go

ddtrace/tracer/civisibility_payload_test.go

ddtrace/tracer/civisibility_writer.go

ddtrace/tracer/civisibility_writer_test.go

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

golangci

ddtrace/tracer/civisibility_transport.go

ddtrace/tracer/civisibility_tslv.go

github-actions · 2024-07-01T15:34:53Z

ddtrace/tracer/civisibility_tslv.go

+func createTestEventFromSpan(span *span) *ciVisibilityEvent {
+	tSpan := createTslvSpan(span)
+	tSpan.SessionID = getAndRemoveMetaToUInt64(span, constants.TestSessionIDTagName)
+	tSpan.ModuleID = getAndRemoveMetaToUInt64(span, constants.TestModuleIDTagName)


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestModuleIDTagName

github-actions · 2024-07-01T15:34:53Z

ddtrace/tracer/civisibility_tslv.go

+	tSpan := createTslvSpan(span)
+	tSpan.SessionID = getAndRemoveMetaToUInt64(span, constants.TestSessionIDTagName)
+	tSpan.ModuleID = getAndRemoveMetaToUInt64(span, constants.TestModuleIDTagName)
+	tSpan.SuiteID = getAndRemoveMetaToUInt64(span, constants.TestSuiteIDTagName)


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestSuiteIDTagName

github-actions · 2024-07-01T15:34:53Z

ddtrace/tracer/civisibility_tslv.go

+	tSpan.SessionID = getAndRemoveMetaToUInt64(span, constants.TestSessionIDTagName)
+	tSpan.ModuleID = getAndRemoveMetaToUInt64(span, constants.TestModuleIDTagName)
+	tSpan.SuiteID = getAndRemoveMetaToUInt64(span, constants.TestSuiteIDTagName)
+	tSpan.CorrelationID = getAndRemoveMeta(span, constants.ItrCorrelationIDTagName)


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.ItrCorrelationIDTagName

github-actions · 2024-07-01T15:34:53Z

ddtrace/tracer/civisibility_tslv.go

+//	A pointer to the created ciVisibilityEvent.
+func createTestSuiteEventFromSpan(span *span) *ciVisibilityEvent {
+	tSpan := createTslvSpan(span)
+	tSpan.SessionID = getAndRemoveMetaToUInt64(span, constants.TestSessionIDTagName)


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestSessionIDTagName

github-actions · 2024-07-01T15:34:53Z

ddtrace/tracer/civisibility_tslv.go

+func createTestSuiteEventFromSpan(span *span) *ciVisibilityEvent {
+	tSpan := createTslvSpan(span)
+	tSpan.SessionID = getAndRemoveMetaToUInt64(span, constants.TestSessionIDTagName)
+	tSpan.ModuleID = getAndRemoveMetaToUInt64(span, constants.TestModuleIDTagName)


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestModuleIDTagName

…isibility-part2-ddtrace-modifications

ddtrace/tracer/civisibility_writer.go

ddtrace/tracer/civisibility_transport_test.go

Co-authored-by: liashenko <[email protected]>

github-actions

Remaining comments which cannot be posted as a review comment to avoid GitHub Rate Limit

golangci

internal/civisibility/integrations/gotesting/testing_test.go|297 col 40| undefined: constants.TestModuleIDTagName (typecheck)
internal/civisibility/integrations/gotesting/testing_test.go|298 col 40| undefined: constants.TestSuiteIDTagName (typecheck)

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/civisibility.go

+func internalCiVisibilityInitialization(tracerInitializer func([]tracer.StartOption)) {
+	ciVisibilityInitializationOnce.Do(func() {
+		// Since calling this method indicates we are in CI Visibility mode, set the environment variable.
+		_ = os.Setenv(constants.CiVisibilityEnabledEnvironmentVariable, "1")


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.CiVisibilityEnabledEnvironmentVariable

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/civisibility.go

+		_ = utils.GetCodeOwners()
+
+		// Preload all CI, Git, and CodeOwners tags.
+		ciTags := utils.GetCiTags()


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: utils.GetCiTags

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api.go

+	}...)
+
+	// Apply CI tags
+	for k, v := range utils.GetCiTags() {


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: utils.GetCiTags

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api_ddtest.go

+
+	span, ctx := tracer.StartSpanFromContext(context.Background(), operationName, testOpts...)
+	if suite.module.session != nil {
+		span.SetTag(constants.TestSessionIDTagName, fmt.Sprint(suite.module.session.sessionID))


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestSessionIDTagName

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api_ddtest.go

+	if suite.module.session != nil {
+		span.SetTag(constants.TestSessionIDTagName, fmt.Sprint(suite.module.session.sessionID))
+	}
+	span.SetTag(constants.TestModuleIDTagName, fmt.Sprint(suite.module.moduleID))


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestModuleIDTagName

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api_ddtest.go

+		span.SetTag(constants.TestSessionIDTagName, fmt.Sprint(suite.module.session.sessionID))
+	}
+	span.SetTag(constants.TestModuleIDTagName, fmt.Sprint(suite.module.moduleID))
+	span.SetTag(constants.TestSuiteIDTagName, fmt.Sprint(suite.suiteID))


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestSuiteIDTagName

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api_ddtest.go

+	}
+
+	file, line := fn.FileLine(fn.Entry())
+	file = utils.GetRelativePathFromCiTagsSourceRoot(file)


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: utils.GetRelativePathFromCiTagsSourceRoot

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api_ddtest.go

+
+	codeOwners := utils.GetCodeOwners()
+	if codeOwners != nil {
+		match := codeOwners.Match("/" + file)


🚫 [golangci] _{reported by reviewdog 🐶}
assignment mismatch: 1 variable but codeOwners.Match returns 2 values

github-actions · 2024-07-05T10:11:47Z

internal/civisibility/integrations/manual_api_ddtestmodule.go

+	span, ctx := tracer.StartSpanFromContext(context.Background(), operationName, testOpts...)
+	moduleID := span.Context().SpanID()
+	if session != nil {
+		span.SetTag(constants.TestSessionIDTagName, fmt.Sprint(session.sessionID))


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestSessionIDTagName

github-actions · 2024-07-05T10:11:48Z

internal/civisibility/integrations/manual_api_ddtestmodule.go

+	if session != nil {
+		span.SetTag(constants.TestSessionIDTagName, fmt.Sprint(session.sessionID))
+	}
+	span.SetTag(constants.TestModuleIDTagName, fmt.Sprint(moduleID))


🚫 [golangci] _{reported by reviewdog 🐶}
undefined: constants.TestModuleIDTagName

github-actions bot reviewed Jun 12, 2024

View reviewed changes

tonyredondo mentioned this pull request Jun 12, 2024

[CI Visibility] Initial internal consts and utilities #2736

Merged

6 tasks

tonyredondo marked this pull request as ready for review June 12, 2024 10:35

tonyredondo requested a review from a team as a code owner June 12, 2024 10:35

ManuelPalenzuelaDD reviewed Jun 13, 2024

View reviewed changes

ddtrace/tracer/civisibility_transport.go Outdated Show resolved Hide resolved

tonyredondo force-pushed the tony/civisibility-part1-consts-and-utils branch from 615ef86 to ccc1a39 Compare June 13, 2024 13:37

tonyredondo requested review from a team as code owners June 13, 2024 13:37

tonyredondo requested a review from romainkomorn-exdatadog June 13, 2024 13:37

tonyredondo force-pushed the tony/civisibility-part2-ddtrace-modifications branch from 47ea235 to 5862d18 Compare June 13, 2024 13:54

tonyredondo mentioned this pull request Jun 13, 2024

[CI Visibility] Manual Api and Go/Testing integration #2742

Merged

6 tasks

ManuelPalenzuelaDD approved these changes Jun 13, 2024

View reviewed changes

darccio requested changes Jun 13, 2024

View reviewed changes

ddtrace/tracer/civisibility_payload.go Outdated Show resolved Hide resolved

ddtrace/tracer/civisibility_payload.go Outdated Show resolved Hide resolved

ddtrace/tracer/civisibility_transport.go Outdated Show resolved Hide resolved

darccio approved these changes Jun 13, 2024

View reviewed changes

ddtrace/tracer/option.go Outdated Show resolved Hide resolved

tonyredondo added 7 commits June 14, 2024 10:00

Initial commit for adding a CI Visibility mode to ddtrace

3922951

Fixes for reviewdog comments

d0ef2b1

Improve the configuration switch

fe0e687

log an error and exit if agentless mode is set but there's no API key

d255e5f

Changes from review

f7fc64b

Changes from review

cdf770b

fix documentation

e3400dd

tonyredondo force-pushed the tony/civisibility-part2-ddtrace-modifications branch from c7de335 to e3400dd Compare June 14, 2024 08:02

ManuelPalenzuelaDD requested a review from dianashevchenko June 27, 2024 08:12

dianashevchenko reviewed Jun 27, 2024

View reviewed changes

PR feedback

f93f2a9

github-actions bot reviewed Jul 1, 2024

View reviewed changes

liashenko added 3 commits July 1, 2024 17:43

Merge branch 'tony/civisibility-part1-consts-and-utils' into tony/civ…

a1f55ad

…isibility-part2-ddtrace-modifications

Fix constants

7d361e3

Remove os.Exit

0e20a0a

Write a test for agentless with no api key

6d9882b

dianashevchenko reviewed Jul 4, 2024

View reviewed changes

ddtrace/tracer/civisibility_writer.go Show resolved Hide resolved

dianashevchenko reviewed Jul 4, 2024

View reviewed changes

ddtrace/tracer/civisibility_transport_test.go Show resolved Hide resolved

[CI Visibility] Manual Api and Go/Testing integration (#2742)

93cfbc8

Co-authored-by: liashenko <[email protected]>

github-actions bot reviewed Jul 5, 2024

View reviewed changes

Fix dependencies

c3be8d0

liashenko merged commit 1aba968 into tony/civisibility-part1-consts-and-utils Jul 8, 2024
156 checks passed

liashenko deleted the tony/civisibility-part2-ddtrace-modifications branch July 8, 2024 09:32

liashenko mentioned this pull request Jul 8, 2024

[CI Visibility] CI Test Visibility integration (#2736) #2774

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI Visibility] Add support for CI Visibility mode in ddtrace #2739

[CI Visibility] Add support for CI Visibility mode in ddtrace #2739

tonyredondo commented Jun 12, 2024 •

edited

Loading

pr-commenter bot commented Jun 12, 2024 •

edited

Loading

ManuelPalenzuelaDD left a comment

darccio left a comment •

edited

Loading

darccio left a comment

github-actions bot left a comment

github-actions bot Jul 1, 2024

github-actions bot Jul 1, 2024

github-actions bot Jul 1, 2024

github-actions bot Jul 1, 2024

github-actions bot Jul 1, 2024

github-actions bot left a comment

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

github-actions bot Jul 5, 2024

[CI Visibility] Add support for CI Visibility mode in ddtrace #2739

[CI Visibility] Add support for CI Visibility mode in ddtrace #2739

Conversation

tonyredondo commented Jun 12, 2024 • edited Loading

What does this PR do?

Motivation

Stacked PRs

Reviewer's Checklist

pr-commenter bot commented Jun 12, 2024 • edited Loading

Benchmarks

scenario:BenchmarkHttpServeTrace-24

ManuelPalenzuelaDD left a comment

Choose a reason for hiding this comment

darccio left a comment • edited Loading

Choose a reason for hiding this comment

darccio left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jul 1, 2024

Choose a reason for hiding this comment

github-actions bot Jul 1, 2024

Choose a reason for hiding this comment

github-actions bot Jul 1, 2024

Choose a reason for hiding this comment

github-actions bot Jul 1, 2024

Choose a reason for hiding this comment

github-actions bot Jul 1, 2024

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

github-actions bot Jul 5, 2024

Choose a reason for hiding this comment

tonyredondo commented Jun 12, 2024 •

edited

Loading

pr-commenter bot commented Jun 12, 2024 •

edited

Loading

darccio left a comment •

edited

Loading