Add Support for Blob Type (s3 and GCS) #879

khrm · 2024-11-10T06:22:09Z

Added support for reading forwarded logs to s3 and GCS.
If vector forwards logs to s3 and GCS, these logs can be read using
Go CDK's blob library.
Also, if result logging mechanism is changed from v1alpha2 to v1alpha3
which doesn't support watcher, we can read old logs if they exist.

Changes

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you review them:

Has Docs included if any changes are user facing
Has Tests included if any functionality added or changed
Tested your changes locally (if this is a code change)
Follows the commit message standard
Meets the Tekton contributor standards (including functionality, content, code)
Has a kind label. You can add a comment on this PR that contains /kind <type>. Valid types are bug, cleanup, design, documentation, feature, flake, misc, question, tep
Release notes block below has been updated with any user-facing changes (API changes, bug fixes, changes requiring upgrade notices or deprecation warnings)
Release notes contain the string "action required" if the change requires additional action from users switching to the new release

Release Notes

Added support for reading forwarded logs to s3 and GCS.
If vector forwards logs to s3 and GCS, these logs can be read using
Go CDK's blob library.
Also, if result logging mechanism is changed from v1alpha2 to v1alpha3
which doesn't support watcher, we can read old logs if they exist.

tekton-robot · 2024-11-10T06:39:32Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.3%	8.4%	0.2
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin_logs/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin_logs/server.go	Do not exist	80.0%
pkg/apis/v1alpha3/types.go	100.0%	0.0%	-100.0
pkg/apis/v1alpha3/types.go	Do not exist	42.9%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

tekton-robot · 2024-11-10T07:31:05Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
cmd/api/main.go	8.3%	8.4%	0.2
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin_logs/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin_logs/server.go	Do not exist	80.0%
pkg/apis/v1alpha3/types.go	100.0%	0.0%	-100.0
pkg/apis/v1alpha3/types.go	Do not exist	42.9%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

khrm

/assign @enarha @vdemeester

tekton-robot · 2024-11-11T09:07:53Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin_logs/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin_logs/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	100.0%	14.3%	-85.7
pkg/apis/v1alpha3/types.go	Do not exist	33.3%
pkg/apis/v1alpha3/types.go	Do not exist	100.0%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

tekton-robot · 2024-11-11T23:05:08Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	0.0%
pkg/apis/v1alpha3/types.go	Do not exist	66.7%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

tekton-robot · 2024-11-11T23:32:20Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

tekton-robot · 2024-11-11T23:51:29Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	0.0%	-38.2
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%

tekton-robot · 2024-11-12T00:25:33Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/gcs.go	42.9%	33.9%	-8.9
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%

khrm · 2024-11-12T00:42:35Z

/test pull-tekton-results-unit-tests

tekton-robot · 2024-11-12T00:47:45Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	26.4%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%

pkg/api/server/v1alpha2/log/gcs_test.go

pkg/api/server/v1alpha2/plugin/plugin_logs.go

enarha · 2024-11-13T08:32:58Z

pkg/api/server/v1alpha2/plugin/plugin_logs.go

+		if legacy {
+			logRec, err = getLogRecord(s.db, parent, rec.ResultID, rec.Name)
+			if err != nil {
+				s.logger.Debugf("error getting legacy log record: %s", err)


why do we continue here?

If err != nil, then logRec will be nil

If legacy is configured which means log type to v1alpha2, why do we continue as v1alpha3?

Yes, fixed that.

Yes, Log Type kind itself can be v1alpha2 or v1alpha3.
In plugin API, there's no LogType record itself.

enarha · 2024-11-13T08:35:50Z

pkg/api/server/v1alpha2/plugin/plugin_logs.go

+				err := json.Unmarshal(logRec.Data, log)
+				if err != nil {
+					err = fmt.Errorf("could not decode Log record: %w", err)
+					s.logger.Error(err)


below you call the .Error() (s.logger.Error(err.Error())) method on the error while logging. I do not mind either, just keep consistent.
Also in the other instances below.

Changed it to use s.logger.Error(err) to everywhere.

enarha · 2024-11-13T08:51:02Z

pkg/api/server/v1alpha2/plugin/server.go

+	}
+
+	s.logger.Debugf("LOGS_TYPE: %s", strings.ToLower(s.config.LOGS_TYPE))
+	// If the logs type is not Loki, we don't need to set up the LogPluginServer


setLogPlugin() with also return true if the log type BlobLogType, so either the comment is outdated or the logic below.

Yes. Comment is outdated.

Updated this.

khrm · 2024-11-13T09:34:15Z

/retest

tekton-robot · 2024-11-13T09:53:22Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	25.6%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%

Added support for reading forwarded logs to s3 and GCS. If vector forwards logs to s3 and GCS, these logs can be read using Go CDK's blob library. Also, if result logging mechanism is changeg from v1alpha2 to v1alpha3 which doesn't support watcher, we can read old logs if they exist.

tekton-robot · 2024-11-13T10:05:51Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	25.6%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	16.7%
pkg/apis/v1alpha3/types.go	Do not exist	0.0%
pkg/apis/v1alpha3/types.go	Do not exist	66.7%

Added documentation for blob plugin logging and added details about CDK.

docs/logging-support.md

tekton-robot · 2024-11-13T10:24:00Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	25.6%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	33.3%
pkg/apis/v1alpha3/types.go	Do not exist	60.0%

Fix typo in documentation (ena)

enarha · 2024-11-13T10:44:50Z

/lgtm

enarha · 2024-11-13T10:45:04Z

/approve

tekton-robot · 2024-11-13T10:45:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enarha, vdemeester

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [enarha,vdemeester]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tekton-robot · 2024-11-13T10:51:01Z

The following is the coverage report on the affected files.
Say /test pull-tekton-results-go-coverage to re-run this coverage report

File	Old Coverage	New Coverage	Delta
pkg/api/server/v1alpha2/log/s3.go	38.2%	37.7%	-0.6
pkg/api/server/v1alpha2/plugin/plugin_logs.go	Do not exist	25.6%
pkg/api/server/v1alpha2/plugin/server.go	Do not exist	80.0%
pkg/api/server/v1alpha2/server.go	75.9%	69.2%	-6.7
pkg/apis/v1alpha3/types.go	66.7%	0.0%	-66.7
pkg/apis/v1alpha3/types.go	Do not exist	50.0%
pkg/apis/v1alpha3/types.go	Do not exist	20.0%
pkg/apis/v1alpha3/types.go	Do not exist	50.0%

khrm · 2024-11-13T17:18:00Z

/kind feature

tekton-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Nov 10, 2024

tekton-robot requested review from dibyom and enarha November 10, 2024 06:22

tekton-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Nov 10, 2024

khrm force-pushed the blobLogs branch from 273dab3 to 955400a Compare November 10, 2024 07:13

khrm force-pushed the blobLogs branch 2 times, most recently from 3f5b1b8 to c91eaaa Compare November 11, 2024 08:31

khrm added this to the v0.13.0 milestone Nov 11, 2024

khrm commented Nov 11, 2024

View reviewed changes

tekton-robot assigned enarha and vdemeester Nov 11, 2024

Vendor Dependencies for Blob Logging

eb72f5d

khrm force-pushed the blobLogs branch from c91eaaa to 9e48acf Compare November 11, 2024 08:40

khrm force-pushed the blobLogs branch from 9e48acf to 3e0ad8e Compare November 11, 2024 22:49

khrm force-pushed the blobLogs branch from 0addfd0 to d4688f4 Compare November 11, 2024 23:35

khrm force-pushed the blobLogs branch from d4688f4 to 1ea49b3 Compare November 12, 2024 00:19

khrm force-pushed the blobLogs branch from 1ea49b3 to d67030f Compare November 12, 2024 00:41

vdemeester approved these changes Nov 12, 2024

View reviewed changes

tekton-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 12, 2024

khrm force-pushed the blobLogs branch from d67030f to a67ad1b Compare November 12, 2024 11:44

enarha reviewed Nov 13, 2024

View reviewed changes

khrm force-pushed the blobLogs branch 2 times, most recently from 7c735d5 to 7d4c40c Compare November 13, 2024 09:46

khrm added 3 commits November 13, 2024 15:29

Add setup for S3 compatible log forwarding in Results

e69c352

Fix Build tests failure in s3 due to deprecation

86aa806

khrm force-pushed the blobLogs branch from 7d4c40c to 86aa806 Compare November 13, 2024 09:59

Add Documentation for Blob Plugin Logging

1e85022

Added documentation for blob plugin logging and added details about CDK.

enarha reviewed Nov 13, 2024

View reviewed changes

docs/logging-support.md Outdated Show resolved Hide resolved

docs/logging-support.md Outdated Show resolved Hide resolved

Apply suggestions from code review

a360752

Fix typo in documentation (ena)

tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 13, 2024

tekton-robot merged commit eb58c78 into tektoncd:main Nov 13, 2024
6 checks passed

khrm deleted the blobLogs branch November 13, 2024 11:35

tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Nov 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Support for Blob Type (s3 and GCS) #879

Add Support for Blob Type (s3 and GCS) #879

khrm commented Nov 10, 2024

tekton-robot commented Nov 10, 2024

tekton-robot commented Nov 10, 2024

khrm left a comment

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 12, 2024

khrm commented Nov 12, 2024

tekton-robot commented Nov 12, 2024

enarha Nov 13, 2024

khrm Nov 13, 2024

enarha Nov 13, 2024

khrm Nov 13, 2024

enarha Nov 13, 2024

khrm Nov 13, 2024

khrm Nov 13, 2024

khrm commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

enarha commented Nov 13, 2024

enarha commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

khrm commented Nov 13, 2024

Add Support for Blob Type (s3 and GCS) #879

Add Support for Blob Type (s3 and GCS) #879

Conversation

khrm commented Nov 10, 2024

Changes

Submitter Checklist

Release Notes

tekton-robot commented Nov 10, 2024

tekton-robot commented Nov 10, 2024

khrm left a comment

Choose a reason for hiding this comment

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 11, 2024

tekton-robot commented Nov 12, 2024

khrm commented Nov 12, 2024

tekton-robot commented Nov 12, 2024

enarha Nov 13, 2024

Choose a reason for hiding this comment

khrm Nov 13, 2024

Choose a reason for hiding this comment

enarha Nov 13, 2024

Choose a reason for hiding this comment

khrm Nov 13, 2024

Choose a reason for hiding this comment

enarha Nov 13, 2024

Choose a reason for hiding this comment

khrm Nov 13, 2024

Choose a reason for hiding this comment

khrm Nov 13, 2024

Choose a reason for hiding this comment

khrm commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

enarha commented Nov 13, 2024

enarha commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

tekton-robot commented Nov 13, 2024

khrm commented Nov 13, 2024