Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add log_schema.kubernetes_key global option #1867

Closed
2 of 4 tasks
binarylogic opened this issue Feb 20, 2020 · 4 comments
Closed
2 of 4 tasks

Add log_schema.kubernetes_key global option #1867

binarylogic opened this issue Feb 20, 2020 · 4 comments
Labels
domain: processing Anything related to processing Vector's events (parsing, merging, reducing, etc.) needs: approval Needs review & approval before work can begin. platform: kubernetes Anything `kubernetes` platform related source: kubernetes_logs Anything `kubernetes_logs` source related type: enhancement A value-adding code change that enhances its existing functionality.

Comments

@binarylogic
Copy link
Contributor

binarylogic commented Feb 20, 2020

Similar to #1823 we need to:

  • Add a log_schema.kubernetes_key option. This should
  • Use this key in the kubernetes source
  • Use this key in the kubernetes_metadata transform
  • Use this key when mapping in the loki and datadog sinks. Any others? Should we open these as separate issues?
@binarylogic binarylogic added this to the Initial containers support milestone Feb 20, 2020
@LucioFranco
Copy link
Contributor

Looks like we can add the first two in this list today but the rest are blocked on the kubernetes_pod_metadata transform.

@binarylogic binarylogic assigned MOZGIII and unassigned LucioFranco and ktff Apr 4, 2020
@binarylogic binarylogic removed this from the Initial Containers Support milestone Jul 26, 2020
@MOZGIII
Copy link
Contributor

MOZGIII commented Aug 3, 2020

I feel like this needs an RFC. We need a way to dynamically extend the schema and refer to the fields in it.

Practically, implementing this PR would be a simple task, yet, I'm a bit hesitant with this approach. As I understand, the main goal would be to automate the interoperation with loki and other sources that follow the same idea of having a semi-manually composed subset of fields that are considered interesting (labels in particular in loki). There are a couple of alternative approaches that seem to be a better fit for that particular task.

I'll elaborate more on this in a dedicated issue.

We can omit this issue from v1 feature set for now.

@binarylogic
Copy link
Contributor Author

I agree. I think all of the field mapping in sinks is better served with metadata about fields instead of a strict a schema.

@binarylogic binarylogic added domain: processing Anything related to processing Vector's events (parsing, merging, reducing, etc.) needs: approval Needs review & approval before work can begin. type: enhancement A value-adding code change that enhances its existing functionality. platform: kubernetes Anything `kubernetes` platform related source: kubernetes_logs Anything `kubernetes_logs` source related labels Aug 3, 2020
@MOZGIII
Copy link
Contributor

MOZGIII commented Sep 15, 2020

Should we close this one?

@jamtur01 jamtur01 closed this as completed Nov 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: processing Anything related to processing Vector's events (parsing, merging, reducing, etc.) needs: approval Needs review & approval before work can begin. platform: kubernetes Anything `kubernetes` platform related source: kubernetes_logs Anything `kubernetes_logs` source related type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Development

No branches or pull requests

5 participants