Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

collect logs from pods from excluded system ns (#1138) #1146

Merged
merged 10 commits into from
Mar 22, 2024
Merged

collect logs from pods from excluded system ns (#1138) #1146

merged 10 commits into from
Mar 22, 2024

Conversation

pfrcks
Copy link
Contributor

@pfrcks pfrcks commented Jan 5, 2024 •
edited
Loading

  • enabled log collection of specific pods from system namespaces.
  • use daemonset and deployment names to identify pods whose logs need to be collected.
  • extracts potential ds and deployment names from podName and matches against user provided input in configmap option include_system_pods

eg. user sets include_system_pods = ["coredns"] where coredns can be deployment name or daemonset name which will result in pod name coredns-77sd7 or coredns-65cbf4f7bc-jcn6z.

@pfrcks pfrcks requested a review from a team as a code owner January 5, 2024 18:04
@pfrcks pfrcks force-pushed the develop/add-excluded-system-ns-pod-log-collection branch from 302299c to dc28f4b Compare January 10, 2024 04:44
@github-actions GitHub Actions
Copy link

This PR is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 5 days.

@github-actions GitHub Actions
Copy link

This PR was closed because it has been stalled for 12 days with no activity.

@github-actions github-actions bot closed this Jan 22, 2024
@ganga1980 ganga1980 reopened this Feb 16, 2024
@github-actions GitHub Actions
Copy link

This PR was closed because it has been stalled for 12 days with no activity.

@github-actions github-actions bot closed this Feb 23, 2024
@pfrcks pfrcks reopened this Mar 18, 2024
@pfrcks pfrcks enabled auto-merge (squash) March 21, 2024 20:14
@pfrcks pfrcks disabled auto-merge March 21, 2024 20:15
ganga1980
ganga1980 previously approved these changes Mar 21, 2024
View reviewed changes
@pfrcks pfrcks force-pushed the develop/add-excluded-system-ns-pod-log-collection branch from aee9e0a to 1732894 Compare March 21, 2024 20:18
@pfrcks pfrcks merged commit 7a14cbe into ci_prod Mar 22, 2024
15 checks passed
@pfrcks pfrcks deleted the develop/add-excluded-system-ns-pod-log-collection branch March 22, 2024 00:02
jatakiajanvi12 added a commit that referenced this pull request Mar 22, 2024
@jatakiajanvi12 @ganga1980
@dependabot @wanlonghenry @pfrcks
Enable disable addon test pipeline (#1208)
e30929c 

* Update daily_addons_enablement_test.yaml for Azure Pipelines

* Bump github.com/docker/docker in /source/plugins/go/input (#1194)

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.6+incompatible to 24.0.7+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v24.0.6...v24.0.7)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* start telegraf only when fluent tcp listener up and running (#1205)

* start telegraf only when fluent tcp listener up and running

* remove duplicate plugin

* update trivy ignore

* consistent with linux

* Containerlogv2 Metadata and annotations based filtering (#1148)

* add fluent-bit kubernetes filter (#1115)

* add fluent-bit kubernetes filter

---------

Co-authored-by: Amol Agrawal <[email protected]>

* add metadata feilds including labels, annotations, uid and image (#1120)

* add metadata feilds including labels, annotations, uid and image

* set the include_fields if customer set it otherwise go with default if enabled

* fix minor bugs

* change placeholder

* add exclude specific kubernetes config

* update tls verify for linux

* make log filtering opt in explicit (#1140)

Co-authored-by: Amol Agrawal <[email protected]>

* Longw/metadata containerlogv2 kubernetes (#1139)

* add metadata feilds including labels, annotations, uid and image

* rename to podUid and add nil check

* set the include_fields if customer set it otherwise go with default if enabled

* add workload to testing clusters

* containerlogv2 metadata backend change with feature flag on

* adjust the marshal and log the output

* address comments add check and scenario for empty list

* remove ADX support

* go fmt for better format

* fix space in tomlparser.rb

* update trivy ignore

* fix  Kube_Tag_Prefix for windows

* update trivy

* add ttl for metadata cache

* update containerlogv2 test workloads

* change typo and update tests

* Longw/metadata containerlogv2 address feedback (#1166)

* add argument for sendMetric

* update SendMetric

* update SendMetric

* Longw/metadata containerlogv2 address feedback2 (#1186)

* extend to support more configs on metadata

* add warning message if feilds not match

* add plugin for geneva path

* add kubernetesMetadataCollection for geneva path

* Longw/metadata containerlogv2 address feedback2 (#1188)

* remove send metric

* update trivy

* Longw/metadata containerlogv2 address feedback3 (#1195)

* update doc for containerlogv2 linux

* add kube_meta_cache_ttl to agent settings

* update logic to adjust colonLocation as the end of the list if it is not found

* address comments and feedback

* update conf file for geneva

* adjust geneva and add new metrics

* address feedback for geneva path

* add telemetry in go

* address comments for geneva conf filter config

---------

Co-authored-by: Amol Agrawal <[email protected]>
Co-authored-by: Amol Agrawal <[email protected]>

* move LOGS_AND_EVENTS_ONLY telemetry to DS (#1212)

* move LOGS_AND_EVENTS_ONLY telemetry to DS

* Add CVE-2024-24557 to trivyignore

---------

Co-authored-by: Amol Agrawal <[email protected]>

* fix input plugin bugs (#1207)

* fix input plugin bugs

---------

Co-authored-by: Amol Agrawal <[email protected]>

* collect logs from pods from excluded system ns (#1138) (#1146)

* collect logs from pods from excluded system ns

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>
Co-authored-by: Ganga Mahesh Siddem <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Long Wan <[email protected]>
Co-authored-by: Amol Agrawal <[email protected]>
Co-authored-by: Amol Agrawal <[email protected]>
pfrcks pushed a commit that referenced this pull request Apr 6, 2024
clean up code + address PR comments
cd83bb9 
move LOGS_AND_EVENTS_ONLY telemetry to DS (#1212)

* move LOGS_AND_EVENTS_ONLY telemetry to DS

* Add CVE-2024-24557 to trivyignore

---------

Co-authored-by: Amol Agrawal <[email protected]>

fix input plugin bugs (#1207)

* fix input plugin bugs

---------

Co-authored-by: Amol Agrawal <[email protected]>

collect logs from pods from excluded system ns (#1138) (#1146)

* collect logs from pods from excluded system ns

Enable disable addon test pipeline (#1208)

* Update daily_addons_enablement_test.yaml for Azure Pipelines

* Bump github.com/docker/docker in /source/plugins/go/input (#1194)

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.6+incompatible to 24.0.7+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v24.0.6...v24.0.7)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* start telegraf only when fluent tcp listener up and running (#1205)

* start telegraf only when fluent tcp listener up and running

* remove duplicate plugin

* update trivy ignore

* consistent with linux

* Containerlogv2 Metadata and annotations based filtering (#1148)

* add fluent-bit kubernetes filter (#1115)

* add fluent-bit kubernetes filter

---------

Co-authored-by: Amol Agrawal <[email protected]>

* add metadata feilds including labels, annotations, uid and image (#1120)

* add metadata feilds including labels, annotations, uid and image

* set the include_fields if customer set it otherwise go with default if enabled

* fix minor bugs

* change placeholder

* add exclude specific kubernetes config

* update tls verify for linux

* make log filtering opt in explicit (#1140)

Co-authored-by: Amol Agrawal <[email protected]>

* Longw/metadata containerlogv2 kubernetes (#1139)

* add metadata feilds including labels, annotations, uid and image

* rename to podUid and add nil check

* set the include_fields if customer set it otherwise go with default if enabled

* add workload to testing clusters

* containerlogv2 metadata backend change with feature flag on

* adjust the marshal and log the output

* address comments add check and scenario for empty list

* remove ADX support

* go fmt for better format

* fix space in tomlparser.rb

* update trivy ignore

* fix  Kube_Tag_Prefix for windows

* update trivy

* add ttl for metadata cache

* update containerlogv2 test workloads

* change typo and update tests

* Longw/metadata containerlogv2 address feedback (#1166)

* add argument for sendMetric

* update SendMetric

* update SendMetric

* Longw/metadata containerlogv2 address feedback2 (#1186)

* extend to support more configs on metadata

* add warning message if feilds not match

* add plugin for geneva path

* add kubernetesMetadataCollection for geneva path

* Longw/metadata containerlogv2 address feedback2 (#1188)

* remove send metric

* update trivy

* Longw/metadata containerlogv2 address feedback3 (#1195)

* update doc for containerlogv2 linux

* add kube_meta_cache_ttl to agent settings

* update logic to adjust colonLocation as the end of the list if it is not found

* address comments and feedback

* update conf file for geneva

* adjust geneva and add new metrics

* address feedback for geneva path

* add telemetry in go

* address comments for geneva conf filter config

---------

Co-authored-by: Amol Agrawal <[email protected]>
Co-authored-by: Amol Agrawal <[email protected]>

* move LOGS_AND_EVENTS_ONLY telemetry to DS (#1212)

* move LOGS_AND_EVENTS_ONLY telemetry to DS

* Add CVE-2024-24557 to trivyignore

---------

Co-authored-by: Amol Agrawal <[email protected]>

* fix input plugin bugs (#1207)

* fix input plugin bugs

---------

Co-authored-by: Amol Agrawal <[email protected]>

* collect logs from pods from excluded system ns (#1138) (#1146)

* collect logs from pods from excluded system ns

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>
Co-authored-by: Ganga Mahesh Siddem <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Long Wan <[email protected]>
Co-authored-by: Amol Agrawal <[email protected]>
Co-authored-by: Amol Agrawal <[email protected]>

Update the pipeline to not trigger for builds (#1214)

Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>

Gangams/add flag to enable service account timebound token (#1213)

* add flag for service account timebound token

* fix pr feedback

* fix pr feedback

Solving pipeline bugs for windows ama url variable (#1215)

Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>

Bump github.com/docker/docker in /source/plugins/go/input (#1209)

Bumps [github.com/docker/docker](https://github.com/docker/docker) from 24.0.7+incompatible to 24.0.9+incompatible.
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v24.0.7...v24.0.9)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Add telemetry for KubernetesMetadata size (#1216)

* add telemetry for KubernetesMetadata size

* go fmt for style

* add condition check with KubernetesMetadataEnabled

use single output

change tags for combined output

fix bugs

update win conf

fix formatting pv metrics

fix formatting pv metrics (#1219)

Co-authored-by: Amol Agrawal <[email protected]>

update logPaths for tests and fix TODOs

Dont start fluentd for MSI mode and fluentbit enabled (#1221)

Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>

address PR comments

change test log extension

Add release notes for 3.1.19 (#1220)

Containerlogv2 Kubernetes Metadata Grafana Dashboard Private Preview (#1218)

* Containerlogv2 Kubernetes Metadata Grafana Dashboard Private Preview

* update dashboard

* dashboard cleanup

* update raw template

* template cleanup

* update

Update the start fluentd condition (#1223)

Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>

add more checks

add debug statement

better logging

Add release notes for 3.1.19 (#1220)

Containerlogv2 Kubernetes Metadata Grafana Dashboard Private Preview (#1218)

* Containerlogv2 Kubernetes Metadata Grafana Dashboard Private Preview

* update dashboard

* dashboard cleanup

* update raw template

* template cleanup

* update

Update the start fluentd condition (#1223)

Co-authored-by: Janvi Jatakia (from Dev Box) <[email protected]>

add more checks

add debug statement

better logging
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ganga1980 ganga1980 ganga1980 approved these changes

Assignees
No one assigned
Labels
no-pr-activity
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@pfrcks @ganga1980