Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Write documentation about how to use tracing filters #970

Open
LucioFranco opened this issue Oct 4, 2019 · 3 comments
Open

Write documentation about how to use tracing filters #970

LucioFranco opened this issue Oct 4, 2019 · 3 comments
Labels
domain: dev Anything related to Vector's development environment domain: internal docs Anything related to Vector's internal documentation domain: observability Anything related to monitoring/observing Vector meta: good first issue Anything that is good for new contributors. type: enhancement A value-adding code change that enhances its existing functionality.

Comments

@LucioFranco
Copy link
Contributor

Motivation

We should provide a good experience when users are trying to inspect why vector may have failed. Vector should provide methods in which developres/users can extract extensive debug information on specific components.

Proposal

Write extensive documentation on how users can use the LOG environment variable to dig deeper into the logs of vector to diagnose certains issues. These issues can range from high level topology issues to low level tokio task issues.

Using the LOG environment variable should follow the way directives are setup within tracing-subscriber. These filters follow roughly what https://docs.rs/env_logger/0.7.0/env_logger/ provides but with additional features to support spans. This additional support is important to allow us to dig deeper into specific components. Possible questions that can be answered are: why are all my sinks failing? why did my http sink fail to gzip my body? why didn't my source task wake up the inner task? All of these questions can be answered by providing the correct filter.

Example

This will only provide trace logs for events created within the vector crate and at the trace level.

$ LOG=vector[sinks]=trace vector -c my_config.toml

This will provide debug logs for just the "http" typed sinks.

$ LOG=vector[sinks{type="http"}]=debug vector -c my_config.toml

This will provide debug logs for our HTTP client from the hyper crate for just the "http" typed sinks.

$ LOG=hyper[sinks{type="http"}]=debug vector -c my_config.toml

Blockers

Only one issue is blocking supporting all of these features: tokio-rs/tracing#367.
@LucioFranco LucioFranco added type: enhancement A value-adding code change that enhances its existing functionality. domain: observability Anything related to monitoring/observing Vector domain: docs labels Oct 4, 2019
@binarylogic binarylogic mentioned this issue Oct 4, 2019
Closed
21 tasks
LucioFranco added a commit that referenced this issue Oct 4, 2019
@LucioFranco
feat(observability): Add rate limited debug messages
9ee6f0a 
This PR contains three major code changes:

- Update all of our tracing dependencies to the latest
versions.
- Update `tracing-limit` to use `tracing_subscriber`'s
`Layer` trait.
- Intrument the main code base with rate limited tracing
events.

This change introduces the ability for tracing events to be
rate limited. The primary reason for including this is for
events that may be important for a user to see but may arrive
in large bursts. This can cause issues around saturating IO and
in general making it difficult to diagnose the issue.

The solution is to allow tracing events to provide a
`rate_limit_secs` attribute that specifies a window at which
we will only see one log and the rest will be counted. This
enables users to know that this event is being recorded but
will not flood their view into vector.

```rust
 INFO basic: hello, world! count=0 rate_limit_secs=5
TRACE basic: this field is not rate limited!
 INFO basic: "hello, world!" is being rate limited. rate_limit_secs=5
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
 INFO basic: 5 "hello, world!" events were rate limited. rate_limit_secs=5
```

Closes #806
Related to #970

Signed-off-by: Lucio Franco <[email protected]>
LucioFranco added a commit that referenced this issue Oct 4, 2019
@LucioFranco
feat(observability): Add rate limited debug messages
ea5e4a8 
This PR contains three major code changes:

- Update all of our tracing dependencies to the latest
versions.
- Update `tracing-limit` to use `tracing_subscriber`'s
`Layer` trait.
- Intrument the main code base with rate limited tracing
events.

This change introduces the ability for tracing events to be
rate limited. The primary reason for including this is for
events that may be important for a user to see but may arrive
in large bursts. This can cause issues around saturating IO and
in general making it difficult to diagnose the issue.

The solution is to allow tracing events to provide a
`rate_limit_secs` attribute that specifies a window at which
we will only see one log and the rest will be counted. This
enables users to know that this event is being recorded but
will not flood their view into vector.

```rust
 INFO basic: hello, world! count=0 rate_limit_secs=5
TRACE basic: this field is not rate limited!
 INFO basic: "hello, world!" is being rate limited. rate_limit_secs=5
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
 INFO basic: 5 "hello, world!" events were rate limited. rate_limit_secs=5
```

Closes #806
Related to #970

Signed-off-by: Lucio Franco <[email protected]>
LucioFranco added a commit that referenced this issue Oct 4, 2019
@LucioFranco
feat(observability): Add rate limited debug messages
1434c84 
This PR contains three major code changes:

- Update all of our tracing dependencies to the latest
versions.
- Update `tracing-limit` to use `tracing_subscriber`'s
`Layer` trait.
- Intrument the main code base with rate limited tracing
events.

This change introduces the ability for tracing events to be
rate limited. The primary reason for including this is for
events that may be important for a user to see but may arrive
in large bursts. This can cause issues around saturating IO and
in general making it difficult to diagnose the issue.

The solution is to allow tracing events to provide a
`rate_limit_secs` attribute that specifies a window at which
we will only see one log and the rest will be counted. This
enables users to know that this event is being recorded but
will not flood their view into vector.

```rust
 INFO basic: hello, world! count=0 rate_limit_secs=5
TRACE basic: this field is not rate limited!
 INFO basic: "hello, world!" is being rate limited. rate_limit_secs=5
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
 INFO basic: 5 "hello, world!" events were rate limited. rate_limit_secs=5
```

Closes #806
Related to #970

Signed-off-by: Lucio Franco <[email protected]>
@LucioFranco LucioFranco mentioned this issue Oct 4, 2019
Merged
@binarylogic binarylogic added this to the Improve setup & debugging milestone Oct 5, 2019
@binarylogic
Copy link
Contributor

Noting, this should be placed in https://docs.vector.dev/usage/administration/monitoring and possibly to https://docs.vector.dev/usage/guides/troubleshooting.

LucioFranco added a commit that referenced this issue Oct 8, 2019
@LucioFranco
feat(observability): Add rate limited debug messages (#971)
b541bb1 
* feat(observability): Add rate limited debug messages

This PR contains three major code changes:

- Update all of our tracing dependencies to the latest
versions.
- Update `tracing-limit` to use `tracing_subscriber`'s
`Layer` trait.
- Intrument the main code base with rate limited tracing
events.

This change introduces the ability for tracing events to be
rate limited. The primary reason for including this is for
events that may be important for a user to see but may arrive
in large bursts. This can cause issues around saturating IO and
in general making it difficult to diagnose the issue.

The solution is to allow tracing events to provide a
`rate_limit_secs` attribute that specifies a window at which
we will only see one log and the rest will be counted. This
enables users to know that this event is being recorded but
will not flood their view into vector.

```rust
 INFO basic: hello, world! count=0 rate_limit_secs=5
TRACE basic: this field is not rate limited!
 INFO basic: "hello, world!" is being rate limited. rate_limit_secs=5
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
TRACE basic: this field is not rate limited!
 INFO basic: 5 "hello, world!" events were rate limited. rate_limit_secs=5
```

Closes #806
Related to #970

Signed-off-by: Lucio Franco <[email protected]>
@binarylogic binarylogic modified the milestones: Local development & testing, Improve observability Oct 19, 2019
@LucioFranco LucioFranco added the meta: blocked Anything that is blocked to the point where it cannot be worked on. label Oct 25, 2019
@LucioFranco
Copy link
Contributor Author

Update on this issue, I have been working with the other tracing maintainers on stabalizing the synatx and getting a release but its moving a bit slower than I had planned due to a couple corner cases. Should have our docs and new version next week before I am off to europe.

@binarylogic binarylogic removed this from the Vector Observability milestone Jul 20, 2020
@binarylogic binarylogic added domain: external docs Anything related to Vector's external, public documentation and removed domain: website labels Jul 29, 2020
@binarylogic binarylogic added domain: dev Anything related to Vector's development environment domain: internal docs Anything related to Vector's internal documentation meta: good first issue Anything that is good for new contributors. and removed domain: external docs Anything related to Vector's external, public documentation labels Aug 7, 2020
@jszwedko
Copy link
Member

jszwedko commented May 4, 2023

We could also update https://vector.dev/docs/reference/cli/#vector_log

@jszwedko jszwedko removed the meta: blocked Anything that is blocked to the point where it cannot be worked on. label May 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain: dev Anything related to Vector's development environment domain: internal docs Anything related to Vector's internal documentation domain: observability Anything related to monitoring/observing Vector meta: good first issue Anything that is good for new contributors. type: enhancement A value-adding code change that enhances its existing functionality.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jszwedko @bruceg @binarylogic @LucioFranco