Skip to content

Commit

Permalink
Introduce tokio-trace (#827)
Browse files Browse the repository at this point in the history 
<!-- Thank you for your Pull Request. Please provide a description above
and review the requirements below.

Bug fixes and new features should include tests.

Contributors guide:
https://github.com/tokio-rs/tokio/blob/master/CONTRIBUTING.md -->

## Motivation

In asynchronous systems like Tokio, interpreting traditional log
messages can often be quite challenging. Since individual tasks are
multiplexed on the same thread, associated events and log lines are
intermixed making it difficult to trace the logic flow. Currently, none
of the available logging frameworks or libraries in Rust offer the
ability to trace logical paths through a futures-based program.

There also are complementary goals that can be accomplished with such a
system. For example, metrics / instrumentation can be tracked by
observing emitted events, or trace data can be exported to a distributed
tracing or event processing system.

In addition, it can often be useful to generate this diagnostic data in
a structured manner that can be consumed programmatically. While prior
art for structured logging in Rust exists, it is not currently
standardized, and is not "Tokio-friendly".

## Solution

This branch adds a new library to the tokio project, `tokio-trace`.
`tokio-trace` expands upon logging-style diagnostics by allowing
libraries and applications to record structured events with additional
information about *temporality* and *causality* --- unlike a log
message, a span in `tokio-trace` has a beginning and end time, may be
entered and exited by the flow of execution, and may exist within a
nested tree of similar spans. In addition, `tokio-trace` spans are
*structured*, with the ability to record typed data as well as textual
messages.

The `tokio-trace-core` crate contains the core primitives for this
system, which are expected to remain stable, while `tokio-trace` crate
provides a more "batteries-included" API. In particular, it provides
macros which are a superset of the `log` crate's `error!`, `warn!`,
`info!`, `debug!`, and `trace!` macros, allowing users to begin the
process of adopting `tokio-trace` by performing a drop-in replacement.

## Notes

Work on this project had previously been carried out in the
[tokio-trace-prototype] repository. In addition to the `tokio-trace` and
`tokio-trace-core` crates, the `tokio-trace-prototype` repo also
contains prototypes or sketches of adapter, compatibility, and utility
crates which provide useful functionality for `tokio-trace`, but these
crates are not yet ready for a release. When this branch is merged, that
repository will be archived, and the remaining unstable crates will be
moved to a new `tokio-trace-nursery` repository. Remaining issues on the
`tokio-trace-prototype` repo will be moved to the appropriate new repo.

The crates added in this branch are not _identical_ to the current head
of the `tokio-trace-prototype` repo, as I did some final clean-up and docs
polish in this branch prior to merging this PR.

[tokio-trace-prototype]: https://github.com/hawkw/tokio-trace-prototype

Closes: #561

Signed-off-by: Eliza Weisman <[email protected]>
  • Loading branch information
@hawkw
hawkw committed Jun 25, 2019
1 parent 767ec8c commit 03d6e80
Show file tree
Hide file tree
Showing 34 changed files with 6,340 additions and 83 deletions.
32 changes: 32 additions & 0 deletions Cargo.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
[workspace]

members = [
".",
"tokio-trace-proc-macros",
"tokio-trace-fmt",
"tokio-trace-futures",
Expand All @@ -13,3 +14,34 @@ members = [
"tokio-trace-subscriber",
"tokio-trace-serde",
]

[package]
name = "tokio-trace"
version = "0.0.1"
authors = ["Eliza Weisman <[email protected]>"]
license = "MIT"
repository = "https://github.com/tokio-rs/tokio"
homepage = "https://tokio.rs"
description = """
A scoped, structured logging and diagnostics system.
"""
categories = ["development-tools::debugging", "asynchronous"]
keywords = ["logging", "tracing"]

# Not yet ready for production.
publish = false

[dependencies]
tokio-trace-core = { path = "tokio-trace-core" }

[dev-dependencies]
ansi_term = "0.11"
humantime = "1.1.1"
futures = "0.1"
log = "0.4"

# These are used for the "basic" example from the tokio-trace-prototype repo,
# which is currently not included as it used the `tokio-trace-log` crate, and
# that crate is currently unstable.
# env_logger = "0.5"
# tokio-trace-log = { path = "../tokio-trace-log" }
38 changes: 21 additions & 17 deletions LICENSE
View file
Original file line number Diff line number Diff line change
@@ -1,21 +1,25 @@
MIT License

Copyright (c) 2019 Tokio Contributors

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
Permission is hereby granted, free of charge, to any
person obtaining a copy of this software and associated
documentation files (the "Software"), to deal in the
Software without restriction, including without
limitation the rights to use, copy, modify, merge,
publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software
is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
The above copyright notice and this permission notice
shall be included in all copies or substantial portions
of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF
ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED
TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT
SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR
IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
220 changes: 154 additions & 66 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,71 +1,161 @@
# tokio-trace-nursery
# tokio-trace

Less-stable utility crates for [`tokio-trace`].

[![MIT licensed][mit-badge]][mit-url]
[![Build Status][travis-badge]][travis-url]
[![Gitter chat][gitter-badge]][gitter-url]

[mit-badge]: https://img.shields.io/badge/license-MIT-blue.svg
[mit-url]: LICENSE
[travis-badge]: https://travis-ci.org/tokio-rs/tokio-trace-nursery.svg?branch=master
[travis-url]: https://travis-ci.org/tokio-rs/tokio-trace-nursery/branches
[gitter-badge]: https://img.shields.io/gitter/room/tokio-rs/tokio.svg
[gitter-url]: https://gitter.im/tokio-rs/tokio

[Website](https://tokio.rs) |
[Chat](https://gitter.im/tokio-rs/tokio)
A scoped, structured logging and diagnostics system.

## Overview

[`tokio-trace`] is a framework for instrumenting Rust programs to collect
structured, event-based diagnostic information. This repository contains a set
of utility and compatibility crates for use with `tokio-trace`.

### Stability

While `tokio-trace` and `tokio-trace-core` have been published on crates.io and
adhere to the same stability policies as the rest of the Tokio project, the
crates in the nursery are generally less stable. Many of these crates are not
yet released and are undergoing active development. Therefore, users are warned
that breaking changes may occur.

In general, when depending on a crate from the nursery as a git dependency,
users are advised to pin to a specific git revision using the [`rev`] Cargo key.
This prevents your build from breaking should a breaking change to that crate be
merged to master.

[`rev`]: https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#specifying-dependencies-from-git-repositories

## Getting Help

First, see if the answer to your question can be found in the API documentation.
If the answer is not there, there is an active community in
the [Tokio Gitter channel][chat]. We would be happy to try to answer your
question. Last, if that doesn't work, try opening an [issue] with the question.

[chat]: https://gitter.im/tokio-rs/tokio
[issue]: https://github.com/tokio-rs/tokio-trace-nursery/issues/new

## Contributing

:balloon: Thanks for your help improving the project! We are so happy to have
you! We have a [contributing guide][guide] to help you get involved in the Tokio
project.

[guide]: CONTRIBUTING.md
<!--
## Project layout
TODO: add this
-->

## Supported Rust Versions

Tokio is built against the latest stable, nightly, and beta Rust releases. The
minimum version supported is the stable release from three months before the
current stable release version. For example, if the latest stable Rust is 1.29,
the minimum version supported is 1.26. The current Tokio version is not
guaranteed to build on Rust versions earlier than the minimum supported version.
`tokio-trace` is a framework for instrumenting Rust programs to collect
structured, event-based diagnostic information.

In asynchronous systems like Tokio, interpreting traditional log messages can
often be quite challenging. Since individual tasks are multiplexed on the same
thread, associated events and log lines are intermixed making it difficult to
trace the logic flow. `tokio-trace` expands upon logging-style diagnostics by
allowing libraries and applications to record structured events with additional
information about *temporality* and *causality* — unlike a log message, a span
in `tokio-trace` has a beginning and end time, may be entered and exited by the
flow of execution, and may exist within a nested tree of similar spans. In
addition, `tokio-trace` spans are *structured*, with the ability to record typed
data as well as textual messages.

The `tokio-trace` crate provides the APIs necessary for instrumenting libraries
and applications to emit trace data.

## Usage

First, add this to your `Cargo.toml`:

```toml
[dependencies]
tokio-trace = { git = "https://github.com/tokio-rs/tokio" }
```

Next, add this to your crate:

```rust
#[macro_use]
extern crate tokio_trace;
```

This crate provides macros for creating `Span`s and `Event`s, which represent
periods of time and momentary events within the execution of a program,
respectively.

As a rule of thumb, _spans_ should be used to represent discrete units of work
(e.g., a given request's lifetime in a server) or periods of time spent in a
given context (e.g., time spent interacting with an instance of an external
system, such as a database). In contrast, _events_ should be used to represent
points in time within a span — a request returned with a given status code,
_n_ new items were taken from a queue, and so on.

`Span`s are constructed using the `span!` macro, and then _entered_
to indicate that some code takes place within the context of that `Span`:

```rust
// Construct a new span named "my span".
let mut span = span!("my span");
span.enter(|| {
// Any trace events in this closure or code called by it will occur within
// the span.
});
// Dropping the span will close it, indicating that it has ended.
```

The `Event` type represent an event that occurs instantaneously, and is
essentially a `Span` that cannot be entered. They are created using the `event!`
macro:

```rust
use tokio_trace::Level;
event!(Level::INFO, "something has happened!");
```

Users of the [`log`] crate should note that `tokio-trace` exposes a set of macros for
creating `Event`s (`trace!`, `debug!`, `info!`, `warn!`, and `error!`) which may
be invoked with the same syntax as the similarly-named macros from the `log`
crate. Often, the process of converting a project to use `tokio-trace` can begin
with a simple drop-in replacement.

Let's consider the `log` crate's yak-shaving example:

```rust
#[macro_use]
extern crate tokio_trace;
use tokio_trace::field;

pub fn shave_the_yak(yak: &mut Yak) {
// Create a new span for this invocation of `shave_the_yak`, annotated
// with the yak being shaved as a *field* on the span.
span!("shave_the_yak", yak = field::debug(&yak)).enter(|| {
// Since the span is annotated with the yak, it is part of the context
// for everything happening inside the span. Therefore, we don't need
// to add it to the message for this event, as the `log` crate does.
info!(target: "yak_events", "Commencing yak shaving");

loop {
match find_a_razor() {
Ok(razor) => {
// We can add the razor as a field rather than formatting it
// as part of the message, allowing subscribers to consume it
// in a more structured manner:
info!({ razor = field::display(razor) }, "Razor located");
yak.shave(razor);
break;
}
Err(err) => {
// However, we can also create events with formatted messages,
// just as we would for log records.
warn!("Unable to locate a razor: {}, retrying", err);
}
}
}
})
}
```

You can find examples showing how to use this crate in the examples directory.

### In libraries

Libraries should link only to the `tokio-trace` crate, and use the provided
macros to record whatever information will be useful to downstream consumers.

### In executables

In order to record trace events, executables have to use a `Subscriber`
implementation compatible with `tokio-trace`. A `Subscriber` implements a way of
collecting trace data, such as by logging it to standard output.

Unlike the `log` crate, `tokio-trace` does *not* use a global `Subscriber` which
is initialized once. Instead, it follows the `tokio` pattern of executing code
in a context. For example:

```rust
#[macro_use]
extern crate tokio_trace;

let my_subscriber = FooSubscriber::new();

tokio_trace::subscriber::with_default(subscriber, || {
// Any trace events generated in this closure or by functions it calls
// will be collected by `my_subscriber`.
})
```

This approach allows trace data to be collected by multiple subscribers within
different contexts in the program. Alternatively, a single subscriber may be
constructed by the `main` function and all subsequent code executed with that
subscriber as the default. Any trace events generated outside the context of a
subscriber will not be collected.

The executable itself may use the `tokio-trace` crate to instrument itself as
well.

The [`tokio-trace-nursery`] repository contains less stable crates designed to
be used with the `tokio-trace` ecosystem. It includes a collection of
`Subscriber` implementations, as well as utility and adapter crates.

[`log`]: https://docs.rs/log/0.4.6/log/

## License

Expand All @@ -76,5 +166,3 @@ This project is licensed under the [MIT license](LICENSE).
Unless you explicitly state otherwise, any contribution intentionally submitted
for inclusion in Tokio by you, shall be licensed as MIT, without any additional
terms or conditions.

[`tokio-trace`]: https://github.com/tokio-rs/tokio/tree/master/tokio-trace
44 changes: 44 additions & 0 deletions benches/no_subscriber.rs
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
#![feature(test)]
#[macro_use]
extern crate tokio_trace;
#[macro_use]
extern crate log;
extern crate test;
use test::Bencher;

#[bench]
fn bench_span_no_subscriber(b: &mut Bencher) {
b.iter(|| {
span!("span");
});
}

#[bench]
fn bench_log_no_logger(b: &mut Bencher) {
b.iter(|| {
log!(log::Level::Info, "log");
});
}

#[bench]
fn bench_costly_field_no_subscriber(b: &mut Bencher) {
b.iter(|| {
span!(
"span",
foo = tokio_trace::field::display(format!("bar {:?}", 2))
);
});
}

#[bench]
fn bench_no_span_no_subscriber(b: &mut Bencher) {
b.iter(|| {});
}

#[bench]
fn bench_1_atomic_load(b: &mut Bencher) {
// This is just included as a baseline.
use std::sync::atomic::{AtomicUsize, Ordering};
let foo = AtomicUsize::new(1);
b.iter(|| foo.load(Ordering::Relaxed));
}
Loading

0 comments on commit 03d6e80

Please sign in to comment.