subscriber: High performance span store #157
Assignees
Labels
crate/subscriber
Related to the `tracing-subscriber` crate
kind/feature
New feature or request
kind/perf
Performance improvements
Comments
hawkw
added
crate/subscriber
This was referenced Jul 7, 2019
|
From a discussion on Gitter with @hawkw. The current front-runner for the global span store is:
|
|
The entire store might not need to be refcounted, as the store would live in a subscriber that is already refcounted by a Dispatcher. |
hawkw
added a commit
that referenced
this issue
Jul 17, 2019
* add `Layer` trait for composing subscribers ### Motivation In many cases, it is valuable to compose multiple consumers for trace events. However, certain aspects of `tracing`'s design --- in particular, that span IDs are assigned by the subscriber, and that a single subscriber collects events from multiple threads --- make it difficult to compose the `Subscriber` trait directly. ## Solution This branch introduces a new trait, `Layer`, in `tracing-subscriber`. `Layer` represents the composable subset of the `Subscriber` functionality --- it recieves notifications of spans and events, and can filter callsites, but it does not assign IDs or manage reference counts, instead being notified on span closure by the underlying subscriber. A `Layer` can thus be wrapped around another subscriber to add additional functionality. In addition, this branch adds functionality for composing layers, and a `filter` module that provides `Layer` implementations for simple filters. ## Future Work To have a complete story for multi-subscriber fanout, we will also want to implement a performant concurrent span store, as I described in #157. To better support the use of subscriber layers, may wish to consider having a "registry" subscriber that implements _only_ span storage, ID generation, and lifecycle management, for composing layers on top of. Finally, we should consider adding `Layer` implementations to other crates that currently only expose `Subscriber` impls. Refs: #135 Signed-off-by: Eliza Weisman <[email protected]> * subscriber: remove ref-counting from Layer (#149) ## Motivation As I mentioned in #136 (comment), the `Layer` trait proposed in #136 should probably _not_ be responsible for managing ref-counts and determining span closures. Instead, the root subscriber should be responsible for this, and `Layer`s should be able to opt into a callback that's notified _when_ a span is closed. ## Solution Adding a callback that's called on closure to `Layer` required modifying the `Subscriber` trait to allow indicating when a span closes. This branch attempts to do this without a breaking change, by adding a new `try_close` trait method to `Subscriber`. `try_close` is identical to `drop_span` except that it returns a boolean value, set to `true` if the span has closed. This function was added in #153. This branch updates `tracing-subscriber::Layer` to use `try_close`. Rather than having `clone_span`/`drop_span` callbacks on `Layer`, we now have a `close` function, which is called when the inner subscriber's `try_close` returns `true`. Signed-off-by: Eliza Weisman <[email protected]> * ensure a Context always refers to a Subscriber This reworks layer composition a bit, and fixes an issue where `Context`s were not actually guaranteed to wrap a `Subscriber`, and thus implemented no methods. Layers now guarantee that `S` implements `Subscriber`. Signed-off-by: Eliza Weisman <[email protected]>
Closed
|
For those of you following along at home, I am still working on this. I know it's been pretty silent, but I just published an experimental first release of a lock-free slab that I believe should offer the right performance characteristics to serve as the core for a span store abstraction. My plan is to replace the storage layer in |
davidbarsky
added a commit
that referenced
this issue
Nov 14, 2019
…Registry + Layers (#420) This branch introduces: - A registry build atop of https://github.com/hawkw/sharded-slab. Layers are expected to consume this registry through the traits `SpanData`, `LookupSpan`, and `LookupMetadata`. Layer-specific data, such as formatted spans and events, are stored in the `Extensions` typemap. Data is writable via the `ExtensionsMut` view struct of the typemap. This enables layers to read and write data that they are specifically interested in. - The `tracing_subscriber::fmt::Subscriber` has been re-implemented in terms of `tracing_subscriber::registry::Registry` and `tracing_subscriber::fmt::Layer`. - The event/field formatters have been modified (in a non-backwards compatible way) to accept a `tracing_subscriber::fmt::FmtContext`. A similar structure existed in `tracing_subscriber::fmt::Subscriber`, but it was not publicly exposed. Resolves #135 Resolves #157 Resolves #391 Signed-off-by: David Barsky <[email protected]> Coauthored-by: Eliza Weisman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Feature Request
Crates
tracing-subscriberMotivation
In many use cases,
Subscriberimplementations will need to store data for each span over the course of that span's lifetime. Since theSubscriberinterface is based on&selfreferences, and a single subscriber is shared across multiple threads, storing mutable per-span data requires some kind of locking.The naïve approach is to store spans in a
RwLock<HashMap<Id, PerSpanData>>or similar structure, but this is not ideal; introducing a global lock means that any access to the span store by the subscriber is a global synchronization point. Instead, it's almost certainly better to use more granular locking, so that modifying the data for unrelated spans on two different threads need not cause the whole program to synchronize.More granular approaches can be more complex to implement than using a simple global lock. So that subscriber implementations need not reinvent the wheel, the
tracing-subscribercrate ought to provide batteries-included structures for storing per-span data.Proposal
The tokio threadpool (and other work-stealing executors) try hard to ensure that individual tasks remain on the same worker thread, and that spans are often attached to a task, we should consider optimizing for a case where span accesses from the same thread are common but spans move across threads infrequently. Given this, a good approach might be one where the locking of span data is sharded by the thread where the span was created, in order to make access fast in the common case. When a span's data is used from another thread, we can "steal" the span from the thread where its data was previously used, a slower operation that's okay for the less common case.
Alternatives
There are other potential solutions to a fast span store. For example, tracing-fmt uses a slab where each entry (span) has its own lock. This may also be worth investigating.
The text was updated successfully, but these errors were encountered: