Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using WithSubscriber with a different registry causes a panic #898

Closed
jeromegn opened this issue Aug 7, 2020 · 3 comments
Closed

Using WithSubscriber with a different registry causes a panic #898

jeromegn opened this issue Aug 7, 2020 · 3 comments
Labels
crate/subscriber Related to the `tracing-subscriber` crate kind/bug Something isn't working

Comments

@jeromegn
Copy link
Contributor

jeromegn commented Aug 7, 2020

Bug Report

Version

├── tracing v0.1.18 (*)
├── tracing-core v0.1.13 (*)
├── tracing-futures v0.2.4
│   └── tracing v0.1.18 (*)
├── tracing-log v0.1.1
│   └── tracing-core v0.1.13 (*)
├── tracing-subscriber v0.2.10
│   ├── tracing-core v0.1.13 (*)
│   ├── tracing-log v0.1.1 (*)
│   └── tracing-serde v0.1.1
│       └── tracing-core v0.1.13 (*)

Platform

Linux 5.7.12-arch1-1

Description

Started using the WithSubscriber trait to optionally use different layers for a specific future. I did this to remove all filters for spans and events inside the future.

my_future
    .with_subscriber(
        tracing_subscriber::registry::Registry::default()
        .with(crate::trace::debug_layer::DebugLayer)
    )

It mostly appeared to do what I wanted, but I got the following:

Panic backtrace 
thread 'tokio-runtime-worker' panicked at 'tried to drop a ref to Id(2251799813685249), but no such span exists!', /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:245:21
stack backtrace:
   0: backtrace::backtrace::libunwind::trace
             at /cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1: backtrace::backtrace::trace_unsynchronized
             at /cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2: std::sys_common::backtrace::_print_fmt
             at src/libstd/sys_common/backtrace.rs:78
   3: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
             at src/libstd/sys_common/backtrace.rs:59
   4: core::fmt::write
             at src/libcore/fmt/mod.rs:1076
   5: std::io::Write::write_fmt
             at src/libstd/io/mod.rs:1537
   6: std::sys_common::backtrace::_print
             at src/libstd/sys_common/backtrace.rs:62
   7: std::sys_common::backtrace::print
             at src/libstd/sys_common/backtrace.rs:49
   8: std::panicking::default_hook::{{closure}}
             at src/libstd/panicking.rs:198
   9: std::panicking::default_hook
             at src/libstd/panicking.rs:218
  10: std::panicking::rust_panic_with_hook
             at src/libstd/panicking.rs:486
  11: rust_begin_unwind
             at src/libstd/panicking.rs:388
  12: std::panicking::begin_panic_fmt
             at src/libstd/panicking.rs:342
  13: <tracing_subscriber::registry::sharded::Registry as tracing_core::subscriber::Subscriber>::try_close
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:245
  14: <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::try_close
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:684
  15: tracing_core::dispatcher::Dispatch::try_close
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-core-0.1.13/src/dispatcher.rs:603
  16: <tracing_subscriber::registry::sharded::Registry as tracing_core::subscriber::Subscriber>::exit::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:208
  17: tracing_core::dispatcher::get_default::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-core-0.1.13/src/dispatcher.rs:352
  18: std::thread::local::LocalKey<T>::try_with
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  19: tracing_core::dispatcher::get_default
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-core-0.1.13/src/dispatcher.rs:339
  20: <tracing_subscriber::registry::sharded::Registry as tracing_core::subscriber::Subscriber>::exit
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:208
  21: <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::exit
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:660
  22: <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::exit
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:660
  23: <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::exit
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:660
  24: tracing_core::dispatcher::Dispatch::exit
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-core-0.1.13/src/dispatcher.rs:533
  25: <tracing::span::Entered as core::ops::drop::Drop>::drop
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-0.1.18/src/span.rs:1199
  26: core::ptr::drop_in_place
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  27: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-futures-0.2.4/src/lib.rs:259
  28: <tracing_futures::WithDispatch<T> as core::future::future::Future>::poll::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-futures-0.2.4/src/lib.rs:430
  29: tracing_core::dispatcher::with_default
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-core-0.1.13/src/dispatcher.rs:230
  30: <tracing_futures::WithDispatch<T> as core::future::future::Future>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-futures-0.2.4/src/lib.rs:430
  31: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/future/future.rs:118
  32: hyper::proto::h2::server::H2Stream<F,B>::poll2
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/hyper-0.13.7/src/proto/h2/server.rs:375
  33: <hyper::proto::h2::server::H2Stream<F,B> as core::future::future::Future>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/hyper-0.13.7/src/proto/h2/server.rs:437
  34: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
  35: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
  36: tokio::runtime::task::core::Core<T,S>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
  37: tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
  38: core::ops::function::FnOnce::call_once
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232
  39: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:318
  40: std::panicking::try::do_call
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
  41: __rust_try
  42: std::panicking::try
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
  43: std::panic::catch_unwind
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
  44: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
  45: tokio::runtime::task::raw::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
  46: tokio::runtime::task::raw::RawTask::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
  47: tokio::runtime::task::Notified<S>::run
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
  48: tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:374
  49: tokio::coop::with_budget::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:127
  50: std::thread::local::LocalKey<T>::try_with
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  51: std::thread::local::LocalKey<T>::with
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  52: tokio::coop::with_budget
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:120
  53: tokio::coop::budget
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:96
  54: tokio::runtime::thread_pool::worker::Context::run_task
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:352
  55: tokio::runtime::thread_pool::worker::Context::run
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:324
  56: tokio::runtime::thread_pool::worker::run::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:309
  57: tokio::macros::scoped_tls::ScopedKey<T>::set
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/macros/scoped_tls.rs:63
  58: tokio::runtime::thread_pool::worker::run
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:306
  59: tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:285
  60: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/task.rs:41
  61: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
  62: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
  63: tokio::runtime::task::core::Core<T,S>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
  64: tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
  65: core::ops::function::FnOnce::call_once
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232
  66: <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:318
  67: std::panicking::try::do_call
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
  68: __rust_try
  69: std::panicking::try
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
  70: std::panic::catch_unwind
             at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
  71: tokio::runtime::task::harness::Harness<T,S>::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
  72: tokio::runtime::task::raw::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
  73: tokio::runtime::task::raw::RawTask::poll
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
  74: tokio::runtime::task::Notified<S>::run
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
  75: tokio::runtime::blocking::pool::Inner::run
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:230
  76: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:210
  77: tokio::runtime::context::enter
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/context.rs:72
  78: tokio::runtime::handle::Handle::enter
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/handle.rs:76
  79: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}
             at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:20

Here's the code for my layer:

DebugLayer 
use tracing_core::span::{Attributes, Id};
use tracing_core::{Event, Subscriber};

use tracing_subscriber::layer::Context;
use tracing_subscriber::registry::{LookupSpan, Registry};
use tracing_subscriber::Layer;

use std::iter::Extend;

/// The debug tracing layer.
#[derive(Debug)]
pub struct DebugLayer;

impl<S> Layer<S> for DebugLayer
where
    S: Subscriber + for<'a> LookupSpan<'a>,
{
    fn new_span(&self, _attrs: &Attributes<'_>, id: &Id, cx: Context<'_, S>) {
        let span = cx.span(id).unwrap();
        if let Some(parent) = span.parent() {
            if parent.extensions().get::<DebugSpan>().is_some() {
                let meta = span.metadata();
                let id = span.id();
                let debug_span = DebugSpan::new(id, meta.name(), meta.target());
                span.extensions_mut().insert(debug_span);
                println!("inserted debug span for {}", meta.name());
            }
        }
    }

    fn on_exit(&self, id: &Id, cx: Context<'_, S>) {
        println!("on exit getting span... {:?}", id);
        let span = cx.span(id).unwrap();

        println!("after getting span");

        // Don't fold the root debug_span into its parent.
        if span.extensions().get::<DebugRoot>().is_some() {
            return;
        };
        println!("checked DebugRoot extension");

        // let name = span.metadata().name();
        let debug_span = match span.extensions_mut().remove::<DebugSpan>() {
            Some(debug_span) => debug_span,
            None => return,
        };

        println!("getting parent span");
        if let Some(parent_id) = span.parent_id() {
            let parent = cx.span(parent_id).expect("parent not found");
            if let Some(parent_debug_span) = parent.extensions_mut().get_mut::<DebugSpan>() {
                parent_debug_span.children.extend(debug_span.flatten());
                println!("flattened children")
            };
        }
    }

    fn on_event(&self, event: &Event, cx: Context<'_, S>) {
        println!("potential event {:?}", event);
        let span = if let Some(parent_id) = event.parent() {
            cx.span(parent_id)
        } else if event.is_contextual() {
            if let Some(parent_id) = cx.current_span().id() {
                println!("got an id! {:?}", parent_id);
                cx.span(parent_id)
            } else {
                return;
            }
        } else {
            return;
        };

        println!("maybe got a span");

        if let Some(span) = span {
            println!("EVENT {:?}", event);

            if let Some(debug_span) = span.extensions_mut().get_mut::<DebugSpan>() {
                println!("got debug span!");
                debug_span.events.push(format!("{:?}", event));
            };
        }
    }
}

/// Indicated the current struct is the root struct.
#[derive(Debug)]
pub struct DebugRoot;

#[derive(Debug)]
pub struct DebugSpan {
    pub target: &'static str,
    pub id: Id,
    pub span_name: &'static str,
    events: Vec<String>,
    children: Vec<Self>,
}

impl DebugSpan {
    fn new(id: Id, span_name: &'static str, target: &'static str) -> Self {
        Self {
            id,
            span_name,
            target,
            events: vec![],
            children: vec![],
        }
    }

    pub fn flatten(mut self) -> Vec<Self> {
        let mut children = vec![];
        std::mem::swap(&mut self.children, &mut children);
        children.push(self);
        children
    }
}

/// A trait ext to cast a Span down to a Registry.
pub trait DebugSpanExt {
    fn with_debug_events(self) -> Self;
    fn get_debug_events(&self) -> Vec<String>;
}

impl DebugSpanExt for tracing::Span {
    fn with_debug_events(self) -> Self {
        self.with_subscriber(|(id, subscriber)| {
            // if let Some(_) = subscriber.downcast_ref::<Metrics>() {
            if let Some(registry) = subscriber.downcast_ref::<Registry>() {
                let span = registry
                    .span(id)
                    .expect("in new_span but span does not exist");
                let meta = span.metadata();
                let data = DebugSpan::new(span.id(), meta.name(), meta.target());
                span.extensions_mut().insert(data);
                span.extensions_mut().insert(DebugRoot);
            }
            // }
        });
        self
    }
    fn get_debug_events(&self) -> Vec<String> {
        self.with_subscriber(|(id, subscriber)| {
            // if let Some(_) = subscriber.downcast_ref::<Metrics>() {
            let registry = subscriber
                .downcast_ref::<Registry>()
                .expect("could not downcast to registry");

            let span = registry
                .span(id)
                .expect("in new_span but span does not exist");
            let exts = span.extensions();

            let debug_span = exts.get::<DebugSpan>().unwrap();
            let mut events = debug_span.events.clone();

            for child in debug_span.children.iter() {
                events.extend(child.events.clone());
            }
            events

            // }
        })
        .unwrap()
    }
}

I then tried a simpler case:

my_future
    .with_subscriber(tracing_subscriber::registry::Registry::default())

and this gave me a different panic (which killed my process and core dumped), maybe more useful.

Reduced example panic 
thread 'tokio-runtime-worker' panicked at 'if a slot can be accessed at the current generation, its value must be `Some`', /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/lib.rs:461:13
stack backtrace:
   0:     0x55be8c1774b5 - backtrace::backtrace::libunwind::trace::h396c07d2071b43af
                               at /cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/libunwind.rs:86
   1:     0x55be8c1774b5 - backtrace::backtrace::trace_unsynchronized::h7aa0e4bb23d9c158
                               at /cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/backtrace-0.3.46/src/backtrace/mod.rs:66
   2:     0x55be8c1774b5 - std::sys_common::backtrace::_print_fmt::hd15ac5d4adcd355b
                               at src/libstd/sys_common/backtrace.rs:78
   3:     0x55be8c1774b5 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hec5354be8ccc3ecc
                               at src/libstd/sys_common/backtrace.rs:59
   4:     0x55be8c19fd9c - core::fmt::write::h3d34909eeb4f225b
                               at src/libcore/fmt/mod.rs:1076
   5:     0x55be8c16fc23 - std::io::Write::write_fmt::h1da287b3de55ed16
                               at src/libstd/io/mod.rs:1537
   6:     0x55be8c179d80 - std::sys_common::backtrace::_print::h4d206838e1ace354
                               at src/libstd/sys_common/backtrace.rs:62
   7:     0x55be8c179d80 - std::sys_common::backtrace::print::h1f778e9940ee5977
                               at src/libstd/sys_common/backtrace.rs:49
   8:     0x55be8c179d80 - std::panicking::default_hook::{{closure}}::h704403a56cbf5783
                               at src/libstd/panicking.rs:198
   9:     0x55be8c179acc - std::panicking::default_hook::ha4567a10dec4ef8d
                               at src/libstd/panicking.rs:218
  10:     0x55be8c17a3b7 - std::panicking::rust_panic_with_hook::h88a1f16ec8a7bb20
                               at src/libstd/panicking.rs:486
  11:     0x55be8c179fbb - rust_begin_unwind
                               at src/libstd/panicking.rs:388
  12:     0x55be8c19e451 - core::panicking::panic_fmt::hbddb7fe6f399b81a
                               at src/libcore/panicking.rs:101
  13:     0x55be8c19e1e3 - core::option::expect_failed::hfee7773bbf1a340a
                               at src/libcore/option.rs:1264
  14:     0x55be8b30497d - core::option::Option<T>::expect::h5cdbf4e73aeed9af
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/option.rs:349
  15:     0x55be8b33e055 - sharded_slab::Slab<T,C>::get::{{closure}}::hc1c556548b4051fe
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/lib.rs:461
  16:     0x55be8b325a8f - sharded_slab::page::slot::Slot<T,C>::get::ha78641bb1406baf3
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/page/slot.rs:152
  17:     0x55be8b325a8f - sharded_slab::page::Shared<T,C>::get::{{closure}}::h67254c76e2a9e964
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/page/mod.rs:189
  18:     0x55be8b3251ea - sharded_slab::sync::inner::UnsafeCell<T>::with::hec4321633d3e118d
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/sync.rs:32
  19:     0x55be8b3251ea - sharded_slab::page::Shared<T,C>::get::hc1bc54235a7fd0f7
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/page/mod.rs:188
  20:     0x55be8b33de6d - sharded_slab::shard::Shard<T,C>::get::h3c3d466f5d745409
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/shard.rs:64
  21:     0x55be8b33de6d - sharded_slab::Slab<T,C>::get::hecdb7240d80d2238
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/sharded-slab-0.0.9/src/lib.rs:460
  22:     0x55be8b312e72 - tracing_subscriber::registry::sharded::Registry::get::ha632865a4188b6e0
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:129
  23:     0x55be8b313780 - <tracing_subscriber::registry::sharded::Registry as tracing_core::subscriber::Subscriber>::try_close::h4481f9b20f55cbfa
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:242
  24:     0x55be8a780f4e - <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::try_close::h47c939807c843749
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:684
  25:     0x55be8a780daa - <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::try_close::h3ea748d52446c5c9
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:684
  26:     0x55be8a780c08 - <tracing_subscriber::layer::Layered<L,S> as tracing_core::subscriber::Subscriber>::try_close::h3a2f6b50d2c4c622
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/layer.rs:684
  27:     0x55be8b2ef535 - tracing_core::dispatcher::Dispatch::try_close::h1eb61b290428baef
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-core-0.1.13/src/dispatcher.rs:603
  28:     0x55be8b313c69 - <tracing_subscriber::registry::sharded::DataInner as core::ops::drop::Drop>::drop::h6e090323060cd8bb
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tracing-subscriber-0.2.10/src/registry/sharded.rs:296
  29:     0x55be8b338ef6 - core::ptr::drop_in_place::hdea3bc96f1de1f9b
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  30:     0x55be8b3368d6 - core::ptr::drop_in_place::h41a947bfc88065ea
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  31:     0x55be8b3386bf - core::ptr::drop_in_place::hb281531e7a116e98
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  32:     0x55be8b338cbf - core::ptr::drop_in_place::hd21e991add5dc02c
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  33:     0x55be8b338736 - core::ptr::drop_in_place::hb92640aa0f4c5a14
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  34:     0x55be8b337a9e - core::ptr::drop_in_place::h82aedae780dc5350
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  35:     0x55be8b336310 - core::ptr::drop_in_place::h209a5d3bcef85f50
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  36:     0x55be8b3389d6 - core::ptr::drop_in_place::hc10938bbf18863ec
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  37:     0x55be8b3394af - core::ptr::drop_in_place::hf9c59fdb9b9467cd
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  38:     0x55be8b3381af - core::ptr::drop_in_place::h986b507a993b1792
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  39:     0x55be8b338416 - core::ptr::drop_in_place::ha148e8c0a771773f
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  40:     0x55be8b338ba9 - core::ptr::drop_in_place::hcd25ffa7282f9437
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  41:     0x55be8b336260 - core::ptr::drop_in_place::h1c3f5d1b003f8320
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  42:     0x55be8b337e4b - core::ptr::drop_in_place::h8902f0b7c7e1c48e
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  43:     0x55be8b3382b9 - core::ptr::drop_in_place::h9a1669961589d51f
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  44:     0x55be8b337470 - core::ptr::drop_in_place::h6436e0e52aca019f
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  45:     0x55be8b335cef - core::ptr::drop_in_place::h00467f0021c96153
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  46:     0x55be8b336fdf - core::ptr::drop_in_place::h58cda6ba935d1ba9
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  47:     0x55be8c1573e0 - core::ptr::drop_in_place::h3e7920fed9e7b5b8
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  48:     0x55be8c1606ee - alloc::sync::Arc<T>::drop_slow::h58a7e0747146b87d
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/sync.rs:868
  49:     0x55be8c160a2c - <alloc::sync::Arc<T> as core::ops::drop::Drop>::drop::hd3800bbfedf2aa64
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/liballoc/sync.rs:1382
  50:     0x55be8c15790f - core::ptr::drop_in_place::hf6178f853565c940
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  51:     0x55be8c15743f - core::ptr::drop_in_place::h447ee41939a124e7
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  52:     0x55be8a7df8ed - core::ptr::drop_in_place::h296588f7a8d64256
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  53:     0x55be8a8007dc - core::ptr::drop_in_place::hf141952e1d00793e
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  54:     0x55be8a7e78ee - core::ptr::drop_in_place::h59b2934ab5300c67
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  55:     0x55be8a7f355c - core::ptr::drop_in_place::h9f73a283471d1c41
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  56:     0x55be8a7f184b - core::ptr::drop_in_place::h9739aa3596f0077a
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  57:     0x55be8a7d97b2 - core::ptr::drop_in_place::h092385240739cd16
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  58:     0x55be8a521512 - tokio::runtime::task::core::Core<T,S>::drop_future_or_output::{{closure}}::h489855b8a6d1403a
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:192
  59:     0x55be8a7580e6 - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::hd6b1f48c90c37b2d
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
  60:     0x55be8a5200d5 - tokio::runtime::task::core::Core<T,S>::drop_future_or_output::h2287e56c9768d4e8
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:190
  61:     0x55be8a7d432d - <tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::Guard<T,S> as core::ops::drop::Drop>::drop::hb21701d6b00ecaff
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:96
  62:     0x55be8a7e832e - core::ptr::drop_in_place::h5e144eb1bb4a2c62
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ptr/mod.rs:184
  63:     0x55be8aa54f28 - tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::h6e7e93b39e057370
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:114
  64:     0x55be8a7d5240 - core::ops::function::FnOnce::call_once::h3f457795262274f8
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232
  65:     0x55be8a6a083b - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h0983b657a21beec6
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:318
  66:     0x55be8a69846a - std::panicking::try::do_call::h7328c6df10c56887
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
  67:     0x55be8a6a076d - __rust_try
  68:     0x55be8a6915c5 - std::panicking::try::h7d2bc8f6ab64ce0c
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
  69:     0x55be8a6a23da - std::panic::catch_unwind::h3061ccb65240cf83
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
  70:     0x55be8aa488c9 - tokio::runtime::task::harness::Harness<T,S>::poll::hb63ffc7f5d9caf76
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
  71:     0x55be8a65cca0 - tokio::runtime::task::raw::poll::h85975d7285b8be5b
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
  72:     0x55be8bd427ff - tokio::runtime::task::raw::RawTask::poll::he8a99997d69f9d5f
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
  73:     0x55be8bd1c4f1 - tokio::runtime::task::Notified<S>::run::h65c90e3c2d7ce5ed
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
  74:     0x55be8bd2c360 - tokio::runtime::thread_pool::worker::Context::run_task::{{closure}}::h6d44a745cbe7323d
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:374
  75:     0x55be8bd0f2b6 - tokio::coop::with_budget::{{closure}}::h620d99ecbce31fbb
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:127
  76:     0x55be8bd326a4 - std::thread::local::LocalKey<T>::try_with::h2b785d6382b0898f
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:263
  77:     0x55be8bd3147e - std::thread::local::LocalKey<T>::with::h1224a2d8b9fe9e9f
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/local.rs:239
  78:     0x55be8bd2bedd - tokio::coop::with_budget::h9cb153673519ab22
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:120
  79:     0x55be8bd2bedd - tokio::coop::budget::hcd3fec7cfce6beca
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/coop.rs:96
  80:     0x55be8bd2bedd - tokio::runtime::thread_pool::worker::Context::run_task::hf4e63c2e1c4b22b5
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:352
  81:     0x55be8bd2b9ef - tokio::runtime::thread_pool::worker::Context::run::hb6786a9e8762edd9
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:331
  82:     0x55be8bd2b643 - tokio::runtime::thread_pool::worker::run::{{closure}}::hcf9c7b7df1dc04d6
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:309
  83:     0x55be8bd0e5c6 - tokio::macros::scoped_tls::ScopedKey<T>::set::h97ce0b68da9ba6dc
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/macros/scoped_tls.rs:63
  84:     0x55be8bd2b547 - tokio::runtime::thread_pool::worker::run::h733e2ff7f61930db
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:306
  85:     0x55be8bd2b3ab - tokio::runtime::thread_pool::worker::Launch::launch::{{closure}}::h16c7b3ecc471ffb4
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/thread_pool/worker.rs:285
  86:     0x55be8bcc6ade - <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll::hca0d19318520c42c
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/task.rs:41
  87:     0x55be8bcdf8ee - tokio::runtime::task::core::Core<T,S>::poll::{{closure}}::hb84836cb1fb898ae
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:173
  88:     0x55be8bcd9ffb - tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut::h3badfd367c217a74
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/loom/std/unsafe_cell.rs:14
  89:     0x55be8bcdf72e - tokio::runtime::task::core::Core<T,S>::poll::h997d8b2676a2db53
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/core.rs:158
  90:     0x55be8bd21775 - tokio::runtime::task::harness::Harness<T,S>::poll::{{closure}}::h9607eb2c3e771833
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:107
  91:     0x55be8bd01880 - core::ops::function::FnOnce::call_once::h488366f4e07e9dbb
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232
  92:     0x55be8bc95bcc - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h93dcd257ba2f28f6
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:318
  93:     0x55be8bd1abdc - std::panicking::try::do_call::h1dd23016ed992bd7
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
  94:     0x55be8bd2100d - __rust_try
  95:     0x55be8bd1aa92 - std::panicking::try::he899cf24e34a5cae
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
  96:     0x55be8bc9611b - std::panic::catch_unwind::h5bf8afea967dfb1e
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
  97:     0x55be8bd21296 - tokio::runtime::task::harness::Harness<T,S>::poll::h7611c1cb7ec5670d
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/harness.rs:89
  98:     0x55be8bd42952 - tokio::runtime::task::raw::poll::h5246be69e1c1d8fa
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:104
  99:     0x55be8bd427ff - tokio::runtime::task::raw::RawTask::poll::he8a99997d69f9d5f
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/raw.rs:66
 100:     0x55be8bd1c561 - tokio::runtime::task::Notified<S>::run::h741f61aa4d784df8
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/task/mod.rs:169
 101:     0x55be8bd43d83 - tokio::runtime::blocking::pool::Inner::run::hff66ef80f18b7df3
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:230
 102:     0x55be8bd43ace - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::{{closure}}::hee8f3a4c9cb5c016
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:210
 103:     0x55be8bcdcb03 - tokio::runtime::context::enter::hcff0e2a91113344f
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/context.rs:72
 104:     0x55be8bcddbed - tokio::runtime::handle::Handle::enter::hd72799c6bd6cccf7
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/handle.rs:76
 105:     0x55be8bd43b5f - tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}::h7b9722a77321dd54
                               at /home/jerome/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tokio-0.2.22/src/runtime/blocking/pool.rs:209
 106:     0x55be8bd008b0 - std::sys_common::backtrace::__rust_begin_short_backtrace::h3a30af436aa0ac05
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/sys_common/backtrace.rs:130
 107:     0x55be8bca7461 - std::thread::Builder::spawn_unchecked::{{closure}}::{{closure}}::h94deff0f0cd85dd0
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/mod.rs:475
 108:     0x55be8bc95ba3 - <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once::h2b34781014b7eec3
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:318
 109:     0x55be8bd1acf4 - std::panicking::try::do_call::hf68e32b2a1b98c1b
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:297
 110:     0x55be8bd2100d - __rust_try
 111:     0x55be8bd1a984 - std::panicking::try::he79e700600f50331
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panicking.rs:274
 112:     0x55be8bc96193 - std::panic::catch_unwind::hd808b585cd26f752
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/panic.rs:394
 113:     0x55be8bca7259 - std::thread::Builder::spawn_unchecked::{{closure}}::h12137f0e1d6500c6
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libstd/thread/mod.rs:474
 114:     0x55be8bd0160f - core::ops::function::FnOnce::call_once{{vtable.shim}}::hdde9320548e37e51
                               at /home/jerome/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/ops/function.rs:232
 115:     0x55be8c17e52a - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::hcf205bcf9b46c587
                               at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/liballoc/boxed.rs:1076
 116:     0x55be8c17e52a - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h2d53e2246128f5d8
                               at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2/src/liballoc/boxed.rs:1076
 117:     0x55be8c17e52a - std::sys::unix::thread::Thread::new::thread_start::h3b6d8a0cd87a87c6
                               at src/libstd/sys/unix/thread.rs:87
 118:     0x7f6659d40422 - start_thread
 119:     0x7f6659c55bf3 - __GI___clone
 120:                0x0 - <unknown>
thread panicked while panicking. aborting.
@hawkw
Copy link
Member

hawkw commented Aug 7, 2020

Oh, I know what's causing this. The registry currently tracks the per-thread stack of which spans have been entered in a thread-local:

tracing/tracing-subscriber/src/registry/stack.rs

Lines 64 to 66 in 63c0f31

thread_local! {
static CONTEXT: RefCell<SpanStack> = RefCell::new(SpanStack::new());
}

Unfortunately, this thread -ocal is shared across all instances of Registry...so when switching between Registry instances on the same thread, it may have thread IDs in it that were generated by a different registry. It's fine if multiple Registry instances are used for different threads, because the thread-local state is different...but it collapses if multiple registries are used on the same thread.

This issue hasn't shown up before because I don't think a lot of folks are doing what you're doing using Registry and WithSubscriber...but we should definitely fix this. I think the easiest solution is just swapping out the thread_local! macro for something like the thread_local crate which allows per-object thread-local storage. We should also add tests for this.

@hawkw hawkw added crate/subscriber Related to the `tracing-subscriber` crate kind/bug Something isn't working labels Aug 7, 2020
@jeromegn jeromegn mentioned this issue Aug 8, 2020
Merged
hawkw pushed a commit that referenced this issue Aug 10, 2020
@jeromegn
subscriber: use the thread_local crate to allow multiple registries (#…
59494e0 
…901)

## Motivation

Fixes part of #898 where creating multiple registries made them fight
over the thread_local-stored span stacks.

## Solution

Use the `thread_local` crate which has a tighter scope.

I believe @hawkw has a test that fails without this PR.
@hawkw
Copy link
Member

hawkw commented Aug 13, 2020

I believe this was fixed by #901, and the filtering issues with multiple subscribers that you ran into are tracked elsewhere (and have been fixed in some places). Is that correct?

@jeromegn
Copy link
Contributor Author

Yes 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
crate/subscriber Related to the `tracing-subscriber` crate kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jeromegn @hawkw