Stack overflow when using a client with a broken connection

[bmwill]

Since #187 landed I figured I would try to see if my simple hello world example would work on the latest master where I use the following client, connect to the helloworld-server, wait for a few RPCs then kill the server.

pub mod hello_world {
    tonic::include_proto!("helloworld");
}

use hello_world::{greeter_client::GreeterClient, HelloRequest};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut client = GreeterClient::connect("http://[::1]:50051").await?;

    loop {
        let request = tonic::Request::new(HelloRequest {
            name: "Tonic".into(),
        });

        let response = client.say_hello(request).await;

        println!("RESPONSE={:?}", response);
        tokio::time::delay_for(std::time::Duration::from_secs(5)).await;
    }

    Ok(())
}

Unexpectedly I instead am greeted by a stack overflow (instead of an error) when I kill the server:

cargo run --bin helloworld-client
    Finished dev [unoptimized + debuginfo] target(s) in 0.15s
     Running `/home/bmwill/w/tonic/target/debug/helloworld-client`
RESPONSE=Ok(Response { metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Sun, 15 Dec 2019 22:46:28 GMT", "grpc-status": "0"} }, message: HelloReply { message: "Hello Tonic!" } })
RESPONSE=Ok(Response { metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Sun, 15 Dec 2019 22:46:33 GMT", "grpc-status": "0"} }, message: HelloReply { message: "Hello Tonic!" } })

thread 'tokio-runtime-worker' has overflowed its stack
fatal runtime error: stack overflow
Aborted (core dumped)

Stack Trace

#20456 0x000055555578e974 in <C as tower_make::make_connection::MakeConnection<Target>>::poll_ready (self=0x555556109598, cx=0x7ffff45b6aa8) at /home/bmwill/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tower-make-0.3.0/src/make_connection.rs:41
#20457 0x000055555578d991 in <tonic::transport::service::connector::Connector<C> as tower_service::Service<http::uri::Uri>>::poll_ready (self=0x555556109598, cx=0x7ffff45b6aa8) at /home/bmwill/w/tonic/tonic/src/transport/service/connector.rs:56
#20458 0x000055555578e974 in <C as tower_make::make_connection::MakeConnection<Target>>::poll_ready (self=0x555556109598, cx=0x7ffff45b6aa8) at /home/bmwill/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/tower-make-0.3.0/src/make_connection.rs:41
#20459 0x000055555578d991 in <tonic::transport::service::connector::Connector<C> as tower_service::Service<http::uri::Uri>>::poll_ready (self=0x555556109598, cx=0x7ffff45b6aa8) at /home/bmwill/w/tonic/tonic/src/transport/service/connector.rs:56

From the stack trace it looks like the Service::poll_ready impl for Connector calls MakeConnection::poll_ready and MakeConnection::poll_ready calls Service::poll_ready resulting in an infinite loop.

I tried to bisect the stack overflow and it appears that
b90c340 (feat(transport): Allow custom IO and UDS example (#184), 2019-12-13) may be the culprit.