Panic when ssh session terminates #1532

noyez · 2022-06-20T14:02:28Z

Hi -- love your project, thank you! i'm trying to figure out the bug below. If you can point in a general direction in the code, i could fix. I am very familiar with rust.

Basic information

zellij --version: 0.30.0
tput lines: 62
tput cols: 242
uname -av or ver(Windows): Linux prometheus-vm 5.4.0-120-generic #136-Ubuntu SMP Fri Jun 10 13:40:48 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

List of programs you interact with as, PROGRAM --version:

No programs running just open zsh sessions.

Further information

SSH into home server, start with zellij --debug
enter the control sequence to close ssh session (~~.). This also happens if you put ssh to sleep via (~~^z) and kill session.
SSH session terminates, zellij-client exists (panics)
SSH back into server, and there are no zellij sessions open.
Reproducible, happens every time.

Specifics about my systems (not sure any info is useful)

Happens when originating from iterm2 or alacritty (on macos)
destination is a linux VM running inside a freebsd host using bhyve as vm host

zellij.log (using --debug)

INFO   |zellij_client            | 2022-06-20 13:50:43.479 [main      ] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:124]: Starting Zellij client!
INFO   |zellij_server            | 2022-06-20 13:50:43.490 [main      ] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-server-0.30.0/src/lib.rs:194]: Starting Zellij server!
INFO   |zellij_server::wasm_vm   | 2022-06-20 13:50:43.547 [wasm      ] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-server-0.30.0/src/wasm_vm.rs:87]: Wasm main thread starts
DEBUG  |status-bar               | 2022-06-20 13:50:43.677 [id: 1     ] quicknav_show_count: 18
DEBUG  |status-bar               | 2022-06-20 13:50:43.682 [id: 1     ] quicknav_show_count: 19
ERROR  |zellij_client            | 2022-06-20 13:50:49.767 [stdin_handler] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: "/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }
INFO   |zellij_server::wasm_vm   | 2022-06-20 13:50:49.776 [wasm      ] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-server-0.30.0/src/wasm_vm.rs:213]: wasm main thread exits

The text was updated successfully, but these errors were encountered:

a-kenji · 2022-06-20T15:08:57Z

Hey @noyez,
thank you for reaching out.
Does this also happen, if you run without plugins?

noyez · 2022-06-20T15:24:36Z

my apologies, but how does one run without plugins? i started with an empty config. and the resulting log is below.

ERROR  |zellij_client            | 2022-06-20 15:22:57.275 [stdin_handler] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: "/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }
INFO   |zellij_server::wasm_vm   | 2022-06-20 15:22:57.293 [wasm      ] [/home/brad/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-server-0.30.0/src/wasm_vm.rs:213]: wasm main thread exits

a-kenji · 2022-06-20T16:00:01Z

With:

 zellij setup --dump-layout default

You can look at the default layout that is loaded.
You can load an empty layout by specifying an empty file:

echo '---' > empty.yaml
zellij --layout ./empty.yaml

noyez · 2022-06-20T16:52:41Z

Still got crash, this time i compiled it from source to generate a debug binary. I used the empty layout.

 rust/zellij  » echo '---' > empty.yaml 
 rust/zellij  » target/debug/zellij --debug --layout ./empty.yaml

INFO   |zellij_client            | 2022-06-20 12:41:39.345 [main      ] [zellij-client/src/lib.rs:124]: Starting Zellij client!
INFO   |zellij_server            | 2022-06-20 12:41:39.351 [main      ] [zellij-server/src/lib.rs:194]: Starting Zellij server!
INFO   |zellij_server::wasm_vm   | 2022-06-20 12:41:39.406 [wasm      ] [zellij-server/src/wasm_vm.rs:87]: Wasm main thread starts
ERROR  |zellij_client            | 2022-06-20 12:41:41.994 [stdin_handler] [zellij-client/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: "zellij-client/src/os_input_output.rs", line: 121, col: 39 } }
INFO   |zellij_client            | 2022-06-20 12:41:41.994 [main      ] [zellij-client/src/lib.rs:369]: Bye from Zellij!
ERROR  |zellij_client            | 2022-06-20 12:41:41.994 [main      ] [zellij-client/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: EIO), location: Location { file: "zellij-client/src/lib.rs", line: 370, col: 32 } }

a-kenji · 2022-06-20T18:25:12Z

Thank you for checking!

tlinford · 2022-06-20T19:22:09Z

@noyez it would be great if you want to take this on!

Did a little digging, the server is closing because when the client panics, this:

zellij/zellij-client/src/lib.rs

Lines 205 to 215 in 29332ca

    
           std::panic::set_hook({ 
        
               use zellij_utils::errors::handle_panic; 
        
               let send_client_instructions = send_client_instructions.clone(); 
        
               let os_input = os_input.clone(); 
        
               Box::new(move |info| { 
        
                   error!("Panic occurred in client:\n{:?}", info); 
        
                   if let Ok(()) = os_input.unset_raw_mode(0) { 
        
                       handle_panic(info, &send_client_instructions); 
        
                   } 
        
               }) 
        
           });

is causing this code:

zellij/zellij-utils/src/errors.rs

Line 98 in 29332ca

let _ = sender.send(T::error(fmt_report(report)));

to send a ClientInstruction::Error, that then gets turned into a ClientToServerMsg::Action(Action::Quit, None) and makes the server exit.

I guess when the ssh connection closes and the read to stdin fails, the client should simply not panic and shutdown gracefully?

@a-kenji maybe this needs to take into account the on_force_close option?

a-kenji · 2022-06-20T19:54:55Z

@tlinford,
Thanks for checking!

maybe this needs to take into account the on_force_close option?

I think that would be a good solution.

noyez · 2022-06-20T23:52:38Z

@noyez it would be great if you want to take this on!

Thanks for the guidance. i'll give it a go over the next few days.

noyez · 2022-06-30T13:08:43Z

It turns out this is not as reproducible as i hoped. when i reported it, my sessions were always lost when the ssh connection terminates every time. My VM rebooted at some point and now, It still happens from time to time, but every time. There must be some other condition to trigger this. i'll keep on trying to reproduce it. It surprised me that no one else has reported this happening, so perhaps its somehow unique to my ssh connections or tasks. I'm going to keep digging.

ahcm · 2022-07-12T11:33:38Z

I assume, I experience a similar problem:

DEBUG  |status-bar               | 2022-07-11 09:22:49.372 [id: 7     ] quicknav_show_count: 15
DEBUG  |status-bar               | 2022-07-11 09:22:49.372 [id: 7     ] quicknav_show_count: 15
ERROR  |zellij_client            | 2022-07-11 10:08:51.843 [stdin_handler] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }
ERROR  |zellij_client            | 2022-07-11 10:08:51.843 [main      ] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(cannot write to stdout: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs", line:341, col: 22 }, can_unwind: true }
ERROR  |zellij_client            | 2022-07-11 10:08:51.845 [router    ] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: "SendError(..)"), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs", line: 282, col: 75 }, can_unwind: true }ERROR  |zellij_client            | 2022-07-11 10:08:51.843 [stdin_handler] [/home/andy/.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }
ERROR  |zellij_client            | 2022-07-11 10:08:51.843 [main      ] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(cannot write to stdout: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs", line:341, col: 22 }, can_unwind: true }
ERROR  |zellij_client            | 2022-07-11 10:08:51.845 [router    ] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: "SendError(..)"), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs", line: 282, col: 75 }, can_unwind: true }

ahcm · 2022-07-14T17:13:23Z

I'm not sure this is a new one. Zellij became stuck, not responding to input.
Quitting ssh with with ~. attaching or ls was hanging:

INFO   |zellij_client            | 2022-07-14 17:03:45.175 [main      ] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:124]: Starting Zellij client!
DEBUG  |status-bar               | 2022-07-14 17:03:45.176 [id: 1     ] quicknav_show_count: 22
ERROR  |zellij_client            | 2022-07-14 17:09:11.136 [stdin_handler] [.../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/lib.rs:205]: Panic occured in client:
PanicInfo { payload: Any { .. }, message: Some(called `Result::unwrap()` on an `Err` value: Os { code: 5, kind: Uncategorized, message: "Input/output error" }), location: Location { file: ".../.cargo/registry/src/jackfan.us.kg-1ecc6299db9ec823/zellij-client-0.30.0/src/os_input_output.rs", line: 121, col: 39 }, can_unwind: true }

00sapo · 2022-09-09T13:56:30Z

Same thing here. SSH disconnected and once connected again zellij is sleeping and all processes in it too. Zellij doesn't respond anymore to any command (I tried: zellij list-sessions and zellij attach).

Here is the last part of the log:

[...]
WARN   |zellij_server::panes::ter| 2022-09-09 14:21:43.753 [screen    ] [zellij-server/src/panes/terminal_character.rs:396]: unhandled csi m code [58, 2, 0, 224, 175, 104] 
WARN   |zellij_server::panes::ter| 2022-09-09 14:21:43.753 [screen    ] [zellij-server/src/panes/terminal_character.rs:396]: unhandled csi m code [4, 3] 
WARN   |zellij_server::panes::ter| 2022-09-09 14:21:43.753 [screen    ] [zellij-server/src/panes/terminal_character.rs:396]: unhandled csi m code [58, 2, 0, 224, 175, 104] 
WARN   |zellij_utils::ipc        | 2022-09-09 15:53:09.490 [server_router] [zellij-utils/src/ipc.rs:202]: Error in IpcReceiver.recv(): InvalidMarkerRead(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }) 
ERROR  |zellij_server::route     | 2022-09-09 15:53:09.490 [server_router] [zellij-server/src/route.rs:585]: Received empty message from client 
ERROR  |zellij_utils::errors     | 2022-09-09 15:53:09.490 [main      ] [zellij-utils/src/errors.rs:94]: Panic occured:
             thread: main
             location: At /build/zellij/src/zellij-0.31.1/zellij-utils/src/ipc.rs:161:69
             message: called `Result::unwrap()` on an `Err` value: InvalidValueWrite(InvalidDataWrite(Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }))

raphCode · 2022-09-13T16:17:55Z

I think the error in the post above me was fixed in v0.31.4:
#1710

Not sure about the hangs or other server exits.

raphCode · 2022-09-13T16:37:36Z

Did a little digging

Thanks @tlinford!

to send a ClientInstruction::Error, that then gets turned into a ClientToServerMsg::Action(Action::Quit, None) and makes the server exit.

This happens here:

zellij/zellij-client/src/lib.rs

Line 359 in 31d7412

let _ = os_input.send_to_server(ClientToServerMsg::Action(Action::Quit, None));

Why do we do this? Shouldn't we leave the server alone when the client crashes?

I traced this line via git blame to this commit:
b8acf19#diff-d018c36dd0b3b1d46b1bcfed749b3f9f09c4d51e79ca63c18f01075172b0fef2R228

I am not familiar with the code at this old revision, but to me it looks like Action::Quit was added in error when refactoring the code.

ahcm · 2022-09-22T08:44:39Z

Making a release for this would be really appreciated.
Thanks!

noyez added the suspected bug label Jun 20, 2022

a-kenji added the stability Issues in relation to stability label Jun 20, 2022

raphCode mentioned this issue Jun 28, 2022

Tracking issue for server stability (old 1) #1100

Closed

18 tasks

raphCode mentioned this issue Jul 7, 2022

Tracking issue for server stability #1568

Open

16 tasks

raphCode mentioned this issue Sep 14, 2022

Don't send the server an exit signal when client crashes #1731

Merged

raphCode closed this as completed in #1731 Sep 14, 2022

raphCode mentioned this issue Sep 27, 2022

I'm not seeing zellij detach on a SIGHUP #1747

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Panic when ssh session terminates #1532

Panic when ssh session terminates #1532

noyez commented Jun 20, 2022

a-kenji commented Jun 20, 2022

noyez commented Jun 20, 2022

a-kenji commented Jun 20, 2022

noyez commented Jun 20, 2022

a-kenji commented Jun 20, 2022

tlinford commented Jun 20, 2022

a-kenji commented Jun 20, 2022

noyez commented Jun 20, 2022 •

edited

Loading

noyez commented Jun 30, 2022

ahcm commented Jul 12, 2022

ahcm commented Jul 14, 2022 •

edited

Loading

00sapo commented Sep 9, 2022 •

edited

Loading

raphCode commented Sep 13, 2022

raphCode commented Sep 13, 2022

ahcm commented Sep 22, 2022

Panic when ssh session terminates #1532

Panic when ssh session terminates #1532

Comments

noyez commented Jun 20, 2022

a-kenji commented Jun 20, 2022

noyez commented Jun 20, 2022

a-kenji commented Jun 20, 2022

noyez commented Jun 20, 2022

a-kenji commented Jun 20, 2022

tlinford commented Jun 20, 2022

a-kenji commented Jun 20, 2022

noyez commented Jun 20, 2022 • edited Loading

noyez commented Jun 30, 2022

ahcm commented Jul 12, 2022

ahcm commented Jul 14, 2022 • edited Loading

00sapo commented Sep 9, 2022 • edited Loading

raphCode commented Sep 13, 2022

raphCode commented Sep 13, 2022

ahcm commented Sep 22, 2022

noyez commented Jun 20, 2022 •

edited

Loading

ahcm commented Jul 14, 2022 •

edited

Loading

00sapo commented Sep 9, 2022 •

edited

Loading