Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Starting multiple sessions for the same host in parallel leads to crash #599

Open
MarcSeebold opened this issue Aug 29, 2023 · 3 comments · May be fixed by #680
Open

Starting multiple sessions for the same host in parallel leads to crash #599

MarcSeebold opened this issue Aug 29, 2023 · 3 comments · May be fixed by #680

Comments

@MarcSeebold
Copy link

Abstract

et sometimes crashes when starting multiple sessions between the same two machines at approximately the same time.

Client

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
$ uname -a
Linux marc-VirtualBox 5.15.0-79-generic #86~20.04.2-Ubuntu SMP Mon Jul 17 23:27:17 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ apt show et
Package: et
Version: 6.2.4-focal1
Priority: optional
Section: net
Maintainer: Jason Gauci <[email protected]>
Installed-Size: 7,685 kB
Depends: libc6 (>= 2.17), libcurl4 (>= 7.16.2), libprotobuf-lite17, libsodium23 (>= 0.6.0), libssl1.1 (>= 1.1.0), libstdc++6 (>= 9), libunwind8, libutempter0 (>= 1.1.5), zlib1g (>= 1:1.1.4), libgcc1 (>= 1:3.3), libtinfo5
Download-Size: 1,184 kB
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/jgmath2000/et/ubuntu focal/main amd64 Packages
Description: Remote terminal for the busy and impatient
 Eternal Terminal is a remote shell that automatically reconnects without interrupting the session.

Server

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.6 LTS
Release:	20.04
Codename:	focal
$ uname -a
Linux mseebold-DT 5.15.0-79-generic #86~20.04.2-Ubuntu SMP Mon Jul 17 23:27:17 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
$ apt show et
Package: et
Version: 6.2.4-focal1
Priority: optional
Section: net
Maintainer: Jason Gauci <[email protected]>
Installed-Size: 7,685 kB
Depends: libc6 (>= 2.17), libcurl4 (>= 7.16.2), libprotobuf-lite17, libsodium23 (>= 0.6.0), libssl1.1 (>= 1.1.0), libstdc++6 (>= 9), libunwind8, libutempter0 (>= 1.1.5), zlib1g (>= 1:1.1.4), libgcc1 (>= 1:3.3), libtinfo5
Download-Size: 1,184 kB
APT-Manual-Installed: yes
APT-Sources: http://ppa.launchpad.net/jgmath2000/et/ubuntu focal/main amd64 Packages
Description: Remote terminal for the busy and impatient
 Eternal Terminal is a remote shell that automatically reconnects without interrupting the session.

Logs/Stacktrace

$ gdb -- et workstation --verbose=9
Excess command line arguments ignored. (--verbose=9)
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from et...
(No debugging symbols found in et)
/home/marc/workstation: No such file or directory.
(gdb) catch throw
Catchpoint 1 (throw)
(gdb) c
The program is not being run.
(gdb) r
Starting program: /usr/bin/et 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[Detaching after vfork from child process 32347]
2023-08-29 10:14:32,517 FATAL [default] Stack Trace: 
[0] 0x155faf et::LogHandler::createLogFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
[1] 0x156f91 et::LogHandler::setupLogFiles(el::Configurations*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)
[2] 0x83e2d main
[3] 0x7ffff76d5083 __libc_start_main
[4] 0x8875e _start
Error: (22): Invalid argument
2023-08-29 10:14:32,517 WARNING [default] Aborting application. Reason: Fatal log at [/build/et-4rfLRV/et-6.2.4/src/base/LogHandler.cpp:97]

Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff76d3859 in __GI_abort () at abort.c:79
#2  0x00005555555d055f in ?? ()
#3  0x000055555567c83d in el::base::Writer::processDispatch() ()
#4  0x00005555556aa06a in et::LogHandler::createLogFile(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#5  0x00005555556aaf91 in et::LogHandler::setupLogFiles(el::Configurations*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) ()
#6  0x00005555555d7e2d in main ()

Logfiles

etclientLogs.tar.gz
etserverLogs.tar.gz
@MarcSeebold
Copy link
Author

My best guess is that createLogFile doesn't check if another process already creates the log file. When it tries to open it, it will fail.

@MarcSeebold
Copy link
Author

Obvious work-around: Wait a bit before running another session.

@MisterTea
Copy link
Owner

Is it an error we can catch and retry?

jamestaylr pushed a commit to jamestaylr/EternalTerminal that referenced this issue Jan 23, 2025
@jamestaylr
Fix multi-session log file race on startup
a61b6a3 
- Fixes MisterTea#599

- Starting multiple sessions simultaneously fails with:
```
2024-11-14 08:20:32,278 FATAL [default] Stack Trace:
[0] 0x0000000104486cab et::LogHandler::createLogFile(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&)
[1] 0x000000010448666d et::LogHandler::setupLogFiles(el::Configurations*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, bool, bool, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>)
[2] 0x00000001043ce52b main
[3] 0x0000000204776345 start
Error: (2): No such file or directory
2024-11-14 08:20:32,278 WARNING [default] Aborting application. Reason: Fatal log at [/tmp/nix-build-eternal-terminal-6.2.8.drv-0/source/src/base/LogHandler.cpp:97]
```

`LogHandler::createLogFile` expects to be the sole owner of the log file and thus [creates the file with `O_EXCL`](https://github.com/MisterTea/EternalTerminal/blob/204612857d233496e7a9c146fd540b7a45599ec7/src/base/LogHandler.cpp#L97). Adding `%f` (microseconds) to `strftime` reduces the probablity of conflict. We don't want to relax the `O_EXCL` constraint (unique log per instance).

Alternative approaches not taken, but could be followed instead:
- Always force `appendPid`
- Append a non-time-based nonce to the path name
@jamestaylr jamestaylr linked a pull request Jan 23, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix multi-session log file race on startup jamestaylr/EternalTerminal
2 participants
@MisterTea @MarcSeebold