Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[yugabyted] don't mess up shell prompt on crash #3676

Closed
jaki opened this issue Feb 18, 2020 · 0 comments
Closed

[yugabyted] don't mess up shell prompt on crash #3676

jaki opened this issue Feb 18, 2020 · 0 comments
Assignees
Labels
kind/improve-ux Issues relating to improving user experience.

Comments

@jaki
Copy link
Contributor

jaki commented Feb 18, 2020

Running ./bin/yugabyted start on commit 06d6d8b (only one I tried) causes a crash. I am creating this issue not to focus on why the crash happens (because it may be for other reasons) but to focus on recovering from the crash.

OS: CentOS 7
commit: 06d6d8b
shells: bash 4.2.46(2)-release (x86_64-redhat-linux-gnu), zsh 5.7.1-test-3 (x86_64-pc-linux-gnu)

After the crash, if I ctrl-C, the "Running system checks" message remains. Pressing Enter or running reset don't help. The python ./bin/yugabyted start process must be killed manually (and SIGTERM doesn't work) to stop polluting the prompt. The prompts get messed up as shown below.

bash-4.2$ ./bin/yugabyted start
Starting yugabyted...
/ Running system checks...yugabyted crashed. For troubleshooting, contact us on https://www.yugabyte.com/slack or check our FAQ at https://docs.yugabyte.com/latest/faq/
\ Running system checks...^Cyugabyted stopped.
- Running system checks...bash-4.2$
| Running system checks...
- Running system checks...
~/code/yugabyte-db/1 arcpatch-D7973~1 % ./bin/yugabyted start --bind_ip 127.0.0.11
Starting yugabyted...
/ Running system checks...yugabyted crashed. For troubleshooting, contact us on https://www.yugabyte.com/slack or check our FAQ at https://docs.yugabyte.com/latest/faq/
\ Running system checks...yugabyted stopped.
- Running system checks...tch-D7973~1 %

Here is var/logs/yugabyted.log:

[yugabyted start] 2020-02-18 10:23:56,940 INFO:  | 0.1s | cmd = start using config file: <HOME>/code/yugabyte-db/1/var/conf/yugabyted.conf (args.config=None)
[yugabyted start] 2020-02-18 10:23:56,940 INFO:  | 0.1s | Starting yugabyted...
[yugabyted start] 2020-02-18 10:23:56,944 INFO:  | 0.1s | Daemon grandchild process begins execution.
[yugabyted start] 2020-02-18 10:23:56,945 INFO:  | 0.1s | yugabyted started running with PID 6955.
[yugabyted start] 2020-02-18 10:23:56,946 INFO:  | 0.1s | Running the post-installation script <HOME>/code/yugabyte-db/1/bin/post_install.sh (may be a no-op)
[yugabyted start] 2020-02-18 10:23:56,952 ERROR:  | 0.1s | Traceback (most recent call last):
  File "./bin/yugabyted", line 1024, in run
    args.func()
  File "./bin/yugabyted", line 150, in start
    self.start_processes()
  File "./bin/yugabyted", line 482, in start_processes
    self.post_install_yb()
  File "./bin/yugabyted", line 659, in post_install_yb
    post_install_script_path, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/share/python-2.7.17/lib/python2.7/subprocess.py", line 394, in __init__
    errread, errwrite)
  File "/usr/share/python-2.7.17/lib/python2.7/subprocess.py", line 1047, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory
@jaki jaki added the kind/improve-ux Issues relating to improving user experience. label Feb 18, 2020
iSignal added a commit that referenced this issue Feb 20, 2020
Summary: We call sys.exit but that doesn't exit the program because of the bg threads still running. Instead let's call log_error_and_exit which does proper teardown

Test Plan:
Run regular case, run a scenario with post_install.sh missing (which earlier hung at the prompt) - we see the following output if post_install.sh is missing

```
15:50 $ bin/yugabyted start
Starting yugabyted...
/ Running system checks...yugabyted crashed. For troubleshooting, contact us on https://www.yugabyte.com/slack or check our FAQ at https://docs.yugabyte.com/latest/faq/
Traceback (most recent call last):
  File "bin/yugabyted", line 1024, in run
    args.func()
  File "bin/yugabyted", line 150, in start
    self.start_processes()
  File "bin/yugabyted", line 482, in start_processes
    self.post_install_yb()
  File "bin/yugabyted", line 659, in post_install_yb
    post_install_script_path, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
    errread, errwrite)
  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

For more information, check the logs in /tmp/ybd-test-708655a4-889c-4a93-a139-43b0fa334104/yugabyte-2.1.0.0/var/logs
```

Reviewers: wesley

Reviewed By: wesley

Subscribers: yugaware

Differential Revision: https://phabricator.dev.yugabyte.com/D8003
@iSignal iSignal closed this as completed Feb 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/improve-ux Issues relating to improving user experience.
Projects
None yet
Development

No branches or pull requests

3 participants