Language Server crash

[Colengms]

    I do have "name" property in my `c_cpp_properties.json`, and the crash still occurs.

The environment is a docker container running Ubuntu 18.04, and I use vscode to SSH-remotely connect to it.

Ever 10 minutes there is a pop up saying "Connection to server got closed. Server will not be restarted."
And in "Output" pane one "cpptools" window is added, this just keeps adding.

{
    "configurations": [
        {
            "name": "Linux",
            "includePath": [
                "${workspaceFolder}/ara-api/**",
            ],
            "compilerPath": "/usr/bin/g++",
            "cStandard": "c99",
            "cppStandard": "c++14",
            "intelliSenseMode": "linux-gcc-x64"
        }
    ],
    "version": 4
}

I tried to debug cpptools with attaching method described here:
https://github.com/microsoft/vscode-cpptools/wiki/Attaching-debugger-to-cpptools-or-cpptools%E2%80%90srv

It is unable to capture the call stack. When crash happens, debugger console window says:
Program terminated with signal SIGKILL, Killed.

C/C++ debug log:

Pastebin log

Originally posted by @zeroxia in #10251 (comment)

[Colengms]

Hi @zeroxia . I moved your comment into a new issue.

From your log, it looks like your issue is a crash that occurs immediately after an IntelliSense update. It should be possible to connect the debugger and catch the crash, to get a call stack.

I just tried the instructions you referred so, and they worked properly for me with 1.13.7 (after updating the program field to refer to the correct file). Could you try that again? Since the issue repro's after processing IntelliSense for a file, try opening the workspace with no files open, connecting the debugger, then opening a source file.

This might be the same issue as reported here: #10269

[sean-mcmanus]

Also, if you're still not able to attach to get a calls stack, you may be able to configure your Linux OS generate a core dump file that has the call stack.

[sean-mcmanus]

Also, if it's a crash you should be seeing something like "Thread 13 "cpptools" received signal SIGSEGV, Segmentation fault." -- I'm not sure what could be triggering a SIGKILL.

[ryanmda]

I'm not the original poster, but I'm seeing a similar issue. Unfortunately, the core dump is too large to attach. Uncompressed it's almost 400 MB; gzipped it was 43 MB and .xz isn't a permitted file format.

Here's the output of file:

core: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from '/home/cfs-user/.vscode-server/extensions/ms-vscode.cpptools-1.13.9/bin/cpptools', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/home/cfs-user/.vscode-server/extensions/ms-vscode.cpptools-1.13.9/bin/cpptools', platform: 'x86_64'

Let me know if you have a suggestion for how to provide this file.

[sean-mcmanus]

@ryanmda We don't need the entire core dump. Can you just run a command to get the call stack of the crashing thread from the core dump?

[ryanmda]

Okay. Connecting with GDB and it reports:

Core was generated by `/home/cfs-user/.vscode-server/extensions/ms-vscode.cpptools-1.13.9/bin/cpptools'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f723b5db876 in __run_exit_handlers (status=4, listp=0x7f723b781718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:77

And backtrace of what I believe is the correct thread:

#0  0x00007f723b5db876 in __run_exit_handlers (status=4, listp=0x7f723b781718 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:77
#1  0x00007f723b5dba60 in __GI_exit (status=<optimized out>) at exit.c:139
#2  0x0000000000a6fbeb in cfe_exit(int) ()
#3  0x0000000000a6fbb7 in exit_compilation(an_error_severity) ()
#4  0x0000000000a6fc16 in term_compilation(an_error_severity) ()
#5  0x0000000000a71e82 in term_on_signal(int) ()
#6  <signal handler called>
#7  0x00007f723b6a300a in __GI___libc_read (nbytes=4096, buf=0x18a9a40, fd=0)
    at ../sysdeps/unix/sysv/linux/read.c:26
#8  __GI___libc_read (fd=0, buf=0x18a9a40, nbytes=4096) at ../sysdeps/unix/sysv/linux/read.c:24
#9  0x00007f723b625b9f in _IO_new_file_underflow (fp=0x7f723b781980 <_IO_2_1_stdin_>)
    at libioP.h:948
#10 0x00007f723b626f86 in __GI__IO_default_uflow (fp=0x7f723b781980 <_IO_2_1_stdin_>)
    at libioP.h:948
#11 0x00007f723b6204f8 in _IO_getc (fp=0x7f723b781980 <_IO_2_1_stdin_>) at getc.c:38
#12 0x0000000000f7e0fd in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::underflow() ()
#13 0x0000000000f4127a in std::basic_istream<char, std::char_traits<char> >& std::getline<char, std::char_traits<char>, std::allocator<char> >(std::basic_istream<char, std::char_traits<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, char) ()
#14 0x00000000004e12c7 in vscode::message_handler::main_loop() ()
#15 0x00000000004e0184 in main ()

[sean-mcmanus]

That isn't a very usable call stack. It shows our main thread waiting on std::getline as normal with some C runtime exit handler running. Could another thread be calling an abort? I don't see where the segmentation fault is happening.

[ryanmda]

Here's the output of info threads. Most of these seem to just be waiting.

  Id   Target Id                        Frame
* 1    Thread 0x7f723b565780 (LWP 6757) 0x00007f723b5db876 in __run_exit_handlers (status=4,
    listp=0x7f723b781718 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true)
    at exit.c:77
  2    Thread 0x7f71e1ffb700 (LWP 6850) futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x1818814 <cond+44>) at ../sysdeps/nptl/futex-internal.h:183
  3    Thread 0x7f71ea7fc700 (LWP 6781) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a14d4)
    at ../sysdeps/nptl/futex-internal.h:183
  4    Thread 0x7f71e27fc700 (LWP 6788) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1898)
    at ../sysdeps/nptl/futex-internal.h:183
  5    Thread 0x7f71e9ffb700 (LWP 6782) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1290)
    at ../sysdeps/nptl/futex-internal.h:183
  6    Thread 0x7f71c3fff700 (LWP 6853) futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x1818810 <cond+40>) at ../sysdeps/nptl/futex-internal.h:183
  7    Thread 0x7f71e37fe700 (LWP 6786) futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f71e37fdde8,
    clockid=<optimized out>, expected=0, futex_word=0x7ffd7e3a17d8) at ../sysdeps/nptl/futex-internal.h:320
  8    Thread 0x7f71e97fa700 (LWP 6783) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1350)
    at ../sysdeps/nptl/futex-internal.h:183
  9    Thread 0x7f71e8ff9700 (LWP 6784) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a165c)
    at ../sysdeps/nptl/futex-internal.h:183
  10   Thread 0x7f71e3fff700 (LWP 6785) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1718)
    at ../sysdeps/nptl/futex-internal.h:183
  11   Thread 0x7f71e2ffd700 (LWP 6787) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1be0)
    at ../sysdeps/nptl/futex-internal.h:183
  12   Thread 0x7f71e17fa700 (LWP 6851) futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x1818810 <cond+40>) at ../sysdeps/nptl/futex-internal.h:183
  13   Thread 0x7f71c37fe700 (LWP 8204) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7f71ac382868)
    at ../sysdeps/nptl/futex-internal.h:183
  14   Thread 0x7f71e0ff9700 (LWP 6852) futex_wait_cancelable (private=<optimized out>, expected=0,
    futex_word=0x1818810 <cond+40>) at ../sysdeps/nptl/futex-internal.h:183
  15   Thread 0x7f71eaffd700 (LWP 6780) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1414)
    at ../sysdeps/nptl/futex-internal.h:183
  16   Thread 0x7f71eb7fe700 (LWP 6779) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a1594)
    at ../sysdeps/nptl/futex-internal.h:183
  17   Thread 0x7f723b564700 (LWP 6758) __pthread_clockjoin_ex (threadid=140126587971328, thread_return=0x0,
    clockid=<optimized out>, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
  18   Thread 0x7f7200ff9700 (LWP 6777) futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7f7200ff89d8,
    clockid=<optimized out>, expected=0, futex_word=0x7ffd7e3a1110) at ../sysdeps/nptl/futex-internal.h:320
  19   Thread 0x7f71ebfff700 (LWP 6778) futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0x7ffd7e3a11d0)
    at ../sysdeps/nptl/futex-internal.h:183

I grabbed thread 17's backtrace since that is a bit different. Let me know if you want another thread or if this helps.

#0  __pthread_clockjoin_ex (threadid=140126587971328, thread_return=0x0, clockid=<optimized out>, abstime=<optimized out>,
    block=<optimized out>) at pthread_join_common.c:145
#1  0x0000000000f22415 in msvc::thread_t::join() ()
#2  0x00000000008ce9a5 in tag_parse_shutdown() ()
#3  0x00000000008ced33 in a_tag_parse_thread::~a_tag_parse_thread() ()
#4  0x00007f723b5db8a7 in __run_exit_handlers (status=4, listp=0x7f723b781718 <__exit_funcs>,
    run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108
#5  0x00007f723b5dba60 in __GI_exit (status=<optimized out>) at exit.c:139
#6  0x0000000000a6fbeb in cfe_exit(int) ()
#7  0x0000000000a6fbb7 in exit_compilation(an_error_severity) ()
#8  0x0000000000a6fc16 in term_compilation(an_error_severity) ()
#9  0x0000000000a71e82 in term_on_signal(int) ()
#10 <signal handler called>
#11 0x00007f723b6b446e in epoll_wait (epfd=3, events=0x7f723b560dd0, maxevents=1024, timeout=-1)
    at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#12 0x0000000000f2d040 in uv.io_poll ()
#13 0x0000000000f2b1a8 in uv_run ()
#14 0x0000000000ef5ffb in msvc::loop_t::run_loop() ()
#15 0x0000000000efb0d5 in void std::__invoke_impl<void, void (*)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t> >(std::__invoke_other, void (*&&)(std::shared_ptr<msvc::loop_t>), std::shared_ptr<msvc::loop_t>&&) ()
#16 0x0000000000f2609a in msvc::thread_helper_t::thread_entry(void*) ()
#17 0x00007f723b78f609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#18 0x00007f723b6b4133 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[ryanmda]

I also ran thread apply all bt to get all thread backtraces. Hopefully something in here is helpful. Attached.

all thread bt.txt

[sean-mcmanus]

I don't see any real work happening on those threads or where a crash is happening. When does this crash occur, i.e. repro steps? Are you shutting down or editing, etc.?

[ryanmda]

It seems to just happen in the background as I'm doing normal work, or I notice it when I come back to my desk after something else. I haven't seen any obvious cause from my end. Maybe it's related to coming in/out of sleep on my machine?

[zeroxia]

@Colengms

I feel that this might be my environment issue. So you may close this issue.

The "SIGKILL" event always happen at "10 minute edge", e.g. 10:00:01, 10:10:02, 10:20:01, the second part might be 1 or 2. If I launch VSCode at 10:28:00, then at 10:30:xx, you expect the SIGKILL event. When cpptools is killed, there is message shown in VSCode:

Connection to server got closed. Server will not be restarted.

And in output pane, at the dropdown menu, you have a new entry named "cpptools", selecting it will show the same message like this:

[Error - 2:40:02 PM] Connection to server got closed. Server will not be restarted.

That is, every 10 minutes, I got a new "cpptools" entry added to the output pane.

The environment is somewhat twisted, it's a Docker container from image based on Ubuntu 16.04, and inside the container, a "schroot" is set up to be Ubuntu 18.04. This is all in a company docker server.

I feel that the SIGKILL is from the kernel, arised by some cron job scheduled every 10 minutes in that docker server.

On another docker container which is native Ubuntu 18.04 and is running in remote docker server as well, I don't have this problem.

Using gdb to attach to "cpptools" process, I'm unable to capture the call stack, as the signal is SIGKILL.

I tried using systemtap or auditd to track the source of SIGKILL, but it does not work in this twisted environment.

Anyway, thank you for your time.

[sean-mcmanus]

@zeroxia Okay, FYI cpptools does have a 5 minute terminate timeout if it detects that is dangling from a VS Code instance, but your issue seems different since it's 10 minutes.

@ryanmda Your issue is different and seems like it might be caused by our 5 minute timeout code going off after your computer re-awakens from sleeping. I've created an issue to track that at #10362 . It's also possible there could be a crash on shutdown in that case due to how it handles the exiting, but the fact that it's trying to exit seems like the root cause.

[sean-mcmanus]

FYI, we're tracking the sigkill issue on cpptools at #11003 .