log-courier admin socket stuck

[sysmonk]

Hi,

We're running lc-admin and log-courier version 2.9.0 and we're seeing that the admin interface sometimes gets stuck.
In example:

# lc-admin -connect tcp:127.0.0.1:12345
Admin version 2.9.0

Setting up client for tcp:127.0.0.1:12345...

There is no response after, and the process is just stuck.

We can, though, connect to the socket:

# nc -v 127.0.0.1 12345
localhost.localdomain [127.0.0.1] 12345 (?) open
GET / HTTP/1.1

HTTP/1.1 400 Bad Request: missing required Host header
Content-Type: text/plain; charset=utf-8
Connection: close

But once the Host header is sent, it's also stuck:

# nc -v 127.0.0.1 12345
localhost.localdomain [127.0.0.1] 12345 (?) open
GET / HTTP/1.1
Host: 127.0.0.1:12345

The log-courier itself does seem to be working ( reading and delivering logs ). Tried changing the log level to debug, but nothing interesting there.

What information can we provide to help with debugging this?

[sysmonk]

I've sent a SIGQUIT signal to log-courier to generate some debug information, but the log-courier STDERR/STDOUT is sent to /dev/null, but i worked around it by grabbing it from strace output and parsing it a bit, so the output might be a bit corrupted, but i hope it can give some useful information.
stuck-2022-10-14.txt

[driskell]

Thanks that dump should help isolate what’s happening - can see admin server stuck waiting for something - probably it’s some old metric collection code still using channels.