Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bug: coroot-node-agent crashes the service rtpengine-daemon with "segmentation fault" #158

Open
nlegyes opened this issue Dec 17, 2024 · 2 comments
Open

Bug: coroot-node-agent crashes the service rtpengine-daemon with "segmentation fault" #158

nlegyes opened this issue Dec 17, 2024 · 2 comments

Comments

@nlegyes
Copy link

nlegyes commented Dec 17, 2024

Hi coroot-node-agent team,

I have currently the problem, that coroot-node-agent crashes the service rtpengine-daemon with "segmentation fault" rtpengine-daemon.service: Main process exited, code=killed, status=11/SEGV.
Originally I have opened this issue at the sipwise/rtpengine issue: 1888 project (if additional context is needed).

Operating System

  • OS: Debian 12 Bookworm
  • Kernel: 6.11.5+bpo-amd64
  • CPU architecture: x86_64

The OS is running as a VM on Proxmox 8.3.
I am using Debian 12 Bookworm Cloud Image.

Application Versions

  • RTPEngine: Version: 13.0.1.6-1~bpo12+1
  • coroot-node-agent: v1.22.10

Steps to reproduce the problem

  1. Install rtpengine-daemon.
  2. Install coroot-node-agent.
  3. Loaded Kernel Module modprobe -v nf_conntrack
  4. RTPEngine crashes after 10 seconds.

Workaround

  1. Uninstall coroot-node-agent.
  2. Loaded Kernel Module modprobe -v nf_conntrack
  3. Start RTPEngine systemctl start rtpengine-daemon
  4. RTPEngine runs now without issues.

Logs Starting coroot-node-agent + rtpengine-daemon

Starting coroot-node-agent and rtpengine-daemon with systemctl start rtpengine-daemon coroot-node-agent && journalctl -u rtpengine-daemon -u coroot-node-agent -f.

Dec 17 17:13:56 scb02proxy02 systemd[1]: Starting rtpengine-daemon.service - NGCP RTP/media Proxy Daemon...
Dec 17 17:13:56 scb02proxy02 systemd[1]: Starting coroot-node-agent.service - Coroot node agent...
Dec 17 17:13:56 scb02proxy02 systemd[1]: Started coroot-node-agent.service - Coroot node agent.
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.718544    6032 net.go:24] whitelisted public IPs: [0.0.0.0/0]
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.718617    6032 net.go:36] ephemeral-port-range: 32768-60999
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724081    6032 cilium.go:30] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_ct4_global: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724130    6032 cilium.go:36] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_ct6_global: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724140    6032 cilium.go:43] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb4_backends_v2: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724149    6032 cilium.go:43] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb4_backends_v3: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724159    6032 cilium.go:52] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb6_backends_v2: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.724167    6032 cilium.go:52] Unable to get object /proc/1/root/sys/fs/bpf/tc/globals/cilium_lb6_backends_v3: no such file or directory
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.727962    6032 main.go:108] agent version: 1.22.10
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728043    6032 main.go:114] hostname: scb02proxy02
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728056    6032 main.go:115] kernel version: 6.11.5+bpo-amd64
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728437    6032 main.go:72] machine-id:  b7076c686366482fa38a1b7837685421
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728506    6032 tracing.go:40] OpenTelemetry traces collector endpoint: https://coroot.int.s24.de.tkelevator.com/v1/traces
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728571    6032 otel.go:30] OpenTelemetry logs collector endpoint: https://coroot.int.s24.de.tkelevator.com/v1/logs
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728717    6032 metadata.go:67] cloud provider:
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728732    6032 collector.go:157] instance metadata: <nil>
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: I1217 17:13:56.728853    6032 profiling.go:55] profiles endpoint: https://coroot.int.s24.de.tkelevator.com/v1/profiles
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [core] compile-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [core] run-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [crypto] Generating new DTLS certificate
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [core] Startup complete, version 13.0.1.6-1~bpo12+1
Dec 17 17:13:56 scb02proxy02 rtpengine[6031]: INFO: [http] Websocket listener thread running
Dec 17 17:13:56 scb02proxy02 systemd[1]: Started rtpengine-daemon.service - NGCP RTP/media Proxy Daemon.
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: W1217 17:13:56.979120    6032 registry.go:86] Cannot connect to the Docker daemon at unix:///proc/1/root/run/docker.sock. Is the docker daemon running?
Dec 17 17:13:56 scb02proxy02 coroot-node-agent[6032]: W1217 17:13:56.979120    6032 registry.go:86] Cannot connect to the Docker daemon at unix:///proc/1/root/run/docker.sock. Is the docker daemon running?
Dec 17 17:14:00 scb02proxy02 coroot-node-agent[6032]: W1217 17:14:00.980919    6032 registry.go:89] couldn't connect to containerd through the following UNIX sockets [/var/snap/microk8s/common/run/containerd.sock,/run/k0s/containerd.sock,/run/k3s/containerd/containerd.sock,/run/containerd/containerd.sock]: failed to dial "/proc/1/root/run/containerd/containerd.sock": context deadline exceeded
Dec 17 17:14:00 scb02proxy02 coroot-node-agent[6032]: W1217 17:14:00.980919    6032 registry.go:89] couldn't connect to containerd through the following UNIX sockets [/var/snap/microk8s/common/run/containerd.sock,/run/k0s/containerd.sock,/run/k3s/containerd/containerd.sock,/run/containerd/containerd.sock]: failed to dial "/proc/1/root/run/containerd/containerd.sock": context deadline exceeded
Dec 17 17:14:00 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:00.981009    6032 crio.go:58] cri-o socket:
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541666    6032 registry.go:323] calculated container id 1 -> /init.scope ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541745    6032 registry.go:328] "ignoring" cg="/init.scope" pid=1
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541789    6032 registry.go:323] calculated container id 2 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541800    6032 registry.go:328] "ignoring" cg="" pid=2
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541830    6032 registry.go:323] calculated container id 3 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541844    6032 registry.go:328] "ignoring" cg="" pid=3
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541870    6032 registry.go:323] calculated container id 4 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541877    6032 registry.go:328] "ignoring" cg="" pid=4
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541898    6032 registry.go:323] calculated container id 5 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541905    6032 registry.go:328] "ignoring" cg="" pid=5
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541925    6032 registry.go:323] calculated container id 6 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541931    6032 registry.go:328] "ignoring" cg="" pid=6
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541950    6032 registry.go:323] calculated container id 7 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541956    6032 registry.go:328] "ignoring" cg="" pid=7
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541976    6032 registry.go:323] calculated container id 10 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.541982    6032 registry.go:328] "ignoring" cg="" pid=10
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542000    6032 registry.go:323] calculated container id 12 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542005    6032 registry.go:328] "ignoring" cg="" pid=12
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542027    6032 registry.go:323] calculated container id 13 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542032    6032 registry.go:328] "ignoring" cg="" pid=13
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542051    6032 registry.go:323] calculated container id 14 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542057    6032 registry.go:328] "ignoring" cg="" pid=14
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542074    6032 registry.go:323] calculated container id 15 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542081    6032 registry.go:328] "ignoring" cg="" pid=15
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542110    6032 registry.go:323] calculated container id 16 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542116    6032 registry.go:328] "ignoring" cg="" pid=16
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542135    6032 registry.go:323] calculated container id 17 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542141    6032 registry.go:328] "ignoring" cg="" pid=17
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542160    6032 registry.go:323] calculated container id 18 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542166    6032 registry.go:328] "ignoring" cg="" pid=18
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542184    6032 registry.go:323] calculated container id 19 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542189    6032 registry.go:328] "ignoring" cg="" pid=19
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542211    6032 registry.go:323] calculated container id 20 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542218    6032 registry.go:328] "ignoring" cg="" pid=20
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542240    6032 registry.go:323] calculated container id 21 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542246    6032 registry.go:328] "ignoring" cg="" pid=21
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542264    6032 registry.go:323] calculated container id 22 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542270    6032 registry.go:328] "ignoring" cg="" pid=22
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542296    6032 registry.go:323] calculated container id 23 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542302    6032 registry.go:328] "ignoring" cg="" pid=23
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542321    6032 registry.go:323] calculated container id 24 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542327    6032 registry.go:328] "ignoring" cg="" pid=24
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542345    6032 registry.go:323] calculated container id 25 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542350    6032 registry.go:328] "ignoring" cg="" pid=25
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542368    6032 registry.go:323] calculated container id 26 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542374    6032 registry.go:328] "ignoring" cg="" pid=26
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542392    6032 registry.go:323] calculated container id 28 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542398    6032 registry.go:328] "ignoring" cg="" pid=28
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542417    6032 registry.go:323] calculated container id 29 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542422    6032 registry.go:328] "ignoring" cg="" pid=29
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542443    6032 registry.go:323] calculated container id 30 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542449    6032 registry.go:328] "ignoring" cg="" pid=30
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542467    6032 registry.go:323] calculated container id 31 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542473    6032 registry.go:328] "ignoring" cg="" pid=31
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542493    6032 registry.go:323] calculated container id 32 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542499    6032 registry.go:328] "ignoring" cg="" pid=32
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542520    6032 registry.go:323] calculated container id 34 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542526    6032 registry.go:328] "ignoring" cg="" pid=34
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542545    6032 registry.go:323] calculated container id 35 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542551    6032 registry.go:328] "ignoring" cg="" pid=35
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542570    6032 registry.go:323] calculated container id 36 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542575    6032 registry.go:328] "ignoring" cg="" pid=36
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542593    6032 registry.go:323] calculated container id 37 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542598    6032 registry.go:328] "ignoring" cg="" pid=37
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542617    6032 registry.go:323] calculated container id 38 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542623    6032 registry.go:328] "ignoring" cg="" pid=38
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542641    6032 registry.go:323] calculated container id 40 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542648    6032 registry.go:328] "ignoring" cg="" pid=40
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542665    6032 registry.go:323] calculated container id 41 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542670    6032 registry.go:328] "ignoring" cg="" pid=41
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542693    6032 registry.go:323] calculated container id 42 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542698    6032 registry.go:328] "ignoring" cg="" pid=42
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542716    6032 registry.go:323] calculated container id 43 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542721    6032 registry.go:328] "ignoring" cg="" pid=43
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542740    6032 registry.go:323] calculated container id 44 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542745    6032 registry.go:328] "ignoring" cg="" pid=44
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542763    6032 registry.go:323] calculated container id 46 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542769    6032 registry.go:328] "ignoring" cg="" pid=46
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542786    6032 registry.go:323] calculated container id 47 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542792    6032 registry.go:328] "ignoring" cg="" pid=47
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542809    6032 registry.go:323] calculated container id 48 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542815    6032 registry.go:328] "ignoring" cg="" pid=48
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542835    6032 registry.go:323] calculated container id 49 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542841    6032 registry.go:328] "ignoring" cg="" pid=49
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542858    6032 registry.go:323] calculated container id 50 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542864    6032 registry.go:328] "ignoring" cg="" pid=50
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542891    6032 registry.go:323] calculated container id 52 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542897    6032 registry.go:328] "ignoring" cg="" pid=52
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542915    6032 registry.go:323] calculated container id 53 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542921    6032 registry.go:328] "ignoring" cg="" pid=53
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542938    6032 registry.go:323] calculated container id 54 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542944    6032 registry.go:328] "ignoring" cg="" pid=54
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542962    6032 registry.go:323] calculated container id 55 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542967    6032 registry.go:328] "ignoring" cg="" pid=55
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542985    6032 registry.go:323] calculated container id 56 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.542990    6032 registry.go:328] "ignoring" cg="" pid=56
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543010    6032 registry.go:323] calculated container id 58 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543016    6032 registry.go:328] "ignoring" cg="" pid=58
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543033    6032 registry.go:323] calculated container id 59 ->  ->
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.543039    6032 registry.go:328] "ignoring" cg="" pid=59
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.643595    6032 tls.go:115] pid=758 libssl_version=v3.0.15: libssl uprobes attached
Dec 17 17:14:01 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:01.845139    6032 tls.go:259] pid=6032 golang_app=/usr/bin/coroot-node-agent golang_version=go1.23.4: crypto/tls uprobes attached
Dec 17 17:14:05 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:05.041006    6032 tls.go:115] pid=602 libssl_version=v3.0.15: libssl uprobes attached
Dec 17 17:14:05 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:05.041178    6032 tls.go:174] pid=602 golang_app=/usr/sbin/zabbix_agent2 golang_version=go1.23.1: no symbol section
Dec 17 17:14:10 scb02proxy02 rtpengine[6031]: INFO: [http] HTTP GET from 10.78.6.26:41630: '/metrics'
Dec 17 17:14:16 scb02proxy02 systemd[1]: rtpengine-daemon.service: Main process exited, code=killed, status=11/SEGV
Dec 17 17:14:16 scb02proxy02 systemd[1]: rtpengine-daemon.service: Failed with result 'signal'.
Dec 17 17:14:16 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:16.978827    6032 registry.go:323] calculated container id 6110 ->  ->
Dec 17 17:14:16 scb02proxy02 coroot-node-agent[6032]: I1217 17:14:16.978861    6032 registry.go:328] "ignoring" cg="" pid=6110

Logs Starting just rtpengine-daemon

Starting rtpengine-daemon with systemctl start rtpengine-daemon && journalctl -u rtpengine-daemon -f.

Dec 17 17:19:05 scb02proxy02 systemd[1]: Starting rtpengine-daemon.service - NGCP RTP/media Proxy Daemon...
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [core] compile-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [core] run-time OpenSSL library: OpenSSL 3.0.15 3 Sep 2024
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [crypto] Generating new DTLS certificate
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [core] Startup complete, version 13.0.1.6-1~bpo12+1
Dec 17 17:19:05 scb02proxy02 rtpengine[6167]: INFO: [http] Websocket listener thread running
Dec 17 17:19:05 scb02proxy02 systemd[1]: Started rtpengine-daemon.service - NGCP RTP/media Proxy Daemon.
Dec 17 17:19:10 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:41972: '/metrics'
Dec 17 17:19:25 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:34238: '/metrics'
Dec 17 17:19:40 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:35786: '/metrics'
Dec 17 17:19:55 scb02proxy02 rtpengine[6167]: INFO: [http] HTTP GET from 10.78.6.26:41378: '/metrics'
@def
Copy link
Member

def commented Dec 18, 2024

I can't reproduce this issue on Debian 12 with kernel version 6.11.5+bpo-amd64

@nlegyes
Copy link
Author

nlegyes commented Dec 25, 2024

Hi @def ,

this is weird.
I was able to reproduce this issue in my HomeLab as well.
In my HomeLab I use Proxmox 8.3.X too.

Steps to reproduce

Here are the exact steps which I did on a fresh Debian 12 Install:

  1. Import Debian 12 Bookworm Cloud Image into Proxmox 8.3.X .

  2. Install Backports Kernel and reboot

sudo apt update && sudo apt full-upgrade -y
sudo apt install -t bookworm-backports linux-image-amd64 linux-headers-amd64 -y
systemctl reboot
  1. Installing RTPEngine:
wget https://dfx.at/rtpengine/latest/pool/main/r/rtpengine-dfx-repo-keyring/rtpengine-dfx-repo-keyring_1.0_all.deb
sudo dpkg -i rtpengine-dfx-repo-keyring_1.0_all.deb
echo "deb [signed-by=/usr/share/keyrings/dfx.at-rtpengine-archive-keyring.gpg] https://dfx.at/rtpengine/latest bookworm main" | sudo tee /etc/apt/sources.list.d/dfx.at-rtpengine.list
sudo apt update && sudo apt install libavcodec-extra rtpengine -y
  1. Installing coroot-node-agent:
curl -sfL https://raw.githubusercontent.com/coroot/coroot-node-agent/main/install.sh | \
 COLLECTOR_ENDPOINT=http://localhost:8080 \
 SCRAPE_INTERVAL=15s \
 sh -
  1. Checking the logs:
journalctl -u rtpengine-daemon -u coroot-node-agent -e

root@test02:/home/nlegyes# journalctl -u rtpengine-daemon -u coroot-node-agent -e
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681789    5264 registry.go:323] calculated container id 20 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681798    5264 registry.go:328] "ignoring" cg="" pid=20
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681830    5264 registry.go:323] calculated container id 21 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681840    5264 registry.go:328] "ignoring" cg="" pid=21
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681872    5264 registry.go:323] calculated container id 22 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681884    5264 registry.go:328] "ignoring" cg="" pid=22
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681912    5264 registry.go:323] calculated container id 23 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681921    5264 registry.go:328] "ignoring" cg="" pid=23
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681971    5264 registry.go:323] calculated container id 24 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.681982    5264 registry.go:328] "ignoring" cg="" pid=24
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682014    5264 registry.go:323] calculated container id 25 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682023    5264 registry.go:328] "ignoring" cg="" pid=25
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682053    5264 registry.go:323] calculated container id 26 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682063    5264 registry.go:328] "ignoring" cg="" pid=26
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682095    5264 registry.go:323] calculated container id 27 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682103    5264 registry.go:328] "ignoring" cg="" pid=27
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682134    5264 registry.go:323] calculated container id 28 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682144    5264 registry.go:328] "ignoring" cg="" pid=28
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682180    5264 registry.go:323] calculated container id 29 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682190    5264 registry.go:328] "ignoring" cg="" pid=29
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682221    5264 registry.go:323] calculated container id 30 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682230    5264 registry.go:328] "ignoring" cg="" pid=30
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682260    5264 registry.go:323] calculated container id 31 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682269    5264 registry.go:328] "ignoring" cg="" pid=31
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682299    5264 registry.go:323] calculated container id 32 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682308    5264 registry.go:328] "ignoring" cg="" pid=32
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682339    5264 registry.go:323] calculated container id 33 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682349    5264 registry.go:328] "ignoring" cg="" pid=33
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682379    5264 registry.go:323] calculated container id 34 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682390    5264 registry.go:328] "ignoring" cg="" pid=34
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682424    5264 registry.go:323] calculated container id 35 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682432    5264 registry.go:328] "ignoring" cg="" pid=35
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682461    5264 registry.go:323] calculated container id 36 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682471    5264 registry.go:328] "ignoring" cg="" pid=36
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682507    5264 registry.go:323] calculated container id 37 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682518    5264 registry.go:328] "ignoring" cg="" pid=37
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682546    5264 registry.go:323] calculated container id 38 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682555    5264 registry.go:328] "ignoring" cg="" pid=38
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682584    5264 registry.go:323] calculated container id 39 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682594    5264 registry.go:328] "ignoring" cg="" pid=39
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682653    5264 registry.go:323] calculated container id 40 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682664    5264 registry.go:328] "ignoring" cg="" pid=40
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682695    5264 registry.go:323] calculated container id 42 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682705    5264 registry.go:328] "ignoring" cg="" pid=42
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682740    5264 registry.go:323] calculated container id 43 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682750    5264 registry.go:328] "ignoring" cg="" pid=43
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682777    5264 registry.go:323] calculated container id 44 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682786    5264 registry.go:328] "ignoring" cg="" pid=44
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682815    5264 registry.go:323] calculated container id 45 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682825    5264 registry.go:328] "ignoring" cg="" pid=45
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682859    5264 registry.go:323] calculated container id 46 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682869    5264 registry.go:328] "ignoring" cg="" pid=46
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682898    5264 registry.go:323] calculated container id 47 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682908    5264 registry.go:328] "ignoring" cg="" pid=47
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682936    5264 registry.go:323] calculated container id 48 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682947    5264 registry.go:328] "ignoring" cg="" pid=48
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.682999    5264 registry.go:323] calculated container id 49 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.683010    5264 registry.go:328] "ignoring" cg="" pid=49
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.683041    5264 registry.go:323] calculated container id 50 ->  ->
Dec 25 18:43:11 test02 coroot-node-agent[5264]: I1225 18:43:11.683052    5264 registry.go:328] "ignoring" cg="" pid=50
Dec 25 18:43:16 test02 coroot-node-agent[5264]: 2024/12/25 18:43:16 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:21 test02 coroot-node-agent[5264]: 2024/12/25 18:43:21 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:26 test02 coroot-node-agent[5264]: 2024/12/25 18:43:26 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:28 test02 systemd[1]: rtpengine-daemon.service: Main process exited, code=killed, status=11/SEGV
Dec 25 18:43:28 test02 systemd[1]: rtpengine-daemon.service: Failed with result 'signal'.
Dec 25 18:43:28 test02 coroot-node-agent[5264]: I1225 18:43:28.948619    5264 wrappers.go:218] level info remote_name 06d942 url http://localhost:8080/v1/metrics msg Done replaying WAL duration 12.216946158s
Dec 25 18:43:29 test02 coroot-node-agent[5264]: I1225 18:43:29.054109    5264 tls.go:259] pid=5264 golang_app=/usr/bin/coroot-node-agent golang_version=go1.23.4: crypto/tls uprobes attached
Dec 25 18:43:31 test02 coroot-node-agent[5264]: 2024/12/25 18:43:31 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:31 test02 coroot-node-agent[5264]: 2024/12/25 18:43:31 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:31 test02 coroot-node-agent[5264]: I1225 18:43:31.764107    5264 wrappers.go:218] level warn remote_name 06d942 url http://localhost:8080/v1/metrics msg Failed to send batch, retrying err Post "http://localhost:8080/v1/metr>
Dec 25 18:43:36 test02 coroot-node-agent[5264]: 2024/12/25 18:43:36 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:41 test02 coroot-node-agent[5264]: 2024/12/25 18:43:41 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:46 test02 coroot-node-agent[5264]: 2024/12/25 18:43:46 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:46 test02 coroot-node-agent[5264]: 2024/12/25 18:43:46 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:51 test02 coroot-node-agent[5264]: 2024/12/25 18:43:51 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:43:56 test02 coroot-node-agent[5264]: 2024/12/25 18:43:56 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:01 test02 coroot-node-agent[5264]: 2024/12/25 18:44:01 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:01 test02 coroot-node-agent[5264]: 2024/12/25 18:44:01 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:06 test02 coroot-node-agent[5264]: 2024/12/25 18:44:06 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:06 test02 coroot-node-agent[5264]: I1225 18:44:06.764793    5264 profiling.go:137] collected 0 profiles in 1ms
Dec 25 18:44:06 test02 coroot-node-agent[5264]: I1225 18:44:06.764832    5264 profiling.go:147] uploaded 0 profiles in 0s
Dec 25 18:44:11 test02 coroot-node-agent[5264]: 2024/12/25 18:44:11 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:16 test02 coroot-node-agent[5264]: 2024/12/25 18:44:16 Post "http://localhost:8080/v1/logs": dial tcp [::1]:8080: connect: connection refused
Dec 25 18:44:16 test02 coroot-node-agent[5264]: 2024/12/25 18:44:16 traces export: Post "http://localhost:8080/v1/traces": dial tcp [::1]:8080: connect: connection refused

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@def @nlegyes