AirRouter with tunnels LOW performance (aka. slugbug)

[dman776]

http://bloodhound.aredn.org/products/AREDN/ticket/218

AirRouter? (non-HP)
tunnels installed and 1 client connected
After a brief period after reboot, the node performance begins to slow dramatically. Appears to be an OOM issue.
Multiple rssi_monitor processes running and it appears that the "iw scan" command is not returning properly causing the rssi_monitor script to hang.
Cron then executes another one a minute later.

---

I have a Nanostation M2 that is sluggish, no tunnel installed nor any additional modules. The common theme is that this unit does not have any RF connections to it currently. dev-171 and dev-172 installed - very little improvement

As noted in #229 this is a memory consumption issue.

One of the first commits that will be going in to resolve this will be to require devices to have at least 64mb of RAM to be permitted to run tunnels (this will rule out the AirRouter? for tunnels in near future unless we decide to use the USB ports for SWAP space, but even then the nodes would be in a degraded state having to swap to USB, performance would be very dependent upon the USB stick)

New node with WiFi? disable - load averages nice and low, but the Tunnel OLSR Not Running bug is still there. Uploading support file.

to test a theory, wifi was disabled (ie. added option disabled 1 to the /etc/wireless file).

Tom applied the "disable wifi" config to one of his nodes:

Reporting back to you regarding the disable of RF deck on AirRouter? tunnel unit. The modification was applied 11 hours ago. So far, the load numbers are .1 or less.

Regarding the access to the node as you mentioned, we have to go to the IP addy of this node :8080.

On N2MH's node that I have, I can report the same findings. All good after 45hrs….

Additional cases of rssi-monitor keeps executing for extended periods of time and on top of itself.

Assigned to Joe to look into.

No one has been able to reproduce this at this time.

This appears to likely be a single device having physical issues vs a platform issue.

Should this be able to be reproduced in the future please re-open the ticket.

Have we had any luck in duplicate this on a second device? + a bump on the remaining questions in comment:7

Ok let's go with the following

1. Has this been duplicated on another air router low power or just this single one?

2. what version of AREDN is the tunnel server (the compression question)

3. What happens if we kill the tunnel? Does the issue keep happening?

Tying to make sure by eliminating the possible issues but mostly this looks like an rssi_monitor problem to me.

Not sure if we're choking on the data (keeping the iw open because of that) or if iw is stalling.

node performance improved slightly after killing iw's as noted below, but, still feels sluggish

Mem: 25128K used, 3444K free, 128K shrd, 1020K buff, 2036K cached
CPU: 0% usr 1% sys 0% nic 98% idle 0% io 0% irq 0% sirq
Load average: 0.66 1.06 1.09 2/60 27338
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
1406 1 root S < 3136 11% 1% vtund[c]: N2MH-AR-TEST-172-31-93-4 tu
27333 27327 root R 1368 5% 1% top
1958 1 root S 2152 8% 0% /usr/sbin/olsrd -f /var/etc/olsrd.con
1428 1 root S 1364 5% 0% {linkled} /bin/sh /usr/local/bin/link
27042 27041 root S 3408 12% 0% {rssi_monitor} /usr/bin/perl /usr/loc
13503 13500 root S 3408 12% 0% {rssi_monitor} /usr/bin/perl /usr/loc
11775 11772 root S 3408 12% 0% {rssi_monitor} /usr/bin/perl /usr/loc
958 1 root S 2504 9% 0% /usr/sbin/snmpd -Lf /dev/null -f
887 1 root S 1636 6% 0% /sbin/netifd
853 1 root S 1532 5% 0% /sbin/rpcd
1 0 root S 1408 5% 0% /sbin/procd
981 1 root S 1396 5% 0% /usr/sbin/uhttpd -f -h /www -r N2MH-A
929 1 root S 1372 5% 0% /usr/sbin/crond -f -c /etc/crontabs -
27327 27321 root S 1368 5% 0% -ash
1469 1 root S 1364 5% 0% {olsrd-namechang} /bin/sh /usr/local/
1468 1 root S 1364 5% 0% {olsrd-watchdog} /bin/sh /usr/local/b
1256 887 root S 1364 5% 0% udhcpc -p /var/run/udhcpc-eth1.pid -s
13500 929 root S 1364 5% 0% /bin/sh -c /usr/local/bin/rssi_monito
27041 929 root S 1364 5% 0% /bin/sh -c /usr/local/bin/rssi_monito
^C772 929 root S 1364 5% 0% /bin/sh -c /usr/local/bin/rssi_monito
root@N2MH-AR-Test:# ps |grep -i iw
11799 root 0 Z [iw]
11810 root 1360 S sh -c /usr/sbin/iw wlan0 scan freq 2412 passive > /d
11811 root 1100 S /usr/sbin/iw wlan0 scan freq 2412 passive
13537 root 0 Z [iw]
13552 root 1360 S sh -c /usr/sbin/iw wlan0 scan freq 2412 passive > /d
13553 root 1100 S /usr/sbin/iw wlan0 scan freq 2412 passive
27077 root 0 Z [iw]
27093 root 1360 S sh -c /usr/sbin/iw wlan0 scan freq 2412 passive > /d
27095 root 1100 S /usr/sbin/iw wlan0 scan freq 2412 passive
27379 root 1360 S grep -i iw
root@N2MH-AR-Test:#

After I kill the (now 3) "iw wlan0 scan freq 2412 passive" commands, here is the top and meminfo:

Mem: 25388K used, 3184K free, 132K shrd, 1360K buff, 3960K cached
CPU: 9% usr 9% sys 0% nic 81% idle 0% io 0% irq 0% sirq
Load average: 0.73 0.94 1.04 1/48 27653
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
27652 27327 root R 1364 5% 9% top
1406 1 root S < 3136 11% 0% vtund[c]: N2MH-AR-TEST-172-31-93-4 tu
958 1 root S 2504 9% 0% /usr/sbin/snmpd -Lf /dev/null -f
1958 1 root S 2152 8% 0% /usr/sbin/olsrd -f /var/etc/olsrd.con
887 1 root S 1636 6% 0% /sbin/netifd
853 1 root S 1532 5% 0% /sbin/rpcd
1 0 root S 1408 5% 0% /sbin/procd
981 1 root S 1396 5% 0% /usr/sbin/uhttpd -f -h /www -r N2MH-A
929 1 root S 1372 5% 0% /usr/sbin/crond -f -c /etc/crontabs -
27327 27321 root S 1368 5% 0% -ash
1428 1 root S 1364 5% 0% {linkled} /bin/sh /usr/local/bin/link
1469 1 root S 1364 5% 0% {olsrd-namechang} /bin/sh /usr/local/
1468 1 root S 1364 5% 0% {olsrd-watchdog} /bin/sh /usr/local/b
1256 887 root S 1364 5% 0% udhcpc -p /var/run/udhcpc-eth1.pid -s
27441 27439 root S 1364 5% 0% {fccid} /bin/sh /usr/local/bin/fccid
27439 929 root S 1364 5% 0% /bin/sh -c /usr/local/bin/fccid
967 1 root S 1360 5% 0% /usr/sbin/telnetd -F -l /bin/login.sh
27644 1469 root S 1356 5% 0% sleep 5
27653 1428 root S 1356 5% 0% sleep 5
^C448 27441 root S 1356 5% 0% sleep 206
root@N2MH-AR-Test:# cat /proc/meminfo
MemTotal: 28572 kB
MemFree: 3344 kB
MemAvailable: 7092 kB
Buffers: 1344 kB
Cached: 3820 kB
SwapCached: 0 kB
Active: 7788 kB
Inactive: 1624 kB
Active(anon): 4344 kB
Inactive(anon): 40 kB
Active(file): 3444 kB
Inactive(file): 1584 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 4264 kB
Mapped: 1300 kB
Shmem: 136 kB
Slab: 5800 kB
SReclaimable: 800 kB
SUnreclaim: 5000 kB
KernelStack: 384 kB
PageTables: 440 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14284 kB
Committed_AS: 10292 kB
VmallocTotal: 1048372 kB
VmallocUsed: 1688 kB
VmallocChunk: 1026680 kB
root@N2MH-AR-Test:#