You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I just moved from AdvancedTomato to FreshTomato a couple days ago. I’ve made sure to check the “Erase all data in NVRAM” when flashing the FreshTomato firmware and re-did all my configuration by hand. Everything seemed to work fine for a few hours, but then at some time, I lost wifi and couldn’t connect anymore and no traffic was going through. I power-cycled the router, and it started working again, until the same thing happened. Over the course of 2 days, I had to power-cycle the router 5 of 6 times.
I setup a remote syslog server to get some details, and it looks like memory exhaustion.
I found traces like this:
Right after boot, the router uses about 40MB of RAM, but when the problem happens, it quickly jumps to over 160MB and the system freezes from there. I’ve checked the list of processes when the problem starts occurring, but couldn't determine the culprit: the sum of individual processes' memory usage doesn’t account for all the memory used, which makes think that this is happening at the kernel level.
I’m not sure if it’s related, but I have CTF enabled.
I had to go back to AdvancedTomato for now, to get a stable connection again, but I thought I would report this, in case it could help.
Comments (38)
M_ars
hi,
can you please provide more infos about your router setup. Connections type, wifi setup, services you use, scripts…
What happens if you turn off CTF ? → please test that case
If CTF On can trigger the crash on your side → Different test - please use the following images (latest 2021-5 release and different ctf binary and latest branch)
Does the crash happen again with CTF turned on ?
https://bitbucket.org/M_ars/freshtomato-arm/downloads/freshtomato-R7000-ARM_NG-2021.5-15082021-ctf_test_v001-AIO-64K.trx
and
https://bitbucket.org/M_ars/freshtomato-arm/downloads/freshtomato-R7000-ARM_NG-2021.5-21082021-ctf_test_v002-AIO-64K.trx
2021-08-17
Kilian reporter
Thanks for the pointers!
can you please provide more infos about your router setup. Connections type, wifi setup, services you use, scripts…
R7000 with a Gigabit WAN connection, 2 wifi radios enabled (2.4 and 5GHz, same SSID with band steering), a guest wifi piggy-backing on the 2.4 interface, pretty default wireless options. No services enabled except 2 OpenVPN clients (each using their own Routing Policy) and a PPTP server. No stats collection (bandwidth monitoring disabled, as it’s pretty much useless with CTF anyway), no web monitoring, no QOS, no Web services/MySQL, no Bittorrent client, no Tor, no Captive portal, no USB, no FTP/CIFS/DLNA.
I’ll test with CTF off, as well as the other ctf binary over the weekend and send the results back.
2021-08-18
Kilian reporter
Just a quick update: I’ve re-flashed FT 2021,5, and I haven’t got to a point where the router crashes with memory allocation failures, but there’s definitely higher-than-expected memory usage. Here’s the full list of processes running:
# top -b -n1
Mem: 122628K used, 132948K free, 2748K shrd, 6704K buff, 24240K cached
CPU: 0.0% usr 0.0% sys 0.0% nic 100% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 0.35 0.36 0.39 1/54 11836
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
5082 1 nobody S 12396 4.8 0 0.0 dnsmasq -c 4096 --log-async
3774 1 root S 4512 1.7 1 0.0 /etc/openvpn/vpnclient1 --cd /etc/openvpn/client1 --config config.ovpn
3795 1 root S 4408 1.7 0 0.0 /etc/openvpn/vpnclient2 --cd /etc/openvpn/client2 --config config.ovpn
3697 1 root S 3716 1.4 1 0.0 httpd
3423 1 root S 1704 0.6 1 0.0 nas
1 0 root S 1676 0.6 0 0.0 /sbin/preinit
3497 1 root S 1672 0.6 1 0.0 blink eth2 5g 10 8192
769 1 root S 1664 0.6 0 0.0 buttons
772 1 root S 1600 0.6 0 0.0 console
3431 1 root S 1428 0.5 1 0.0 crond
30397 30396 root S 1412 0.5 1 0.0 -sh
4029 1 root S 1412 0.5 0 0.0 udhcpc -i vlan2 -b -s dhcpc-event -H rtr -m -S -p /var/run/udhcpc-wan.pid
3325 1 root S 1408 0.5 1 0.0 syslogd -D -R nas:514 -L -s 50 -O /var/log/messages -b 1
774 772 root S 1408 0.5 0 0.0 /bin/sh
11836 30397 root R 1408 0.5 1 0.0 top -b -n1
3327 1 root S 1400 0.5 0 0.0 klogd
3435 1 root S 1376 0.5 1 0.0 rstats
3466 1 root S 1204 0.4 1 0.0 /usr/sbin/bsd
30396 950 root S 1112 0.4 1 0.0 dropbear -p 22 -a
950 1 root S 1092 0.4 0 0.0 dropbear -p 22 -a
3421 1 root S 1048 0.4 1 0.0 eapd
4024 1 root S 832 0.3 0 0.0 miniupnpd -f /etc/upnp/config
3436 3434 root S 672 0.2 0 0.0 /usr/sbin/bcrelay -i disable -o ppp[0-9].* -n
732 1 root S 656 0.2 1 0.0 hotplug2 --persistent --no-coldplug
3434 1 root S 656 0.2 0 0.0 pptpd -c /etc/vpn/pptpd.conf -o /etc/vpn/pptpd_options -C 50
3 2 root SW 0 0.0 0 0.0 [ksoftirqd/0]
9 2 root SW 0 0.0 1 0.0 [ksoftirqd/1]
264 2 root SW 0 0.0 1 0.0 [mtdblock3]
299 2 root SW 0 0.0 0 0.0 [kworker/0:1]
781 2 root SWN 0 0.0 0 0.0 [jffs2_gcd_mtd6]
7 2 root SW 0 0.0 1 0.0 [migration/1]
6 2 root SW 0 0.0 0 0.0 [migration/0]
59 2 root SW 0 0.0 0 0.0 [sync_supers]
300 2 root SW 0 0.0 1 0.0 [kworker/1:1]
5 2 root SW 0 0.0 0 0.0 [kworker/u:0]
173 2 root SW< 0 0.0 1 0.0 [crypto]
117 2 root SW 0 0.0 1 0.0 [kswapd0]
259 2 root SW 0 0.0 1 0.0 [mtdblock2]
303 2 root SW 0 0.0 1 0.0 [mtdblock5]
62 2 root SW< 0 0.0 1 0.0 [kblockd]
313 2 root SW 0 0.0 1 0.0 [mtdblock7]
2 0 root SW 0 0.0 0 0.0 [kthreadd]
165 2 root SW 0 0.0 1 0.0 [fsnotify_mark]
4 2 root SW 0 0.0 0 0.0 [kworker/0:0]
249 2 root SW 0 0.0 1 0.0 [mtdblock0]
254 2 root SW 0 0.0 1 0.0 [mtdblock1]
802 2 root SW 0 0.0 1 0.0 [khubd]
8 2 root SW 0 0.0 1 0.0 [kworker/1:0]
269 2 root SW 0 0.0 1 0.0 [mtdblock4]
10 2 root SW< 0 0.0 1 0.0 [khelper]
11 2 root SW 0 0.0 1 0.0 [kworker/u:1]
14 2 root SW< 0 0.0 1 0.0 [netns]
308 2 root SW 0 0.0 1 0.0 [mtdblock6]
61 2 root SW 0 0.0 1 0.0 [bdi-default]
Taken all together, they use maybe 40MB of memory, while the system reports about 100MB used. So the difference is likely taken by the kernel, but I’m not sure how to pinpoint that more precisely. Global usage continue to grow overt time, though (that capture is with ~1h uptime).
I’ll go ahead and flash the test images, to see if they behave the same way.
2021-08-21
Kilian reporter
So I tested both test images (test_v001 and test_v002) and they both leak the same way. See this top output with test_v002:
Mem: 177168K used, 78408K free, 2736K shrd, 6596K buff, 12308K cached
CPU: 4.5% usr 4.5% sys 0.0% nic 86.3% idle 0.0% io 0.0% irq 4.5% sirq
Load average: 0.32 0.28 0.28 1/57 17816
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
2838 1 nobody S 12388 4.8 0 4.5 dnsmasq -c 4096 --log-async
17816 15652 root R 1408 0.5 0 4.5 top -n 1 -b
1364 1 root S 4468 1.7 1 0.0 /etc/openvpn/vpnclient1 --cd /etc/openvpn/client1 --config config.ovpn
1386 1 root S 4408 1.7 0 0.0 /etc/openvpn/vpnclient2 --cd /etc/openvpn/client2 --config config.ovpn
1186 1 root S 3716 1.4 1 0.0 httpd
947 1 root S 1704 0.6 1 0.0 nas
1 0 root S 1676 0.6 1 0.0 /sbin/preinit
1063 1 root S 1672 0.6 1 0.0 blink eth2 5g 10 8192
769 1 root S 1664 0.6 1 0.0 buttons
772 1 root S 1600 0.6 0 0.0 console
958 1 root S 1428 0.5 0 0.0 crond
1172 1 root S 1424 0.5 0 0.0 /usr/sbin/ntpd -l -t -S /sbin/ntpd_synced -d6
3868 3867 root S 1424 0.5 1 0.0 -sh
15652 15538 root S 1412 0.5 0 0.0 -sh
1345 1 root S 1412 0.5 1 0.0 udhcpc -i vlan2 -b -s dhcpc-event -H rtr -m -S -p /var/run/udhcpc-wan.pid
782 1 root S 1408 0.5 1 0.0 syslogd -D -R nas:514 -L -s 50 -O /var/log/messages -b 1
774 772 root S 1408 0.5 0 0.0 /bin/sh
784 1 root S 1400 0.5 0 0.0 klogd
961 1 root S 1376 0.5 1 0.0 rstats
1009 1 root S 1192 0.4 1 0.0 /usr/sbin/bsd
15538 943 root S 1112 0.4 0 0.0 dropbear -p 22 -a
3867 943 root S 1112 0.4 1 0.0 dropbear -p 22 -a
943 1 root S 1092 0.4 0 0.0 dropbear -p 22 -a
945 1 root S 1048 0.4 1 0.0 eapd
1338 1 root S 832 0.3 1 0.0 miniupnpd -f /etc/upnp/config
965 963 root S 672 0.2 0 0.0 /usr/sbin/bcrelay -i disable -o ppp[0-9].* -n
732 1 root S 656 0.2 0 0.0 hotplug2 --persistent --no-coldplug
963 1 root S 656 0.2 1 0.0 pptpd -c /etc/vpn/pptpd.conf -o /etc/vpn/pptpd_options -C 50
3 2 root SW 0 0.0 0 0.0 [ksoftirqd/0]
9 2 root SW 0 0.0 1 0.0 [ksoftirqd/1]
264 2 root SW 0 0.0 1 0.0 [mtdblock3]
6204 2 root SWN 0 0.0 1 0.0 [jffs2_gcd_mtd6]
299 2 root SW 0 0.0 0 0.0 [kworker/0:1]
6 2 root SW 0 0.0 0 0.0 [migration/0]
117 2 root SW 0 0.0 0 0.0 [kswapd0]
7 2 root SW 0 0.0 1 0.0 [migration/1]
59 2 root SW 0 0.0 0 0.0 [sync_supers]
300 2 root SW 0 0.0 1 0.0 [kworker/1:1]
5 2 root SW 0 0.0 1 0.0 [kworker/u:0]
173 2 root SW< 0 0.0 1 0.0 [crypto]
795 2 root SW 0 0.0 1 0.0 [khubd]
313 2 root SW 0 0.0 1 0.0 [mtdblock7]
2 0 root SW 0 0.0 1 0.0 [kthreadd]
303 2 root SW 0 0.0 1 0.0 [mtdblock5]
249 2 root SW 0 0.0 1 0.0 [mtdblock0]
254 2 root SW 0 0.0 1 0.0 [mtdblock1]
259 2 root SW 0 0.0 1 0.0 [mtdblock2]
4 2 root SW 0 0.0 0 0.0 [kworker/0:0]
269 2 root SW 0 0.0 1 0.0 [mtdblock4]
10 2 root SW< 0 0.0 1 0.0 [khelper]
11 2 root SW 0 0.0 1 0.0 [kworker/u:1]
8 2 root SW 0 0.0 1 0.0 [kworker/1:0]
308 2 root SW 0 0.0 1 0.0 [mtdblock6]
61 2 root SW 0 0.0 1 0.0 [bdi-default]
62 2 root SW< 0 0.0 1 0.0 [kblockd]
14 2 root SW< 0 0.0 1 0.0 [netns]
165 2 root SW 0 0.0 1 0.0 [fsnotify_mark]
Sum of VSZ is 57,180K, mem used is 177,168K after 57 min of uptime.
2021-08-21
Kilian reporter
With CTF disabled on the default image (stock 2021.5), memory usage is stable at under 40MB after 1h.
But of course, the router can only reach ~250-300Mbps on a 1Gbps connection, so disabling CTF is not a great solution 😕
I also tried to disable Jumbo Frames, but that doesn’t seem to make much difference.
So, to summarize:
* 2021.5 (stock, or test_v001 or test_v002) w/ CTF enabled: memory usage grows over time, until memory allocation errors make the router unusable
* 2021.5 w/ CTF disabled and jumbo frames enabled: memory usage is stable
* 2021.5 w/ CTF enabled and jumbo frames disabled: memory usage grows over time (albeit more slowly)
* AdvancedTomato v3.5-140 w/ CTF enabled and jumbo frames enabled: memory usage is stable
Hope this can help pinpoint the problem and hopefully fix it.
Thanks!
2021-08-21
M_ars
mhh ok. I have no problems with CTF on my side. I also asked a few more people. Can you crash the router with both test_v001 or test_v002 images ?
What functions/features else do you use? First time you mentioned Jumbo frames?
→ Jumbo Frames do not work on Advanced Tomato (or latest shibby 140) except you adjusted something. How do you check if its working ?
2021-08-22
Kilian reporter
mhh ok. I have no problems with CTF on my side.
What’s your memory usage? Especially after a day or 2 of runtime? Is it stable over time, or is it increasing?
Can you crash the router with both test_v001 or test_v002 images ?
I haven’t waited long enough, but given they were both displaying the same pattern, it’s quite certain yes: memory usage started with both at ~40MB right after boot, and then steadily increased over time, to reach ~100MB after one hour, without starting any additional service.
What functions/features else do you use?
You can see all the processes that run on the router from the top capture I posted above. I can also share a sanitized nvram dump if it could help. Any particular variable of interest?
First time you mentioned Jumbo frames?
Yeah, I didn’t think it would make much of a difference, and just thought about it because it’s on the same page as the CTF option.
→ Jumbo Frames do not work on Advanced Tomato (or latest shibby 140) except you adjusted something. How do you check if its working ?
I’m running stock Advanced Tomato from here: https://advancedtomato.com/downloads/router/r7000 (R7000-AT-ARM-3.5-140-AIO-64K.trx) without any modification. I don’t know how to check it is really working, but what I meant is that I’ve had the “Jumbo Frames” box checked in Advanced Tomato for the last 3 years.
Thanks!
2021-08-22
Kilian reporter
Here’s an example of the memory leak, with test_v001: memory usage goes from 48k to 56k in just 10mn, without any notable network activity:
root@rtr:/tmp/home/root# while true; do printf "%s : " "$(date)"; free | awk '/Mem:/ {print $3}'; sleep 10; done
Sun Aug 22 15:51:34 PDT 2021 : 48088
Sun Aug 22 15:51:44 PDT 2021 : 48080
Sun Aug 22 15:51:54 PDT 2021 : 48840
Sun Aug 22 15:52:04 PDT 2021 : 48872
Sun Aug 22 15:52:14 PDT 2021 : 48796
Sun Aug 22 15:52:24 PDT 2021 : 48984
Sun Aug 22 15:52:34 PDT 2021 : 49036
Sun Aug 22 15:52:44 PDT 2021 : 49020
Sun Aug 22 15:52:54 PDT 2021 : 49508
Sun Aug 22 15:53:04 PDT 2021 : 49600
Sun Aug 22 15:53:14 PDT 2021 : 49696
Sun Aug 22 15:53:24 PDT 2021 : 49696
Sun Aug 22 15:53:34 PDT 2021 : 49684
Sun Aug 22 15:53:45 PDT 2021 : 49684
Sun Aug 22 15:53:55 PDT 2021 : 50492
Sun Aug 22 15:54:05 PDT 2021 : 50492
Sun Aug 22 15:54:15 PDT 2021 : 50972
Sun Aug 22 15:54:25 PDT 2021 : 50908
Sun Aug 22 15:54:35 PDT 2021 : 50904
Sun Aug 22 15:54:45 PDT 2021 : 50876
Sun Aug 22 15:54:55 PDT 2021 : 51380
Sun Aug 22 15:55:05 PDT 2021 : 51396
Sun Aug 22 15:55:15 PDT 2021 : 51460
Sun Aug 22 15:55:25 PDT 2021 : 51384
Sun Aug 22 15:55:35 PDT 2021 : 51396
Sun Aug 22 15:55:45 PDT 2021 : 51464
Sun Aug 22 15:55:55 PDT 2021 : 52036
Sun Aug 22 15:56:05 PDT 2021 : 52100
Sun Aug 22 15:56:15 PDT 2021 : 52180
Sun Aug 22 15:56:25 PDT 2021 : 52320
Sun Aug 22 15:56:35 PDT 2021 : 52324
Sun Aug 22 15:56:45 PDT 2021 : 52336
Sun Aug 22 15:56:55 PDT 2021 : 52832
Sun Aug 22 15:57:05 PDT 2021 : 52816
Sun Aug 22 15:57:15 PDT 2021 : 52804
Sun Aug 22 15:57:25 PDT 2021 : 52924
Sun Aug 22 15:57:35 PDT 2021 : 52928
Sun Aug 22 15:57:45 PDT 2021 : 52976
Sun Aug 22 15:57:55 PDT 2021 : 53660
Sun Aug 22 15:58:05 PDT 2021 : 53692
Sun Aug 22 15:58:15 PDT 2021 : 53692
Sun Aug 22 15:58:25 PDT 2021 : 53876
Sun Aug 22 15:58:35 PDT 2021 : 53804
Sun Aug 22 15:58:45 PDT 2021 : 53880
Sun Aug 22 15:58:55 PDT 2021 : 54328
Sun Aug 22 15:59:05 PDT 2021 : 54376
Sun Aug 22 15:59:15 PDT 2021 : 54420
Sun Aug 22 15:59:25 PDT 2021 : 54516
Sun Aug 22 15:59:35 PDT 2021 : 54488
Sun Aug 22 15:59:45 PDT 2021 : 54504
Sun Aug 22 15:59:55 PDT 2021 : 55220
Sun Aug 22 16:00:05 PDT 2021 : 55244
Sun Aug 22 16:00:15 PDT 2021 : 55220
Sun Aug 22 16:00:25 PDT 2021 : 55376
Sun Aug 22 16:00:35 PDT 2021 : 55444
Sun Aug 22 16:00:45 PDT 2021 : 55440
Sun Aug 22 16:00:55 PDT 2021 : 55952
Sun Aug 22 16:01:05 PDT 2021 : 55952
Sun Aug 22 16:01:15 PDT 2021 : 56112
Sun Aug 22 16:01:25 PDT 2021 : 56076
^C
root@rtr:/tmp/home/root# grep AIO /etc/motd
FreshTomato 2021.5 -15082021-ctf_test_v001 K26ARM USB AIO-64K
and here’s the comparison between global memory usage, and the sum of memory usage of all user-land processes:
root@rtr:/tmp/home/root# while true; do printf "%s | total used: %s | sum apps : %s\n" "$(date)" "$(free | awk '/Mem:/ {print $3}')" "$(top -b -n1 | awk '/root|nobody/ {s+=$5} END {print s}')"; sleep 10; done
Sun Aug 22 16:17:08 PDT 2021 | total used: 71260 | sum apps : 55952
Sun Aug 22 16:17:18 PDT 2021 | total used: 71392 | sum apps : 55952
Sun Aug 22 16:17:28 PDT 2021 | total used: 71304 | sum apps : 55952
Sun Aug 22 16:17:38 PDT 2021 | total used: 71772 | sum apps : 55952
Sun Aug 22 16:17:48 PDT 2021 | total used: 72008 | sum apps : 55952
Sun Aug 22 16:17:59 PDT 2021 | total used: 73044 | sum apps : 55976
Sun Aug 22 16:18:09 PDT 2021 | total used: 73044 | sum apps : 55976
Sun Aug 22 16:18:19 PDT 2021 | total used: 73044 | sum apps : 55976
Sun Aug 22 16:18:29 PDT 2021 | total used: 73152 | sum apps : 55976
Sun Aug 22 16:18:39 PDT 2021 | total used: 73204 | sum apps : 55976
Sun Aug 22 16:18:49 PDT 2021 | total used: 73696 | sum apps : 55976
Sun Aug 22 16:18:59 PDT 2021 | total used: 73692 | sum apps : 55976
Sun Aug 22 16:19:10 PDT 2021 | total used: 73696 | sum apps : 55976
Sun Aug 22 16:19:20 PDT 2021 | total used: 73868 | sum apps : 55976
Sun Aug 22 16:19:30 PDT 2021 | total used: 73820 | sum apps : 55976
Sun Aug 22 16:19:40 PDT 2021 | total used: 74164 | sum apps : 55976
Sun Aug 22 16:19:50 PDT 2021 | total used: 74496 | sum apps : 55976
Sun Aug 22 16:20:00 PDT 2021 | total used: 74616 | sum apps : 55976
Sun Aug 22 16:20:10 PDT 2021 | total used: 74816 | sum apps : 55976
Sun Aug 22 16:20:20 PDT 2021 | total used: 74812 | sum apps : 55976
Sun Aug 22 16:20:31 PDT 2021 | total used: 74692 | sum apps : 55976
Sun Aug 22 16:20:41 PDT 2021 | total used: 74784 | sum apps : 55976
Sun Aug 22 16:20:51 PDT 2021 | total used: 75296 | sum apps : 55976
Sun Aug 22 16:21:01 PDT 2021 | total used: 75276 | sum apps : 55976
Sun Aug 22 16:21:11 PDT 2021 | total used: 75232 | sum apps : 55976
Sun Aug 22 16:21:21 PDT 2021 | total used: 75264 | sum apps : 55976
Sun Aug 22 16:21:31 PDT 2021 | total used: 75380 | sum apps : 55976
Sun Aug 22 16:21:41 PDT 2021 | total used: 75504 | sum apps : 55980
Sun Aug 22 16:21:52 PDT 2021 | total used: 76264 | sum apps : 55980
Sun Aug 22 16:22:02 PDT 2021 | total used: 76180 | sum apps : 55980
Sun Aug 22 16:22:12 PDT 2021 | total used: 76212 | sum apps : 55980
You can see that the user-land applications memory footprint stays globally stable, while the overall memory usage keeps climbing. Which likely indicates that the kernel is the source of usage increase.
2021-08-22
Kilian reporter
Here are the exact same measurements with CTF disabled (still used test_v001)
root@rtr:/tmp/home/root# while true; do printf "%s | total used: %s | sum apps : %s\n" "$(date)" "$(free | awk '/Mem:/ {print $3}')" "$(top -b -n1 | awk '/root|nobody/ {s+=$5} END {print s}')"; sleep 10; done
Sun Aug 22 16:56:23 PDT 2021 | total used: 36360 | sum apps : 55944
Sun Aug 22 16:56:33 PDT 2021 | total used: 36304 | sum apps : 55944
Sun Aug 22 16:56:43 PDT 2021 | total used: 36364 | sum apps : 55944
Sun Aug 22 16:56:53 PDT 2021 | total used: 36364 | sum apps : 55944
Sun Aug 22 16:57:04 PDT 2021 | total used: 36364 | sum apps : 55968
Sun Aug 22 16:57:14 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 16:57:24 PDT 2021 | total used: 36420 | sum apps : 55968
Sun Aug 22 16:57:34 PDT 2021 | total used: 36364 | sum apps : 55968
Sun Aug 22 16:57:44 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 16:57:54 PDT 2021 | total used: 36468 | sum apps : 55968
Sun Aug 22 16:58:04 PDT 2021 | total used: 36408 | sum apps : 55968
Sun Aug 22 16:58:14 PDT 2021 | total used: 36364 | sum apps : 55968
Sun Aug 22 16:58:25 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 16:58:35 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 16:58:45 PDT 2021 | total used: 36436 | sum apps : 55968
Sun Aug 22 16:58:55 PDT 2021 | total used: 36364 | sum apps : 55968
Sun Aug 22 16:59:05 PDT 2021 | total used: 36408 | sum apps : 55968
Sun Aug 22 16:59:15 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 16:59:25 PDT 2021 | total used: 36420 | sum apps : 55968
Sun Aug 22 16:59:36 PDT 2021 | total used: 36424 | sum apps : 55968
Sun Aug 22 16:59:46 PDT 2021 | total used: 36408 | sum apps : 55968
Sun Aug 22 16:59:56 PDT 2021 | total used: 36428 | sum apps : 55968
Sun Aug 22 17:00:06 PDT 2021 | total used: 36500 | sum apps : 55968
Sun Aug 22 17:00:16 PDT 2021 | total used: 36508 | sum apps : 55968
Sun Aug 22 17:00:26 PDT 2021 | total used: 36492 | sum apps : 55968
Sun Aug 22 17:00:36 PDT 2021 | total used: 36420 | sum apps : 55968
Sun Aug 22 17:00:46 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:00:57 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:01:07 PDT 2021 | total used: 36520 | sum apps : 55968
Sun Aug 22 17:01:17 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:01:27 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:01:37 PDT 2021 | total used: 36464 | sum apps : 55968
Sun Aug 22 17:01:47 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:01:57 PDT 2021 | total used: 36476 | sum apps : 55968
Sun Aug 22 17:02:07 PDT 2021 | total used: 36476 | sum apps : 55968
Sun Aug 22 17:02:18 PDT 2021 | total used: 36460 | sum apps : 55968
Sun Aug 22 17:02:28 PDT 2021 | total used: 36460 | sum apps : 55968
Sun Aug 22 17:02:38 PDT 2021 | total used: 36404 | sum apps : 55968
Sun Aug 22 17:02:48 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:02:58 PDT 2021 | total used: 36460 | sum apps : 55968
Sun Aug 22 17:03:08 PDT 2021 | total used: 36416 | sum apps : 55968
Sun Aug 22 17:03:18 PDT 2021 | total used: 36420 | sum apps : 55968
Sun Aug 22 17:03:28 PDT 2021 | total used: 36464 | sum apps : 55968
Sun Aug 22 17:03:39 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 17:03:49 PDT 2021 | total used: 36376 | sum apps : 55968
Sun Aug 22 17:03:59 PDT 2021 | total used: 36452 | sum apps : 55968
Sun Aug 22 17:04:09 PDT 2021 | total used: 36448 | sum apps : 55968
Sun Aug 22 17:04:19 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 17:04:29 PDT 2021 | total used: 36420 | sum apps : 55968
Sun Aug 22 17:04:39 PDT 2021 | total used: 36408 | sum apps : 55968
Sun Aug 22 17:04:49 PDT 2021 | total used: 36376 | sum apps : 55968
Sun Aug 22 17:05:00 PDT 2021 | total used: 36336 | sum apps : 55968
Sun Aug 22 17:05:10 PDT 2021 | total used: 36392 | sum apps : 55968
Sun Aug 22 17:05:20 PDT 2021 | total used: 36276 | sum apps : 55968
Sun Aug 22 17:05:30 PDT 2021 | total used: 36336 | sum apps : 55968
Sun Aug 22 17:05:40 PDT 2021 | total used: 36220 | sum apps : 55968
Sun Aug 22 17:05:50 PDT 2021 | total used: 36276 | sum apps : 55968
Sun Aug 22 17:06:00 PDT 2021 | total used: 36188 | sum apps : 55968
Sun Aug 22 17:06:10 PDT 2021 | total used: 36288 | sum apps : 55968
Sun Aug 22 17:06:21 PDT 2021 | total used: 36188 | sum apps : 55968
Here, the total memory usage stays stable at around 36k. The sum of memory usage for the user-land applications is very similar as with CTF enabled (55-56k).
So I guess the logical conclusion is that the memory leak comes from the kernel, and happens when CTF is enabled.
Doesn’t mean that CTF is the direct culprit, though, it may just be the enabling factor. I found a number of very similar memory leak reports with the stock R7000 firmware, like this: https://community.netgear.com/t5/Nighthawk-WiFi-Routers/Nighthawk-R7000-Firmware-1-0-9-64-memory-leak-lose-internet-and/m-p/1713956
There’s also this, which seem to incriminate the 5GHz radio: https://forum.dd-wrt.com/phpBB2/viewtopic.php?p=1172469 (but I tried the suggested workaround of disabling and re-enabling the 5GHz radio right after boot, but it didn’t help with this memory leak, unfortunately).
2021-08-22
Kilian reporter
Is there any other information I could provide to help diagnose, and hopefully fix that memory leak?
2021-09-07
Jerry
Same issue on 2021.8.
EA6700, normal starts with round 24MB memory usage, and raised to 160MB after large file transferring with CTF enabled. Then, all connections are down include 2.4 & 5G wifi. Router stuck, only cold reboot helps.
But tested with CTF disabled, all goes well, even if more than 30GB files are transferred to NAS. Memory still stays on 25MB or around.
2022-01-14
Jerry
BTW, I tested on cleaned NVRAM, with CTF on, others in default, no leak at all.
Then, I set other services on for normally use, pppoe, PPTP server, port forwarding including NAS port forwards. Memory leaked when I transferred some movies to NAS, files over 20GB.
Can not tell the bug exactly, seems like it relates not only CTF option, should be some combinations with multiple options.
I now changed to DD-WRT. Some web services need to go on.
2022-01-15
M_ars
Can you repeate with minimum config and find the combination with CTF? File Transfer from where to where using pppoe or pptp or … ?
2022-01-16
Jerry
I had been using 2021.8 for weeks, and versions bellow before. Never encountered the bug so badly until PPTP server was enabled. I did not watch the memory usage before because the router still worked.
CTF was always on, transferred movies from laptop to NAS under the same router, exactly, from 192.168.0.10 to 192.168.0.12.
But still need to find the minimal combination. CTF and PPTP server are the settings need to check. I’ll test it some day.
2022-01-16
Shahnewaz Ahmed
I thought my router was busted till I enabled the logs and found the same kernel warnings and memory leak as mentioned above.
For me I recently enabled PPTP Server on my side, and the memory leaks started shortly thereafter. Could be coincidence, but I can certainly do some testing.
FreshTomato version: 2021.8
2022-02-13
M_ars
Please test again with PPTP Server disabled. thx
2022-02-14
M_ars
what router do you use? I would like to test a small change
2022-02-14
Jerry
It reminds me that, it did not happen until PPTP server was on. But, sorry, I have been looking for a job for months, not tested it up till now.
2022-02-14
Shahnewaz Ahmed
I am using the same router as the reporter (Netgear R7000). I would be interested in testing.
2022-02-14
M_ars
for testing - small change to bypass CTF
https://bitbucket.org/M_ars/freshtomato-arm/downloads/freshtomato-R7000-ARM_NG-2021.8-07022022_pptp_bypass_v001-VPN-64K.trx
2022-02-15
Shahnewaz Ahmed
Just tested it for about a day. The memory usage creeps up more slowly, but by the next night, it reaches capacity and leads to the same errors as shown above.
2022-02-17
M_ars
thx for feedback - next attempt
please test “v002”
https://bitbucket.org/M_ars/freshtomato-arm/downloads/freshtomato-R7000-ARM_NG-2021.8-07022022_pptp_bypass_v002-VPN-64K.trx
2022-02-18
Shahnewaz Ahmed
I am dirty flashing these builds, would that hamper testing or is it okay?
For this build, same result. Memory usage creeps up till the router chokes.
2022-02-19
M_ars
I am dirty flashing these builds, would that hamper testing or is it okay?
thats ok - no need to clean nvram.
thx for feedback. further investigation needed for pptp server code
Support for PPP over IPv4.(Point-to-Point Tunneling Protocol)
This driver requires pppd plugin to work in client mode or
modified pptpd (poptop) to work in server mode.
See http://accel-pptp.sourceforge.net/ for information how to
utilize this module.
2022-02-21
Jerry
Confirmed, CTF on and PPTP server on will lead to memory leak. Only CTF on, all is OK.
2022-02-27
M_ars
thx for feedback. I see one patch from linksys src. it may can help here. Will add it later
2022-02-28
pedro repo owner
changed title to PPTP server with CTF enabled causes kernel memory leak?
2022-05-22
Marius D
i would remove pptp support in favor of wireguard - no need to ipsec either
2022-08-02
Marius D
@sysres - wireguard is available in FT ARM https://www.linksysinfo.org/index.php?threads/wireguard-on-freshtomato.76295/ but ipsec it’s not (by default - needs extra modules)
2022-08-04
Marius D
I believe this bug should be closed with “won’t fix” - to force users off pptp.
Of course PR/MR from any pptp user are welcome
2022-08-05
Kilian reporter
I agree that having a GUI interface to configure WiregGuard would be the ideal way forward.
Now, PPTP tunnels are already deprecated and can't be used on Android anymore, so I expect the user base to dwindle over time.
In the meantime, maybe displaying a warning on the PPTP server page when CTF is enabled (or just plain disable the option) would be a good start? A bit like the existing warnings for CTF and QOS.
2022-08-05
M_ars
Can someone please test 2022.5 release for this topic/problem.
→ There have been a lot of kernel updates and also to pptp code
thx & BR
2022-08-06
Jerry
Just tested, still exists in 2022.5.
2022-09-30
M_ars
thx for testing
2022-09-30
Jerry
But seems going more slowly than versions before.
Coping 6G data to other PC via local network, with PPTP on and CTF on, memory rises to 90MB from 24MB. But with PPTP on, CTF off, it stays on 24-25MB.
2022-09-30
Jerry
Still exists in 2024.1
2024-04-05
pedro repo owner
Because we have now newer pptpd 1.5.0 and pppd 2.5.1 in repo can someone test @M_ars images from here (or ask for a new one)?: https://www.linksysinfo.org/index.php?threads/freshtomato-arm-mips-ax-images-for-testing-latest-branch.78916/
2024-11-13
Jerry
Still exists in 2024.5.
Running PPTP Server with CTF on, memory usage goes from 33MB to 156MB just in few hours when copying large files via SMB. Then, my EA 6900 died in minutes. WIFI is down, failed to login, lost IP, etc…
2024-12-30
The text was updated successfully, but these errors were encountered:
Ref. https://bitbucket.org/pedro311/freshtomato-arm/issues/147/pptp-server-with-ctf-enabled-causes-kernel
Kilian created an issue 2021-08-16
Hi!
FW: 2021.5 (2021.08.14) freshtomato-R7000-ARM_NG-2021.5-AIO-64K.trx
Model: netgear R7000
I just moved from AdvancedTomato to FreshTomato a couple days ago. I’ve made sure to check the “Erase all data in NVRAM” when flashing the FreshTomato firmware and re-did all my configuration by hand. Everything seemed to work fine for a few hours, but then at some time, I lost wifi and couldn’t connect anymore and no traffic was going through. I power-cycled the router, and it started working again, until the same thing happened. Over the course of 2 days, I had to power-cycle the router 5 of 6 times.
I setup a remote syslog server to get some details, and it looks like memory exhaustion.
I found traces like this:
or
and
Right after boot, the router uses about 40MB of RAM, but when the problem happens, it quickly jumps to over 160MB and the system freezes from there. I’ve checked the list of processes when the problem starts occurring, but couldn't determine the culprit: the sum of individual processes' memory usage doesn’t account for all the memory used, which makes think that this is happening at the kernel level.
I’m not sure if it’s related, but I have CTF enabled.
I had to go back to AdvancedTomato for now, to get a stable connection again, but I thought I would report this, in case it could help.
Comments (38)
The text was updated successfully, but these errors were encountered: