Network | Connection test timeout

[1985kasper]

Creating a bug report/issue

Details:

• Date | Sun May 19 17:08:23 CEST 2019
• Bug report | N/A
• DietPi version | v6.20.0 (Fourdee/master)
• Img creator | DietPi Core Team
• Pre-image | Raspbian Lite
• SBC device | RPi 3 Model B+ (armv7l) (index=3)
• Kernel version | Letsencrypt supports Free Noip.com Dynamic DNS #1159 SMP Sun Nov 4 17:50:20 GMT 2018
• Distro | stretch (index=4)
• Command | Connection test: http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian
• Exit code | 4
• Software title | DietPi-Login

Additional Information (if applicable)

• Software title | Dietpi
• Was the software title installed freshly or updated/migrated? Fresh image downloaded today. Rpi 3 B+
• Can this issue be replicated on a fresh installation of DietPi?

• Bug report ID | sed -n 5p /DietPi/dietpi/.hw_model

Steps to reproduce

1. Bootup a rpi with a fresh image
2. setup fails because of connection errors with the mirrors

Expected behaviour

• setup should continue without error

Actual behaviour

• setup stops

Extra details

• 3. tried global and UK mirror to rule out mirror being down

4. tried wifi/ethernet DHCP and static with google dns open dns and custom dns set to ISP
5. able to resolve websites through wget
6. internet connection test through dietpi menus fails
7. apt-get update is working as expected

[MichaIng]

@1985kasper
Please test how long the connection test would take:

time wget --spider http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian

Our default timeout is 5s which should be more than sufficient. But lately we received multiple reports where it was not, with connection times 6-15 seconds and such. Totally unacceptable an no idea so far where this comes from, perhaps some bad package update, not sure.

For this we implemented the possibility to manually adjust the timeout, sadly with v6.23. So you need to manually raise it:

sed -i 's/timeout:-5/timeout:-20/' /DietPi/dietpi/func/dietpi-globals

Then rerun: /DietPi/dietpi/login

Since the update includes a kernel upgrade to v4.19, it would be interesting if that solved the issue, who knows. So retest connection time:

time wget --spider http://mirror.ox.ac.uk/sites/archive.raspbian.org/archive/raspbian

If it is still > 5 seconds, adjust it in: dietpi-config > Network Options: Misc

[1985kasper]

jep tried with ftp.mirror.transip.net/raspbian/raspbian (since i am in NL i thought i might as well take the closest one). Can't copy paste the result cause i am doing this physicly with screen/keyboard but it's 0m5.296s so indeed just over 5s

[MichaIng]

@1985kasper
I guess it will be the same for different mirrors then as well, but at least a method to compare them. However the solution above should work around the issue and perhaps the kernel upgrade or some other package one fixes the issue magically 😉.

[1985kasper]

Yeah it fixed it! It turned me crazy! thought my network was failing me...

btw offtopic: I strongly believe in security. The current root enabled wide open default setup is a bit dangerous for novice users enthusiastically opening up their port 22 to the internet for example.

Do you mind if i open a feauture request to suggest some hardening? Maybe this can be an optional choice at first boot. But i do believe root should be disables at all times and replaced by a sudo user etc. just basic security 101

[1985kasper]

The same failure is now happening for github:

https://raw.githubusercontent.com/MichaIng/DietPi/master/dietpi.txt

Adjusted timeout time to 15 seconds in the config network misc but not sure if it applies to github as well as the mirror.
Suggestion to do it manually?

[MichaIng]

@1985kasper
But the update went through regardless right?
EDIT: Ah makes sense that the same error re-appears, since the update overwrites the manually applied code change 😉. However no big issue.

Please apply the step that was failing manually (includes timeout raise to 20s, just to be failsafe):

sudo -s # sudo shell required to run G_CONFIG_INJECT commands
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_ATTEMPTS=' 'CONFIG_G_CHECK_URL_ATTEMPTS=3' /DietPi/dietpi.txt
exit # exit sudo shell
sudo /DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt

Then re-run the update, just to be failsafe: dietpi-update -1

[1985kasper]

This was a first boot of a fresh image. But the setup continued after showing the error.

[MichaIng]

I think I have to collect all reports about >5s timeouts and see if we find any similarities. If I remember right it is actually the DNS resolving step, not the connection to the bare IP itself. But trying out different DNS nameservers didn't change it. So anything related to the DNS resolving mechanism system wise must be going wrong. But I could never replicate it, have connections times < 0.5s here on all systems and machines... 🤔

---

About your sudo suggestion:

• Actually I never really understood the practical benefit of logging in as sudo capable user vs root.
• With root disabled, when an attacker hacks the sudo capable user, it can just do exactly the same...
• So actually sudo gives for my impression a false security feeling. Users tend to use less strong passwords and less secure authentication mechanisms for their users and such.
• So I prefer to use root user only (also for comfort) and use a hardened SSH authentication via passphrased key, instead of user/pw.
• And about SSH in general: If a user forwards port 22, then (s)he should know exactly what this means anyway. And I would anyway never recommend to use port 22, since you will just face Chinese brute-force attempts all the day, blowing your logs. Better use any random port and forward network internally to port 22 of the server. This in combination with key authentication makes your SSH bomb safe, while disabling root has practically IMO not any security benefit.
• Ahh, not to forget fail2ban as a MUST have then 😉.
• And actually I would never forward anything to internal port 22. When you REALLY need to SSH from outside your local network, use a VPN server!
• The ONLY benefit of using non-root IMO is, that you cannot that easily accidentally destroy your system, e.g. doing a mistype like rm -R /* while rm -R ./* was meant or such 😆.

Perhaps we should collect such security relevant infos and make them available for users, either within the DietPi system on first run setup or at our online docs.

[MichaIng]

Made a start with security recommendations: https://github.com/MichaIng/DietPi/wiki/Security-recommendation

[1985kasper]

Awesome! I understand your argument against sudo. It is just a safety precaution i guess. the root user is the first try for hackers. By disabling root login on outside world facing machines to a random username the chances of getting in go down for random attackers. It's the same with bike locks and front door locks. Someone that wants to get in will always find a way. However the more time it takes the more annoyed they get the sooner they'll give up.
And regarding the port 22. I often see people asking questions on messageboards/forums etc. Imagine someone giving opening port 22 as advice (yes this does happen in the real world) without the user knowing what they're doing.

As an addition i would say google authentication is a nice option to have setup in addition to password and keys. As far as i was able to find this does require a SSH server change to opensshd from dropbear.

I do just use 22 and don't bother with changing the port BUT use key+password+google authenticator. and of course fail2ban

[MichaIng]

@1985kasper

the root user is the first try for hackers

Hmm yeah that is actually true. The same argument than for using a different external SSH port. Although I never feared that one could hit my password, especially since I use key authentication with passphrase, I was just annoyed by those attempts filling my logs. I literally never faced any single brute-force attempts (at least based on regular check/review) after switching to a random port for ~1.5 years. Now I have not opened SSH from web at all, since I simply don't require it. However IF some bot tries your SSH port, then in most cases it will try as root user, that is true!

Okay but for completeness indeed then it makes sense to add the recommendation to disable root user login (at least for SSH) completely when opening it to web, even that key authentication makes bot brute-force success practically impossible already.

About Two-factor authentication:

• Of course a second requirement further enhances security.
• I just don't like the fact that you rely on an external provider (and proprietary method) in most cases.
• And it is completely outside of comfort.
• I personally like to keep it simple, using the FOSS UNIX internal features, also to have no external provider general access meta-data. Practically those make random attack success impossible already.
• If you fear targeted attacks, e.g. employees that take care company secrets and are target of industrial espionage, or political oppositions in countries with suppressive government , then the topic is again a totally different one. But I think there is no alternative to VPN then with a well thought-through key exchange and update schedule + company/network-internally handled 2-factor authentication.

[1985kasper]

I don't open up 22 on all my LAN machines. I use one machine (in this case a rpi running dietpi that is why i brought it up :-)) a rpi purely because it is the most energy efficient i had available. This is my "jumphost" and the only one that can be accessed from 1 external IP, a cloud VPS. Both the Cloud VPS and the rpi are setup with key+2fa+password.
This is an alternative to vpn because vpn is a pain to setup (it is for example also not allowed on my work workstation) if you need it on different machines. Now i can just quickly ssh into my lan to reboot something when i am at a friends house or something. I can also use portforwarding with SSH to access any web apps running at home that i don't want to open directly without the need for a VPN
If the cloud vps gets compromised i simply destroy it and provision a new one in 10 minutes

[MichaIng]

@1985kasper
Sounds reasonable. DietPi-Software should provide a moreless painless setup of VPN, especially with WireGuard (which makes setup generally much easier than OpenVPN does). Theoretically if some network does not allow the default VPN port, one can change it to e.g. 22 or 443 or whatever 😄. But yeah, already the need to install and configure a client on all machines where you want to access from is a hurdle. WireGuard just a week ago or such released a Windows client as experimental/beta, so this was not even possible desktops/notebooks before. But e.g. the Android client works quite well according to my tests. You can easily configure it to only tunnel access to the servers local network while regular internet access remains outside the VPN. So it can simply stay enabled on the phone.

[1985kasper]

Maybe if the Windows version has that option it would be nice. But because the workstation is already on a VPN to access certain work systems i can't risk not being able to access those anymore. And unfortunately some stuff just doesn't work well on a phone for example my routers GUI and apps such as sonarr/radarr don't display very nice on my phone.
Same goes for my NAS, there's an app but the functionality is limited. To avoid opening the NAS directly SSH port forwarding is a nice alternative :-)

But anyway it is not so much work to turn a dietpi rpi into the wayi want it to use the above setup. My comment was just more because there might be users that might not understand the power of root and the dangers that come with it especially in combination with a open SSH port.

Also i would say ufw is a nice alternative for iptables for the simple stuff. iptables might be throwing people off. It's a pain to remember the syntax even when using it daily :-)

[MichaIng]

@1985kasper
Okay, back to topic: DietPi-Update finally succeeded with the above steps? #2831 (comment)

[1985kasper]

Yes it all went through. Didn't actually have to do any additional steps. After the github failure it worked eventually.

I am on 6.24 now. So all looks good

[MichaIng]

@1985kasper
You should still apply the failing step:

sudo -s # sudo shell required to run G_CONFIG_INJECT commands
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_TIMEOUT=' 'CONFIG_G_CHECK_URL_TIMEOUT=20' /DietPi/dietpi.txt
G_CONFIG_INJECT 'CONFIG_G_CHECK_URL_ATTEMPTS=' 'CONFIG_G_CHECK_URL_ATTEMPTS=3' /DietPi/dietpi.txt
exit # exit sudo shell
sudo /DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt

It is a snake biting it's own tail:

• With v6.23 we introduced the new custom connection timeout settings, exactly due to multiple reports with long connection times, like in your case.
• Since the new code got downloaded, it expects these settings now.
• Since due to connection timeout, adding these new settings failed, DietPi scripts will fail to do connection tests. EDIT: Ah no actually it is handles failsafe: When the settings do not exist, it will fall back to default 5 seconds at least. However since this might be not sufficient....
• So /DietPi/dietpi/func/dietpi-set_software verify_dietpi.txt which aims to add new dietpi.txt settings in general, fails as well 😄.
• So add them manually, then run the script to add further new settings.

[1985kasper]

I ran the code. No errors so all seems to be in order. Thanks for the very nice and quick help!

Can you possibly looking at my comment on the vlan issue commit aa4a36f? Not entirely sure how it's supposed to work with these drop-ins

[MichaIng]

@1985kasper
Okay I mark this issue as closed then. Feel free to reopen if required. I will have a look at the VLAN topic.

[1985kasper]

Actually i am not sure if this is because of the commands above or a seperate issue. But PHP (installed during the intallation of pihole PRIOR to running these commands) seems to have disappeared. Pihole gives a 503 unavailable error and pihole -d (debug) hangs at php version
At boot there is also a php related issue

[MichaIng]

@1985kasper

journalctl -u php7.3-fpm
journalctl -u lighttpd
cat /var/log/php7.3-fpm.log
cat /var/log/lighttpd/error.log

You are using Lighttpd, right?

[1985kasper]

Yes it's lighthttpd

DietPi:~ $ sudo journalctl -u php7.3-fpm
-- Logs begin at Sun 2019-05-19 22:33:39 CEST, end at Sun 2019-05-19 22:43:33 CEST. --
May 19 22:34:14 DietPi systemd[1]: Starting The PHP 7.3 FastCGI Process Manager...
May 19 22:35:44 DietPi systemd[1]: php7.3-fpm.service: Start operation timed out. Terminating.
May 19 22:35:44 DietPi systemd[1]: Failed to start The PHP 7.3 FastCGI Process Manager.
May 19 22:35:44 DietPi systemd[1]: php7.3-fpm.service: Unit entered failed state.
May 19 22:35:44 DietPi systemd[1]: php7.3-fpm.service: Failed with result 'timeout'.

DietPi:~ $ sudo journalctl -u lighttpd
-- Logs begin at Sun 2019-05-19 22:33:39 CEST, end at Sun 2019-05-19 22:44:09 CEST. --
May 19 22:35:44 DietPi systemd[1]: Starting Lighttpd Daemon...
May 19 22:35:45 DietPi systemd[1]: Started Lighttpd Daemon.

@DietPi:~ $ sudo cat /var/log/lighttpd/error.log
2019-05-19 22:35:45: (log.c.217) server started
2019-05-19 22:35:59: (mod_fastcgi.c.1926) connect failed: No such file or directory on unix:/run/php/php7.3-fpm.sock
2019-05-19 22:35:59: (mod_fastcgi.c.2842) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
2019-05-19 22:35:59: (mod_fastcgi.c.1689) all handlers for /admin/index.php? on .php are down.
2019-05-19 22:36:01: (mod_fastcgi.c.2686) fcgi-server re-enabled: unix:/run/php/php7.3-fpm.sock
2019-05-19 22:36:08: (mod_fastcgi.c.1926) connect failed: No such file or directory on unix:/run/php/php7.3-fpm.sock
2019-05-19 22:36:08: (mod_fastcgi.c.2842) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
2019-05-19 22:36:08: (mod_fastcgi.c.1689) all handlers for /admin/index.php? on .php are down.
2019-05-19 22:36:10: (mod_fastcgi.c.2686) fcgi-server re-enabled: unix:/run/php/php7.3-fpm.sock
2019-05-19 22:39:25: (mod_fastcgi.c.1926) connect failed: No such file or directory on unix:/run/php/php7.3-fpm.sock
2019-05-19 22:39:25: (mod_fastcgi.c.2842) backend died; we'll disable it for 1 seconds and send the request to another backend instead: reconnects: 0 load: 1
2019-05-19 22:39:25: (mod_fastcgi.c.1689) all handlers for /index.php? on .php are down.
2019-05-19 22:39:27: (mod_fastcgi.c.2686) fcgi-server re-enabled: unix:/run/php/php7.3-fpm.sock

[MichaIng]

@1985kasper
/var/log/php7.3-fpm.log does not exist?

Does systemctl restart php7.3-fpm finally succeed but take a long time?
In case, check out: #2806

• On some systems (especially headless ones) since a while entropy level seems to be low. Note sure where this comes from, perhaps some systemd update a while ago or such.
• Entropy pool can be filled by HAVEGE: G_AGI haveged
• Since in the other case, as well php7.3-fpm failed to start with timeout, perhaps it is for the same reason.

[1985kasper]

Well got it fixed by a purge of PHP and reinstalling. What do you mean by entropy in this context?

[1985kasper]

Actually... a reboot killed it again. I'll try the entropy thing

[MichaIng]

@1985kasper

Well got it fixed by a purge of PHP and reinstalling.

Ah okay great.... EDIT: ...not. Okay at least it looks very similar to the other issue.

What do you mean by entropy in this context?

A source for getting random strings basically, required for e.g. cryptography tasks and such. Can be retrieved via /dev/random or /dev/urandom. I also don't know much details about it, but generally this is used by many things. And hangs on boot + some errors messages on boot, containing "random" or failing/long taking service (re)starts where reported and solved by installing the haveged package.

[1985kasper]

@DietPi:~ $ sudo systemctl restart php7.3-fpm
[sudo] password for :
Job for php7.3-fpm.service failed because a timeout was exceeded.
See "systemctl status php7.3-fpm.service" and "journalctl -xe" for details.
@DietPi:~ $ G_AGI haveged
[FAILED] Root privileges required. Please run the command with "sudo" or "G_SUDO".

@DietPi:~ $ sudo G_AGI haveged
sudo: G_AGI: command not found

[MichaIng]

@1985kasper
Looks like you used an "a" inside where it should be an "e"? haveged

The G_* commands don't work with sudo, because global functions are only loaded in interactive shells.
Use: G_SUDO G_AGI haveged
Or simply: sudo apt install haveged

---

Ah lol indeed it seems the package is not available on Raspbian... Ehm let me check the other topic again, I am pretty sure it was RPi as well. On Debian it's available: https://packages.debian.org/stretch/haveged EDIT: Ah just forgot to run apt update after reboot (APT lists moved to RAM)

[1985kasper]

Woops... been at it all day... eyes are getting tired. OK it's installed now