Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

keepalived crashed when reloading, thanks. #118

Closed
icymoon opened this issue Jan 29, 2018 · 5 comments
Closed

keepalived crashed when reloading, thanks. #118

icymoon opened this issue Jan 29, 2018 · 5 comments
Assignees
@mscbg

Comments

@icymoon
Copy link
Collaborator

icymoon commented Jan 29, 2018 •
edited by beacer
Loading

Start keepalived with keepalived.conf.1 and reload it with keepalived.conf.2.
It will crashed with segment fault.

# cat /etc/keepalived/keepalived.conf.1
! Configuration File for keepalived

global_defs {
    router_id DPVS_TESTING
}

local_address_group laddr_g1 {
    172.16.10.10 dpdk0    # use DPDK interface
    172.16.10.11 dpdk0    # use DPDK interface
    172.16.10.12 dpdk0    # use DPDK interface
    172.16.10.13 dpdk0    # use DPDK interface
    172.16.10.14 dpdk0    # use DPDK interface
}

#
# Virtual Server Section
#

virtual_server_group 192.168.10.x_http2 {
    192.168.10.5 80
}

virtual_server_group 192.168.10.x_dns1 {
    192.168.10.4 53
}
virtual_server_group 192.168.10.x_dns2 {
    192.168.10.5 53
}

virtual_server group 192.168.10.x_dns1 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol TCP       # Protocol TCP
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 53 { # real-server
        weight 100
        inhibit_on_failure
       MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.3"
           misc_timeout 3
       }
    }
   real_server 172.16.10.4 53 { # real-server
       weight 100
       inhibit_on_failure
       MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.4"
           misc_timeout 5
       }
  }
}
virtual_server group 192.168.10.x_dns2 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol UDP       # Protocol TCP
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 53 { # real-server
        weight 100
        inhibit_on_failure
        MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.3"
           misc_timeout 5
       }
   }
   real_server 172.16.10.4 53 { # real-server
       weight 100
       inhibit_on_failure
       MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.4"
           misc_timeout 5
       }
  }
}
virtual_server group 192.168.10.x_http1 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol TCP       # Protocol TCP
    !syn_proxy
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.3 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.3 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
    real_server 172.16.10.4 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.4 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.4 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
}
virtual_server group 192.168.10.x_http2 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol TCP       # Protocol TCP
    syn_proxy
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.3 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.3 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
    real_server 172.16.10.4 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.4 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.4 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
}

# cat /etc/keepalived/keepalived.conf.2
! Configuration File for keepalived

global_defs {
    router_id DPVS_TESTING
}

local_address_group laddr_g1 {
    172.16.10.10 dpdk0    # use DPDK interface
    172.16.10.11 dpdk0    # use DPDK interface
    172.16.10.12 dpdk0    # use DPDK interface
    172.16.10.13 dpdk0    # use DPDK interface
    172.16.10.14 dpdk0    # use DPDK interface
}

#
# Virtual Server Section
#
virtual_server_group 192.168.10.x_http1 {
    192.168.10.4 80
}
virtual_server_group 192.168.10.x_http2 {
    92.168.10.5 80
}

virtual_server_group 192.168.10.x_dns1 {
    92.168.10.xss 53
}
virtual_server_group 192.168.10.x_dns2 {
    192.168.10.5 53
}

virtual_server group 192.168.10.x_dns1 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol UDP       # Protocol TCP
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 53 { # real-server
        weight 100
        inhibit_on_failure
       MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.3"
           misc_timeout 5
       }
    }
   real_server 172.16.10.4 53 { # real-server
       weight 100
       inhibit_on_failure
       MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.4"
           misc_timeout 5
       }
  }
}
virtual_server group 192.168.10.x_dns2 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol UDP       # Protocol TCP
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 53 { # real-server
        weight 100
        inhibit_on_failure
        MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.3"
           misc_timeout 5
       }
   }
   real_server 172.16.10.4 53 { # real-server
       weight 100
       inhibit_on_failure
       MISC_CHECK {
           misc_path "nslookup tdomain.cn 172.16.10.4"
           misc_timeout 5
       }
  }
}
virtual_server group 192.168.10.x_http1 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol TCP       # Protocol TCP
    !syn_proxy
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.3 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.3 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
    real_server 172.16.10.4 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.4 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.4 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
}
virtual_server group 192.168.10.x_http2 {
    delay_loop 3
    lb_algo rr         # scheduling algorithm Round-Robin
    lb_kind FNAT       # Forwarding Mode Full-NAT
    protocol TCP       # Protocol TCP
    syn_proxy
    alpha

    laddr_group_name laddr_g1   # Local IP group-ID

    real_server 172.16.10.3 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.3 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.3 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
    real_server 172.16.10.4 8080 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8080
        }
    }
    real_server 172.16.10.4 8081 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8081
        }
    }
    real_server 172.16.10.4 8082 { # real-server
        weight 100
        inhibit_on_failure
        TCP_CHECK {    # health check
            nb_sock_retry 2
            connect_timeout 3
            connect_port 8082
        }
    }
}

# gdb ./keepalived ./core.18781
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /home/tianyan/source/dpvs_all/dpvs/bin/keepalived...done.
[New LWP 18781]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `./keepalived -d -f /etc/keepalived/keepalived.conf -S 6'.
Program terminated with signal 11, Segmentation fault.
#0  vs_exist (old_vs=old_vs@entry=0x22f2e20) at ipwrapper.c:551
551		if (!clear_diff_vsge(old->addr_ip, new->addr_ip, old_vs))
Missing separate debuginfos, use: debuginfo-install glibc-2.17-196.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libselinux-2.5-11.el7.x86_64 nss-softokn-freebl-3.28.3-8.el7_4.x86_64 openssl-libs-1.0.2k-8.el7.x86_64 pcre-8.32-17.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  vs_exist (old_vs=old_vs@entry=0x22f2e20) at ipwrapper.c:551
#1  0x000000000040f36c in clear_diff_services () at ipwrapper.c:834
#2  0x0000000000405775 in start_check () at check_daemon.c:128
#3  0x0000000000405889 in reload_check_thread (thread=<optimized out>) at check_daemon.c:212
#4  0x000000000042136d in thread_call (thread=0x7ffd58501d70) at scheduler.c:759
#5  launch_scheduler () at scheduler.c:782
#6  0x00000000004058cd in start_check_child () at check_daemon.c:311
#7  0x0000000000403127 in start_keepalived () at main.c:80
#8  main (argc=<optimized out>, argv=<optimized out>) at main.c:303
@mscbg
Copy link
Collaborator

mscbg commented Jan 29, 2018

@icymoon ，You have a wrong config in keepalived.conf1. As virtual_server_group 192.168.10.x_http1 is not configured, it is still used. Anyway, keepalived really has bug here. I will fix it later.

@icymoon
Copy link
Collaborator Author

icymoon commented Jan 29, 2018

I run the test with several wrong config files.. keepalived shouldn't crash with an unused configuration.
Thanks a lot.

@mscbg
Copy link
Collaborator

mscbg commented Jan 29, 2018

Yes，you are right. But the cofigured svc is in memory. When keepalived is reload, old svc will just try to find their group to compare new group. Bug is just here. You can start keepalived using wrong configured file. That's why i say there is bug here.

@mscbg
Copy link
Collaborator

mscbg commented Jan 29, 2018

You can just test this by delete unused svc. I have already tried this.

@icymoon
Copy link
Collaborator Author

icymoon commented Jan 29, 2018

Thank you very much ：）

@mscbg mscbg closed this as completed Feb 26, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mscbg mscbg

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@icymoon @mscbg