Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failures in functional tests #759

Closed
krizhanovsky opened this issue Jul 2, 2017 · 2 comments · Fixed by #783
Closed

Failures in functional tests #759

krizhanovsky opened this issue Jul 2, 2017 · 2 comments · Fixed by #783
Assignees
@intelfx
Labels
bug test
Milestone
0.5 alpha

Comments

@krizhanovsky
Copy link
Contributor

There are many failures in functional tests running in my #757 . The complete environment you can find in my VM, basically I just ran ./run_tests.py.

    Error: Can't start TempestaFW on localhost

I had a look at functional/sched/test_http.py line 110, but there is nothing special. I.e. it's hard to understand what the problem is.

Secondly there are messages like

    Error: Can't get stats of Nginx on localhost:8000

which seems the test bug, since Nginx is installed from standard package.

Third, it seems you need to set ulimit in the script and check that it's run by root:

    Error:  Deproxy: Server 127.0.0.1:8027: [Errno 24] Too many open files

The test output is at the below. Firstly, there are cryptic error reports like:

    # ./run_tests.py 
    
    ----------------------------------------------------------------------
    Running functional tests...
    ----------------------------------------------------------------------
    
    E...EE.E..........................EE....x.x.EEEEE...................................................F..............
    ======================================================================
    ERROR: test_scheduler (sched.test_http.HttpRules)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_http.py", line 110, in test_scheduler
        self.init()
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_http.py", line 104, in init
        self.tempesta.start()
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 297, in start
        self.node.run_cmd(cmd, err_msg=(self.err_msg % 'start'))
      File "/root/tempesta/tempesta_fw/t/functional/helpers/remote.py", line 64, in run_cmd
        error.bug(err_msg)
      File "/root/tempesta/tempesta_fw/t/functional/helpers/error.py", line 28, in bug
        raise Error(msg)
    Error: Can't start TempestaFW on localhost
    
    ======================================================================
    ERROR: test_ratio (sched.test_ratio.FairLoadEqualConns)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_ratio.py", line 54, in test_ratio
        self.generic_test_routine('cache 0;\n')
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 141, in generic_test_routine
        self.assert_servers()
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_ratio.py", line 40, in assert_servers
        self.servers_get_stats()
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 121, in servers_get_stats
        control.servers_get_stats(self.servers)
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 405, in servers_get_stats
        pool.map(Nginx.get_stats, servers)
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
        raise self._value
    Error: Can't get stats of Nginx on localhost:8000
    
    ======================================================================
    ERROR: test_ratio (sched.test_ratio.FairLoadRandConns)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_ratio.py", line 54, in test_ratio
        self.generic_test_routine('cache 0;\n')
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 141, in generic_test_routine
        self.assert_servers()
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_ratio.py", line 40, in assert_servers
        self.servers_get_stats()
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 121, in servers_get_stats
        control.servers_get_stats(self.servers)
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 405, in servers_get_stats
        pool.map(Nginx.get_stats, servers)
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
        raise self._value
    Error: Can't get stats of Nginx on localhost:8000
    
    ======================================================================
    ERROR: test_hash (sched.test_hash_stress.BindToServer)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_hash_stress.py", line 49, in test_hash
        self.generic_test_routine('cache 0;\n')
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 141, in generic_test_routine
        self.assert_servers()
      File "/root/tempesta/tempesta_fw/t/functional/sched/test_hash_stress.py", line 38, in assert_servers
        self.servers_get_stats()
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 121, in servers_get_stats
        control.servers_get_stats(self.servers)
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 405, in servers_get_stats
        pool.map(Nginx.get_stats, servers)
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
        raise self._value
    Error: Can't get stats of Nginx on localhost:8005
    
    ======================================================================
    ERROR: test_on_close (regression.test_srv_failovering.FailoveringTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_srv_failovering.py", line 44, in test_on_close
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 656, in loop
      File "/usr/lib/python2.7/asyncore.py", line 201, in poll2
      File "/usr/lib/python2.7/asyncore.py", line 117, in readwrite
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 586, in handle_error
      File "/root/tempesta/tempesta_fw/t/functional/helpers/error.py", line 28, in bug
        raise Error(msg)
    Error: 	Deproxy: Server 127.0.0.1:8027: [Errno 24] Too many open files
    
    ======================================================================
    ERROR: test_on_shutdown (regression.test_srv_failovering.FailoveringTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_srv_failovering.py", line 55, in test_on_shutdown
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 656, in loop
      File "/usr/lib/python2.7/asyncore.py", line 201, in poll2
      File "/usr/lib/python2.7/asyncore.py", line 117, in readwrite
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 586, in handle_error
      File "/root/tempesta/tempesta_fw/t/functional/helpers/error.py", line 28, in bug
        raise Error(msg)
    Error: 	Deproxy: Server 127.0.0.1:8025: [Errno 24] Too many open files
    
    ======================================================================
    ERROR: test_shutdown (regression.test_shutdown.ShutdownTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_shutdown.py", line 53, in test_shutdown
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 656, in loop
      File "/usr/lib/python2.7/asyncore.py", line 201, in poll2
      File "/usr/lib/python2.7/asyncore.py", line 117, in readwrite
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 586, in handle_error
      File "/root/tempesta/tempesta_fw/t/functional/helpers/error.py", line 28, in bug
        raise Error(msg)
    Error: 	Deproxy: Server 127.0.0.1:8024: [Errno 24] Too many open files
    
    ======================================================================
    ERROR: test_shutdown (regression.test_shutdown.ShutdownTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_shutdown.py", line 29, in tearDown
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 305, in stop
        self.node.run_cmd(cmd, timeout=timeout, err_msg=(self.err_msg % 'stop'))
      File "/root/tempesta/tempesta_fw/t/functional/helpers/remote.py", line 56, in run_cmd
        stderr=stderr_pipe) as p:
      File "/usr/lib/python2.7/dist-packages/subprocess32.py", line 792, in __init__
      File "/usr/lib/python2.7/dist-packages/subprocess32.py", line 1273, in _get_handles
    OSError: [Errno 24] Too many open files
    
    ======================================================================
    ERROR: test_shutdown_with_traffic (regression.test_shutdown.ShutdownTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_shutdown.py", line 60, in test_shutdown_with_traffic
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_shutdown.py", line 43, in init
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_shutdown.py", line 38, in create_servers
      File "/root/tempesta/tempesta_fw/t/functional/testers/functional.py", line 47, in create_servers_helper
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 544, in __init__
      File "/root/tempesta/tempesta_fw/t/functional/helpers/deproxy.py", line 547, in setup
      File "/usr/lib/python2.7/asyncore.py", line 297, in create_socket
      File "/usr/lib/python2.7/socket.py", line 191, in __init__
    error: [Errno 24] Too many open files
    
    ======================================================================
    ERROR: test_shutdown_with_traffic (regression.test_shutdown.ShutdownTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/regression/test_shutdown.py", line 29, in tearDown
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 305, in stop
        self.node.run_cmd(cmd, timeout=timeout, err_msg=(self.err_msg % 'stop'))
      File "/root/tempesta/tempesta_fw/t/functional/helpers/remote.py", line 56, in run_cmd
        stderr=stderr_pipe) as p:
      File "/usr/lib/python2.7/dist-packages/subprocess32.py", line 792, in __init__
      File "/usr/lib/python2.7/dist-packages/subprocess32.py", line 1273, in _get_handles
    OSError: [Errno 24] Too many open files
    
    ======================================================================
    ERROR: test_cache_bypass_all (cache.test_cache.TestCacheDisabled)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/cache/test_cache.py", line 36, in test_cache_bypass_all
        self.generic_test_routine(config, self.chain(cache_alowed=False))
      File "/root/tempesta/tempesta_fw/t/functional/testers/functional.py", line 97, in generic_test_routine
        self.tempesta.start()
      File "/root/tempesta/tempesta_fw/t/functional/helpers/control.py", line 297, in start
        self.node.run_cmd(cmd, err_msg=(self.err_msg % 'start'))
      File "/root/tempesta/tempesta_fw/t/functional/helpers/remote.py", line 64, in run_cmd
        error.bug(err_msg)
      File "/root/tempesta/tempesta_fw/t/functional/helpers/error.py", line 28, in bug
        raise Error(msg)
    Error: Can't start TempestaFW on localhost
    
    ======================================================================
    FAIL: test (sessions.test_sticky_sess_stress.OneClient)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/root/tempesta/tempesta_fw/t/functional/sessions/test_sticky_sess_stress.py", line 56, in test
        self.generic_test_routine(self.config)
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 139, in generic_test_routine
        self.assert_clients()
      File "/root/tempesta/tempesta_fw/t/functional/testers/stress.py", line 92, in assert_clients
        self.assertEqual(err, 0, msg='HTTP client detected errors')
    AssertionError: HTTP client detected errors
    
    ----------------------------------------------------------------------
    Ran 113 tests in 584.163s
    
    FAILED (failures=1, errors=11, expected failures=2)
@krizhanovsky krizhanovsky added this to the 0.5.0 Web Server milestone Jul 2, 2017
@krizhanovsky
Copy link
Contributor Author

One more problem with the test is that it doesn't clean environment after running. The failed test run mentioned above has left this configuration:

    # cat tempesta/etc/tempesta_fw.conf
    sched ratio;
    server 127.0.0.1:8000 conns_n=1;
    server 127.0.0.1:8001 conns_n=1;
    server 127.0.0.1:8002 conns_n=1;
    server 127.0.0.1:8003 conns_n=1;
    server 127.0.0.1:8004 conns_n=1;
    server 127.0.0.1:8005 conns_n=1;
    server 127.0.0.1:8006 conns_n=1;
    server 127.0.0.1:8007 conns_n=1;
    server 127.0.0.1:8008 conns_n=1;
    server 127.0.0.1:8009 conns_n=1;
    server 127.0.0.1:8010 conns_n=1;
    server 127.0.0.1:8011 conns_n=1;
    server 127.0.0.1:8012 conns_n=1;
    server 127.0.0.1:8013 conns_n=1;
    server 127.0.0.1:8014 conns_n=1;
    server 127.0.0.1:8015 conns_n=1;
    server 127.0.0.1:8016 conns_n=1;
    server 127.0.0.1:8017 conns_n=1;
    server 127.0.0.1:8018 conns_n=1;
    server 127.0.0.1:8019 conns_n=1;
    server 127.0.0.1:8020 conns_n=1;
    server 127.0.0.1:8021 conns_n=1;
    server 127.0.0.1:8022 conns_n=1;
    server 127.0.0.1:8023 conns_n=1;
    server 127.0.0.1:8024 conns_n=1;
    server 127.0.0.1:8025 conns_n=1;
    server 127.0.0.1:8026 conns_n=1;
    server 127.0.0.1:8027 conns_n=1;
    server 127.0.0.1:8028 conns_n=1;
    server 127.0.0.1:8029 conns_n=1;
    server 127.0.0.1:8030 conns_n=1;
    server 127.0.0.1:8031 conns_n=1;
    sticky_sessions;
    cache 0;
    sticky enforce;
    sticky_secret "f00)9eR59*_/22";

I.e. my original configuration was replaced by the config. The test must leave original configuration after it's exit, i.e. it should copy tempesta_fw.conf somewhere and restore it on exit.

@vankoven
Copy link
Contributor

vankoven commented Jul 6, 2017

Currently there is the option to preserve configuration files after test clean up: https://github.com/tempesta-tech/tempesta/blob/master/tempesta_fw/t/functional/helpers/remote.py#L16 . The option is very helpful to spawn Tempesta and nginx config files which was used in a failed test for a further manual checks. As described in comment above test Tempesta confing file is the only file that doesn't respect the option.

Copying/restoring the original config file is not a reliable approach, e.g. it is impossible to restore the original config after a kernel crash. So variable TFW_CFG_PATH must be set when the test suite runs the Tempesta start/stop script. The variable must be defined in tests_config.ini with default value of /tmp/tempesta_fw.conf. Just a couple of changes to helpers.control.Tempesta class will be required.

@krizhanovsky krizhanovsky assigned intelfx and unassigned vankoven Jul 19, 2017
@intelfx intelfx mentioned this issue Aug 1, 2017
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@intelfx intelfx

Labels
bug test
Projects
None yet
Milestone
0.5 alpha
Development

Successfully merging a pull request may close this issue.

Various fixes to functional tests tempesta-tech/tempesta
3 participants
@krizhanovsky @intelfx @vankoven