Improve behave functional test, add /test/run-functional-tests

[liangxin1300]

Motivation

During developing, when specific GH CI case failed, it's not easy to reproduce it locally:

• setup N nodes cluster, and setup passwordless between nodes
• install python3-behave package on one node
• copy crmsh/test/features/ to one node
• run behave <specific_test_file.feature>

It will be more convenient to have an one-liner tool to do all above steps, in one step.

Features

• Setup N nodes cluster quickly
• Setup N nodes without cluster configured and started
• List existing test cases
• Run specific test case
• Cleanup what already deployed

Changes in /test/run-functional-tests

Help info

# ./test/run-functional-tests -h
Usage: run-functional-tests [OPTIONS]|[TESTCASE INDEX]
run-functional-tests is a tool for developers to setup the cluster in containers to run functional tests

OPTIONS:
  -h, --help	Show this help message and exit
  -l            List existing functional test cases and exit
  -n NUM        Only setup a cluster with NUM nodes(containers)
  -c            Don't config corosync on containers(with -n option)
  -d            Cleanup the cluster containers

Setup N nodes cluster quickly

# ./test/run-functional-tests -n 2
INFO: Loading docker image liangxin1300/hatbw...
INFO: Create ha specific docker network "ha_network_first"...
INFO: Create ha specific docker network "ha_network_second"...
INFO: Setup cluster...
INFO: Deploying "hanode1"...
INFO: Building crmsh codes on "hanode1"...
INFO: Deploying "hanode2"...
INFO: Building crmsh codes on "hanode2"...
INFO: Starting pacemaker...done
INFO: Cluster services started on hanode1
INFO: Starting pacemaker...done
INFO: Cluster services started on hanode2
[root@docker1 crmsh]# docker exec hanode1 crm_mon -1
Cluster Summary:
  * Stack: corosync
  * Current DC: hanode1 (version 2.1.2+20220126.16c1bab10-1.2-2.1.2+20220126.16c1bab10) - partition with quorum
  * Last updated: Sun Mar 27 14:53:39 2022
  * Last change:  Sun Mar 27 14:53:28 2022 by hacluster via crmd on hanode1
  * 2 nodes configured
  * 0 resource instances configured

Node List:
  * Online: [ hanode1 hanode2 ]

Active Resources:
  * No active resources

Cleanup what already deployed

# ./test/run-functional-tests -d
INFO: Cleanup container "hanode2"...
INFO: Cleanup container "hanode1"...
INFO: Cleanup ha specific docker network "ha_network_first"...
INFO: Cleanup ha specific docker network "ha_network_second"...

Setup N nodes without cluster configured and started

# ./test/run-functional-tests -c -n 2
INFO: Loading docker image liangxin1300/hatbw...
INFO: Create ha specific docker network "ha_network_first"...
INFO: Create ha specific docker network "ha_network_second"...
INFO: Setup cluster...
INFO: Deploying "hanode1"...
INFO: Building crmsh codes on "hanode1"...
INFO: Deploying "hanode2"...
INFO: Building crmsh codes on "hanode2"...

List existing test cases

# ./test/run-functional-tests -l
Index|File Name|Description
  1    bootstrap_bugs.feature                    Regression test for bootstrap bugs                         
  2    bootstrap_init_join_remove.feature        crmsh bootstrap process - init, join and remove            
  3    bootstrap_options.feature                 crmsh bootstrap process - options                          
  4    configure_bugs.feature                    Functional test for configure sub level                    
  5    constraints_bugs.feature                  Verify constraints(order/colocation/location) bug          
  6    crm_report_bugs.feature                   crm report functional test for verifying bugs              
  7    geo_setup.feature                         geo cluster                                                
  8    qdevice_options.feature                   corosync qdevice/qnetd options                             
  9    qdevice_setup_remove.feature              corosync qdevice/qnetd setup/remove process                
 10    qdevice_usercase.feature                  Verify usercase master survive when split-brain            
 11    qdevice_validate.feature                  corosync qdevice/qnetd options validate                    
 12    resource_failcount.feature                Use "crm resource failcount" to manage failcounts          
 13    resource_set.feature                      Use "crm configure set" to update attributes and operations
 14    regression test                           Original regression test

Run test case 1 and 2

<pre># ./test/run-functional-tests 1 2
ERROR: HA specific network &quot;ha_network_first&quot; already exists
[root@docker1 crmsh]# ./test/run-functional-tests -d
INFO: Cleanup container &quot;hanode2&quot;...
INFO: Cleanup container &quot;hanode1&quot;...
INFO: Cleanup ha specific docker network &quot;ha_network_first&quot;...
INFO: Cleanup ha specific docker network &quot;ha_network_second&quot;...
[root@docker1 crmsh]# ./test/run-functional-tests 1 2
INFO: Loading docker image liangxin1300/hatbw...
INFO: Create ha specific docker network &quot;ha_network_first&quot;...
INFO: Create ha specific docker network &quot;ha_network_second&quot;...
INFO: Setup cluster...
INFO: Deploying &quot;hanode1&quot;...
INFO: Building crmsh codes on &quot;hanode1&quot;...
INFO: Deploying &quot;hanode2&quot;...
INFO: Building crmsh codes on &quot;hanode2&quot;...
INFO: Deploying &quot;hanode3&quot;...
INFO: Building crmsh codes on &quot;hanode3&quot;...
@bootstrap
Feature: Regression test for bootstrap bugs<font color="#555753"> # app/test/features/bootstrap_bugs.feature:2</font>
  Tag @clean means need to stop cluster service if the service is available
  Need nodes: hanode1 hanode2 hanode3
  @clean
  Scenario: Set placement-strategy value as &quot;default&quot;(bsc#1129462) 
  ...
  ...
  ...