fib & decap tests failed because of KeyError: 'members'

[ysmanman]

Description

Observed the following KeyError in fib & decap test in T2 testing.

common/plugins/log_section_start/__init__.py:61:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                                                                                                                                                                                                                                                                                                                                                                                           
common/fixtures/fib_utils.py:284: in fib_info_files                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
    fib_info = get_t2_fib_info(duthosts, duts_config_facts, duts_minigraph_facts)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
    def get_t2_fib_info(duthosts, duts_cfg_facts, duts_mg_facts):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        """Get parsed FIB information from redis DB for T2 topology.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        Args:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
            duthosts (DutHosts): Instance of DutHosts for interacting with DUT hosts                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
            duts_cfg_facts (dict): Running config facts of all DUT hosts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
            duts_mg_facts (dict): Minigraph facts of all DUT hosts.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        Returns:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
            dict: Map of prefix to PTF ports that are connected to DUT output ports.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                {                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                    '192.168.0.0/21': [],                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                    '192.168.8.0/25': [[58 59] [62 63] [66 67] [70 71]],                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                    '192.168.16.0/25': [[58 59] [62 63] [66 67] [70 71]],                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                    ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                    '20c0:c2e8:0:80::/64': [[58 59] [62 63] [66 67] [70 71]],                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                    '20c1:998::/64': [[58 59] [62 63] [66 67] [70 71]],                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                    ...                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                }                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
        """                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        timestamp = datetime.now().strftime('%Y-%m-%d-%H:%M:%S')                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        fib_info = {}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        # Collect system neighbors, inband intf and port channel info to resolve ptf ports                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
        # for system neigh or lags.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
        dut_inband_intfs = {}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
        dut_port_channels = {}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
        switch_type = ''                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
        for duthost in duthosts.frontend_nodes:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
            cfg_facts = duts_cfg_facts[duthost.hostname]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
            for asic_cfg_facts in cfg_facts:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                if duthost.facts['switch_type'] == "voq":                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                    switch_type = "voq"                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                    dut_inband_intfs.setdefault(duthost.hostname,[]).extend(asic_cfg_facts['VOQ_INBAND_INTERFACE'])                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                dut_port_channels.setdefault(duthost.hostname,{}).update(asic_cfg_facts.get('PORTCHANNEL', {}))                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
        sys_neigh = {}                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
        if switch_type == "voq":                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
            voq_db = VoqDbCli(duthosts.supervisor_nodes[0])                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
            for entry in voq_db.dump_neighbor_table():                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                neigh_key = entry.split('|')                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                neigh_ip = neigh_key[-1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
                sys_neigh[neigh_ip] = {'duthost_name' : neigh_key[-4], 'intf' : neigh_key[-2]}                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
        for duthost in duthosts.frontend_nodes:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
            cfg_facts = duts_cfg_facts[duthost.hostname]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
            mg_facts = duts_mg_facts[duthost.hostname]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
            for asic_index, asic_cfg_facts in  enumerate(cfg_facts):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                asic = duthost.asic_instance(asic_index)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                asic.shell("{} redis-dump -d 0 -k 'ROUTE*' -y > /tmp/fib.{}.txt".format(asic.ns_arg, timestamp))                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                duthost.fetch(src="/tmp/fib.{}.txt".format(timestamp), dest="/tmp/fib")                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                po = asic_cfg_facts.get('PORTCHANNEL', {})                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                ports = asic_cfg_facts.get('PORT', {})                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                with open("/tmp/fib/{}/tmp/fib.{}.txt".format(duthost.hostname, timestamp)) as fp:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                    fib = json.load(fp)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                    for k, v in fib.items():                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                        skip = False                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                        prefix = k.split(':', 1)[1]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                        ifnames = v['value']['ifname'].split(',')                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
                        nh = v['value']['nexthop']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                        nh_ips = nh.split(',')                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                        oports = []                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                        for idx, ifname in enumerate(ifnames):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
                            if ifname in po:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                # ignore the prefix, if the prefix nexthop is not a frontend port                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                if 'members' in po[ifname]:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       
                                    if 'role' in ports[po[ifname]['members'][0]] and ports[po[ifname]['members'][0]]['role'] == 'Int':                                                                                                                                                                                                                                                                                                                                                                                                    
                                        if len(oports) == 0:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                            skip = True                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                    else:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
                                        oports.append([str(mg_facts[asic_index]['minigraph_ptf_indices'][x]) for x in po[ifname]['members']])                                                                                                                                                                                                                                                                                                                                                                                                                           
                                        skip = False                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                            else:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                if ifname in ports:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         
                                    if 'role' in ports[ifname] and ports[ifname]['role'] == 'Int':                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                        if len(oports) == 0:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                
                                            skip = True                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
                                    elif 'role' in ports[ifname] and ports[ifname]['role'] == 'Inb':                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                                        if nh == '0.0.0.0' or nh == '::':                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                                            # This is a system or inband neighbor.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                                            neigh_ip = prefix                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                                        else:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                                            neigh_ip = nh_ips[idx]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          
                                        remote_duthost_name = sys_neigh[neigh_ip]['duthost_name']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 
                                        remote_neigh_intf = sys_neigh[neigh_ip]['intf']                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                        # Skip route for inband neighbors.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                        if remote_neigh_intf in dut_inband_intfs[remote_duthost_name]:                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                            skip =True                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      
                                            continue                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                        remote_dut_mg_facts = duts_mg_facts[remote_duthost_name]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            
                                        if remote_neigh_intf.startswith('PortChannel'):                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                            # The nexthop is a system lag.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  
                                            if dut_port_channels[remote_duthost_name].has_key(remote_neigh_intf):                                                                                                                                                                                                                                                                                                                                                                                                                                                                           
                                                oport_list = []                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
>                                               for a_member in dut_port_channels[remote_duthost_name][remote_neigh_intf]['members']:                                                                                                                                                                                                                                                                                                                                                                                                                                          
E                                               KeyError: 'members'                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               
                                                                                                                          

Steps to reproduce the issue:

1. Run fib or decap test on T2 topo.

Describe the results you received:

Describe the results you expected:

Additional information you deem important:

**Output of `show version`:**

```
(paste your output here)
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```

[ysmanman]

The error is likely because sonic-net/sonic-buildimage#14028 removed the members collection from PORTCHANNEL table in config_db.

[ysmanman]

Add @arlakshm @wenyiz2021 for visibility.

[bingwang-ms]

This issue has been fixed by PR #7668

[yxieca]

@ysmanman can you verify and close this issue?

[abdosi]

@anamehra for viz.

[abdosi]

@vperumal for viz.

[wenyiz2021]

it would be good to have #7660 as well, 7668 is a workaround for remaining tests that still uses members key, we can eventually remove 7668 if no more tests depends on 7668