Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error logs seen when executing "show interface counters" cli in Sonic 202111 #6678

Open
sshankar-marvell opened this issue Oct 31, 2022 · 0 comments

Comments

@sshankar-marvell
Copy link

Description

The below logs are seen whenever "show interface counters" cli is executed

WARNING python3: :- operator(): Key 'RATES:oid:0x1000000000248' field 'RX_UTIL' unavailable in database 'COUNTERS_DB'
WARNING python3: :- operator(): Key 'RATES:oid:0x1000000000248' field 'TX_UTIL' unavailable in database 'COUNTERS_DB'

The root cause of this is as follows

The "show interface counters" is implemented by the portstat.py script (src/sonic-utilities/scripts/portstat)

    rates_key_list = [ 'RX_BPS', 'RX_PPS', 'RX_UTIL', 'TX_BPS', 'TX_PPS', 'TX_UTIL' ]
    def get_rates(table_id):
        """
            Get the rates from specific table.
        """
        fields = ["0","0","0","0","0","0"]
        for pos, name in enumerate(rates_key_list):
            full_table_id = RATES_TABLE_PREFIX + table_id
            counter_data =  self.db.get(self.db.COUNTERS_DB, full_table_id, name)
            if counter_data is None:
                fields[pos] = STATUS_NA
            elif fields[pos] != STATUS_NA:
                fields[pos] = float(counter_data)
        cntr = RateStats._make(fields)
        return cntr

The get_rates() API in the portstat script tries to read RX_UTIL, TX_UTIL fields from the RATES table in the COUNTERS_DB database in RedisDB whenever "show interface counters" is run. But the problem here is that the fields RX_UTIL, TX_UTIL fields (which are present in the rates_key_list structure) are NOT present in RATES table in COUNTERS_DB database in RedisDB. The values of these fields in the portstat script are calculated based on the RX_BPS, TX_BPS fields in the RATES table (format_util(rates.rx_bps, port_speed), format_util(rates.tx_bps, port_speed)). But since the get_rates() API tries to read these non existing fields in the RATES table itresults in the Sonic error logs in the syslog file

Steps to reproduce the issue:

  1. Login to a Sonic device running 202111 version
  2. Run the "show interface counters" CLI
  3. Once the cli execution is complete check the /var/log/syslog file
  4. The following error logs will be seen

WARNING python3: :- operator(): Key 'RATES:oid:0x1000000000248' field 'RX_UTIL' unavailable in database 'COUNTERS_DB'
WARNING python3: :- operator(): Key 'RATES:oid:0x1000000000248' field 'TX_UTIL' unavailable in database 'COUNTERS_DB'

Describe the results you received:

Executed the "show interface counters" CLI on a Sonic 202111 device. Once the cli execution
is complete checked the /var/log/syslog file. In this file the error logs were seen

WARNING python3: :- operator(): Key 'RATES:oid:0x1000000000248' field 'RX_UTIL' unavailable in database 'COUNTERS_DB'
WARNING python3: :- operator(): Key 'RATES:oid:0x1000000000248' field 'TX_UTIL' unavailable in database 'COUNTERS_DB'

Describe the results you expected:

When the "show interface counters" is executed no error logs should be seen in /var/log/syslog

Additional information you deem important:

Though the same get_rates() API was backported to portstat script in the SONIC 202012 version, this issue is not seen on the device running sonic version 202012. The reason for the error not logged in Sonic 202012, but logged in Sonic 202111 is because:

• In 202012, the DB interfacing is through python (sonic-py-swsssdk/src/swsssdk/interface.py) and the exception is logged only when the DBInterface operation is requested as blocking. In the case of the portstat script, the request to access the RATES table is non-blocking.

• In 202111, the DB interfacing is through SonicV2Connector in C++ (sonic-net/sonic-swss-common#387). The motive here is to move the python-based framework in sonic-py-swsssdk to a C++ based framework in sonic-swss-common. As part of this migration, there’s been a forced logging irrespective of the blocking status. (https://github.com/sonic-net/sonic-swss-common/blame/202111/common/dbinterface.cpp -> DBInterface::get)

**Output of `show version`:**

```

show version:

SONiC Software Version: SONiC.202111.Innovium.0-dirty-20220805.084100
Distribution: Debian 11.4
Kernel: 5.10.0-8-2-amd64
Build commit: 9d1ef336f
Build date: Fri Aug 5 17:10:56 UTC 2022
Built by: admin@sonic
```

**Attach debug file `sudo generate_dump`:**

```
(paste your output here)
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant