[Arista][T2] Seeing ports linkDown when PMON startup isn't delayed

[arista-nwolfe]

#19657 recently merged to not delay the startup of pmon on SpineRouter (T2) systems.

This has resulted in a subset of ports to stay link down on our multi-asic LCs:

root@cmp214-5:~# show interfaces status
     Interface            Lanes    Speed    MTU    FEC         Alias            Vlan    Oper    Admin             Type    Asym PFC
--------------  ---------------  -------  -----  -----  ------------  --------------  ------  -------  ---------------  ----------
     Ethernet0      72,73,74,75     100G   9100     rs   Ethernet1/1          routed    down     down  QSFP28 or later         off
     Ethernet8      80,81,82,83     100G   9100     rs   Ethernet2/1          routed    down     down  QSFP28 or later         off
    Ethernet16      88,89,90,91     100G   9100     rs   Ethernet3/1          routed    down     down              N/A         off
    Ethernet24      96,97,98,99     100G   9100     rs   Ethernet4/1          routed    down     down              N/A         off
    Ethernet32  104,105,106,107     100G   9100     rs   Ethernet5/1  PortChannel102    down       up  QSFP28 or later         off
    Ethernet40  112,113,114,115     100G   9100     rs   Ethernet6/1          routed    down     down              N/A         off
    Ethernet48  120,121,122,123     100G   9100     rs   Ethernet7/1  PortChannel102    down       up  QSFP28 or later         off
    Ethernet56  128,129,130,131     100G   9100     rs   Ethernet8/1          routed    down     down              N/A         off
    Ethernet64  136,137,138,139     100G   9100     rs   Ethernet9/1          routed    down     down              N/A         off
    Ethernet72      64,65,66,67     100G   9100     rs  Ethernet10/1          routed    down       up  QSFP28 or later         off

The set of ports which are down varies from boot to boot.

One thing that I noticed is that there are attributes missing from the ASIC_DB of the ports which are linkDown.
For example, Ethernet240 (oid:0x10100000000000d) was linkUp and had the following ASIC_DB attributes:

root@cmp214-6:~# sonic-db-cli -n asic1 ASIC_DB hgetall ASIC_STATE:SAI_OBJECT_TYPE_PORT:oid:0x10100000000000d
{'NULL': 'NULL', 'SAI_PORT_ATTR_ADMIN_STATE': 'true', 'SAI_PORT_ATTR_AUTO_NEG_MODE': 'false', 'SAI_PORT_ATTR_SPEED': '100000', 'SAI_PORT_ATTR_MTU': '9122', 'SAI_PORT_ATTR_FEC_MODE': 'SAI_PORT_FEC_MODE_RS', 'SAI_PORT_ATTR_QOS_DSCP_TO_TC_MAP': 'oid:0x1140000000008e3', 'SAI_PORT_ATTR_QOS_TC_TO_QUEUE_MAP': 'oid:0x1140000000008e8', 'SAI_PORT_ATTR_QOS_TC_TO_PRIORITY_GROUP_MAP': 'oid:0x1140000000008e7', 'SAI_PORT_ATTR_QOS_PFC_PRIORITY_TO_QUEUE_MAP': 'oid:0x1140000000008e4', 'SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL': '24'}

However Ethernet144 (oid:0x101000000000001) was linkDown and had the following ASIC_DB attributes:

root@cmp214-6:~# sonic-db-cli -n asic1 ASIC_DB hgetall ASIC_STATE:SAI_OBJECT_TYPE_PORT:oid:0x101000000000001
{'NULL': 'NULL', 'SAI_PORT_ATTR_ADMIN_STATE': 'true', 'SAI_PORT_ATTR_AUTO_NEG_MODE': 'false', 'SAI_PORT_ATTR_SPEED': '100000', 'SAI_PORT_ATTR_MTU': '9122', 'SAI_PORT_ATTR_QOS_DSCP_TO_TC_MAP': 'oid:0x1140000000008e3', 'SAI_PORT_ATTR_QOS_TC_TO_QUEUE_MAP': 'oid:0x1140000000008e8', 'SAI_PORT_ATTR_QOS_TC_TO_PRIORITY_GROUP_MAP': 'oid:0x1140000000008e7', 'SAI_PORT_ATTR_QOS_PFC_PRIORITY_TO_QUEUE_MAP': 'oid:0x1140000000008e4', 'SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL': '24'}

Note that 'SAI_PORT_ATTR_FEC_MODE': 'SAI_PORT_FEC_MODE_RS' is missing from Ethernet144.

But I see that the APPL_DB of both Ethernet144 and Ethernet240 are identical, suggesting orchagent didn't make propagate the state from APPL_DB to ASIC_DB properly for Ethernet144.

If I remove and re-add this fec attribute from Ethernet144's CONFIG_DB entry I see it reflected in ASIC_DB and the port then comes up:

sonic-db-cli -n asic1 CONFIG_DB hdel "PORT|Ethernet144" fec
sonic-db-cli -n asic1 CONFIG_DB hset "PORT|Ethernet144" fec rs

root@cmp214-6:~# sonic-db-cli -n asic1 ASIC_DB hgetall ASIC_STATE:SAI_OBJECT_TYPE_PORT:oid:0x101000000000001
{'NULL': 'NULL', 'SAI_PORT_ATTR_ADMIN_STATE': 'true', 'SAI_PORT_ATTR_AUTO_NEG_MODE': 'false', 'SAI_PORT_ATTR_SPEED': '100000', 'SAI_PORT_ATTR_MTU': '9122', 'SAI_PORT_ATTR_QOS_DSCP_TO_TC_MAP': 'oid:0x1140000000008e3', 'SAI_PORT_ATTR_QOS_TC_TO_QUEUE_MAP': 'oid:0x1140000000008e8', 'SAI_PORT_ATTR_QOS_TC_TO_PRIORITY_GROUP_MAP': 'oid:0x1140000000008e7', 'SAI_PORT_ATTR_QOS_PFC_PRIORITY_TO_QUEUE_MAP': 'oid:0x1140000000008e4', 'SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL': '24', 'SAI_PORT_ATTR_FEC_MODE': 'SAI_PORT_FEC_MODE_RS'}

  Interface            Lanes    Speed    MTU    FEC         Alias    Vlan    Oper    Admin             Type    Asym PFC
-----------  ---------------  -------  -----  -----  ------------  ------  ------  -------  ---------------  ----------
Ethernet144      72,73,74,75     100G   9100     rs  Ethernet19/1  routed      up       up  QSFP28 or later         off

[rlhui]

does it relate to port speed change (handled in voq system as port del and then port add)?

[arista-nwolfe]

Root Cause Analysis:
doPortTask in portsorch waits for the buffer configuration to be complete by bufferorch before syncing the port attributes from APPL_DB to ASIC_DB:
https://github.com/sonic-net/sonic-swss/blob/master/orchagent/portsorch.cpp#L3949

if (!gBufferOrch->isPortReady(pCfg.key))
{
    // buffer configuration hasn't been applied yet. save it for future retry
    m_pendingPortSet.emplace(pCfg.key);
    it++;
    continue;
}

In parallel, with PMON not delayed, PMON is processing the media_settings.json file and pushing the tuning values to PORT_TABLE in APPL_DB. It turns out that the media_settings.json file for SKU x86_64-arista_7800r3a_36d2_lc has negative values, which are unsupported, and causing errors in portsorch:
https://github.com/sonic-net/sonic-buildimage/blob/master/device/arista/x86_64-arista_7800r3a_36d2_lc/media_settings.json

2024 Aug 19 20:27:39.784399 cmp206-4 ERR swss0#orchagent: :- doPortTask: KEY: Ethernet72, OP: SET
2024 Aug 19 20:27:39.784831 cmp206-4 ERR swss0#orchagent: :- parsePortSerdes: Failed to parse field(pre1): Invalid argument: '-0x6'

This error causes the port event to be removed from portsorch's taskmap:
https://github.com/sonic-net/sonic-swss/blob/master/orchagent/portsorch.cpp#L3850

if (!parsePortFvs(fvMap))
{
    it = taskMap.erase(it);
    continue;
}

This removal of the port from the taskmap causes the updates for the other attributes in APPL_DB to be lost as well, leaving ASIC_DB out of sync from APPL_DB.

TLDR:
If bufferorch declares the port ready before PMON processes the erroneous media_settings.json file then the ports will come up fine.
If PMON processes the erroneous media_settings.json file before bufferorch declares the port ready then the ports won't come up.

Solution:
We will update the media_settings.json file for x86_64-arista_7800r3a_36d2_lc to use 2's complement values instead of negative values.

Follow up questions:
What's the reason we don't fail more aggressively if the media_settings loaded in the APPL_DB are incorrect?
Only emitting a trace seems like it's destined to be ignored or missed, like this one, for many years until it manifests as a different failure.

[abdosi]

Issue is fixed with updating media_setting.json. Subsequent PR might be needed if their is bad input in media_setting.json. take better failure actions.

[arista-nwolfe]

The media_settings.json file has been fixed by these PRs so closing:
Master - #20308
202405 - #20283