Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SONiC Broadcom AS7816-64X]: Orchagent and syncd crash seen when Asym PFC is enabled in an interface #481

Open
afzalashfak opened this issue Oct 8, 2019 · 2 comments
Open

[SONiC Broadcom AS7816-64X]: Orchagent and syncd crash seen when Asym PFC is enabled in an interface #481

afzalashfak opened this issue Oct 8, 2019 · 2 comments

Comments

@afzalashfak
Copy link

I am using AS7816-64X switch. SONiC version is 20190924.
Orchagent and syncd crash seen when Asym PFC is enabled in an interface.

CLI:
pfc config asymmetric Ethernet0 on

Crash Logs:

orchagent.1478758153.76.core.gz

Nov 10 06:09:13.270432 sonic NOTICE swss#orchagent: :- doPortTask: Set port Ethernet0 asymmetric PFC to on
Nov 10 06:09:13.270488 sonic INFO lldp#lldpmgrd: Port name Ethernet0 oper status: up
Nov 10 06:09:13.271436 sonic ERR syncd#syncd: brcm_sai_set_port_attribute:1027 Unknown port attribute 74 passed
Nov 10 06:09:13.271436 sonic ERR syncd#syncd: :- processEvent: VID: oid:0x1000000000012 RID: oid:0x100000022
Nov 10 06:09:13.271629 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.271744 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.272216 sonic ERR syncd#syncd: :- processEvent: attr: SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL_MODE: SAI_PORT_PRIORITY_FLOW_CONTROL_MODE_SEPARATE
Nov 10 06:09:13.272216 sonic ERR syncd#syncd: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0
Nov 10 06:09:13.272216 sonic ERR syncd#syncd: :- syncd_main: Runtime error: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0
Nov 10 06:09:13.272216 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: sending switch_shutdown_request notification to OA
Nov 10 06:09:13.272508 sonic NOTICE swss#orchagent: :- handle_switch_shutdown_request: switch shutdown request
Nov 10 06:09:13.272734 sonic NOTICE swss#portmgrd: :- doTask: Configure Ethernet0 MTU to 9100
Nov 10 06:09:13.272734 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: notification send successfull
Nov 10 06:09:13.275208 sonic INFO swss#supervisord: orchagent terminate called after throwing an instance of 'std::invalid_argument'
Nov 10 06:09:13.275208 sonic INFO swss#supervisord: orchagent what(): parse error - unexpected end of input
Nov 10 06:09:13.276925 sonic NOTICE swss#portmgrd: :- doTask: Configure Ethernet0 admin status to up
Nov 10 06:09:13.283521 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.283521 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.292829 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.292874 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.301471 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.301498 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.309988 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.309988 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.317396 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.317396 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.325146 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.325146 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.332736 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.332736 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.340707 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.340707 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:13.348443 sonic INFO lldp#lldpmgrd: Unable to retrieve description for port 'Ethernet0'. Not adding port description
Nov 10 06:09:13.348443 sonic DEBUG lldp#lldpmgrd: Running command: 'lldpcli configure ports Ethernet0 lldp portidsubtype local hundredGigE1'
Nov 10 06:09:14.460835 sonic INFO swss#supervisor-proc-exit-listener: Process orchagent exited unxepectedly. Terminating supervisor...
Nov 10 06:09:20.642954 sonic INFO containerd[512]: time="2016-11-10T06:09:20.642830813Z" level=info msg="shim reaped" id=28a3cfff32772bc530fa245719f439b75623f57893ac82134525a2b75f67e41c
Nov 10 06:09:20.652922 sonic INFO dockerd[547]: time="2016-11-10T06:09:20.652784418Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 06:09:20.774943 sonic INFO swss.sh[5221]: 0
Nov 10 06:09:20.782956 sonic NOTICE root: Stopping swss service...
Nov 10 06:09:20.786862 sonic NOTICE root: Locking /tmp/swss-syncd-lock from swss service
Nov 10 06:09:20.791343 sonic NOTICE root: Locked /tmp/swss-syncd-lock (10) from swss service
Nov 10 06:09:21.199234 sonic NOTICE root: Warm boot flag: swss false.
Nov 10 06:09:21.244675 sonic INFO swss.sh[22837]: swss
Nov 10 06:09:21.249579 sonic NOTICE root: Stopped swss service...
Nov 10 06:09:21.253467 sonic NOTICE root: Unlocking /tmp/swss-syncd-lock (10) from swss service
Nov 10 06:09:21.260178 sonic INFO systemd[1]: Stopping syncd service...
Nov 10 06:09:21.264662 sonic NOTICE root: Stopping syncd service...
Nov 10 06:09:21.268833 sonic NOTICE root: Locking /tmp/swss-syncd-lock from syncd service
Nov 10 06:09:21.273405 sonic NOTICE root: Locked /tmp/swss-syncd-lock (10) from syncd service
Nov 10 06:09:21.695354 sonic NOTICE root: Warm boot flag: syncd false.
Nov 10 06:09:21.699772 sonic NOTICE root: cold shutdown syncd process ...
Nov 10 06:09:21.839122 sonic NOTICE syncd#syncd_request_shutdown: :- main: requested COLD shutdown
Nov 10 06:09:21.839310 sonic NOTICE syncd#syncd: :- syncd_main: is asic queue empty: 0
Nov 10 06:09:21.839310 sonic ERR syncd#syncd: brcm_sai_set_port_attribute:1027 Unknown port attribute 77 passed
Nov 10 06:09:21.839310 sonic ERR syncd#syncd: :- processEvent: VID: oid:0x1000000000012 RID: oid:0x100000022
Nov 10 06:09:21.839310 sonic ERR syncd#syncd: :- processEvent: attr: SAI_PORT_ATTR_PRIORITY_FLOW_CONTROL_TX: 120
Nov 10 06:09:21.839310 sonic ERR syncd#syncd: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0
Nov 10 06:09:21.839336 sonic ERR syncd#syncd: :- syncd_main: Runtime error: :- processEvent: failed to execute api: set, key: SAI_OBJECT_TYPE_PORT:oid:0x1000000000012, status: SAI_STATUS_ATTR_NOT_SUPPORTED_0
Nov 10 06:09:21.839336 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: sending switch_shutdown_request notification to OA
Nov 10 06:09:21.839246 sonic INFO syncd.sh[22999]: requested COLD shutdown
Nov 10 06:09:21.839477 sonic NOTICE syncd#syncd: :- notify_OA_about_syncd_exception: notification send successfull
Nov 10 06:09:21.839477 sonic NOTICE syncd#syncd: :- syncd_main: Removing the switch gSwitchId=0xb970112100000000
Nov 10 06:09:24.003084 sonic WARNING kernel: [159188.373052] linux-kernel-bde (9483): _interrupt_disconnect d 0
Nov 10 06:09:24.003109 sonic WARNING kernel: [159188.373055] linux-kernel-bde (9483): _interrupt_disconnect: isr_active = 1
Nov 10 06:09:24.003111 sonic WARNING kernel: [159188.373055] linux-kernel-bde (9483): disconnect primary isr
Nov 10 06:09:24.012107 sonic INFO syncd#supervisord: syncd 0:soc_shutdown: soc_shutdown: all units detached#015
Nov 10 06:09:24.015107 sonic WARNING kernel: [159188.384102] linux-bcm-knet (10179): Next DMA events (0x00000001)
Nov 10 06:09:24.015969 sonic NOTICE syncd#syncd: :- syncd_main: remove switch took 1.654644 sec
Nov 10 06:09:24.015969 sonic NOTICE syncd#syncd: :- syncd_main: calling api uninitialize
Nov 10 06:09:24.015969 sonic NOTICE syncd#syncd: :- syncd_main: uninitialize finished
Nov 10 06:09:24.860710 sonic NOTICE syncd#syncd: :- threadFunction: ending timer watchdog thread
Nov 10 06:09:24.886226 sonic NOTICE syncd#dsserve: child /usr/bin/syncd exited status: 0
Nov 10 06:09:24.886497 sonic INFO syncd#supervisord: syncd [5] child /usr/bin/syncd exited status: 0
Nov 10 06:09:25.267600 sonic NOTICE root: Finished cold shutdown syncd process ...
Nov 10 06:09:26.300595 sonic INFO syncd#supervisord 2016-11-10 06:09:24,886 INFO exited: syncd (exit status 0; expected)
Nov 10 06:09:26.300595 sonic INFO syncd#supervisord 2016-11-10 06:09:25,360 WARN received SIGTERM indicating exit request
Nov 10 06:09:26.300595 sonic INFO syncd#supervisord 2016-11-10 06:09:25,361 INFO waiting for rsyslogd to die
Nov 10 06:09:27.105062 sonic WARNING snmp#snmp-subagent [sonic_ax_impl] WARNING: No managment ports found in b'MGMT_PORT|'
Nov 10 06:09:27.539052 sonic INFO containerd[512]: time="2016-11-10T06:09:27.538891458Z" level=info msg="shim reaped" id=6335d8395a21594ccdeaf7c8c2be5a0f619f16fb23e7adfbc992a55abc1d1caf
Nov 10 06:09:27.548767 sonic INFO dockerd[547]: time="2016-11-10T06:09:27.548654301Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 06:09:27.634112 sonic INFO syncd.sh[22999]: syncd
Nov 10 06:09:27.634655 sonic INFO syncd.sh[9332]: 0
Nov 10 06:09:27.640219 sonic NOTICE root: Stopped syncd service...
Nov 10 06:09:27.645312 sonic NOTICE root: Unlocking /tmp/swss-syncd-lock (10) from syncd service
Nov 10 06:09:27.651296 sonic INFO systemd[1]: Stopped syncd service.
Nov 10 06:09:27.656606 sonic INFO systemd[1]: Stopping DHCP relay container...
Nov 10 06:09:27.901079 sonic INFO containerd[512]: time="2016-11-10T06:09:27.900974047Z" level=info msg="shim reaped" id=6574c4e328d49dba89dd78a5f7c1dc0a41ffb7d35784b805e633195548807c85
Nov 10 06:09:27.911610 sonic INFO dockerd[547]: time="2016-11-10T06:09:27.911485369Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 06:09:27.996899 sonic INFO dhcp_relay.sh[23652]: dhcp_relay
Nov 10 06:09:27.997703 sonic INFO dhcp_relay.sh[10065]: 0
Nov 10 06:09:28.002436 sonic INFO systemd[1]: Stopped DHCP relay container.
Nov 10 06:09:28.003321 sonic INFO systemd[1]: Stopping TEAMD container...
Nov 10 06:09:28.292591 sonic INFO containerd[512]: time="2016-11-10T06:09:28.292421480Z" level=info msg="shim reaped" id=cc602baadada95bf7d8cd1aa41a2de72efcbe0338dde255c23ccfe9abec6cfd4
Nov 10 06:09:28.302508 sonic INFO dockerd[547]: time="2016-11-10T06:09:28.302404002Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 06:09:28.404531 sonic INFO teamd.sh[23691]: teamd
Nov 10 06:09:28.405110 sonic INFO teamd.sh[8270]: 0
Nov 10 06:09:28.412014 sonic INFO systemd[1]: Stopped TEAMD container.
Nov 10 06:09:28.419018 sonic INFO systemd[1]: Stopping Router advertiser container...
Nov 10 06:09:29.675716 sonic INFO containerd[512]: time="2016-11-10T06:09:29.675589177Z" level=info msg="shim reaped" id=b767be4ec3eb0a555a2365b5d10bc698011215c4d55a0c0f7ecac96654138b0e
Nov 10 06:09:29.685852 sonic INFO dockerd[547]: time="2016-11-10T06:09:29.685727099Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Nov 10 06:09:29.786365 sonic INFO radv.sh[23732]: radv
Nov 10 06:09:29.787249 sonic INFO radv.sh[8298]: 0
Nov 10 06:09:29.793899 sonic INFO systemd[1]: Stopped Router advertiser container.
[orchagent.1478758153.76.core.gz]
@pollyhsu2git
Copy link

pollyhsu2git commented Oct 8, 2019 •
edited
Loading

@afzalashfak
You may try the workaround of the AS7816-64X issue report via sonic-net/sonic-buildimage#3508.

@afzalashfak
Copy link
Author

afzalashfak commented Oct 10, 2019 •
edited
Loading

@pollyhsu2git
This issue is particularly observed when we enable asymmetric pfc in an interface. It is not related to sonic-buildimage. It is a configuration issue.
FYI, I'd loaded the build using the workaround mentioned in sonic-net/sonic-buildimage#3508, before trying the configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pollyhsu2git @afzalashfak