[bgpd] Static route removal failure with "Too few "nexthop" instances" error log

[vivekrnv]

This is a statistical issue seen in SONIC images >= 202411

Only seen twice in our regression while running the test test_everflow_remove_unused_ecmp_next_hop[cli-upstream-default

We see the following error log:

ERR bgp#mgmtd[33]: [KF39R-NRP86] mgmt_txn_notify_be_cfgdata_reply: ERROR: CFGDATA_CREATE_REQ sent to 'staticd' failed txn-id: 91 err: YANG error(s):#012 Path: Schema location "/frr-routing:routing/control-plane-protocols/control-p
2025 Jan 21 21:55:43.831074 r-leopard-72 ERR bgp#mgmtd[33]: [KF39R-NRP86] mgmt_txn_notify_be_cfgdata_reply: ERROR: CFGDATA_CREATE_REQ sent to 'staticd' failed txn-id: 91 err: YANG error(s):#012 Path: Schema location "/frr-routing:routing/control-plane-protocols/control-plane-protocol/frr-staticd:staticd/route-list/path-list/frr-nexthops/nexthop".#012 Error: Too few "nexthop" instances.
2025 Jan 21 21:55:43.832525 r-leopard-72 ERR bgp#mgmtd[33]: [GGJTQ-VTT01] SET_CONFIG request for client 0x16c failed, Error: 'YANG error(s):#012 Path: Schema location "/frr-routing:routing/control-plane-protocols/control-plane-protocol/frr-staticd:staticd/route-list/path-list/frr-nexthops/nexthop".#012 Error: Too few "nexthop" instances.#012'

This is unfortunately not reproduced manually. Suggestion to add debug logs is being asked in the FRR community and is tracked here https://frrouting.slack.com/archives/C95S79K7T/p1738004753526379

[hasan-brcm]

This needs to be followed up in frr community.

sharpd
  11:05 AM
If mgmtd is spitting out this error:


sharpd
  [11:05 AM](https://frrouting.slack.com/archives/C95S79K7T/p1738004735293459)
2025 Jan 23 05:14:07.588242 r-leopard-44 ERR bgp#mgmtd[39]: [KF39R-NRP86] mgmt_txn_notify_be_cfgdata_reply: ERROR: CFGDATA_CREATE_REQ sent to 'staticd' failed txn-id: 8 err: YANG error(s):#012 Path: Schema location "/frr-routing:routing/control-plane-protocols/control-plane-protocol/frr-staticd:staticd/route-list/path-list/frr-nexthops/nexthop".#012 Error: Too few "nexthop"
instances.
 
2025 Jan 23 05:14:07.588350 r-leopard-44 ERR bgp#mgmtd[39]: [SG2F2-M78GE] COMMIT_CONFIG request for client 0x42 failed, Error: 'YANG error(s):#012 Path: Schema location "/frr-routing:routing/control-plane-protocols/control-plane-protocol/frr-staticd:staticd/route-list/path-list/frr-nexthops/nexthop".#012 Error: Too few "nexthop" instances.#012'


vivek reddy
  Thursday at 1:31 PM
Input for this is static route removal. When this error log is seen, the route is failed to be removed.


sharpd
  [Thursday at 1:31 PM](https://frrouting.slack.com/archives/C95S79K7T/p1738272672402969?thread_ts=1738004753.526379&cid=C95S79K7T)
@chopps
 or 
@idryzhov
 do you have suggestions on what is needed to debug here?


idryzhov
  Friday at 10:44 AM
What are you trying to do? The error is clear - there are less then minimum allowed nexthops in a route (I guess 0). If you’re using API to remove the nexthop, then you should instead remove the whole route. If you’re using vtysh, then it’s a bug in vtysh.

[hasan-brcm]

@vivekrnv , please collect techsupport and upload so that we can see how this config was applied.

[hasan-brcm]

@dgsudharsan to follow up with @vivekrnv. Please collect TS once this issue is hit, and upload here.