libyang: upgrade to v3 step 1 -- import libyang3 / libyang3-py3, remove stale versions

[bradh352]

PR Tracking Table

All future PRs in this series use commits stacked on top of this one. So the PRs may be larger, care will need to be taken to look at the individual commits to see what is new.

PR	description	dependencies
#21679	This PR. remove libyang 1.0.184. build libyang 3.7.8, libyang-python 3.0.3. Migrate FRR to libyang 3. Update sonic-yang-models test cases specifically to make it easier to see libyang3-specific changes once those land.	None
#21719	Port sonic-yang-models test cases	#21679
#21716	Port sonic-yang-mgmt.	#21679, #21719
#21718	Update all subprojects that use libyang1 to depend on libyang3	#21679, #21719, #21716
sonic-net/sonic-utilities#3766	sonic-utilities	#21679, #21719, #21716, #21718
#21752, sonic-net/sonic-swss-common#973	swss-common	#21679, #21719, #21716, #21718
#21789, sonic-net/sonic-mgmt-common#157	sonic-mgmt-common (GOlang)	#21679, #21718
sonic-net/sonic-platform-daemons#589 , sonic-net/sonic-swss#3518 , sonic-net/sonic-dbsyncd#74 , sonic-net/sonic-restapi#157 , sonic-net/sonic-snmpagent#348 , sonic-net/sonic-platform-common#544	azure pipelines: update various external repos to pull in the libyang3 debs that are otherwise not in need of modification	all of the above
#21790	cleanup / remove libyang1	all of the above

Why I did it

SONiC currently uses 3 versions of libyang: 1.0.73, 1.0.184, and 2.1.148. All are end of life and potentially have security issues. The current release is 3.7.8 and that is what we are targeting here.

There are also numerous libyang related bugs that have been "worked around" in SONiC, such as PR #21078. Hopefully upgrading to a modern version will resolve such issues.

This PR has been extracted from a larger changeset to make it easier to review. Once this is merged, a follow-up PR will be made to port any existing 1.0.73 code to libyang3 (this is mostly done already, but spans multiple repositories). There are some test case modifications included to simplify the review of the upgrade to the libyang3, it should be straight forward to review these changes.

I sent an email to the mailing list about this here:
https://lists.sonicfoundation.dev/g/sonic-dev/message/938

This PR is must be applied first, but there is no need to wait since this just starts setting the groundwork for the changes.

Work item tracking

N/A

How I did it

This PR gets rid of 1.0.184 as it appears unused, and upgrades 2.1.148 to 3.7.8 (including backporting FRR patches for libyang support). It also generates a build for the new libyang3 python bindings as they are no longer included in the libyang sources but are instead in an external repository. Libyang1 and Libyang3 are designed that both can be installed simultaneously (except the dev packages).

I have also already started upstreaming patches to libyang and libyang-python for inclusion for issues that have been found that impact SONiC use-cases.

Please review each commit separately in this PR, it would be easier than reviewing it as a whole. The commit messages should explain what is going on and why it is needed.

How to verify it

See that the build still succeeds and existing test cases still pass

Which release branch to backport (provide reason below if selected)

N/A

Tested branch (Please provide the tested image version)

master as of 20250207

Description for the changelog

libyang: upgrade to v3 step 1

Link to config_db schema for YANG module changes

N/A

A picture of a cute animal (not mandatory but encouraged)

Signed-off-by: Brad House (@bradh352)

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[bradh352]

@ganglyu @qiluo-msft please review ... this needs to be merged before I can submit PRs for the migration of the libyang 1.0.73 code to v3 ...

[wen587]

Hi @bradh352 , could you help me understand why src/sonic-yang-models/tests/yang_model_tests/tests/ is impacted? Aren't they dependent on libyang 1.0.73?

[ganglyu]

@bradh352

Thank you very much for your contribution!

We have encountered numerous issues with libyang 1.0.73, which is currently used by sonic-yang-models. On the other hand, libyang 1.0.184 and libyang 2.1.148, which are used by FRR, have not presented any issues to our knowledge.

My suggestion is to install libyang3 in this PR without replacing the existing dependency.

[ganglyu]

Why do we need to modify the unit test?

[bradh352]

From the commit message:

sonic-yang-models: Update test cases in preparation for libyang3
Convert some yang model test cases that use eStr for failure matching
to use eStrKey instead to standardize expected failure matching so
when messsages change they can be updated in one place instead of all
over.

Also, split out some new eStrKey values that are needed for libyang3
due to different messages, and document the libyang3 values needed once
upgraded but keep libyang1 values for now.

Finally, some tests were using both eStr and eStrKey and the eStr
portion is not needed to perform a meaningful validation, plus the
eStr in these cases are not valid for libyang3.

So basically, its just to make the review of the upgrade to libyang3 easier. I've actually got libyang3 working and passing these test cases so I wanted a bulk of the test case changes done here to make it easier to review the more complex porting of the libyang1 python swig module to libyang3 python cffi.

[bradh352]

@ganglyu

@bradh352

Thank you very much for your contribution!

We have encountered numerous issues with libyang 1.0.73, which is currently used by sonic-yang-models. On the other hand, libyang 1.0.184 and libyang 2.1.148, which are used by FRR, have not presented any issues to our knowledge.

My suggestion is to install libyang3 in this PR without replacing the existing dependency.

I honestly believe not having multiple versions of libyang to be extremely important. I think their API is stabilizing. The only thing required was to apply some commits already upstream in FRR 10.2 to support 3.7.8. From the commit message:

This also backports a series of commits from FRR to support libyang3,
upgrading FRR itself isn't currently in scope:

• FRRouting/frr@290c6e3
• FRRouting/frr@8e2b2ca
• FRRouting/frr@22eccbf
• FRRouting/frr@87c9060

I think we'd be foolish to kick the can down the road while we're working on it now. I also haven't verified if libyang2 and libyang3 can be installed simultaneously like libyang1 and libyang3 can.

Of course if you tell me there's no way you could possibly accept this PR because of this you leave me little choice but I don't see a sufficient argument at this time against it.

[bradh352]

Hi @bradh352 , could you help me understand why src/sonic-yang-models/tests/yang_model_tests/tests/ is impacted? Aren't they dependent on libyang 1.0.73?

From the commit message:

sonic-yang-models: Update test cases in preparation for libyang3
Convert some yang model test cases that use eStr for failure matching
to use eStrKey instead to standardize expected failure matching so
when messsages change they can be updated in one place instead of all
over.

Also, split out some new eStrKey values that are needed for libyang3
due to different messages, and document the libyang3 values needed once
upgraded but keep libyang1 values for now.

Finally, some tests were using both eStr and eStrKey and the eStr
portion is not needed to perform a meaningful validation, plus the
eStr in these cases are not valid for libyang3.

So basically, its just to make the review of the upgrade to libyang3 easier. I've actually got libyang3 working and passing these test cases so I wanted a bulk of the test case changes done here to make it easier to review the more complex porting of the libyang1 python swig module to libyang3 python cffi.

[bradh352]

Guess I need to figure out python wheel packaging for cross compilation to support marvell armhf (its an architecture-specific package since it uses cffi)? I clearly am not testing such a use case currently in my local environment. Anyone have any pointers for this?

[ganglyu]

@ganglyu

@bradh352
Thank you very much for your contribution!
We have encountered numerous issues with libyang 1.0.73, which is currently used by sonic-yang-models. On the other hand, libyang 1.0.184 and libyang 2.1.148, which are used by FRR, have not presented any issues to our knowledge.
My suggestion is to install libyang3 in this PR without replacing the existing dependency.

I honestly believe not having multiple versions of libyang to be extremely important. I think their API is stabilizing. The only thing required was to apply some commits already upstream in FRR 10.2 to support 3.7.8. From the commit message:

This also backports a series of commits from FRR to support libyang3,
upgrading FRR itself isn't currently in scope:

• FRRouting/frr@290c6e3
• FRRouting/frr@8e2b2ca
• FRRouting/frr@22eccbf
• FRRouting/frr@87c9060

I think we'd be foolish to kick the can down the road while we're working on it now. I also haven't verified if libyang2 and libyang3 can be installed simultaneously like libyang1 and libyang3 can.

Of course if you tell me there's no way you could possibly accept this PR because of this you leave me little choice but I don't see a sufficient argument at this time against it.

I will engage the owner of SONiC FRR to review the related changes.

[ganglyu]

Guess I need to figure out python wheel packaging for cross compilation to support marvell armhf (its an architecture-specific package since it uses cffi)? I clearly am not testing such a use case currently in my local environment. Anyone have any pointers for this?

I'm not an expert, but I believe the Sonic build pipeline uses the armhf system to build the Sonic image for Marvell armhf. Therefore, we shouldn't encounter any cross-compilation issues in the Sonic build pipeline.

[bradh352]

Guess I need to figure out python wheel packaging for cross compilation to support marvell armhf (its an architecture-specific package since it uses cffi)? I clearly am not testing such a use case currently in my local environment. Anyone have any pointers for this?

Looks like azure isn't actually cross compiling at all, its running an armhf image that is advertising armv8l support which I think just means the kernel knows its a 64bit cpu ... but the system was compiled for armv7l/armhf, so python is I think determining the system based on uname -m.

I tried using a 64bit arm vm on my macbook to build sonic, but it immediately fails as it thinks I need to use QEMU multiarch ... even hacking the makefile to disable that and it fails later on. Parallels doesn't allow me to directly install an armhf image, so I guess I'm dead in the water trying to test this on my own hardware right now. I might need to pick up an rpi5 or something to build but that seems like it might be really really slow....

So I'll probably try to throw in some test commits to see if a simple 'setarch armhf' works around the issue.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[bradh352]

@ganglyu now that all PRs have been submitted in this series, can we start getting some eyes on this and get things moving?

This PR and the follow-up PR #21719 are low risk as they don't actually touch any code (other than backporting some FRR patches that are already upstream).

[ganglyu]

Let's wait for the check pipeline to complete.

[bradh352]

Let's wait for the check pipeline to complete.

It failed due to an issue in make configure unrelated to this patch. I'll rebase it so it forces a rebuild.

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[bradh352]

@ganglyu
the t1-lag test failed on both step1 and step2 PRs due to timeout, all other tests failed ... this doesn't appear in any way related to this patch (don't see how it could). going to rebase again to force another build, but the automated test system has been really flaky lately in general.

[mssonicbld]

/azp run Azure.sonic-buildimage

[mssonicbld]

/azp run Azure.sonic-buildimage

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).