Version cache #10613

Kalimuthu-Velappan · 2022-04-19T11:50:29Z

Support for package version caching

During the SONiC build, it downloads and installs packages for
multiple build components which includes debian packages, pip modules,
python packages, source code packages, git modules and docker images.
The package versioning feature provides an infrastructure to select
and freeze a particular version of a package.

The package version caching feature provides an infrastructure to cache
the package with the given version into a local storage, so that
subsequent installation will not download from web instead it will load
from local cache path.

The following variable controls the verion caching feature. By default
it is truned off.
SONIC_VERSION_CACHE_METHOD = [none/cache]
SONIC_VERSION_CACHE_SOURCE = /path/to/local/storage

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

lgtm-com · 2022-04-19T12:15:45Z

This pull request introduces 1 alert when merging 0c0f35999358becd3d1b5c60d14c83c1d7455499 into 598ab99 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

lgtm-com · 2022-04-19T17:49:55Z

This pull request introduces 1 alert when merging 0010001239d1846beaf4e01cb25061e388ddb073 into 8b5d908 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

lgtm-com · 2022-04-28T19:32:35Z

This pull request introduces 1 alert when merging 5063102635fd8fe81df671c75c05005856233be0 into 1d84e0d - view on LGTM.com

new alerts:

1 for Modification of parameter with default

lgtm-com · 2022-05-05T13:36:45Z

This pull request introduces 1 alert when merging 8a98c4ee09ad925aad87ccdb4fddb0200edc7bcd into 7c4ee43 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

lgtm-com · 2022-05-06T05:38:26Z

This pull request introduces 1 alert when merging af492ce46684e7a0c0c802b5b4fe5b424aa104a8 into 9ae17e6 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

Kalimuthu-Velappan · 2022-05-06T12:22:37Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-05-06T12:22:39Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-05-06T12:22:51Z

Azure Pipelines successfully started running 1 pipeline(s).

liushilongbuaa · 2022-05-07T03:00:56Z

Makefile.work

-	    { echo Image $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) not found. Building... ; \
-	      $(DOCKER_BASE_BUILD) ; \
-	      scripts/collect_docker_version_files.sh $(SLAVE_BASE_IMAGE):$(SLAVE_BASE_TAG) target ; }
+	@$(DOCKER_BUILD)


sonic-slave-base-build is different from sonic-slave-build.
Why put them together?

liushilongbuaa · 2022-05-07T03:03:40Z

Makefile.work

-	@docker inspect --type image $(SLAVE_IMAGE):$(SLAVE_TAG) &> /dev/null || \
-	    { echo Image $(SLAVE_IMAGE):$(SLAVE_TAG) not found. Building... ; \
-	    $(DOCKER_BUILD) ; }
+	@$(DOCKER_BUILD)


Because of Calling DOCKER_BUILD twice, you get this PR checker error:

docker tag sonic-slave-bullseye:00952e8de3f 00952e8de3f sonicdev-microsoft.azurecr.io/sonic-slave-bullseye:latest

Thanks for your guidance. Fixed it

liushilongbuaa · 2022-05-07T05:12:35Z

src/sonic-build-hooks/hooks/apt-get

+        set -x
+        if [[ "${FORCE_UPDATE}" == Y  ]]; then
+            UPDATE=y
+        elif $(set -- /var/lib/apt/lists/*_debian_dists_${DISTRO}_InRelease; test -e "$1"); then


"$1" is "update"?

Yes, $1 is an update arg which is coming from 'apt-get update'. During apt-get update, it will cache the debian package release files. so that subsequent build will be loaded from cache.

liushilongbuaa · 2022-05-07T05:20:20Z

Makefile.work

+SONIC_OVERRIDE_BUILD_VARS += SONIC_VERSION_CACHE_SOURCE=$(SONIC_VERSION_CACHE_SOURCE)
+export SONIC_VERSION_CACHE SONIC_VERSION_CACHE_SOURCE
+$(shell test -d $(SONIC_VERSION_CACHE_SOURCE) || \
+    mkdir -p $(SONIC_VERSION_CACHE_SOURCE) && chmod -f 777 $(SONIC_VERSION_CACHE_SOURCE) 2>/dev/null )


I find error log in the PR validation.
mkdir: cannot create directory '/vcache/sonic-slave-bullseye': Permission denied

It is a merge issue. Fixed it.

liushilongbuaa · 2022-05-07T05:32:44Z

[ FAIL LOG END ] [ target/python-wheels/buster/scapy-2.4.5-py2.py3-none-any.whl ]
make: *** [slave.mk:754: target/python-wheels/buster/scapy-2.4.5-py2.py3-none-any.whl] Error 1
make: *** Waiting for unfinished jobs....
PR validation failed. But it shows success. There must be something wrong.

liushilongbuaa · 2022-05-07T05:37:21Z

Makefile.cache

@@ -71,6 +71,8 @@ SONIC_COMMON_FLAGS_LIST :=  $(CONFIGURED_PLATFORM) \
                            $(CONFIGURED_ARCH) \
                            $(BLDENV) \
                            $(SONIC_DEBUGGING_ON) \
+                            $(SONIC_VERSION_CONTROL_COMPONENTS) \


Makefile.cache runs before pip/apt-get. If source code and dependencies not changed, it will copy package from local path. This feature shouldn't change Makefile.cache.

liushilongbuaa · 2022-05-07T05:45:56Z

Makefile.work

@@ -294,26 +314,61 @@ endif

 endif

-DOCKER_BASE_BUILD = docker build --no-cache \
+DOCKER_AUTH:=docker login -u sonicbrcm -p 4b5d1f28-6f43-41da-a794-88805ee8fc2d 


It's not safe to push to github.
And why we should login to dockerhub using special account?

Added for the local build, accidentally it got merged into PR code. Just removed it.

lgtm-com · 2022-05-09T04:25:31Z

This pull request introduces 1 alert when merging e194e412f0d30751f756e6068fde0872fb6bca38 into 71a515e - view on LGTM.com

new alerts:

1 for Modification of parameter with default

Kalimuthu-Velappan · 2022-05-11T17:34:01Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-05-11T17:34:02Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-05-11T17:34:14Z

Azure Pipelines successfully started running 1 pipeline(s).

Kalimuthu-Velappan · 2022-05-12T11:23:56Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-05-12T11:23:58Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-05-12T11:24:10Z

Azure Pipelines successfully started running 1 pipeline(s).

Kalimuthu-Velappan · 2022-05-16T11:12:34Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-05-16T11:12:35Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-05-16T11:12:48Z

Azure Pipelines successfully started running 1 pipeline(s).

Kalimuthu-Velappan · 2022-05-17T13:33:55Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-05-17T13:33:57Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-05-17T13:34:11Z

Azure Pipelines successfully started running 1 pipeline(s).

lgtm-com · 2022-05-23T12:38:53Z

This pull request introduces 1 alert when merging 900eb928251f4bf633b9d87bc4c3c6d26d616810 into b313563 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

zhangyanzhao · 2022-07-26T20:29:45Z

build failure blocks the merge.

During the SONiC build, it downloads and installs packages for multiple build components which includes debian packages, pip modules, python packages, source code packages, git modules and docker images. The package versioning feature provides an infrastructure to select and freeze a particular version of a package. The package version caching feature provides an infrastructure to cache the package with the given version into a local storage, so that subsequent installation will not download from web instead it will load from local cache path. The following variable controls the verion caching feature. By default it is truned off. SONIC_VERSION_CACHE_METHOD = [none/cache] SONIC_VERSION_CACHE_SOURCE = /path/to/local/storage

linux-foundation-easycla · 2022-08-03T10:55:30Z

The committers listed above are authorized under a signed CLA.

✅ login: Kalimuthu-Velappan (f2da2aa)

lgtm-com · 2022-08-03T11:22:39Z

This pull request introduces 1 alert when merging f2da2aa into 45ded68 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

liushilongbuaa · 2022-08-05T09:36:24Z

It is better to split this big PR into several little PRs.

Setup version cache inside slave docker.
- version cache for git
- version cache for apt/apt-get
- version cache for pip/pip3
- ...
Setup version cache when building slave docker.
Setup version cache when building SONiC image.

Kalimuthu-Velappan · 2022-08-16T17:02:00Z

I agree that it would be better to handle the PR for each functionality. But it is very difficult to split them into multiple PRs as it's all interdependent with each other on the caching framework. It would be easy for tracking and testing the complete framework if it is a single PR.

Do you really want to split them into multiple?

Kalimuthu-Velappan · 2022-08-17T03:41:43Z

/azpw run Azure.sonic-buildimage

mssonicbld · 2022-08-17T03:41:44Z

/AzurePipelines run Azure.sonic-buildimage

azure-pipelines · 2022-08-17T03:41:56Z

Azure Pipelines successfully started running 1 pipeline(s).

liushilongbuaa · 2022-08-17T07:57:22Z

I agree that it would be better to handle the PR for each functionality. But it is very difficult to split them into multiple PRs as it's all interdependent with each other on the caching framework. It would be easy for tracking and testing the complete framework if it is a single PR.

Do you really want to split them into multiple?

I suggest keeping original branch and split them.
Change will be clear:
1. Change Makefile.work and slave.mk to pass config variable
2. Mount cache dir.
3. Change hooks logic:
i. if not exist; then pip download; fi; pip install locally
ii. if not exist; then apt-get download; fi; apt-get install locally

liushilongbuaa · 2022-08-17T07:58:11Z

@xumia , please help review.

Kalimuthu-Velappan · 2022-08-18T03:54:24Z

As requested, I will try to split into multiple PRs and raise a separate list of PRs.

zhangyanzhao · 2022-12-07T23:02:18Z

Is the PR splitted? Can you please update the HLD PR with the new code PRs listed? @Kalimuthu-Velappan

Kalimuthu-Velappan · 2022-12-08T12:35:47Z

Updated the code PR list in the HLD PR. sonic-net/SONiC#941

…

On Thu, Dec 8, 2022 at 4:32 AM Yanzhao Zhang ***@***.***> wrote: Is the PR splitted? Can you please update the HLD PR with the new code PRs listed? @Kalimuthu-Velappan <https://github.com/Kalimuthu-Velappan> — Reply to this email directly, view it on GitHub <#10613 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AM2UC2UBFCAPAVLMV74KMODWMEJQJANCNFSM5TYSYRQQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.

Kalimuthu-Velappan requested review from qiluo-msft, xumia and lguohan as code owners April 19, 2022 11:50

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch 2 times, most recently from 4fc95ab to 0010001 Compare April 19, 2022 17:09

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch from 0010001 to 5063102 Compare April 28, 2022 19:11

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch from 5063102 to 8a98c4e Compare May 5, 2022 12:48

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch 2 times, most recently from 604ee1f to af492ce Compare May 6, 2022 05:13

liushilongbuaa reviewed May 7, 2022

View reviewed changes

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch 4 times, most recently from 89d4361 to e194e41 Compare May 9, 2022 04:03

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch from e194e41 to 3027bce Compare May 10, 2022 00:53

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch from 45ef6a2 to 900eb92 Compare May 23, 2022 12:11

Kalimuthu-Velappan requested a review from a team as a code owner June 10, 2022 02:01

Kalimuthu-Velappan force-pushed the VERSION_CACHE branch from 900eb92 to f2da2aa Compare August 3, 2022 10:55

Version cache #10613

Are you sure you want to change the base?

Version cache #10613

Conversation

Kalimuthu-Velappan commented Apr 19, 2022

Why I did it

How I did it

How to verify it

Which release branch to backport (provide reason below if selected)

Description for the changelog

Link to config_db schema for YANG module changes

A picture of a cute animal (not mandatory but encouraged)

lgtm-com bot commented Apr 19, 2022

lgtm-com bot commented Apr 19, 2022

lgtm-com bot commented Apr 28, 2022

lgtm-com bot commented May 5, 2022

lgtm-com bot commented May 6, 2022

Kalimuthu-Velappan commented May 6, 2022

mssonicbld commented May 6, 2022

azure-pipelines bot commented May 6, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liushilongbuaa commented May 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liushilongbuaa May 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lgtm-com bot commented May 9, 2022

Kalimuthu-Velappan commented May 11, 2022

mssonicbld commented May 11, 2022

azure-pipelines bot commented May 11, 2022

Kalimuthu-Velappan commented May 12, 2022

mssonicbld commented May 12, 2022

azure-pipelines bot commented May 12, 2022

Kalimuthu-Velappan commented May 16, 2022

mssonicbld commented May 16, 2022

azure-pipelines bot commented May 16, 2022

Kalimuthu-Velappan commented May 17, 2022

mssonicbld commented May 17, 2022

azure-pipelines bot commented May 17, 2022

lgtm-com bot commented May 23, 2022

zhangyanzhao commented Jul 26, 2022

linux-foundation-easycla bot commented Aug 3, 2022 • edited Loading

lgtm-com bot commented Aug 3, 2022

liushilongbuaa commented Aug 5, 2022

Kalimuthu-Velappan commented Aug 16, 2022

Kalimuthu-Velappan commented Aug 17, 2022

mssonicbld commented Aug 17, 2022

azure-pipelines bot commented Aug 17, 2022

liushilongbuaa commented Aug 17, 2022

liushilongbuaa commented Aug 17, 2022

Kalimuthu-Velappan commented Aug 18, 2022

zhangyanzhao commented Dec 7, 2022

Kalimuthu-Velappan commented Dec 8, 2022 via email

liushilongbuaa May 7, 2022 •

edited

Loading

linux-foundation-easycla bot commented Aug 3, 2022 •

edited

Loading