Update reproducible comparing tests on linux (not docker image) #3687

sophia-guo · 2024-03-05T20:49:49Z

Part of #3683

SCM_REF build issue Reproducible comparing using sbom json file got fatal: Not a valid object name jdk-21.0.3+6_adopt-266-ge01a50b2cac #3710
sbom with personal build branch and sha
Retrieving boot JDK 21.0.2+13-LTS not available -- BOOTJDK version is not abvailable by API https://ci.adoptium.net/view/Test_grinder/job/Grinder/9300/ -- remove suffix -LTS if bootjdk include -LTS
error: pathspec '"e95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b"' did not match any file(s) known to git. Update reproducible comparing tests on linux (not docker image) #3687 (comment)
Support running with upsteam jenkins jobs by mount test resources hit the user permission issue test-ibmcloud-ubuntu1604-x64-1 cannot clean workspace (Docker test issue) infrastructure#3490

The text was updated successfully, but these errors were encountered:

sophia-guo · 2024-03-05T20:56:30Z

Trying in adopitum jenkins with some necessary updates.
On test-ibmcloud-rhel7-x64-1 https://ci.adoptium.net/view/Test_system/job/Test_openjdk21_hs_dev.system_x86-64_linux/21/console

15:19:17  TESTING:
15:19:18  Loaded plugins: product-id, search-disabled-repos, subscription-manager
15:19:18  You need to be root to perform this command.

That is in linux_repro_build_compare.sh if it's rhel that some extra tools are needed to install https://github.com/adoptium/temurin-build/blob/master/tooling/reproducible/linux_repro_build_compare.sh#L27-L37. However jenkins doesn't have the permission.

sophia-guo · 2024-03-05T20:59:19Z

For non-rhel for now jq is need to install, which also has the permission issue. Can we ask the infra to install jq, which is also required for windows. @andrew-m-leonard @sxa

sophia-guo · 2024-03-18T14:30:12Z

SCM_REF build issue #3710

sophia-guo · 2024-03-22T20:30:44Z

Upstream build is triggered by personal temrurin-build ( could be the old repo openjdk-build) branch and sha, which will cause build failed -

   {
      "name" : "Temurin Build Ref",
      "value" : "https://github.com/andrew-m-leonard/openjdk-build/commit/532e5d1f4c39c121b42303c8f29d69e007ce96fe"
    },

15:32:44  Cloning into 'temurin-build'...
15:32:45  error: pathspec '"532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe\n532e5d1f4c39c121b42303c8f29d69e007ce96fe"' did not match any file(s) known to git.

@andrew-m-leonard should personal branch be supported?

sophia-guo · 2024-03-25T12:56:15Z

@andrew-m-leonard see #3687 (comment)

sophia-guo · 2024-03-25T20:55:13Z

#3687 (comment)

This happened if Temurin Build Ref is referred develper's personal repo.

However I've tried the release version with main repo got the similar issue. - Using release jdk22 linux-x64 https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json/*view*/ I got similar error

16:36:01  error: pathspec '"e95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b\ne95140bf52915bda6c5ff271538789ed9a55162b"' did not match any file(s) known to git.
16:36:02  reproducibleCompare

https://ci.adoptium.net/view/Test_grinder/job/Grinder/9302/consoleFull

sophia-guo · 2024-03-25T21:06:16Z

e95140b is the commit of branch https://github.com/adoptium/temurin-build/commits/v2024.03.01 , which is not in the master. Hence can't be found. Suppose reproducible comparing should support the release? @andrew-m-leonard

andrew-m-leonard · 2024-03-26T10:09:29Z

e95140b is the commit of branch https://github.com/adoptium/temurin-build/commits/v2024.03.01 , which is not in the master. Hence can't be found. Suppose reproducible comparing should support the release? @andrew-m-leonard

Should it matter where the commitSHA is from ?

sophia-guo · 2024-03-26T13:01:28Z

Shouldn't be. But, according to the test the sha can't be found.
https://ci.adoptium.net/view/Test_grinder/job/Grinder/9302/consoleFull

andrew-m-leonard · 2024-03-26T14:29:34Z

I suspect this needs changing to not assume a git "branch" ? https://github.com/sophia-guo/openjdk-build/blob/33e187ebdab191756b3f6584ef3a24862e529d4c/test/system/reproducibleCompare/build.xml#L61
probably needs to clone, then do a checkout by SHA

sophia-guo · 2024-03-27T04:25:27Z

The build.xml only pre stages all shell scripts. https://github.com/sophia-guo/openjdk-build/blob/33e187ebdab191756b3f6584ef3a24862e529d4c/test/system/reproducibleCompare/build.xml#L61.
The reproducible build params are coming from sbom.json file.

The issue can be easily reproduced with master branch by following command

docker run -v "$(TEST_RESROOT):/home/jenkins/test" -w "/home/jenkins/test" --name reproducibleCompare centos:7 /bin/bash /home/jenkins/test/linux_repro_build_compare.sh  /home/jenkins/jdkbinary/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json /home/jenkins/jdkbinary/OpenJDK22U-jdk_x64_linux_hotspot_22_36.tar.gz

export $(TEST_RESROOT) to local temurin-build/tooling/reproducible. Download the jdkfile and sbom file to same directory. Update linux_repro_build_compare.sh to support local files.

andrew-m-leonard · 2024-03-27T08:09:29Z

The reproducible build params are coming from sbom.json file.

yeah, I am wondering if the SCM_REF for these examples in the SBOM should be empty ?
and in which case the git checkout should then checkout by openjdk_commit

sophia-guo · 2024-03-27T12:36:48Z

The files are release files
https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-jdk_x64_linux_hotspot_22_36.tar.gz

https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json

        {
          "name" : "SCM Ref",
          "value" : "jdk-22+36_adopt"
        },
        {
          "name" : "OpenJDK Source Commit",
          "value" : "https://github.com/adoptium/jdk22/commit/1aa79b171e3f8b58852a01e08d82230565a01989"
        },
        {
          "name" : "Temurin Build Ref",
          "value" : "https://github.com/adoptium/temurin-build/commit/e95140bf52915bda6c5ff271538789ed9a55162b"

With linux it should be easily reproduced by

linux_repo_build_compare.sh https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-sbom_x64_linux_hotspot_22_36.json https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/5/artifact/target/linux/x64/temurin/OpenJDK22U-jdk_x64_linux_hotspot_22_36.tar.gz

andrew-m-leonard · 2024-03-27T14:46:31Z

@sophia-guo The jq command needs fixing, the git checkout command it is running is:

(cd temurin-build && git checkout e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b e95140bf52915bda6c5ff271538789ed9a55162b)

because jq is returning:

andrew@andrew:~/temurin-build/tooling/reproducible$ jq -r '.components[] | .properties[] | select (.name == "Temurin Build Ref") | .value' "OpenJDK22U-sbom_x64_linux_hotspot_22_36.json" | awk -F/ '{print $NF}'
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b
e95140bf52915bda6c5ff271538789ed9a55162b

sophia-guo · 2024-03-27T16:39:44Z

#3687 (comment)

I see. Seems SBOM file content changed. For this specific sbomfile there are 4 components ( jdk, jre, static-lib, debug). All properties of those 4 components are duplicate except filename itself.

sophia-guo · 2024-04-02T18:16:35Z

The diff for reproducing release jdk22 is as following

diff -r jdk-22+36/release compare.1/jdk-22+36/release
11,13d10
< BUILD_SOURCE="git:e95140bf52915bda6c5ff271538789ed9a55162b"
< BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build.git"
< SOURCE_REPO="https://github.com/adoptium/jdk22.git"

That is for reproducing one the release file doesn't have those information, which is minor but not sure if it is expected. @andrew-m-leonard

andrew-m-leonard · 2024-04-03T14:46:40Z

The diff for reproducing release jdk22 is as following
diff -r jdk-22+36/release compare.1/jdk-22+36/release
11,13d10
< BUILD_SOURCE="git:e95140bf52915bda6c5ff271538789ed9a55162b"
< BUILD_SOURCE_REPO="https://github.com/adoptium/temurin-build.git"
< SOURCE_REPO="https://github.com/adoptium/jdk22.git"
That is for reproducing one the release file doesn't have those information, which is minor but not sure if it is expected. @andrew-m-leonard

I thought that should be clean'd by this

temurin-build/tooling/reproducible/repro_common.sh

Line 349 in 646a79f

sed -i '/^BUILD_SOURCE=.*$/d' "${DIR}/release"

sophia-guo · 2024-04-03T15:03:34Z

Linux_repro_build_compare.sh doesn't use the repro_common.sh

andrew-m-leonard · 2024-04-03T15:08:31Z

Linux_repro_build_compare.sh doesn't use the repro_common.sh

Ah yes, so we should re-use that clean method to make these compare if possible?

sophia-guo · 2024-04-03T15:11:10Z

We can do in separate issue or pr later.

sophia-guo self-assigned this Mar 5, 2024

sophia-guo added this to 2024 1Q Adoptium Plan Mar 5, 2024

github-actions bot added the x-linux Issues that affect or relate to the x64/x32 LINUX OS label Mar 5, 2024

sophia-guo mentioned this issue Mar 5, 2024

Add reproducible comparison tests for windows and update tests for linux running out of docker image #3683

Closed

2 tasks

jiekang moved this to In Progress in 2024 1Q Adoptium Plan Mar 6, 2024

sophia-guo mentioned this issue Mar 6, 2024

Ansible request for jq for test machines adoptium/infrastructure#3453

Closed

github-actions bot added jenkins Issues that enhance or fix our jenkins server testing Issues that enhance or fix our test suites labels Mar 28, 2024

sophia-guo mentioned this issue Mar 29, 2024

Update reproducible comparing tests with centos:7 #3739

Merged

sophia-guo closed this as completed in #3739 Apr 5, 2024

github-project-automation bot moved this from In Progress to Done in 2024 1Q Adoptium Plan Apr 5, 2024

sophia-guo removed this from 2024 1Q Adoptium Plan Apr 5, 2024

sophia-guo added this to 2024 2Q Adoptium Plan Apr 5, 2024

sophia-guo moved this to Done in 2024 2Q Adoptium Plan Apr 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update reproducible comparing tests on linux (not docker image) #3687

Update reproducible comparing tests on linux (not docker image) #3687

sophia-guo commented Mar 5, 2024 •

edited

Loading

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 18, 2024

sophia-guo commented Mar 22, 2024 •

edited

Loading

sophia-guo commented Mar 25, 2024

sophia-guo commented Mar 25, 2024 •

edited

Loading

sophia-guo commented Mar 25, 2024

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 26, 2024 •

edited

Loading

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 27, 2024

andrew-m-leonard commented Mar 27, 2024 •

edited

Loading

sophia-guo commented Mar 27, 2024 •

edited

Loading

andrew-m-leonard commented Mar 27, 2024

sophia-guo commented Mar 27, 2024

sophia-guo commented Apr 2, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

Update reproducible comparing tests on linux (not docker image) #3687

Update reproducible comparing tests on linux (not docker image) #3687

Comments

sophia-guo commented Mar 5, 2024 • edited Loading

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 5, 2024

sophia-guo commented Mar 18, 2024

sophia-guo commented Mar 22, 2024 • edited Loading

sophia-guo commented Mar 25, 2024

sophia-guo commented Mar 25, 2024 • edited Loading

sophia-guo commented Mar 25, 2024

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 26, 2024 • edited Loading

andrew-m-leonard commented Mar 26, 2024

sophia-guo commented Mar 27, 2024

andrew-m-leonard commented Mar 27, 2024 • edited Loading

sophia-guo commented Mar 27, 2024 • edited Loading

andrew-m-leonard commented Mar 27, 2024

sophia-guo commented Mar 27, 2024

sophia-guo commented Apr 2, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

andrew-m-leonard commented Apr 3, 2024

sophia-guo commented Apr 3, 2024

sophia-guo commented Mar 5, 2024 •

edited

Loading

sophia-guo commented Mar 22, 2024 •

edited

Loading

sophia-guo commented Mar 25, 2024 •

edited

Loading

sophia-guo commented Mar 26, 2024 •

edited

Loading

andrew-m-leonard commented Mar 27, 2024 •

edited

Loading

sophia-guo commented Mar 27, 2024 •

edited

Loading