Support Azul Zulu builds of OpenJDK publishing at the Adoptium marketplace

[sashapepin]

Azul wants to publish Azul Zulu builds of OpenJDK at the Adoptium marketplace.

The aqa-tests results for Linux platform build (zulu8.54.0.21-ca-jdk8.0.292-linux_x64) for sanity.system, extended.system, special.system, sanity.functional, extended.functional, special.functional, sanity.openjdk, extended.openjdk and special.openjdk test suites can be found here:
https://cdn.azul.com/aqavit/zulu8.54.0.21-ca-jdk8.0.292-linux_x64-results

[sxa]

We do not have the marketplace fully set up yet, but since Azul is a working group member we expect they'll be available via the marketplace in due course once it is ready :-)

[ppetrosh]

To clarify, @sxa, do the posted result look good/acceptable to you? Thanks.

[sxa]

To clarify, @sxa, do the posted result look good/acceptable to you? Thanks.

I'll defer to @smlambert on that one :-)

[smlambert]

Looks pretty good, from a quick glance though there are a few test case failures that are unexpected to see. Some may be configuration issues, would have to look more closely.

sanity.openjdk

• jdk_util
  extended.openjdk
• hotspot_jre, jdk_other, jdk_net, jdk_nio, jdk_security1, jdk_security2, jdk_text, jdk_jmx

We are currently working on an easy way to pull tagged version of test material for specific repositories. This will be what is expected for vendors wanting to list in the marketplace, and should be ready soon. (this work essentially adds scripts that make it easier for users, it is currently possible to do it now but requires some extra steps that we are working on simplifying).

I'd like to get a run where we test a release build against a GA tagged version of openjdk test material (to rule out that some failures could be due to version skew, a mismatch between test materials and code under test).

[sashapepin]

Hi @smlambert @sxa , is there any progress with the simplifying process you are working on?
I have already looked through https://github.com/adoptium/marketplace-data
and https://marketplace-api.adoptium.net/q/swagger-ui/
What should we do now to proceed with publishing Zulu (Azul) builds in the marketplace?
Thanks!

[sxa]

Hi @sashapepin Yes there are now tagged versions of the test suites - I believe the correct version for the current releases is v0.9.1 in the AQA test repositories but I'll leave it to @smlambert to confirm as I haven't been so closely involved with the marketplace certification discussions this time.

[smlambert]

If you are verifying the April builds, you are going to use aqa-tests v0.9.1-release branch and set the environment variable USE_TESTENV_PROPERTIES=true (which will pin the other test material to specific tagged versions).

Please use this doc as the basis to follow:
https://adoptium.net/docs/aqavit-verification/

For command line runs, looks like:

git clone --depth 1 --branch v0.9.1-release https://github.com/adoptium/aqa-tests.git

export TEST_JDK_HOME=
export USE_TESTENV_PROPERTIES=true
export JDK_VERSION=17
export JDK_IMPL=hotspot
export BUILD_LIST=functional // where this matches the directory of the test material you want to compile

cd aqa-tests
./get.sh
./compile.sh
cd TKG
make _sanity.functional
…
make _extended.system

Collect *.tap file and metadata file
Archive .zip
Publish .zip

The errors listed in the tap files shared in this issue, it is worth noting that you need to also supply the native testimage for the sanity.openjdk and extended.openjdk targets.

If you are using our Jenkins pipeline this can be done by setting SDK_RESOURCE=customized and CUSTOMIZED_SDK_URL=url to jdk binary url to native test image

where they are space separated. If you are running via commandline, you can export an environment variable called TESTIMAGE_PATH (which would be what we'd pass to jtreg on the -nativepath option).

In the event that there are Azul specific features that cause some of the tagged openjdk test material to fail, there is an option for temporary exclusion while we sort it out. (We want to establish the baseline set of tests that are applicable to all, and then we can thoughtfully add more tests as we understand different distributors/vendor implementations.

Happy to answer more questions and help in any way to prep you for marketplace launch. Cheers!

[sashapepin]

@smlambert thanks a lot for the detailed answer!

[sashapepin]

@smlambert could you please clarify:
Collect *.tap file and metadata file - what is a metadata file? Where it could be found?
In aqa-tests/testenv/testenv.properties I can see
AQA_REQUIRED_TARGETS=sanity.functional,extended.functional,special.functional,sanity.openjdk,extended.openjdk,sanity.system,extended.system,sanity.perf,extended.perf
Can I somehow use AQA_REQUIRED_TARGETS property to run all the reqired tests at once?
Thanks!

[smlambert]

re: #2826 (comment)

@sashapepin - you can run all the targets at once, but we typically do not recommend it since running them serially is going to take a long while...

but having said that, it is 'possible'... and if many folks are running via commandline, we should add some convenience scripts I suppose.

You will have to set
export BUILD_LIST=functional,openjdk,system,perf to ensure that all of the test material gets compiled, then presumably

And assuming I've created a script called runRequiredTargets.sh that looked like this and put it in the TKG dir:

echo "Running AQA_REQUIRED_TARGETS: $AQA_REQUIRED_TARGETS"

for i in $(echo $AQA_REQUIRED_TARGETS | sed "s/,/ /g")
do
    # execute the top-level test targets
    make  "_$i"
done

---

Updating the instructions I shared in a previous comment change to something like:

git clone --depth 1 --branch v0.9.1-release https://github.com/adoptium/aqa-tests.git

export TEST_JDK_HOME=<location of your JDK under test>
export USE_TESTENV_PROPERTIES=true
export JDK_VERSION=17
export JDK_IMPL=hotspot
export BUILD_LIST=functional,openjdk,system,perf // where these are all the directories whose test material we wish to use
export AQA_REQUIRED_TARGETS=sanity.functional,extended.functional,special.functional,sanity.openjdk,extended.openjdk,sanity.system,extended.system,sanity.perf,extended.perf // I will plant to create a PR to export this var during test setup in future

cd aqa-tests
./get.sh
./compile.sh
cd TKG
./runRequiredTargets.sh

// Collect all *.tap file from the TKG/output_`timestamp` dir and the **AQACert.log and SHA.txt file** from the TKG dir
// Archive .zip
// Publish .zip

[smlambert]

When we run in Jenkins, we archive the tap and AQACert.log and SHA.txt file, see an example of a test job here:
https://ci.adoptopenjdk.net/job/Test_openjdk18_hs_extended.system_s390x_linux/107/ (which we actually broke down even further into 3 test jobs to complete the run faster). Sharing to show the artifacts I would gather for extended.system, one of the 9 top-level targets that are in the AQA_REQUIRED_TARGETS list.

If you are running serially, there would be a single tap file for extended.system (instead of 3 with _testList_0, 1, 2 suffixes).

[sashapepin]

Thank you very much @smlambert !

[sashapepin]

@smlambert could you please have a look at this test results
archive.zip
Does it look good to you to be published?
Thanks!

[smlambert]

hi @sashapepin - good start!
so, if you intend to publish jdk8 x64 linux build, described by:

DETECTED_JAVA_VERSION=openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Zulu 8.62.0.19-CA-linux64) (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Zulu 8.62.0.19-CA-linux64) (build 25.332-b09, mixed mode)
DETECTED_RELEASE_INFO=JAVA_VERSION="1.8.0_332"
OS_NAME="Linux"
OS_VERSION="2.6"
OS_ARCH="amd64"
SOURCE="git:f4b2b4c5882e"

the contents of this archive.zip will contain 8 other .tap files also (which are possibly in other output_xxtimestamp directories):
Test_openjdk8_hotspot_extended.system_x86-64_linux.tap 👍 (good, all test targets 'ok', no presence of 'not ok')
AQACert.log 👍
SHA.txt 👍
Test_openjdk8_hotspot_sanity.system_x86-64_linux.tap ❔
Test_openjdk8_hotspot_sanity.functional_x86-64_linux.tap ❔
Test_openjdk8_hotspot_extended.functional_x86-64_linux.tap ❔
Test_openjdk8_hotspot_special.functional_x86-64_linux.tap ❔
Test_openjdk8_hotspot_sanity.perf_x86-64_linux.tap ❔
Test_openjdk8_hotspot_extended.perf_x86-64_linux.tap ❔
Test_openjdk8_hotspot_sanity.openjdk_x86-64_linux.tap ❔
Test_openjdk8_hotspot_extended.openjdk_x86-64_linux.tap ❔

[sashapepin]

Thanks a lot @smlambert !
I know about other .tap files :) I just wanted to be sure that the archived information is full and correct to be published.
Thanks again!

[sashapepin]

Hi @smlambert !
Here are all our test results for Zulu8 on Linux x64
functional_extended.zip
functional_sanity.zip
functional_special.zip
openjdk_extended.zip
openjdk_sanity.zip
perf_extended.zip
perf_sanity.zip
system_extended.zip
system_sanity.zip

All tests passed except 12 tests in openjdk.extended test suite. I've found only 5 test failures in your test run on the same platform https://ci.adoptopenjdk.net/view/Test_openjdk/job/Test_openjdk8_hs_extended.openjdk_x86-64_linux_testList_0/42/tapResults/
As I'm not familiar so far with openjdk tests could you please look at our results and probably give any hints.
Thanks!

[smlambert]

@sashapepin - this looks pretty good.

@sophia-guo is kindly putting together a PR #3723 to exclude the few remaining failing test cases. Once the PR is merged and cherry-picked to the v0.9.1-release branch you can rerun the failing targets and append the TAP files from the rerun to complete the verification work.

We can look more closely at the failing tests to see if they should be permanently excluded, or whether there are improvements to the tests or the product code before reinclusion, but will not need to block.

[sashapepin]

Thank you very much @smlambert and @sophia-guo !
I very appreciate your help!
I will re-run the tests tonight and will send you new results.

[smlambert]

@sophia-guo - can you put a note when your PR is cherrypicked and put into v0.9.1-release branch, as no point for @sashapepin to rerun until its backported.

[sophia-guo]

Done #3743

[sashapepin]

@smlambert and @sophia-guo thank you very much for your efforts!
Fortunately openjdk extended test suite started right after the backport push had happened and all tests passed this time.
Here are the latest test results:
functional_sanity.zip
functional_extended.zip
functional_special.zip
system_sanity.zip
system_extended.zip
openjdk_sanity.zip
openjdk_extended.zip
perf_sanity.zip
perf_extended.zip

[smlambert]

Great!

All TAP files can be added to a single archive (nested zip is fine or flattened) and pointed to in the aqavit_results_link in the json file for listing in the Marketplace (as described in https://adoptium.net/docs/marketplace-listing/ and exemplified by the example.json here).

azul_tap.zip is sufficient to show that the product described by contents of AQACert.log is now AQAvit verified:

DETECTED_JAVA_VERSION=openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Zulu 8.62.0.19-CA-linux64) (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Zulu 8.62.0.19-CA-linux64) (build 25.332-b09, mixed mode)
DETECTED_RELEASE_INFO=JAVA_VERSION="1.8.0_332"
OS_NAME="Linux"
OS_VERSION="2.6"
OS_ARCH="amd64"
SOURCE="git:f4b2b4c5882e"

We can and should take a look at the excluded tests to see what actions can happen next to fix/reinclude if applicable.

[sashapepin]

Thank you very much @smlambert !
What should be my next step? Should I prepare a set of json and test data files or create a pull request?
I looked through the files in azul_tap.zip which you created and found output.html there which was obviously generated by some script. Should I create a such file or it will be generated automatically? I also noticed that SHA.txt files for different test configs contain different set of components with hashes e.g. system contain STF and aqa-systemtest hashes. So I do not understand how in this case test results zip file can be flattened.
Thanks!

[smlambert]

You do not need to create the output.html file, I just included it, so it was easy for you and others to see and navigate the results (described here how it was created: #3713 (comment)).

Yes, for this release, you would not flatten, as the different test groups include different information in the SHA.txt file. Going forward (for July and onward), we are going to include the metadata information at the top of the TAP files so that the additional metadata files (AQACert.log and SHA.txt) will no longer be required.

Your next step would be to prepare the json file (as per the https://adoptium.net/docs/marketplace-guide, and see the 'Providing Publisher Information' and opening a "New Marketplace Publisher issue".

Tagging @tellison who will see that new Marketplace Publisher issue and can provide guidance should you need.

[smlambert]

I believe the AQAvit piece is complete, so we can likely close this issue. The remaining activity is outlined in #2826 (comment).

[sashapepin]

Hi @smlambert !
Sorry for the delayed response. I'm currently on vacation.
Yes, I think the issue can be closed!
Thank you very much for your help!