AdoptOpenJDK Quality Assurance (AQA) 1.0 - Goals and Definition

[smlambert]

Verification of AdoptOpenJDK binaries should:

• be open & transparent
• consist of a diverse & robust set of test suites
• evolve alongside implementations
• have continual investment
• have process to modify
• be measured by codecov & other metrics
• allow for comparative analysis
• be portable
• get tagged & published alongside binaries (build confidence, results transparency)

This epic will collect the various issues related to this overall goal, and also link to a separate Google document entitled AdoptOpenJDK Quality Assurance (AQA) 1.0 where the community can discuss and refine our goals and the definition of this quality bar together.

[smlambert]

Related: #186

[smlambert]

Incorporated feedback into the google doc.

For the upcoming release, we will use it as an initial trial of mechanisms required for AQA (bill of materials that include SHAs of all repos housing test materials used to verify builds).

[smlambert]

Issues that need to be completed to deliver AQA v1.0:

• identify initial set of test targets that will form v1.0

• document rationale behind choices of test targets in v1.0

• auto-gen test jobs (replacing hand-built test jobs at Adopt (to ensure jobs are up-to-date, with correct params), including those that are not yet running nightly/weekly (extended.perf, special.functional, etc)

• update build pipelines to launch newly auto-generated test jobs

• produce a file that can be incorporated into the releases repos (alongside binaries) which details the SHAs of each test repo used and also summarizes test results (with link to TRSS view)

• potentially add badge to website indicating AQA verified (though this can come in a later version of AQA), with link to TRSS

[smlambert]

I will close this issue as completed

2 pieces to be discussed and if kept, to be moved to next drop of AQA (#1522) are:

• pushing SHA.txt to release repo (currently archived with test results on Jenkins at Adopt, and in artifactory at other projects
• the possible addition of a badge on the website