Add native Apple Silicon target to build workflow #2245
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… build job matrix The "Arduino IDE" GitHub Actions workflow uses a job matrix to make the builds for each host target in parallel. The same steps are used for each job, but some configuration adjustments must be made on a per-target basis. This is done through various attributes in the matrix configuration. Previously the `os` attribute was used for two distinct things: - The machine identifier of the GitHub Actions runner machine of the job. - The differentiator in the human-targeted job name. The attribute name "os" (for "operating system") was misleading because runners are differentiated by more than only the operating system on the machine. The use of a machine identifier as a differentiator in the human-targeted job name was a bad idea because these identifiers would only effectively communicate the nature of a build to humans who are quite knowledgeable about the GitHub Actions workflow syntax. The impact of these poor decisions has not been too severe previously due to there only being a single job for each operating system. However, there is a need for multiple jobs per operating system in order to support multiple host architectures (e.g., macOS x86 and ARM). The solution is to: - Use an appropriate name for the runner identifier attribute. - Use a dedicated attribute for the human friendly job name differentiator.
The "Arduino IDE" GitHub Actions workflow is used to generate several distinct types of builds: - Tester builds of commits - Nightly builds - Release builds Different actions must be performed depending on which type of build is being produced. The workflow contains code that uses various criteria to determine the build type. Previously that code was duplicated in multiple places: - The packaging job - The changelog generation job - The nightly build publishing job - The release publishing job This duplication is avoided by moving the code to a dedicated job that makes the build type information available to all subsequent jobs via outputs.
per1234
added
type: enhancement
|
For https://github.com/arduino/arduino-ide/actions/runs/6409533621, I see some inconsistency in the file naming:
|
|
Do you think this PR could cover #2117? I see arduino-ide/electron-app/scripts/package.js Lines 108 to 123 in 8f4bcc8 |
kittaakos
reviewed
Oct 5, 2023
per1234
force-pushed
the
gha-apple-silicon-build
branch
from
2788aca to
f076580
Compare
kittaakos
reviewed
Oct 5, 2023
On every release tag, and manual trigger when the "Include builds on non-free runners" checkbox is checked, make the Arduino IDE build for native Apple Silicon host in addition to the builds that are always generated by the "Arduino IDE" GitHub Actions workflow. Previously, the build workflow only produced a build for x86-64 (AKA "Intel") macOS hosts. Although it is possible to use those builds on Apple Silicon machines via the Rosetta 2 translation software, the performance is significantly inferior to a native build so we must also provide Apple Silicon native builds. Previously the Apple Silicon builds were produced manually. The reason for using that inefficient and error-prone approach instead of the automated continuous deployment system used for other builds was that GitHub did not provide the necessary Apple Silicon runner machines and Arduino was not capable of setting up such self-hosted machines in a manner that would make them feasible for the project maintainers to use. GitHub hosted Apple Silicon runner machines are now available so we can add the target to the build workflow. GitHub gives unlimited use of the basic runner machines for workflow runs in public repositories. However, the macOS ARM architecture is only provided in runner machines which are classified as "larger runner". Use of these runners is charged on a per-minute basis, without any of the free allowances GitHub provides for the normal runners. In order to avoid unnecessary expenditures, native Apple Silicon builds must be generated only when there is compelling reason to do so. Such a build is needed for every release, so the workflow is configured to always generate the builds when triggered by a tag. In addition to releases, Apple Silicon tester builds for pull requests that might have special implications for this target. For this reason, the workflow is configured to allow Apple Silicon builds to be triggered manually by a repository maintainer. The workflow uses a job matrix to run the build for each target on the appropriate runner machine in parallel, using the universally applicable workflow code for all jobs. It uses another job matrix to generate individual workflow artifacts for the tester builds of each target. Previously it was possible to always use the same matrix configurations for all workflow runs. With the addition of the selectively run macOS ARM job, it is now necessary to generate these matrixes on the fly. The electron-updater package used by Arduino IDE's auto-update capability uses a data file (known as the "channel update info file") to check for the availability of updates. A single "channel update info file" is used for the data of the macOS x86 and ARM builds. Since a separate job is used to produce each of those builds, this means the "channel update info file" produced by each of the macOS build jobs must be merged into a single file.
per1234
force-pushed
the
gha-apple-silicon-build
branch
from
a121dd8 to
284d39f
Compare
I gave it a try but unfortunately found it is not possible when I got to the stage of testing the effect on auto-update. I added an explanation of the situation in #2117.
The origin of the is GitHub Actions: https://docs.github.com/en/actions/learn-github-actions/contexts#runner-context
There is a comment in the script that links to that information: arduino-ide/scripts/merge-channel-files.js Lines 45 to 46 in 284d39f Even though the inconsistency is unfortunate, I don't think it is significant enough make it worth adding the necessary code to the workflow to translate the GitHub Actions identifier to the electron-updater identifier. |
The "Arduino IDE" GitHub Actions workflow uploads the nightly and release builds to Amazon S3, from which they are downloaded by the auto-update as well as directly by users via the links on the "Software" page of arduino.cc. The workflow can also be useful in forks. Either by those who want to test contributions staged in their fork prior to submitting a PR to the parent repo, or by those maintaining a hard fork of the project. Even though these forks wouldn't (and couldn't due to lack of access to the encrypted credential secrets only available to the workflow when ran in a trusted context in Arduino's repo)credentials stored in Arduino's repo) use the S3 upload component of the workflow, they may still find it valuable for continuous integration as well as continuous deployment via the tester builds and release builds the workflow also publishes to the GitHub repository it runs in. For this reason, the workflow contains code to determine whether it should attempt the S3 uploads. Previously that code was duplicated in both the nightly and release publishing jobs of the workflow. Since the workflow already contains a job specifically for the purpose of determining the characteristics of the build being performed and making that information available from single source for use throughout the rest of the workflow, it makes sense to also move the S3 upload determination code to that job.
…workflow The "Arduino IDE" GitHub Actions workflow uploads the nightly and release builds to Amazon S3, from which they are downloaded by the auto-update as well as directly by users via the links on the "Software" page of arduino.cc. The workflow can also be useful in forks. Either by those who want to test contributions staged in their fork prior to submitting a PR to the parent repo, or by those maintaining a hard fork of the project. Even though these forks wouldn't (and couldn't due to lack of access to the encrypted credential secrets only available to the workflow when ran in a trusted context in Arduino's repo)credentials stored in Arduino's repo) use the S3 upload component of the workflow, they may still find it valuable for continuous integration as well as continuous deployment via the tester builds and release builds the workflow also publishes to the GitHub repository it runs in. For this reason, the workflow contains code to determine whether it should attempt the S3 uploads. Previously the repository name was used as the criteria in that code. The project specificity of that approach makes the workflow less easily reusable. A more generally applicable criterion is whether the encrypted credential certificate is defined. The new criterion allows the workflow to be used in any repository where the administrator has created an encrypted secret containing their AWS credentials. That might be other projects owned by Arduino, or even 3rd party projects where the owners want to take a similar build publishing approach using their own AWS account.
Thanks! |
kittaakos
approved these changes
Oct 6, 2023
4 tasks
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On every release tag, and manual trigger when the "Include builds on non-free runners" checkbox is checked, make the Arduino IDE build for native Apple Silicon host in addition to the builds that are always generated by the "Arduino IDE" GitHub Actions workflow.
Background
Previously, the build workflow only produced a build for x86-64 (AKA "Intel") macOS hosts. Although it is possible to use those builds on Apple Silicon machines via the Rosetta 2 translation software, the performance is significantly inferior to a native build so Arduino must also provide Apple Silicon native builds (#408).
Previously the Apple Silicon builds were produced manually. The reason for using that inefficient and error-prone approach instead of the automated continuous deployment system used for other builds was that GitHub did not provide the necessary Apple Silicon runner machines and Arduino was not capable of setting up such a self-hosted machine in a manner that would make them feasible for the project maintainers to use.
Details
GitHub hosted Apple Silicon runner machines are now available so we can add the target to the build workflow.
GitHub gives unlimited use of the basic runner machines for workflow runs in public repositories. However, the macOS ARM architecture is only provided in runner machines which are classified as "larger runner". Use of these runners is charged on a per-minute basis, without any of the free allowances GitHub provides for the normal runners. In order to avoid unnecessary expenditures, native Apple Silicon builds must be generated only when there is compelling reason to do so. Such a build is needed for every release, so the workflow is configured to always generate the builds when triggered by a tag. In addition to releases, Apple Silicon tester builds will also be useful for pull requests that might have special implications for this target. For this reason, the workflow is configured to allow Apple Silicon builds to be triggered manually by a repository maintainer:
Job Matrix Generation
The workflow uses a job matrix to run the build for each target on the appropriate runner machine in parallel, using the universally applicable workflow code for all jobs. It uses another job matrix to generate individual workflow artifacts for the tester builds of each target. Previously it was possible to always use the same matrix configurations for all workflow runs. With the addition of the selectively run macOS ARM job, it is now necessary to generate these matrixes on the fly.
Merging Channel Files
The electron-updater package used by Arduino IDE's auto-update capability uses a data file (known as the "channel update info file") to check for the availability of updates. A single "channel update info file" is used for the data of the macOS x86 and ARM builds. Since a separate job is used to produce each of those builds, this means the "channel update info file" produced by each of the macOS build jobs must be merged into a single file.
Notes for Reviewers
The first two commits do some minor refactoring of the workflow in preparation for the work to add the new build target. See the individual commit messages for details.
Demos
Demonstration release I did in my fork using the updated workflow:
https://github.com/per1234/arduino-ide/releases/tag/0.0.0-rc.26
Demonstration of manually triggered run with the "Include builds on non-free runners" option enabled:
https://github.com/arduino/arduino-ide/actions/runs/6409533621
Demonstration of manually triggered run with the "Include builds on non-free runners" option disabled:
https://github.com/arduino/arduino-ide/actions/runs/6409535219
Follow Up
After this is merged, I will submit a PR for the adjustments to #2129 reflecting the change.
UPDATE: PR submitted #2248
Resolves #408