Skip to content

Commit

Permalink
Build wrappers on Github Actions (#2267)
Browse files Browse the repository at this point in the history
* First action attempt

* Wrong path to action

* Renamed action

* Sadly the action is brought back to do 1 thing only

* Some mispelling

* Created JS action to build wrapper

* Renamed folder in main.yml

* Reshuffled things

* Hopefully nodejs 14 and not 12

* Few fixes

* Removed nodejs 14 stuff

* All compacted into index.js

* First iteration of build_wrappers action

* Added some info text

* Forgot to update action path

* Moved from core.debug to core.info

* Updated string to use template literals

* Correct error to return success

* Forgot to build

* I like it more with the dash

* Forgot to build again

* Fixed some checks on string content

* Build forgotten

* Live output for built cmd + removed exit code when error

* Reverted to cp.exec

* Better output folding

* Check for error at caching

* Some fixes + lots of prints

* Changed from exec to spawn

* Try to fix spawn and env var

* Small improvements

* Async method should now be fixed

* Final test to check correctness

* Small code style changes

* First attempt at making caching and building general

* Forgotten meta build action

* Fixed some mispelling in commands in main

* Refactored lots of stuff

* Json in 1 line in main.yml workflow

* Added test framework, added tests for input parsing, split utils in multiple files

* Added a TODO and better logging in spawned child

* Removed hash options, it doesn't seem necessary

* Unit test added for all common functionalities

* Extracted main functionalities from action

* Updated action.yml name and description

* Folder rename

* Small cleanup main.yml

* Extrapolated core code in it's own folder

* Forgotten to change path to action in workflow

* Fix path for action

* This time path to action should be fixed

* Minor fixes

* Temporary test with external repo for core code

* Just a test with types in lib

* Small error correction

* Test with checkout action

* Attempt moving checkout js in same dist of build_executor

* Test with double checkout

* Postponment of checkout action inclued

* Fixed some wrong build command

* Added package folder (lib) and changed input parsing for paths

* Updated path to packages in main

* Added optional hash prefix

* HashKey used as index by the cache function is now returned as output

* Cmds in input aren't anymore a json array of json objects

* Reshaped tasks.json, added scripts in package.json and updated structure in packages

* Eslinter and prettifier setup and run

* Removed result struct and reintroduced exceptions

* As far as I can go with the linter

* Made utils internal, linted and prettified action

* Final adjustments for PR review

* Added one more test to check prefix on hash

* Fixed wrongly passed hash prefix

* Returned wrong value from main

* Some rephrasing

* Improved comments

* Added settings for debugging source code and tests

* Reshaped hashing function to better account json structure + improved test speed on hashing test

* Rephrased a log

* Final minor corrections before PR review

* Forgotten to build and lint/prettify

* Better fail message

* Improved input_parsing and expanded tests

* More unit testing, more safety checks and few other minor changes

* Fixes for the GA package (#2306)

* Add typescript to dependencies
* Add some basic eslint rules (will extend later)
* Add typings to package.json
* Add __tests__ to tsc includes
* Disable the unused var warnings for tests

* Indent code with 4 spaces (#2307)

* Indent code with 4 spaces

* Increase print-width

* Simplified according to PR review

* Fixed input in action

* Compacted code in only 1 file

* Added options to debug typescript files directly

* Forgot to remove a comment

* Migrate GA to a pure workflow (#2322)

* wip

* Supply input and output paths to the action

* Add ncc

* Fix the action metadata

* Tweak the build wrappers command

* Try to simplify configuration passing to bash builds

* Bump down ubuntu version for android

* Try and use ytt for templates

* Remove the `null` from PR

* Try #3

* Try #4

* Try 5

* Try 6

* Try to use pure workflow

* Remove extra brace

* Yet another brace

* cacheKey -> key

* Add the cache condition to any intermediate steps

* Revert files that are unrelated to the workflow change

* More tweaks to the template

* Test building iOS wrappers

* Try to build iOS #2

* Bring back the cmake configuration check in ios and android build scripts

* Try building linux on bare metal

* Use correct ubuntu image

* Try to add Android

* Put matrix under strategy

* Setup NDK env variables

* Add $

* Don't download the ndk

* Added windows to templates

* Try to cache vcpkg

* Trigger uncached build

* Trigger another uncached build

* Another build

* Another uncached run

* Revert dummy changes

* Added UWP to template

* Fixed small detail

* Added saving output

* Always output cacheKey from building wrappers

* Removed TODO comment

* Ac/retrieve multiple caches ga (#2325)

* Added fetching multiple bins

* Store binaries as artifacts

* Store binaries as artifacts - 2

* Store binaries as artifacts - 3

* Store binaries as artifacts - 4

* Store binaries as artifacts - 5

* PR ready for review

Co-authored-by: Andrea Catalini <[email protected]>

* Remove the typescript action (#2326)

* remove some leftovers

* Remove process.sh

Co-authored-by: Nikola Irinchev <[email protected]>
  • Loading branch information
LaPeste and nirinchev authored Apr 9, 2021
1 parent 5199757 commit b42cebc
Show file tree
Hide file tree
Showing 10 changed files with 492 additions and 15 deletions.
12 changes: 12 additions & 0 deletions .github/templates/Readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# Preprocessing the GA workflows

We're using [ytt](https://github.com/vmware-tanzu/carvel-ytt) to generate the github actions workflows.

## Prerequisites

1. Download the executable for your platform from the [releases page](https://github.com/vmware-tanzu/carvel-ytt/releases).
1. Rename it to `ytt`.
1. Place it in your PATH.

## Building the docs
1. In `$SolutionDir/.github/templates` run `./process.sh`.
187 changes: 187 additions & 0 deletions .github/templates/main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,187 @@
#@ load("@ytt:template", "template")
#@ load("@ytt:overlay", "overlay")

#@ configuration = "Release"
#@ androidABIs = [ 'armeabi-v7a', 'arm64-v8a', 'x86', 'x86_64' ]
#@ windowsArchs = [ 'Win32', 'x64' ]
#@ windowsUWPArchs = [ 'Win32', 'x64', 'ARM' ]
#@ wrappersCacheCondition = "steps.check-cache.outputs.cache-hit != 'true'"

#@ def checkoutCode():
name: Checkout code
uses: actions/checkout@v2
with:
submodules: recursive
#@ end

#@ def checkCache(key):
name: Check cache
id: check-cache
uses: actions/cache@v2
with:
path: ./wrappers/build/**
key: #@ key
#@ end

#@ def cacheVcpkg():
name: Check Vcpkg cache
id: check-vcpkg-cache
uses: actions/cache@v2
with:
path: 'C:\src'
key: vcpkg
if: #@ wrappersCacheCondition
#@ end

#@ def setupVcpkg():
name: Setup Vcpkg
run: |
Write-Output 'Beginning download...'
Invoke-WebRequest -Uri https://static.realm.io/downloads/vcpkg.zip -OutFile C:\vcpkg.zip
Write-Output ((Get-Item C:\vcpkg.zip).length/1MB)
Expand-Archive -Path C:\vcpkg.zip -DestinationPath C:\
Write-Output 'Completed!'
shell: powershell
if: #@ wrappersCacheCondition + " && steps.check-vcpkg-cache.outputs.cache-hit != 'true'"
#@ end

#@ def buildWrappers(cmd, outputVar, intermediateSteps = []):
#@ configurationParam = " --configuration=" + configuration
#@ ltoParam = " -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=${{ github.event_name != 'pull_request' && 'ON' || 'OFF' }}"
#@ if cmd.startswith("powershell"):
#@ configurationParam = " -Configuration " + configuration
#@ ltoParam = "${{ github.event_name != 'pull_request' && ' -EnableLTO' || '' }}"
#@ end
#@ cacheKey = outputVar + "-" + configuration + "-${{ github.event_name != 'pull_request' && 'ON' || 'OFF' }}-${{hashFiles('./wrappers/**')}}"
#@ actualCommand = cmd + configurationParam + ltoParam

steps:
- #@ checkoutCode()
- #@ checkCache(cacheKey)
#@ for step in intermediateSteps:
- #@ step
#@ end
- name: Build wrappers
run: #@ actualCommand
if: #@ wrappersCacheCondition
- name: Store artifacts
uses: actions/upload-artifact@v2
with:
name: #@ outputVar
path: wrappers/build/**
retention-days: 1
#@ end

---
name: build and test
"on":
push:
branches:
- main
- master
pull_request:
jobs:
build-wrappers-macos:
runs-on: macos-latest
name: Build macOS wrappers
_: #@ template.replace(buildWrappers("./wrappers/build-macos.sh", "wrappers-macos"))
build-wrappers-ios:
runs-on: macos-latest
name: Build iOS wrappers
_: #@ template.replace(buildWrappers("./wrappers/build-ios.sh", "wrappers-ios"))
build-wrappers-linux:
runs-on: ubuntu-20.04
name: Build Linux wrappers
_: #@ template.replace(buildWrappers("./wrappers/build.sh", "wrappers-linux"))
build-wrappers-android:
runs-on: ubuntu-20.04
name: Build Android wrappers
strategy:
matrix:
arch: #@ androidABIs
_: #@ template.replace(buildWrappers("./wrappers/build-android.sh --ARCH=${{ matrix.arch }}", "wrappers-android-${{ matrix.arch }}"))
build-wrappers-windows:
runs-on: windows-latest
name: Build Windows wrappers
strategy:
matrix:
arch: #@ windowsArchs
_: #@ template.replace(buildWrappers("powershell ./wrappers/build.ps1 Windows -Platforms ${{ matrix.arch }}", "wrappers-windows-${{ matrix.arch }}", [ cacheVcpkg(), setupVcpkg() ]))
build-wrappers-windows-uwp:
runs-on: windows-latest
name: Build Windows UWP wrappers
strategy:
matrix:
arch: #@ windowsUWPArchs
_: #@ template.replace(buildWrappers("powershell ./wrappers/build.ps1 WindowsStore -Platforms ${{ matrix.arch }}", "wrappers-windows-uwp-${{ matrix.arch }}", [ cacheVcpkg(), setupVcpkg() ]))
build-package:
runs-on: windows-latest
name: Build packages
needs:
- build-wrappers-windows
- build-wrappers-macos
- build-wrappers-ios
- build-wrappers-android
- build-wrappers-linux
- build-wrappers-windows-uwp
steps:
- name: Fetch Artifacts MacOS
uses: actions/download-artifact@v2
with:
name: wrappers-macos
path: wrappers/build
- name: Fetch Artifacts iOS
uses: actions/download-artifact@v2
with:
name: wrappers-ios
path: wrappers/build
- name: Fetch Artifacts Linux
uses: actions/download-artifact@v2
with:
name: wrappers-linux
path: wrappers/build
- name: Fetch Artifacts Android armeabi-v7a
uses: actions/download-artifact@v2
with:
name: wrappers-android-armeabi-v7a
path: wrappers/build
- name: Fetch Artifacts Android arm64-v8a
uses: actions/download-artifact@v2
with:
name: wrappers-android-arm64-v8a
path: wrappers/build
- name: Fetch Artifacts Android x86
uses: actions/download-artifact@v2
with:
name: wrappers-android-x86
path: wrappers/build
- name: Fetch Artifacts Android x86_64
uses: actions/download-artifact@v2
with:
name: wrappers-android-x86_64
path: wrappers/build
- name: Fetch Artifacts Windows 32bit
uses: actions/download-artifact@v2
with:
name: wrappers-windows-Win32
path: wrappers/build
- name: Fetch Artifacts Windows 64bit
uses: actions/download-artifact@v2
with:
name: wrappers-windows-x64
path: wrappers/build
- name: Fetch Artifacts Windows UWP 32bit
uses: actions/download-artifact@v2
with:
name: wrappers-windows-uwp-Win32
path: wrappers/build
- name: Fetch Artifacts Windows UWP 64bit
uses: actions/download-artifact@v2
with:
name: wrappers-windows-uwp-x64
path: wrappers/build
- name: Fetch Artifacts Windows UWP ARM
uses: actions/download-artifact@v2
with:
name: wrappers-windows-uwp-ARM
path: wrappers/build
2 changes: 1 addition & 1 deletion .github/workflows/benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ on:
workflow_dispatch:
push:
branches:
-master
- master
env:
# Branch for Github Pages
gh-pages-branch: gh-pages
Expand Down
Loading

0 comments on commit b42cebc

Please sign in to comment.