Skip to content

Commit

Permalink
Add upload rerun_c to ci and show artifacts build results & releases (#…
Browse files Browse the repository at this point in the history
…3028)

* Part of #2919

### What

Adds ci jobs to upload rerun_c for linux x64, mac x64, mac aarch64,
windows x64 and shows those both on the build summary html as well as
the github developer release

Unrelated changes
* remove unnecessary dependency from rerun_c - part of #2905
* no longer show full changelog on Dev releases - this got often very
long and isn't all that useful, also doesn't understand our minor
releases


### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3028) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3028)
- [Docs
preview](https://rerun.io/preview/pr%3Aandreas%2Fcpp%2Frerun_c-ci-build/docs)
- [Examples
preview](https://rerun.io/preview/pr%3Aandreas%2Fcpp%2Frerun_c-ci-build/examples)

---------

Co-authored-by: Jan Procházka <[email protected]>
  • Loading branch information
Wumpf and jprochazk authored Aug 18, 2023
1 parent 5a562d3 commit 6cf7020
Show file tree
Hide file tree
Showing 7 changed files with 293 additions and 12 deletions.
12 changes: 10 additions & 2 deletions .github/workflows/on_pull_request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ jobs:
RRD_ARTIFACT_NAME: linux-rrd-fast
secrets: inherit

build-rerun_c-and-upload:
name: "Build & Upload rerun_c (Linux x64)"
uses: ./.github/workflows/reusable_build_and_upload_rerun_c.yml
with:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
PLATFORM: linux
UPLOAD_COMMIT_OVERRIDE: ${{ github.event.pull_request.head.sha }}
secrets: inherit

build-web:
name: "Build Web"
uses: ./.github/workflows/reusable_build_web.yml
Expand Down Expand Up @@ -85,7 +94,7 @@ jobs:

save-pr-summary:
name: "Save PR Summary"
needs: [upload-web, run-notebook]
needs: [upload-web, run-notebook, build-rerun_c-and-upload]
uses: ./.github/workflows/reusable_pr_summary.yml
with:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
Expand All @@ -99,4 +108,3 @@ jobs:
CONCURRENCY: pr-${{ github.event.pull_request.number }}
PR_NUMBER: ${{ github.event.pull_request.number }}
secrets: inherit

69 changes: 61 additions & 8 deletions .github/workflows/on_push_main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,14 @@ jobs:
WHEEL_ARTIFACT_NAME: linux-wheel
secrets: inherit

track-sizes:
name: "Track Sizes"
needs: [build-web-demo]
uses: ./.github/workflows/reusable_track_size.yml
with:
CONCURRENCY: push-${{ github.ref_name }}
secrets: inherit

upload-web-demo:
name: "Upload Web Demo"
needs: [build-web-demo]
Expand All @@ -68,12 +76,40 @@ jobs:
MARK_PRERELEASE_FOR_MAINLINE: true
secrets: inherit

track-sizes:
name: "Track Sizes"
needs: [build-web-demo]
uses: ./.github/workflows/reusable_track_size.yml
build-rerun_c-and-upload-linux:
needs: [checks]
name: "Linux-x64: Build & Upload rerun_c"
uses: ./.github/workflows/reusable_build_and_upload_rerun_c.yml
with:
CONCURRENCY: push-${{ github.ref_name }}
CONCURRENCY: push-linux-${{ github.ref_name }}
PLATFORM: linux
secrets: inherit

build-rerun_c-and-upload-macos-intel:
needs: [checks]
name: "Mac-Intel: Build & Upload rerun_c"
uses: ./.github/workflows/reusable_build_and_upload_rerun_c.yml
with:
CONCURRENCY: push-macos-intel-${{ github.ref_name }}
PLATFORM: macos-intel
secrets: inherit

build-rerun_c-and-upload-macos-arm:
needs: [checks]
name: "Mac-Arm: Build & Upload rerun_c"
uses: ./.github/workflows/reusable_build_and_upload_rerun_c.yml
with:
CONCURRENCY: push-macos-arm-${{ github.ref_name }}
PLATFORM: macos-arm
secrets: inherit

build-rerun_c-and-upload-windows:
needs: [checks]
name: "Windows-x64: Build & Upload rerun_c"
uses: ./.github/workflows/reusable_build_and_upload_rerun_c.yml
with:
CONCURRENCY: push-windows-${{ github.ref_name }}
PLATFORM: windows
secrets: inherit

build-linux:
Expand Down Expand Up @@ -179,7 +215,15 @@ jobs:
concurrency:
group: push-${{ github.ref_name }}-prerelease
cancel-in-progress: true
needs: [upload-web, generate-pip-index]
needs:
[
upload-web,
generate-pip-index,
build-rerun_c-and-upload-linux,
build-rerun_c-and-upload-macos-intel,
build-rerun_c-and-upload-macos-arm,
build-rerun_c-and-upload-windows,
]
runs-on: "ubuntu-latest"
steps:
- name: Add SHORT_SHA env property with commit short sha
Expand Down Expand Up @@ -207,16 +251,25 @@ jobs:
## Example Hosted App
https://app.rerun.io/commit/${{ env.SHORT_SHA }}
## Demo App
https://demo.rerun.io/commit/${{ env.SHORT_SHA }}
## Wheels can be installed with:
```
pip install --pre -f https://build.rerun.io/commit/${{ env.SHORT_SHA}}/wheels --upgrade rerun-sdk
```
## Static libraries for rerun_c
* [Windows x64](https://build.rerun.io/commit/${{ env.SHORT_SHA }}/rerun_c/windows/rerun_c.lib)
* [Linux x64](https://build.rerun.io/commit/${{ env.SHORT_SHA }}/rerun_c/linux/librerun_c.a)
* [Mac Arm64](https://build.rerun.io/commit/${{ env.SHORT_SHA }}/rerun_c/macos-arm/librerun_c.a)
* [Mac Intel](https://build.rerun.io/commit/${{ env.SHORT_SHA }}/rerun_c/macos-intel/librerun_c.a)
prerelease: true
name: "Development Build"
tag: "prerelease"
token: ${{ secrets.GITHUB_TOKEN }}
generateReleaseNotes: true
generateReleaseNotes: false
allowUpdates: true
removeArtifacts: true
replacesArtifacts: true

196 changes: 196 additions & 0 deletions .github/workflows/reusable_build_and_upload_rerun_c.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
name: Reusable Rerun-c Build

on:
workflow_call:
inputs:
CONCURRENCY:
required: true
type: string
PLATFORM:
required: true
type: string
RELEASE_VERSION:
required: false
type: string
default: "prerelease"
UPLOAD_COMMIT_OVERRIDE:
required: false
type: string
default: ""
UPLOAD_COMMIT:
required: false
type: boolean
default: true
ADHOC_NAME:
required: false
type: string
default: ""

workflow_dispatch:
inputs:
ADHOC_NAME:
required: true
type: string
description: "Name of the adhoc build, used for upload directory"
PLATFORM:
type: choice
options:
- linux
- windows
- macos-arm
- macos-intel
description: "Platform to build for"
required: true
CONCURRENCY:
required: false
type: string
default: "adhoc"
RELEASE_VERSION:
required: false
type: string
default: "prerelease"
UPLOAD_COMMIT_OVERRIDE:
required: false
type: string
default: ""
UPLOAD_COMMIT:
required: false
type: boolean
default: true

concurrency:
group: ${{ inputs.CONCURRENCY }}-build-rerun_c
cancel-in-progress: true

env:
# web_sys_unstable_apis is required to enable the web_sys clipboard API which egui_web uses
# https://rustwasm.github.io/wasm-bindgen/api/web_sys/struct.Clipboard.html
# https://rustwasm.github.io/docs/wasm-bindgen/web-sys/unstable-apis.html
RUSTFLAGS: --cfg=web_sys_unstable_apis --deny warnings

# See https://github.com/ericseppanen/cargo-cranky/issues/8
RUSTDOCFLAGS: --deny warnings --deny rustdoc::missing_crate_level_docs

permissions:
contents: "read"
id-token: "write"

jobs:
set-config:
name: Set Config (${{ inputs.PLATFORM }})
runs-on: ubuntu-latest-16-cores
outputs:
RUNNER: ${{ steps.set-config.outputs.runner }}
TARGET: ${{ steps.set-config.outputs.target }}
RUN_TESTS: ${{ steps.set-config.outputs.run_tests }}
CONTAINER: ${{ steps.set-config.outputs.container }}
LIB_NAME: ${{ steps.set-config.outputs.lib_name }}
steps:
- name: Set runner and target based on platform
id: set-config
run: |
case "${{ inputs.PLATFORM }}" in
linux)
runner="ubuntu-latest-16-cores"
target="x86_64-unknown-linux-gnu"
run_tests="true"
container="{'image': 'rerunio/ci_docker:0.8'}"
lib_name="librerun_c.a"
;;
windows)
runner="windows-latest-8-cores"
target="x86_64-pc-windows-msvc"
run_tests="true"
container="null"
lib_name="rerun_c.lib"
;;
macos-arm)
runner="macos-latest"
target="aarch64-apple-darwin"
run_tests="false"
container="null"
lib_name="librerun_c.a"
;;
macos-intel)
runner="macos-latest"
target="x86_64-apple-darwin"
run_tests="false"
container="null"
lib_name="librerun_c.a"
;;
*) echo "Invalid platform" && exit 1 ;;
esac
echo "runner=$runner" >> "$GITHUB_OUTPUT"
echo "target=$target" >> "$GITHUB_OUTPUT"
echo "run_tests=$run_tests" >> "$GITHUB_OUTPUT"
echo "container=$container" >> "$GITHUB_OUTPUT"
echo "lib_name=$lib_name" >> "$GITHUB_OUTPUT"
rs-build-rerun_c:
name: Build rerun_c (${{ needs.set-config.outputs.RUNNER }})

needs: [set-config]

runs-on: ${{ needs.set-config.outputs.RUNNER }}
container: ${{ fromJson(needs.set-config.outputs.CONTAINER) }}

steps:
- name: Show context
run: |
echo "GITHUB_CONTEXT": $GITHUB_CONTEXT
echo "JOB_CONTEXT": $JOB_CONTEXT
echo "INPUTS_CONTEXT": $INPUTS_CONTEXT
echo "ENV_CONTEXT": $ENV_CONTEXT
env:
ENV_CONTEXT: ${{ toJson(env) }}
GITHUB_CONTEXT: ${{ toJson(github) }}
JOB_CONTEXT: ${{ toJson(job) }}
INPUTS_CONTEXT: ${{ toJson(inputs) }}

- uses: actions/checkout@v3

- name: Set up Rust
uses: ./.github/actions/setup-rust
with:
cache_key: "build-${{ inputs.PLATFORM }}"
save_cache: false
workload_identity_provider: ${{ secrets.GOOGLE_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}

- name: Build rerun_c (release)
uses: actions-rs/cargo@v1
with:
command: build
args: --locked -p rerun_c --release

- id: "auth"
uses: google-github-actions/auth@v1
with:
workload_identity_provider: ${{ secrets.GOOGLE_WORKLOAD_IDENTITY_PROVIDER }}
service_account: ${{ secrets.GOOGLE_SERVICE_ACCOUNT }}

- name: Add SHORT_SHA env property with commit short sha
id: "short-sha"
run: |
if [ -z "${{ inputs.UPLOAD_COMMIT_OVERRIDE }}" ]; then
USED_SHA=${{ github.sha }}
else
USED_SHA=${{ inputs.UPLOAD_COMMIT_OVERRIDE }}
fi
echo "SHORT_SHA=$(echo $USED_SHA | cut -c1-7)" >> $GITHUB_OUTPUT
- name: "Upload web-viewer (commit)"
if: ${{ inputs.UPLOAD_COMMIT }}
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "./target/release/${{ needs.set-config.outputs.LIB_NAME }}"
destination: "rerun-builds/commit/${{ steps.short-sha.outputs.SHORT_SHA }}/rerun_c/${{ inputs.PLATFORM }}"
parent: false

- name: "Upload web-viewer (adhoc)"
if: ${{ inputs.ADHOC_NAME != '' }}
uses: google-github-actions/upload-cloud-storage@v1
with:
path: "./target/release/${{ needs.set-config.outputs.LIB_NAME }}"
destination: "rerun-builds/adhoc/${{inputs.ADHOC_NAME}}/rerun_c/${{ inputs.PLATFORM }}"
parent: false
1 change: 0 additions & 1 deletion Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 0 additions & 1 deletion crates/rerun_c/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ test = false
[dependencies]
re_log.workspace = true
re_sdk.workspace = true
re_types.workspace = true

arrow2.workspace = true
ahash.workspace = true
Expand Down
12 changes: 12 additions & 0 deletions scripts/ci/generate_pr_summary.py
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,18 @@ def generate_pr_summary(github_token: str, github_repository: str, pr_number: in
print(f"Found web assets commit: {commit_short}")
found["hosted_app"] = f"https://app.rerun.io/commit/{commit_short}"

# Check if there are rerun_c libraries
rerun_libraries_blobs = [
builds_bucket.blob(f"commit/{commit_short}/rerun_c/windows/rerun_c.lib"),
builds_bucket.blob(f"commit/{commit_short}/rerun_c/linux/librerun_c.a"),
builds_bucket.blob(f"commit/{commit_short}/rerun_c/macos-arm/librerun_c.a"),
builds_bucket.blob(f"commit/{commit_short}/rerun_c/macos-intel/librerun_c.a"),
]
rerun_libraries = [f"https://build.rerun.io/{blob.name}" for blob in rerun_libraries_blobs if blob.exists()]
if rerun_libraries:
print(f"Found rerun_c libraries for commit: {commit_short}")
found["rerun_c_libraries"] = rerun_libraries

# Check if there are benchmark results
bench_blob = builds_bucket.blob(f"commit/{commit_short}/bench_results.txt")
if bench_blob.exists():
Expand Down
14 changes: 14 additions & 0 deletions scripts/ci/templates/pr_results_summary.html
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,10 @@
margin-left: 20px;
}

.rerun_c-libraries {
margin-left: 20px;
}

.wheel-list {
margin-left: 20px;
}
Expand All @@ -35,6 +39,16 @@ <h3>Hosted App:</h3>
<a href="{{ build.hosted_app }}" target="_blank">{{ build.hosted_app }}</a>
</div>
{% endif %}
{% if build.rerun_c_libraries %}
<div class="rerun_c-libraries">
<h3>Rerun C build:</h3>
<ul>
{% for library in build.rerun_c_libraries %}
<li><a href="{{ library }}" target="_blank">{{ library }}</a></li>
{% endfor %}
</ul>
</div>
{% endif %}
{% if build.bench_results %}
<h3><a href="{{ build.bench_results }}" target="_blank">Benchmark Results</a></h3>
{% endif %}
Expand Down

0 comments on commit 6cf7020

Please sign in to comment.