Skip to content

Commit

Permalink
Dev build (#2936)
Browse files Browse the repository at this point in the history
# Motivation
We would like to add API functions to the nns-dapp that are present only
in test deployments. For this, we need a separate test build.

This is not the first case of a wasm having build flavours. For example,
[Internet Identity has production, test and dev
flavours](https://github.com/dfinity/internet-identity/releases/tag/release-2023-07-21).

# Changes
* Add a feature flag called `test` to the nns-dapp crate. The feature
flag will be used to gate API methods available only in test builds.
* Build with `test` in the dockerfile.
* Add the test build to the list of wasms published in releases.
* For consistency, the `noassets` build, used for testnets, is also
published. The `noassets` build could also be named the `tiny` build but
that is outside the scope of this PR.
* Note: This gives us three build flavours: `production`, `test` and
`tiny` aka `noassets`
* Note: For consistency, the wasms are named:
`nns-dapp_${flavour}.wasm.gz`
* Create a separate API method list for each build flavour. The API
method list is used to check that the wasm API is as expected, and we
expect the APIs to diverge.

# Tests
- See a toy release:
https://github.com/dfinity/nns-dapp/releases/tag/dev-build-test-tag

# Todos

- [x] Add entry to changelog (if necessary).

---------

Co-authored-by: Max Murphy-Skvorzov <[email protected]>
  • Loading branch information
bitdivine and bitdivine authored Jul 26, 2023
1 parent 7a36d1c commit 6fda486
Show file tree
Hide file tree
Showing 12 changed files with 131 additions and 11 deletions.
4 changes: 2 additions & 2 deletions .github/actions/release_nns_dapp/action.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ runs:
cd "${{ inputs.assets_dir }}"
daily_build_name="nns-dapp-$(git rev-parse HEAD).wasm.gz"
cp nns-dapp.wasm.gz "$daily_build_name"
artefacts=(nns-dapp.wasm.gz sns_aggregator.wasm.gz sns_aggregator_dev.wasm.gz assets.tar.xz "$daily_build_name" nns-dapp-arg-{mainnet,local}.{did,bin})
artefacts=(nns-dapp.wasm.gz nns-dapp_noassets.wasm.gz nns-dapp_production.wasm.gz nns-dapp_test.wasm.gz sns_aggregator.wasm.gz sns_aggregator_dev.wasm.gz assets.tar.xz "$daily_build_name" nns-dapp-arg-{mainnet,local}.{did,bin})
# For backwards compatibility we publish the wasms with a .wasm suffix as well:
for file in "${artefacts[@]}" ; do
[[ "${file}" == "${file%.wasm.gz}" ]] || {
Expand All @@ -34,7 +34,7 @@ runs:
: Creates or updates a release for the tag
if gh release view "$tag"
then gh release upload --repo dfinity/nns-dapp --clobber "$tag" "${artefacts[@]}" || true
else gh release create --title "Release for tags/$tag" --draft --notes "Build artefacts from tag: $tag" "$tag" "${artefacts[@]}"
else gh release create --title "Release for tags/$tag" --draft --notes "$("$GITHUB_ACTION_PATH/template" --tag "$tag" --dir .)" "$tag" "${artefacts[@]}"
fi
: If the tag is for a proposal or nightly, make it public
[[ "$tag" != proposal-* ]] && [[ "$tag" != nightly-* ]] || { echo "Making release public" ; gh release edit "$tag" --draft=false ; }
Expand Down
1 change: 1 addition & 0 deletions .github/actions/release_nns_dapp/clap.bash
50 changes: 50 additions & 0 deletions .github/actions/release_nns_dapp/template
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
#!/usr/bin/env bash
set -euo pipefail
SOURCE_DIR="$(dirname "$(realpath "${BASH_SOURCE[0]}")")"
PATH="$SOURCE_DIR:$PATH"

print_help() {
cat <<-EOF
Creates the text for GitHub releases.
EOF
}

# Source the clap.bash file ---------------------------------------------------
source "$SOURCE_DIR/clap.bash"
# Define options
clap.define short=t long=tag desc="The git tag of the release" variable=DFX_RELEASE_NAME default="${GITHUB_REF_NAME:-unnamed}"
clap.define short=d long=dir desc="The asset dir" variable=DFX_ASSET_DIR default="out"
# Source the output file ----------------------------------------------------------
source "$(clap.build)"

GITHUB_REPOSITORY="${GITHUB_REPOSITORY:-dfinity/nns-dapp}"
GITHUB_SHA="${GITHUB_SHA:-$(git rev-parse HEAD)}"

cat <<EOF
This is a release of the $(nns-dapp) and $(sns_aggregator) canisters.
Release: [$DFX_RELEASE_NAME](https://github.com/$GITHUB_REPOSITORY/releases/tag/$DFX_RELEASE_NAME)
Commit: [$GITHUB_SHA](https://github.com/$GITHUB_REPOSITORY/commit/$GITHUB_SHA)
| Filename | sha256 |
|---|---|
$(
# Prints a row in the table describing a wasm, if it exists.
file_row() {
local filename sha
filename="${canister}${flavor:+_}${flavor}.wasm.gz"
if test -e "$DFX_ASSET_DIR/$filename"; then
sha="$(sha256sum "$DFX_ASSET_DIR/$filename" | awk '{print $1}')"
printf "| %s | %s |\n" "[$filename](https://github.com/$GITHUB_REPOSITORY/releases/download/$DFX_RELEASE_NAME/$filename)" "$sha"
fi
}
for canister in nns-dapp sns_aggregator; do
for flavor in "" production test dev noassets; do
file_row
done
done
)
EOF
14 changes: 14 additions & 0 deletions BUILD.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,20 @@ Our CI also performs these steps; you can compare the SHA256 with the output the

TODO: When we make a proposal, we should have a corresponding release that voters can download. E.g. https://github.com/dfinity/nns-dapp/releases/tag/release-candidate exists but it doesn't have build artefacts.
### Build flavors
The build creates several different `nns-dapp` and `sns_aggregator` wasms. These builds target specific use cases:
| Flavor | Description | |
| --- | --- | :---: |
| Production | This is the production build deployed to https://nns.ic0.app |
| Dev | This is a build for testing integration with third party canisters. If you wish to include `nns-dapp` or the `sns_aggregator` in your CI, this build is designed to make your testing easy and convenient. |
| Test | This is a build for testing internal functionality. If you are changing `nns-dapp` or `sns_aggregator` functionality and need to access or modify the internal state of the nns-dapp to test, this is the build for you. |
| Noassets | This is a build of the nns-dapp that does not include web assets. This reduces the size of the wasm, which may be convenient in some test scenarios. Note that web assets can be uploaded to the nns-dapp after deployment by the canister controller. |
TODO: Document how to make the `sns_aggregator` collect data quickly.
TODO: Document how to create accounts in the nns-dapp.
## Development
Development relies on the presence of a testnet that is setup with the II, governance, ledger, and cycle minting canisters. Fully local development is unfortunately not yet supported and the tools for setting up a testnet are not yet available publicly. It is on the roadmap to make these tools available publicly for developers.
Expand Down
1 change: 1 addition & 0 deletions CHANGELOG-Nns-Dapp.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ The NNS Dapp is released through proposals in the Network Nervous System. Theref

#### Added

* A separate build of nns-dapp for testing. See: https://github.com/dfinity/nns-dapp/releases/tag/dev-build-test-tag
* A test that state is preserved in downgrade-upgrade tests.
* Support SNS neuron permission in fake SNS governance API.
* Support selective pausing and resuming in API fakes.
Expand Down
9 changes: 7 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ COPY ./rs/backend /build/rs/backend
COPY ./scripts/nns-dapp/test-exports /build/scripts/nns-dapp/test-exports
COPY ./scripts/clap.bash /build/scripts/clap.bash
COPY ./build-backend.sh /build/
COPY ./scripts/nns-dapp/flavours.bash /build/scripts/nns-dapp/flavours.bash
COPY ./build-rs.sh /build/
COPY ./Cargo.toml /build/
COPY ./Cargo.lock /build/
Expand All @@ -149,7 +150,8 @@ RUN touch --no-create rs/backend/src/main.rs rs/backend/src/lib.rs
RUN ./build-backend.sh
COPY ./scripts/dfx-wasm-metadata-add /build/scripts/dfx-wasm-metadata-add
ARG COMMIT
RUN scripts/dfx-wasm-metadata-add --commit "$COMMIT" --canister_name nns-dapp --verbose
RUN . scripts/nns-dapp/flavours.bash && for flavour in "${NNS_DAPP_BUILD_FLAVOURS[@]}" ; do scripts/dfx-wasm-metadata-add --commit "$COMMIT" --canister_name nns-dapp --wasm "nns-dapp_$flavour.wasm.gz" --verbose ; done
RUN scripts/dfx-wasm-metadata-add --commit "$COMMIT" --canister_name nns-dapp --wasm nns-dapp.wasm.gz --verbose

# Title: Image to build the nns-dapp backend without assets.
FROM builder AS build_nnsdapp_without_assets
Expand All @@ -158,6 +160,7 @@ COPY ./rs/backend /build/rs/backend
COPY ./scripts/nns-dapp/test-exports /build/scripts/nns-dapp/test-exports
COPY ./scripts/clap.bash /build/scripts/clap.bash
COPY ./build-backend.sh /build/
COPY ./scripts/nns-dapp/flavours.bash /build/scripts/nns-dapp/flavours.bash
COPY ./build-rs.sh /build/
COPY ./Cargo.toml /build/
COPY ./Cargo.lock /build/
Expand Down Expand Up @@ -212,7 +215,9 @@ COPY --from=configurator /build/nns-dapp-arg* /
# Note: The frontend/.env is kept for use with test deployments only.
COPY --from=configurator /build/frontend/.env /frontend-config.sh
COPY --from=build_nnsdapp /build/nns-dapp.wasm.gz /
COPY --from=build_nnsdapp_without_assets /build/nns-dapp.wasm.gz /nns-dapp_noassets.wasm.gz
COPY --from=build_nnsdapp /build/nns-dapp_test.wasm.gz /
COPY --from=build_nnsdapp /build/nns-dapp_production.wasm.gz /
COPY --from=build_nnsdapp_without_assets /build/nns-dapp_test.wasm.gz /nns-dapp_noassets.wasm.gz
COPY --from=build_nnsdapp /build/assets.tar.xz /
COPY --from=build_frontend /build/sourcemaps.tar.xz /
COPY --from=build_aggregate /build/sns_aggregator.wasm.gz /
Expand Down
21 changes: 16 additions & 5 deletions build-backend.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,23 @@
set -euo pipefail

TOPLEVEL="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# Defines NNS_DAPP_BUILD_FLAVOURS:
source "${TOPLEVEL}/scripts/nns-dapp/flavours.bash"

###############
# backend # (output: nns-dapp.wasm.gz)
# backend # (output: nns-dapp_test.wasm.gz, nns-dapp_production.wasm.gz and, for backwards compatibility, nns-dapp.wasm.gz)
###############
echo Compiling rust package
"$TOPLEVEL/build-rs.sh" nns-dapp

echo Sanity check
scripts/nns-dapp/test-exports
for DFX_FLAVOUR in "${NNS_DAPP_BUILD_FLAVOURS[@]}"; do

echo "Compiling nns-dapp backend, $DFX_FLAVOUR build..."
"$TOPLEVEL/build-rs.sh" nns-dapp --features "${DFX_FLAVOUR}"

mv "nns-dapp.wasm.gz" "nns-dapp_${DFX_FLAVOUR}.wasm.gz"

echo "Sanity checking $DFX_FLAVOUR build..."
scripts/nns-dapp/test-exports --flavour "$DFX_FLAVOUR"
done

# For backwards compatibility:
cp nns-dapp_production.wasm.gz nns-dapp.wasm.gz
5 changes: 5 additions & 0 deletions rs/backend/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -56,3 +56,8 @@ regex = "1.7.1"
[dev-dependencies]
maplit = "1.0.2"
anyhow = "1.0.68"

[features]
production=[]
test=[]
# TODO: Define noassets as a feature flag, to keep different build flavours consistent.
File renamed without changes.
23 changes: 23 additions & 0 deletions rs/backend/nns-dapp-exports-test.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
canister_global_timer
canister_heartbeat
canister_init
canister_post_upgrade
canister_pre_upgrade
canister_query get_account
canister_query get_canisters
canister_query get_stats
canister_query get_transactions
canister_query http_request
canister_update <ic-cdk internal> timer_executor
canister_update add_account
canister_update add_assets_tar_xz
canister_update add_pending_notify_swap
canister_update add_stable_asset
canister_update attach_canister
canister_update create_sub_account
canister_update detach_canister
canister_update get_proposal_payload
canister_update register_hardware_wallet
canister_update rename_canister
canister_update rename_sub_account
main
3 changes: 3 additions & 0 deletions scripts/nns-dapp/flavours.bash
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# TODO: Add tiny/noassets to this list once it is built with a cargo feature flag.
# shellcheck disable=SC2034
NNS_DAPP_BUILD_FLAVOURS=( production test )
11 changes: 9 additions & 2 deletions scripts/nns-dapp/test-exports
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
set -euo pipefail
SOURCE_DIR="$(dirname "$(realpath "${BASH_SOURCE[0]}")")/.."
PATH="$SOURCE_DIR:$PATH"
# Defines NNS_DAPP_BUILD_FLAVOURS:
source "${SOURCE_DIR}/nns-dapp/flavours.bash"

print_help() {
cat <<-EOF
Expand All @@ -10,15 +12,20 @@ print_help() {
EOF
}

# shellcheck disable=SC2016
DFX_WASM_PATH_DEFAULT_TEMPLATE='nns-dapp_${DFX_FLAVOUR}.wasm.gz'

# Source the clap.bash file ---------------------------------------------------
source "$SOURCE_DIR/clap.bash"
# Define options
clap.define short=w long=wasm desc="The location of the nns-dapp wasm." variable=DFX_WASM_PATH default="nns-dapp.wasm.gz"
clap.define short=w long=wasm desc="The location of the nns-dapp wasm." variable=DFX_WASM_PATH default="$DFX_WASM_PATH_DEFAULT_TEMPLATE"
clap.define short=u long=update-golden desc="Update the exports golden file" variable=UPDATE_GOLDEN nargs=0 default="false"
clap.define short=f long=flavour desc="The build flavour. One of: ${NNS_DAPP_BUILD_FLAVOURS[*]}" variable=DFX_FLAVOUR default="production"
# Source the output file ----------------------------------------------------------
source "$(clap.build)"

GOLDEN_FILE="rs/backend/nns-dapp-exports.txt"
[[ "$DFX_WASM_PATH" != "$DFX_WASM_PATH_DEFAULT_TEMPLATE" ]] || DFX_WASM_PATH="$(eval echo "${DFX_WASM_PATH_DEFAULT_TEMPLATE}")"
GOLDEN_FILE="rs/backend/nns-dapp-exports-${DFX_FLAVOUR}.txt"

wasm_exports() {
ic-wasm <(gunzip <"$1") info | sed -nE '/^Exported methods:/,/^]/p' | sed '1d;$d' | sed -E 's/.*"(.*)",/\1/'
Expand Down

0 comments on commit 6fda486

Please sign in to comment.