Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sec fixes, build-time optimizations, dependency optimizations, CI migration, etc #1699

Merged
merged 55 commits into from
Mar 23, 2023
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
44c6397
upgrade crossbeam from vulnerable version
onur-ozkan Mar 7, 2023
f8bf71e
disable solana
onur-ozkan Mar 7, 2023
ce99a1c
save dev state
onur-ozkan Mar 7, 2023
6b721e0
save dev state
onur-ozkan Mar 7, 2023
1a3bc45
update jemalloc config, enable `share-generics` nightly feat, use lld
onur-ozkan Mar 8, 2023
c4305a1
comment out lld configs
onur-ozkan Mar 8, 2023
e179834
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 8, 2023
667e976
create check, fmt, lint multi-os pipelines
onur-ozkan Mar 8, 2023
34be3ac
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 8, 2023
da636a2
improve checks for x86
onur-ozkan Mar 8, 2023
1f73e89
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 8, 2023
1ac04ed
update pipeline
onur-ozkan Mar 8, 2023
7b283d0
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 8, 2023
0bbfc4e
exclude check step
onur-ozkan Mar 8, 2023
c110fce
update pipeline
onur-ozkan Mar 8, 2023
98d0594
add wasm lint step
onur-ozkan Mar 8, 2023
0058f71
improvide wasm stack and add wasm pipeline
onur-ozkan Mar 9, 2023
a3b3cba
initialize build workflow
onur-ozkan Mar 9, 2023
11091bf
increase fmt & lint CI timeout
onur-ozkan Mar 9, 2023
4eb1557
compute MM_VERSION and build for wasm
onur-ozkan Mar 10, 2023
148366b
add arm builds
onur-ozkan Mar 10, 2023
07a9515
add tests
onur-ozkan Mar 14, 2023
376e383
swtich docker deamon in win
onur-ozkan Mar 14, 2023
a5fc6f8
test
onur-ozkan Mar 14, 2023
f01a704
test
onur-ozkan Mar 14, 2023
7df9405
test
onur-ozkan Mar 14, 2023
4d2a8cd
run docker tests in seperated job
onur-ozkan Mar 14, 2023
6f01ced
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 14, 2023
ac07984
get rid of docker in mac
onur-ozkan Mar 14, 2023
29e28bc
update test docs
onur-ozkan Mar 14, 2023
6a47670
test
onur-ozkan Mar 14, 2023
abcd891
disable `test_metadata_list_and_json_deserialization` for `ci` profile
onur-ozkan Mar 14, 2023
c6b07e1
optimize workflow configs
onur-ozkan Mar 14, 2023
bf4ba65
disable incompatible tests
onur-ozkan Mar 14, 2023
0b752f1
check shared runners
onur-ozkan Mar 14, 2023
b8d7d75
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 14, 2023
d1cd769
enable cross target linting
onur-ozkan Mar 14, 2023
f343573
apply linting for all targets
onur-ozkan Mar 14, 2023
9050e0a
add wasm lint
onur-ozkan Mar 14, 2023
3b23222
update macos test
onur-ozkan Mar 14, 2023
cc0ff30
fix some clippy warnings
onur-ozkan Mar 14, 2023
540f764
seperate tests
onur-ozkan Mar 14, 2023
07b5791
ignore unstable tests
onur-ozkan Mar 14, 2023
fd7f710
disable mm2 integration tests for mac due to concurrency limits
onur-ozkan Mar 14, 2023
ee37070
implement container build and push on docker registry
onur-ozkan Mar 15, 2023
8cdac94
add release builds pipelines
onur-ozkan Mar 15, 2023
1391e8e
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 17, 2023
78d3307
add changelog entries
onur-ozkan Mar 17, 2023
6c706ce
test
onur-ozkan Mar 20, 2023
e9f1265
Merge branch 'dev' of github.com:KomodoPlatform/atomicDEX-API into de…
onur-ozkan Mar 20, 2023
d084b62
implement build uploads from CI
onur-ozkan Mar 20, 2023
72e9819
build dylib for x86-64 mac
onur-ozkan Mar 21, 2023
e71e2cf
statically link libusb
onur-ozkan Mar 21, 2023
53b2986
add changelog entry
onur-ozkan Mar 22, 2023
e031b29
fix `test_tendermint_activation_and_balance` test
onur-ozkan Mar 22, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion .cargo/config
Original file line number Diff line number Diff line change
@@ -1,3 +1,27 @@
[env]
JEMALLOC_SYS_WITH_MALLOC_CONF = "percpu_arena:percpu,oversize_threshold:0,background_thread:true,metadata_thp:auto,dirty_decay_ms:5000,muzzy_decay_ms:5000"

[target.'cfg(all())']
rustflags = [ "-Zshare-generics=y" ]

# # Install lld using package manager
# [target.x86_64-unknown-linux-gnu]
# linker = "clang"
# rustflags = [ "-Clink-arg=-fuse-ld=lld" ]
#
# # `brew install llvm`
# [target.x86_64-apple-darwin]
# rustflags = [
# "-C",
# "link-arg=-fuse-ld=/usr/local/opt/llvm/bin/ld64.lld",
# ]
#
# [target.aarch64-apple-darwin]
# rustflags = [
# "-C",
# "link-arg=-fuse-ld=/opt/homebrew/opt/llvm/bin/ld64.lld",
# ]

[target.wasm32-unknown-unknown]
runner = 'wasm-bindgen-test-runner'
rustflags = ["--cfg=web_sys_unstable_apis"]
rustflags = [ "--cfg=web_sys_unstable_apis" ]
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ cmake-build-debug
# Ignoring the Dockerfile allows us to change it without invalidating the "COPY ." step.
/Dockerfile

/target
!/target/release/mm2
!/target/ci/mm2
/mm2src/*/target

Expand Down
189 changes: 189 additions & 0 deletions .github/workflows/dev-build.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
name: Development builds
on:
push:
branches:
- dev
pull_request:
branches:
- dev

env:
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
MANUAL_MM_VERSION: true

jobs:
linux-x86-64:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29

- name: Build
run: |
VERSION=$(git rev-parse --short=7 HEAD)
rm -f ./MM_VERSION
echo $VERSION > ./MM_VERSION
cargo build --bin mm2 --profile ci

- name: Login to dockerhub
if: github.event_name != 'pull_request'
run: docker login --username ${{ secrets.DOCKER_HUB_USERNAME }} --password ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }} docker.io

- name: Build and push container image
if: github.event_name != 'pull_request'
run: |
CONTAINER_TAG="dev-$(git rev-parse --short=7 HEAD)"
docker build -t komodoofficial/atomicdexapi:"$CONTAINER_TAG" -t komodoofficial/atomicdexapi:dev-latest -f Dockerfile.dev-release .
docker push komodoofficial/atomicdexapi:"$CONTAINER_TAG"
docker push komodoofficial/atomicdexapi:dev-latest

mac-x86-64:
timeout-minutes: 30
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29

- name: Build
run: |
VERSION=$(git rev-parse --short=7 HEAD)
rm -f ./MM_VERSION
echo $VERSION > ./MM_VERSION
cargo build --bin mm2 --profile ci --target x86_64-apple-darwin

win-x86-64:
timeout-minutes: 30
runs-on: windows-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29

- name: Build
run: |
$VERSION="$(git rev-parse --short=7 HEAD)"
if (test-path "./MM_VERSION") {
remove-item "./MM_VERSION"
}
echo $VERSION > ./MM_VERSION
cargo build --bin mm2 --profile ci

wasm:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29
rustup target add wasm32-unknown-unknown

- name: Install wasm-pack
run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh

- name: Build
run: |
VERSION=$(git rev-parse --short=7 HEAD)
rm -f ./MM_VERSION
echo $VERSION > ./MM_VERSION
wasm-pack build mm2src/mm2_bin_lib --target web --out-dir wasm_build/deps/pkg/

ios-aarch64:
timeout-minutes: 30
runs-on: macos-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29
rustup target add aarch64-apple-ios

- name: Build
run: |
VERSION=$(git rev-parse --short=7 HEAD)
rm -f ./MM_VERSION
echo $VERSION > ./MM_VERSION
cargo rustc --target aarch64-apple-ios --lib --profile ci --package mm2_bin_lib --crate-type=staticlib

android-aarch64:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29
rustup target add aarch64-linux-android

- name: Setup NDK
run: ./android-ndk.sh x86 21

- name: Build
run: |
VERSION=$(git rev-parse --short=7 HEAD)
rm -f ./MM_VERSION
echo $VERSION > ./MM_VERSION

export PATH=$PATH:/android-ndk/bin
CC_aarch64_linux_android=aarch64-linux-android21-clang CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android21-clang cargo rustc --target=aarch64-linux-android --lib --profile ci --crate-type=staticlib --package mm2_bin_lib

android-armv7:
timeout-minutes: 30
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal
rustup default nightly-2022-10-29
rustup target add armv7-linux-androideabi

- name: Setup NDK
run: ./android-ndk.sh x86 21

- name: Build
run: |
VERSION=$(git rev-parse --short=7 HEAD)
rm -f ./MM_VERSION
echo $VERSION > ./MM_VERSION

export PATH=$PATH:/android-ndk/bin
CC_armv7_linux_androideabi=armv7a-linux-androideabi21-clang CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER=armv7a-linux-androideabi21-clang cargo rustc --target=armv7-linux-androideabi --lib --profile ci --crate-type=staticlib --package mm2_bin_lib

deployment-commitment:
if: github.event_name != 'pull_request'
needs: linux-x86-64
timeout-minutes: 15
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- name: Activate SSH
uses: webfactory/[email protected]
with:
ssh-private-key: ${{ secrets.ATOMICDEX_DEPLOYMENTS_SSH }}

- name: Commitment
run: |
git clone [email protected]:KomodoPlatform/atomicdex-deployments.git
if [ -d "atomicdex-deployments/atomicDEX-API" ]; then
cd atomicdex-deployments/atomicDEX-API
sed -i "1s/^.*$/${GITHUB_SHA::9}/" .commit
git config --global user.email "[email protected]"
git config --global user.name "linuxci"
git add .commit
git commit -m "[atomicDEX-API] ${GITHUB_SHA::9} is committed for git & container registry"
git push
fi
44 changes: 44 additions & 0 deletions .github/workflows/fmt-and-lint.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Format and Lint
on:
push:
branches:
- main
- dev
pull_request:
branches:
- main
- dev

jobs:
fmt-and-lint:
timeout-minutes: 45
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal --component rustfmt clippy
rustup default nightly-2022-10-29

- name: fmt check
run: cargo fmt -- --check

- name: x86-64 code lint
run: cargo clippy --all-targets --profile ci -- --D warnings

wasm-lint:
timeout-minutes: 45
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Install toolchain
run: |
rustup toolchain install nightly-2022-10-29 --no-self-update --profile=minimal --component clippy
rustup default nightly-2022-10-29
rustup target add wasm32-unknown-unknown

- name: wasm code lint
run: cargo clippy --target wasm32-unknown-unknown --profile ci -- --D warnings
Loading