Enabling ARM64 CI #430

brev · 2019-04-27T00:33:50Z

Enabling ARM64 CI

Enabling ARM64 CI on CircleCI using Docker + QEMU.

TODO

~~workflow config too? researching.~~
~~can we mark a job/platform(this arm64) in circle to "be able to fail"? (per @breznak)~~
Add final badge and supported text to top of README
Build will be failing until ARM64 support #131 is resolved

Changes

Added docs (inline and readme) for the changes below.
Moved CircleCI config from old 1.x location to new 2.x location.
- Added CircleCI config for ARM64 Docker+QEMU build.
Updated Dockerfile
- Migrated from nupic.core to nupic.cpp
- Add support from command-line argument variable pass-in (arch)
- Modernize and prep to work for both Linux x86_64 and ARM64
Aligned badges on top of README
Removed some ugly blank lines and my editor removed some orphan spaces.

Local Test

# developing on mac os/x
brew install circleci
cd nupic.cpp/
circleci local execute --job arm64-build-test

Context

#134 * Moving CircleCI from old 1.x config style (`./circle.yml`) to the newer 2.x style (`./.circleci/config.yml`). * Adding a 2nd job to the existing CircleCI config (Darwin), to try and build an ARM64 Docker build. * Updates to Dockerfile for `nupic.cpp` instead of older `nupic.core`. With these changes (so far), one can start a local CircleCI build of NuPIC Docker on ARM64. We get to our first error (which seems like the kind of error I'd expect going from `x86_64` -> `arm64`. I have no idea what to do with it, though: ``` [ 63%] Building CXX object src/test/CMakeFiles/unit_tests.dir/unit/ntypes/BasicTypeTest.cpp.o /usr/local/src/nupic.cpp/src/test/unit/ntypes/BasicTypeTest.cpp:115:49: error: narrowing conversion of '-128' from 'int' to 'nupic::Byte {aka char}' inside { } [-Wnarrowing] Byte bufByte[8] = {0, 1, 2, 3, 4, 5, -128, 127}; ^ src/test/CMakeFiles/unit_tests.dir/build.make:521: recipe for target 'src/test/CMakeFiles/unit_tests.dir/unit/ntypes/BasicTypeTest.cpp.o' failed make[2]: *** [src/test/CMakeFiles/unit_tests.dir/unit/ntypes/BasicTypeTest.cpp.o] Error 1 CMakeFiles/Makefile2:436: recipe for target 'src/test/CMakeFiles/unit_tests.dir/all' failed make[1]: *** [src/test/CMakeFiles/unit_tests.dir/all] Error 2 Makefile:151: recipe for target 'all' failed make: *** [all] Error 2 setup.py: Calling /usr/local/src/nupic.cpp/bindings/py/packaging/setup.py ```

- https://github.com/brev/nupic.cpp/pull/1 - #134 (comment)

brev · 2019-04-27T01:20:07Z

Can we mark a job/platform(this arm64) in circle to "be able to fail"?

@breznak I'm not seeing any settings from the CircleCI UI or Config for this, but it appears we can just force a success exit code in the script?

numpy and boost are slooow.

breznak

Thank you @brev , this is a great contribution! 💯 I'll be very happy to see ARM64 officially working!

seems the "make arm64 CI jobs ok-to-fail until fixed" problem is fine, as CircleCI is smart enough and runs the checks as separate jobs, so you can see OSX is still required, while ARM is not 👌
there's a weird problem saying for our OSX job that it does not run with comment "Please select OSX plan". I don't know why this PR would cause it? Maybe the change to v2 format. If you are unaware of the problem/fix as well, I'll have to email their support, they were really responsive and helpful. I've mailed them to enable OSX for us for free, as is their offering for open-source, maybe the system just changed with the move to v2.

So the only issue is the OSX not running, which I'll also try to resolve by mailing support.

.circleci/config.yml

breznak · 2019-04-27T09:43:35Z

.circleci/config.yml

+      - run: $CI_SUDO docker run --rm --privileged multiarch/qemu-user-static:register --reset
+      - run:
+          command: $CI_SUDO docker build --build-arg arch=arm64 .
+          no_output_timeout: 2h


question, how long to the tests (on CI in docker...) take to run?

@breznak arm64 is 1 hour 45 min, mac is 25 min.

breznak · 2019-04-27T09:45:02Z

README.md

 ran against this extention library. If you are porting your code to this codebase, please review [API Changelog](API_CHANGELOG.md).

-### New Features 
+### New Features


ok now, but for the future please fix your editor to avoid these "automated" whitespace changes. It makes the review more difficult.

breznak · 2019-04-27T09:56:59Z

Dockerfile

+
+# Multiarch Ubuntu Bionic 18.04. arches: x86_64, arm64, etc.
+#   https://hub.docker.com/r/multiarch/ubuntu-core/tags/
+FROM multiarch/ubuntu-core:$arch-bionic

 RUN apt-get update && \


please try if this works without the update, only install? As after updating, much more packages will be rebuilt and it takes much longer.

update is needed, some packages are no longer downloadable otherwise. This is a bummber, would have been faster to use just what is in the docker img.

This reverts commit 868e7a8.

This reverts commit 3868bfa.

…into enabling-arm64-ci

This reverts commit e8f1a6d.

breznak · 2019-05-02T17:53:26Z

here's a weird problem saying for our OSX job that it does not run with comment "Please select OSX plan". I don't know why this PR would cause it?

For some reason this PR kept running in Brev's CI, and not under htm-community (we have open-source deal with CircleCI to allow us run OSX CI for free 👍 )
Opening the duplicate PR #438 seems to have triggered the correct build. I'll let both run and merge this PR ASAP.

breznak · 2019-05-02T18:57:08Z

I've merged this great contribution from @brev 👏 that:

enables CI for ARM64
updates CircleCI to version 2.0 format
TODO iron out ARM64 build -> ARM64 support #131
optimize the CI setup for docker

Thank you!! 👍

dkeeney · 2019-05-02T23:57:23Z

Very good. Thanks @brev

brev · 2019-05-03T01:59:36Z

thanks @breznak @dkeeney !

brev added 4 commits April 21, 2019 20:11

CircleCI ARM64 Docker/QEMU build.

7d26713

- https://github.com/brev/nupic.cpp/pull/1 - #134 (comment)

fix

dbc719e

nit

5b5ba7e

brev mentioned this pull request Apr 27, 2019

Find CI for ARM64 #134

Closed

brev added 3 commits April 26, 2019 17:54

nudges

4c14331

trying to trigger my personal circleci build of this branch

26111fe

Make it GO in CircleCI (add to Workflow)

1c0bf86

update circleci timeout for our build

4d2e633

numpy and boost are slooow.

breznak assigned brev Apr 27, 2019

breznak added enhancement New feature or request platform labels Apr 27, 2019

Merge branch 'master' into enabling-arm64-ci

b155601

breznak requested changes Apr 27, 2019

View reviewed changes

breznak reviewed Apr 27, 2019

View reviewed changes

breznak and others added 5 commits May 2, 2019 17:09

CI: circleCI rename to OSX-build-and-test

3868bfa

Docker: avoid apt-update

868e7a8

Docker: try rm some dependencies

e8f1a6d

Merge branch 'master' into enabling-arm64-ci

fc812bd

Revert "Docker: avoid apt-update"

dafece1

This reverts commit 868e7a8.

breznak closed this May 2, 2019

breznak reopened this May 2, 2019

breznak added 3 commits May 2, 2019 17:27

Revert "CI: circleCI rename to OSX-build-and-test"

a000899

This reverts commit 3868bfa.

Merge branch 'enabling-arm64-ci' of https://github.com/brev/nupic.cpp …

ef51ecb

…into enabling-arm64-ci

Revert "Docker: try rm some dependencies"

8ac758f

This reverts commit e8f1a6d.

breznak merged commit 2586440 into htm-community:master May 2, 2019

breznak added the community label May 2, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling ARM64 CI #430

Enabling ARM64 CI #430

brev commented Apr 27, 2019 •

edited

Loading

brev commented Apr 27, 2019

breznak left a comment •

edited

Loading

breznak Apr 27, 2019

brev May 3, 2019

breznak Apr 27, 2019

breznak Apr 27, 2019

breznak May 2, 2019

breznak commented May 2, 2019

breznak commented May 2, 2019 •

edited

Loading

dkeeney commented May 2, 2019

brev commented May 3, 2019

Enabling ARM64 CI #430

Enabling ARM64 CI #430

Conversation

brev commented Apr 27, 2019 • edited Loading