Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix build for case-insensitive FS and add CI test for OSX #1951

Merged
merged 2 commits into from
Aug 23, 2019
Merged

Fix build for case-insensitive FS and add CI test for OSX #1951

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
50e3011
[python] Fix build for case-insensitive FS :apple:
opacam Aug 17, 2019
5a4484b
[ci] Add Mac OSX CI's test & refactor android's NDK/SDK installation
opacam Aug 22, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 28 additions & 11 deletions .travis.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,11 +16,6 @@ before_install:
- git remote set-branches --add origin develop
- git fetch

env:
global:
- ANDROID_SDK_HOME=/opt/android/android-sdk
- ANDROID_NDK_HOME=/opt/android/android-ndk

jobs:
include:
- &linting
Expand Down Expand Up @@ -65,19 +60,41 @@ jobs:
# case that the travis log doesn't produce any output for more than 10 minutes
- while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done &
script:
- docker run -e CI -e TRAVIS_JOB_ID="$TRAVIS_JOB_ID" -e TRAVIS_BRANCH="$TRAVIS_BRANCH" p4a /bin/sh -c "$COMMAND"
- >
docker run
-e CI
-e TRAVIS_JOB_ID
-e TRAVIS_BRANCH
-e ANDROID_SDK_HOME="/home/user/.android/android-sdk"
-e ANDROID_NDK_HOME="/home/user/.android/android-ndk"
p4a /bin/sh -c "$COMMAND"
after_script:
# kill the background process started before run docker
- kill %1
name: Python 3 armeabi-v7a
name: Python 3 arm64-v8a
# overrides requirements to skip `peewee` pure python module, see:
# https://github.com/kivy/python-for-android/issues/1263#issuecomment-390421054
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=armeabi-v7a
env:
COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=arm64-v8a
- <<: *testing
name: Python 3 arm64-v8a
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python3_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools' --arch=arm64-v8a
name: Python 3 armeabi-v7a
os: osx
osx_image: xcode11 # since xcode1.3, python3 is the default interpreter
before_script:
# installs java 1.8, android's SDK/NDK and p4a
- make -f ci/makefiles/osx.mk
- export JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just wondering, is needed for building the app in the script part below?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we need it because the default java version for travis's OS X image (xcode11) has another version of java

See also: kivy/buildozer#945

# Run a background process (like we do with linux tests)
- while sleep 540; do echo "==== Still running (travis, don't kill me) ===="; done &
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might not be needed for the OSX build actually, because I don't think it would run more than 20 or 30 minutes without log printing.
travis_wait waits 20 minutes by default and can be increased up to 30 minutes by passing time as parameter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I maintained the background process...I think that could be useful..., but I removed one line of the above comment (since we already commented before)...but we can remove it if you feel this is useless...just let me know 😉

script:
- >
cd testapps && python3 setup_testapp_python3_sqlite_openssl.py apk
--sdk-dir $HOME/.android/android-sdk
--ndk-dir $HOME/.android/android-ndk
--requirements libffi,sdl2,pyjnius,kivy,python3,openssl,requests,sqlite3,setuptools
--arch=armeabi-v7a
- <<: *testing
name: Python 2 basic
name: Python 2 armeabi-v7a (with numpy)
env: COMMAND='. venv/bin/activate && cd testapps/ && python setup_testapp_python2_sqlite_openssl.py apk --sdk-dir $ANDROID_SDK_HOME --ndk-dir $ANDROID_NDK_HOME --requirements sdl2,pyjnius,kivy,python2,openssl,requests,sqlite3,setuptools,numpy'
- <<: *testing
name: Rebuild updated recipes
Expand Down
62 changes: 11 additions & 51 deletions Dockerfile.py2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,64 +37,17 @@ ENV RETRY="retry -t 3 --"
RUN curl https://raw.githubusercontent.com/kadwanev/retry/1.0.1/retry \
--output /usr/local/bin/retry && chmod +x /usr/local/bin/retry

ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
ENV ANDROID_NDK_VERSION="17c"
ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"

# get the latest version from https://developer.android.com/ndk/downloads/index.html
ENV ANDROID_NDK_ARCHIVE="android-ndk-r${ANDROID_NDK_VERSION}-linux-x86_64.zip"
ENV ANDROID_NDK_DL_URL="https://dl.google.com/android/repository/${ANDROID_NDK_ARCHIVE}"

# download and install Android NDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_NDK_DL_URL}" \
--output "${ANDROID_NDK_ARCHIVE}" \
&& mkdir --parents "${ANDROID_NDK_HOME_V}" \
&& unzip -q "${ANDROID_NDK_ARCHIVE}" -d "${ANDROID_HOME}" \
&& ln -sfn "${ANDROID_NDK_HOME_V}" "${ANDROID_NDK_HOME}" \
&& rm -rf "${ANDROID_NDK_ARCHIVE}"


ENV ANDROID_SDK_HOME="${ANDROID_HOME}/android-sdk"

# get the latest version from https://developer.android.com/studio/index.html
ENV ANDROID_SDK_TOOLS_VERSION="4333796"
ENV ANDROID_SDK_BUILD_TOOLS_VERSION="28.0.2"
ENV ANDROID_SDK_TOOLS_ARCHIVE="sdk-tools-linux-${ANDROID_SDK_TOOLS_VERSION}.zip"
ENV ANDROID_SDK_TOOLS_DL_URL="https://dl.google.com/android/repository/${ANDROID_SDK_TOOLS_ARCHIVE}"

# download and install Android SDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_SDK_TOOLS_DL_URL}" \
--output "${ANDROID_SDK_TOOLS_ARCHIVE}" \
&& mkdir --parents "${ANDROID_SDK_HOME}" \
&& unzip -q "${ANDROID_SDK_TOOLS_ARCHIVE}" -d "${ANDROID_SDK_HOME}" \
&& rm -rf "${ANDROID_SDK_TOOLS_ARCHIVE}"

# update Android SDK, install Android API, Build Tools...
RUN mkdir --parents "${ANDROID_SDK_HOME}/.android/" \
&& echo '### User Sources for Android SDK Manager' \
> "${ANDROID_SDK_HOME}/.android/repositories.cfg"

# Download and accept Android licenses (JDK necessary!)
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" > /dev/null

# Set avdmanager permissions (executable)
RUN chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"


ENV USER="user"
ENV HOME_DIR="/home/${USER}"
ENV ANDROID_HOME="${HOME_DIR}/.android"
ENV WORK_DIR="${HOME_DIR}" \
PATH="${HOME_DIR}/.local/bin:${PATH}"

# install system dependencies
RUN ${RETRY} apt -y install -qq --no-install-recommends \
python virtualenv python-pip wget lbzip2 patch sudo \
&& apt -y autoremove
&& apt -y autoremove \
&& apt -y clean

# build dependencies
# https://buildozer.readthedocs.io/en/latest/installation.html#android-on-ubuntu-16-04-64bit
Expand All @@ -113,6 +66,10 @@ RUN ${RETRY} apt -y install -qq --no-install-recommends \
&& apt -y autoremove \
&& apt -y clean

# Install Java, set JAVA_HOME (to accept android's SDK licenses) and clean
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove && apt -y clean
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

# prepare non root env
RUN useradd --create-home --shell /bin/bash ${USER}
Expand All @@ -124,9 +81,12 @@ RUN echo "%sudo ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

WORKDIR ${WORK_DIR}
COPY --chown=user:user . ${WORK_DIR}
RUN chown --recursive ${USER} ${ANDROID_SDK_HOME}
RUN mkdir ${ANDROID_HOME} && chown --recursive ${USER} ${ANDROID_HOME}
USER ${USER}

# Download and install android's NDK/SDK
RUN make -f ci/makefiles/android.mk target_os=linux

# install python-for-android from current branch
RUN virtualenv --python=python venv \
&& . venv/bin/activate \
Expand Down
61 changes: 9 additions & 52 deletions Dockerfile.py3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@

FROM ubuntu:18.04

ENV ANDROID_HOME="/opt/android"

# configure locale
RUN apt update -qq > /dev/null && apt install -qq --yes --no-install-recommends \
locales && \
Expand All @@ -37,57 +35,9 @@ ENV RETRY="retry -t 3 --"
RUN curl https://raw.githubusercontent.com/kadwanev/retry/1.0.1/retry \
--output /usr/local/bin/retry && chmod +x /usr/local/bin/retry

ENV ANDROID_NDK_HOME="${ANDROID_HOME}/android-ndk"
ENV ANDROID_NDK_VERSION="17c"
ENV ANDROID_NDK_HOME_V="${ANDROID_NDK_HOME}-r${ANDROID_NDK_VERSION}"

# get the latest version from https://developer.android.com/ndk/downloads/index.html
ENV ANDROID_NDK_ARCHIVE="android-ndk-r${ANDROID_NDK_VERSION}-linux-x86_64.zip"
ENV ANDROID_NDK_DL_URL="https://dl.google.com/android/repository/${ANDROID_NDK_ARCHIVE}"

# download and install Android NDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_NDK_DL_URL}" \
--output "${ANDROID_NDK_ARCHIVE}" \
&& mkdir --parents "${ANDROID_NDK_HOME_V}" \
&& unzip -q "${ANDROID_NDK_ARCHIVE}" -d "${ANDROID_HOME}" \
&& ln -sfn "${ANDROID_NDK_HOME_V}" "${ANDROID_NDK_HOME}" \
&& rm -rf "${ANDROID_NDK_ARCHIVE}"


ENV ANDROID_SDK_HOME="${ANDROID_HOME}/android-sdk"

# get the latest version from https://developer.android.com/studio/index.html
ENV ANDROID_SDK_TOOLS_VERSION="4333796"
ENV ANDROID_SDK_BUILD_TOOLS_VERSION="28.0.2"
ENV ANDROID_SDK_TOOLS_ARCHIVE="sdk-tools-linux-${ANDROID_SDK_TOOLS_VERSION}.zip"
ENV ANDROID_SDK_TOOLS_DL_URL="https://dl.google.com/android/repository/${ANDROID_SDK_TOOLS_ARCHIVE}"

# download and install Android SDK
RUN ${RETRY} curl --location --progress-bar --insecure \
"${ANDROID_SDK_TOOLS_DL_URL}" \
--output "${ANDROID_SDK_TOOLS_ARCHIVE}" \
&& mkdir --parents "${ANDROID_SDK_HOME}" \
&& unzip -q "${ANDROID_SDK_TOOLS_ARCHIVE}" -d "${ANDROID_SDK_HOME}" \
&& rm -rf "${ANDROID_SDK_TOOLS_ARCHIVE}"

# update Android SDK, install Android API, Build Tools...
RUN mkdir --parents "${ANDROID_SDK_HOME}/.android/" \
&& echo '### User Sources for Android SDK Manager' \
> "${ANDROID_SDK_HOME}/.android/repositories.cfg"

# Download and accept Android licenses (JDK necessary!)
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "build-tools;${ANDROID_SDK_BUILD_TOOLS_VERSION}" > /dev/null
RUN yes | "${ANDROID_SDK_HOME}/tools/bin/sdkmanager" "platforms;android-27" > /dev/null

# Set avdmanager permissions (executable)
RUN chmod +x "${ANDROID_SDK_HOME}/tools/bin/avdmanager"


ENV USER="user"
ENV HOME_DIR="/home/${USER}"
ENV ANDROID_HOME="${HOME_DIR}/.android"
ENV WORK_DIR="${HOME_DIR}" \
PATH="${HOME_DIR}/.local/bin:${PATH}"

Expand All @@ -114,6 +64,10 @@ RUN ${RETRY} apt -y install -qq --no-install-recommends \
&& apt -y autoremove \
&& apt -y clean

# Install Java and set JAVA_HOME (to accept android's SDK licenses)
RUN ${RETRY} apt -y install -qq --no-install-recommends openjdk-8-jdk \
&& apt -y autoremove && apt -y clean
ENV JAVA_HOME /usr/lib/jvm/java-8-openjdk-amd64

# prepare non root env
RUN useradd --create-home --shell /bin/bash ${USER}
Expand All @@ -127,9 +81,12 @@ RUN pip2 install --upgrade Cython==0.28.6

WORKDIR ${WORK_DIR}
COPY --chown=user:user . ${WORK_DIR}
RUN chown --recursive ${USER} ${ANDROID_SDK_HOME}
RUN mkdir ${ANDROID_HOME} && chown --recursive ${USER} ${ANDROID_HOME}
USER ${USER}

# Download and install android's NDK/SDK
RUN make -f ci/makefiles/android.mk target_os=linux
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! I'm just wondering, this layer would be cached entirely then correct? This is fine, I wasn't sure if it would be cached or not

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...this layer would be cached entirely then correct? This is fine, I wasn't sure...

I hope so...but I don't know for sure...so...:no_mouth:


# install python-for-android from current branch
RUN virtualenv --python=python3 venv \
&& . venv/bin/activate \
Expand Down
68 changes: 68 additions & 0 deletions ci/makefiles/android.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Downloads and installs the Android SDK depending on supplied platform: darwin or linux
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice DRY file! Pretty useful 👏


# We must provide a platform (darwin or linux) and we need JAVA_HOME defined
ifndef target_os
$(error target_os is not set...aborted!)
endif

# Those android NDK/SDK variables can be override when running the file
ANDROID_NDK_VERSION ?= 17c
ANDROID_SDK_TOOLS_VERSION ?= 4333796
ANDROID_SDK_BUILD_TOOLS_VERSION ?= 28.0.2
ANDROID_HOME ?= $(HOME)/.android
ANDROID_API_LEVEL ?= 27

ANDROID_SDK_HOME=$(ANDROID_HOME)/android-sdk
ANDROID_SDK_TOOLS_ARCHIVE=sdk-tools-$(target_os)-$(ANDROID_SDK_TOOLS_VERSION).zip
ANDROID_SDK_TOOLS_DL_URL=https://dl.google.com/android/repository/$(ANDROID_SDK_TOOLS_ARCHIVE)

ANDROID_NDK_HOME=$(ANDROID_HOME)/android-ndk
ANDROID_NDK_FOLDER=$(ANDROID_HOME)/android-ndk-r$(ANDROID_NDK_VERSION)
ANDROID_NDK_ARCHIVE=android-ndk-r$(ANDROID_NDK_VERSION)-$(target_os)-x86_64.zip
ANDROID_NDK_DL_URL=https://dl.google.com/android/repository/$(ANDROID_NDK_ARCHIVE)

$(info Target install OS is : $(target_os))
$(info Android SDK home is : $(ANDROID_SDK_HOME))
$(info Android NDK home is : $(ANDROID_NDK_HOME))
$(info Android SDK download url is : $(ANDROID_SDK_TOOLS_DL_URL))
$(info Android NDK download url is : $(ANDROID_NDK_DL_URL))
$(info Android API level is : $(ANDROID_API_LEVEL))
$(info Android NDK version is : $(ANDROID_NDK_VERSION))
$(info JAVA_HOME is : $(JAVA_HOME))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great idea to log this!


all: install_sdk install_ndk

install_sdk: download_android_sdk extract_android_sdk update_android_sdk

install_ndk: download_android_ndk extract_android_ndk

download_android_sdk:
curl --location --progress-bar --continue-at - \
$(ANDROID_SDK_TOOLS_DL_URL) --output $(ANDROID_SDK_TOOLS_ARCHIVE)

download_android_ndk:
curl --location --progress-bar --continue-at - \
$(ANDROID_NDK_DL_URL) --output $(ANDROID_NDK_ARCHIVE)

# Extract android SDK and remove the compressed file
extract_android_sdk:
mkdir -p $(ANDROID_SDK_HOME) \
&& unzip -q $(ANDROID_SDK_TOOLS_ARCHIVE) -d $(ANDROID_SDK_HOME) \
&& rm -f $(ANDROID_SDK_TOOLS_ARCHIVE)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor for later.
I'm wondering if we do want to clean it or use it as a target for not re-downloading it next time. So basically the idea wold be to have something like:

$(ANDROID_SDK_TOOLS_ARCHIVE):
	curl --location --progress-bar --continue-at - \
	$(ANDROID_SDK_TOOLS_DL_URL) --output $(ANDROID_SDK_TOOLS_ARCHIVE

download_android_sdk: $(ANDROID_SDK_TOOLS_ARCHIVE)

That way running the target twice won't run the download twice if already present. Same story for other targets/dependencies

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking in keeping docker's layer smallest as possible...since we don't to re use the removed files right now...so ...maybe better to change/test this if we need it, right? 😉

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes let's keep it that way 😄



# Extract android NDK and remove the compressed file
extract_android_ndk:
mkdir -p $(ANDROID_NDK_FOLDER) \
&& unzip -q $(ANDROID_NDK_ARCHIVE) -d $(ANDROID_HOME) \
&& ln -sfn $(ANDROID_NDK_FOLDER) $(ANDROID_NDK_HOME) \
&& rm -f $(ANDROID_NDK_ARCHIVE)

# updates Android SDK, install Android API, Build Tools and accept licenses
update_android_sdk:
touch $(ANDROID_HOME)/repositories.cfg
yes | $(ANDROID_SDK_HOME)/tools/bin/sdkmanager --licenses > /dev/null
$(ANDROID_SDK_HOME)/tools/bin/sdkmanager "build-tools;$(ANDROID_SDK_BUILD_TOOLS_VERSION)" > /dev/null
$(ANDROID_SDK_HOME)/tools/bin/sdkmanager "platforms;android-$(ANDROID_API_LEVEL)" > /dev/null
# Set avdmanager permissions (executable)
chmod +x $(ANDROID_SDK_HOME)/tools/bin/avdmanager
23 changes: 23 additions & 0 deletions ci/makefiles/osx.mk
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# installs java 1.8, android's SDK/NDK, cython and p4a

# The following variable/s can be override when running the file
ANDROID_HOME ?= $(HOME)/.android

all: install_java upgrade_cython install_android_ndk_sdk install_p4a

install_java:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is to think about later, but installing system requirements could actually be part Linux Makefile as well

brew tap adoptopenjdk/openjdk
brew cask install adoptopenjdk8
/usr/libexec/java_home -V

upgrade_cython:
pip3 install --upgrade Cython==0.28.6

install_android_ndk_sdk:
mkdir -p $(ANDROID_HOME)
make -f ci/makefiles/android.mk target_os=darwin JAVA_HOME=`/usr/libexec/java_home -v 1.8`

install_p4a:
# check python version and install p4a
python3 --version
pip3 install -e .
Loading