Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

makefiles/docker.inc.mk: handle building in git worktree [backport 2018.10] #10307

Merged
merged 2 commits into from
Nov 2, 2018
Merged

makefiles/docker.inc.mk: handle building in git worktree [backport 2018.10] #10307

merged 2 commits into from
Nov 2, 2018

Conversation

cladmi
Copy link
Contributor

@cladmi cladmi commented Oct 31, 2018

Backport of #10303

Contribution description

When building from a worktree, the common git directory was not mounted in docker.

This lead to the version not being set and issues with git-cache in
ubuntu bionic that could not execute the 'git hash-object' command.

Implementation details

I just test that .git is a file starting with gitdir to know it is in a worktree.
Any other solution is welcomed.

I then mounted the whole --git-common-dir in the same location.
Using the gitdir pointed by the .git file was not enough to make it valid.

I aslo tested overwriting the .git file with the .git repository but docker did not allow that.

Testing procedure

Original behavior untouched

In a normal repository, no .git directory is mounted when building with docker.

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all

# There should be no output with `-v path_to_dir/RIOT/.git:path_to_dir/RIOT/.git`

New behavior

This must be run from a git worktree (replace riot/master to the upstream master branch if different)

git worktree add ../git_test_worktree riot/master
cd ../git_test_worktree
Testing the RIOT_VERSION calculation from git

With riot/master we get an UNKNOWN version, (see the last line).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all && strings examples/hello-world/bin/native/hello-world.elf  | grep "This is RIOT"
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
     \
    -w '/data/riotbuild/riotproject/examples/hello-world/' \
    'riot/riotbuild:latest' make all
Building application "hello-world" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
  20641     372   47684   68697   10c59 /data/riotbuild/riotproject/examples/hello-world/bin/native/hello-world.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
main(): This is RIOT! (Version: UNKNOWN (builddir: /data/riotbuild/riotbase))

With this PR the version is correctly detected (see the last line).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/hello-world/ clean all && strings examples/hello-world/bin/native/hello-world.elf  | grep "This is RIOT"
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/harter/work/git/RIOT/.git:/home/harter/work/git/RIOT/.git \
     \
    -w '/data/riotbuild/riotproject/examples/hello-world/' \
    'riot/riotbuild:latest' make all 
Building application "hello-world" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
  20649     372   47684   68705   10c61 /data/riotbuild/riotproject/examples/hello-world/bin/native/hello-world.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/hello-world'
main(): This is RIOT! (Version: 2018.10-RC1-41-g72f71-pr/make/docker/handle_worktree)
Testing the behavior with ubuntu:bionic docker image

You must have an initialized git cache

./dist/tools/git/git-cache init

Build a docker image from RIOT-OS/riotdocker#42
I used docker build . -t riot/bionic from the repository.

Compiling a package using the bionic image fails in master as git hash-object tries to detect the current directory as a git directory.

rm -rf examples/filesystem/bin/; BUILD_IN_DOCKER=1 DOCKER="sudo docker" DOCKER_IMAGE="riot/bionic:latest" RIOT_CI_BUILD=1  make -C examples/filesystem/ clean all
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'
make[1]: Nothing to be done for 'Makefile.include'.
make: Nothing to be done for 'clean'.
Launching build container using image "riot/bionic:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache \
    -e 'RIOT_CI_BUILD=1' \
    -w '/data/riotbuild/riotproject/examples/filesystem/' \
    'riot/bionic:latest' make all
make[1]: Nothing to be done for 'Makefile.include'.
Building application "filesystem" for "native" with MCU "native".

rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
fatal: not a git repository: /home/harter/work/git/RIOT/.git/worktrees/riot_master
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
fatal: Invalid refspec 'refs/tags//*:refs/tags/*'
/data/riotbuild/riotbase/pkg/pkg.mk:33: recipe for target '/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded' failed
make[1]: *** [/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded] Error 128
/data/riotbuild/riotbase/Makefile.include:487: recipe for target 'pkg-prepare' failed
make: [pkg-prepare] Error 2 (ignored)
rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
fatal: not a git repository: /home/harter/work/git/RIOT/.git/worktrees/riot_master
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
fatal: Invalid refspec 'refs/tags//*:refs/tags/*'
/data/riotbuild/riotbase/pkg/pkg.mk:33: recipe for target '/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded' failed
make[1]: *** [/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded] Error 128
/data/riotbuild/riotbase/Makefile.include:491: recipe for target 'pkg-build-littlefs' failed
make: *** [pkg-build-littlefs] Error 2
/home/harter/work/git/worktree/riot_master/makefiles/docker.inc.mk:100: recipe for target '..in-docker-container' failed
make: *** [..in-docker-container] Error 2
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'

It works with this PR

rm -rf examples/filesystem/bin/; BUILD_IN_DOCKER=1 DOCKER="sudo docker" DOCKER_IMAGE="riot/bionic:latest" RIOT_CI_BUILD=1  make -C examples/filesystem/ clean all
make: Entering directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'
make[1]: Nothing to be done for 'Makefile.include'.
make: Nothing to be done for 'clean'.
Launching build container using image "riot/bionic:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/worktree/riot_master/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/worktree/riot_master/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/worktree/riot_master/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/worktree/riot_master:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/harter/work/git/RIOT/.git:/home/harter/work/git/RIOT/.git \
    -e 'RIOT_CI_BUILD=1' \
    -w '/data/riotbuild/riotproject/examples/filesystem/' \
    'riot/bionic:latest' make all
make[1]: Nothing to be done for 'Makefile.include'.
Building application "filesystem" for "native" with MCU "native".

rm -Rf /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
mkdir -p /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs
/data/riotbuild/riotbase/dist/tools/git/git-cache clone "https://github.com/geky/littlefs.git" "0bb1f7af17755bd792f0c4966877fb1886dfc802" "/data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs"
git-cache: cloning from cache. tag=commit0bb1f7af17755bd792f0c4966877fb1886dfc802
touch /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs/.git-downloaded
"make" -C /data/riotbuild/riotproject/examples/filesystem/bin/pkg/native/littlefs -f /data/riotbuild/riotbase/pkg/littlefs/Makefile.littlefs
   text    data     bss     dec     hex filename
  87035    1180   49336  137551   2194f /data/riotbuild/riotproject/examples/filesystem/bin/native/filesystem.elf
make: Leaving directory '/home/harter/work/git/worktree/riot_master/examples/filesystem'

Issues/PRs references

The RIOT_VERSION issue was detected in #9645 (comment) and the git-cache issue was detected when running Release compilation with bionic image RIOT-OS/Release-Specs#76 (comment)

@cladmi
makefiles/docker.inc.mk: handle building in git worktree
1e2498b 
When building from a worktree, the common git directory was not mounted
in docker.

This lead to the version not being set and issues with git-cache in
ubuntu bionic that could not execute the 'git hash-object' command.

(cherry picked from commit 61a3e5d)
@cladmi cladmi added Area: build system Area: Build system Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … Area: tools Area: Supplementary tools CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: release backport Integration Process: The PR is a release backport of a change previously provided to master Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation labels Oct 31, 2018
@cladmi cladmi requested a review from jcarrano October 31, 2018 14:58
@cladmi cladmi mentioned this pull request Oct 31, 2018
Merged
@cladmi
Copy link
Contributor Author

cladmi commented Oct 31, 2018

I will re-run the tests with this one.

@cladmi
Copy link
Contributor Author

cladmi commented Oct 31, 2018

Both test indeed failed with the release branch and worked with this PR.

@cladmi cladmi added this to the Release 2018.10 milestone Nov 1, 2018
jcarrano
jcarrano approved these changes Nov 1, 2018
View reviewed changes
Copy link
Contributor

@jcarrano jcarrano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works.

@jia200x
Copy link
Member

jia200x commented Nov 2, 2018

let's wait for Murdock. Thanks for the backport and testing!

@cladmi
Copy link
Contributor Author

cladmi commented Nov 2, 2018

Murdock is green. Thank you for the review of the backport too.

@cladmi cladmi merged commit 488771f into RIOT-OS:2018.10-branch Nov 2, 2018
@cladmi cladmi deleted the backport/2018.10/pr/make/docker/handle_worktree branch November 2, 2018 11:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jcarrano jcarrano jcarrano approved these changes

Assignees
No one assigned
Labels
Area: build system Area: Build system Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … Area: tools Area: Supplementary tools CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Process: release backport Integration Process: The PR is a release backport of a change previously provided to master Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation
Projects
None yet
Milestone
Release 2018.10
Development

Successfully merging this pull request may close these issues.
3 participants
@cladmi @jia200x @jcarrano