Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Docker and Dev Container setup using Buildkit #4392

Draft
wants to merge 66 commits into
base: main
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
063ab99
Comment out old CI for migration
ruffsl Jun 3, 2024
a4dff33
Simplify dockerignore by inverting matching set
ruffsl Jun 3, 2024
b070400
Refactor Dockerfile using Buildkit and bake files
ruffsl Jun 3, 2024
7a09d15
Change COLCON_HOME in Dockerfile for now
ruffsl Jun 3, 2024
cdbe165
Refactor Dev Container using Buildkit and bake files
ruffsl Jun 3, 2024
acec643
Refactor Action Workflows using Buildkit and bake files
ruffsl Jun 4, 2024
b0dd131
Comment out CI trigger while WIP
ruffsl Jun 4, 2024
35d28c7
Rollback RunsOn cache action to use GitHub's cache
ruffsl Jun 4, 2024
e78549e
Roll back RunsOn runner tags
ruffsl Jun 4, 2024
e9c2769
Roll back AWS ECR changes to use GHCR
ruffsl Jun 4, 2024
44c45d4
Update retention to use sanitized tag names
ruffsl Jun 4, 2024
f27f3a2
Add debugger and releaser bake targets
ruffsl Jun 4, 2024
37887b9
Clean ECR comments
ruffsl Jun 4, 2024
dc16919
Remove debugging command for test workflow
ruffsl Jun 4, 2024
b4ff763
Move docker stuff into .docker path
ruffsl Jun 8, 2024
2b0bae0
Update docker paths
ruffsl Jun 8, 2024
667a8ff
Mount user home as volume
ruffsl Jun 8, 2024
18bf8a4
vcs import from underlay.repos file
ruffsl Jun 13, 2024
ba682a1
Simplify given cache should only be saved
ruffsl Jul 6, 2024
a6497ad
Use separate cache step to save via if always
ruffsl Jul 6, 2024
c1a9aab
Update docker bake action from v4 to v5
ruffsl Jul 6, 2024
76110c0
Fix FromAsCasing warnings
ruffsl Jul 6, 2024
88e34e1
Merge remote-tracking branch 'origin/main' into buildkit
ruffsl Jul 6, 2024
a2652cd
Set ccache key via septate step
ruffsl Jul 6, 2024
cee6ac0
Pass ccache_cache_key between jobs
ruffsl Jul 6, 2024
3df9fb0
Restore ccache for build prod image jobs
ruffsl Jul 6, 2024
248b33c
Update path to Dockerfile
ruffsl Jul 6, 2024
5736762
Uncomment main caller workflow
ruffsl Jul 6, 2024
df9842a
Trigger on any change to the .docker path
ruffsl Jul 6, 2024
eb26b41
Change workflow action to current branch to test CI
ruffsl Jul 6, 2024
894126c
Fix typo to use GITHUB_TOKEN
ruffsl Jul 6, 2024
aef79ad
Omit OCI configuration used for AWS ECR
ruffsl Jul 6, 2024
5d8c00f
Image images using org + repo name
ruffsl Jul 6, 2024
ae8a274
Use gha for buildkit cache backend instead of S3
ruffsl Jul 6, 2024
de79d04
Bake underlay source into base image
ruffsl Jul 6, 2024
45f00f2
Revert "Bake underlay source into base image"
ruffsl Jul 6, 2024
140d1b3
Add nav2_minimal_turtlebot_simulation as submodules
ruffsl Jul 6, 2024
27e8ceb
Remove clone step using vcstool
ruffsl Jul 6, 2024
54f57aa
Fix OVERLAY_WS ENV to match Dockerfile
ruffsl Jul 7, 2024
c09f746
Try building prod image regardless of test results
ruffsl Jul 7, 2024
5121a30
Enable docker-outside-docker
ruffsl Jul 9, 2024
2c8bcc3
Reorder mounts
ruffsl Jul 9, 2024
65c0194
Enable gh CLI
ruffsl Jul 9, 2024
f22b3cb
Add svg extension to view Dockerfile graphs
ruffsl Jul 9, 2024
50b8f7c
Rename nav2 ws volume to be more descriptive
ruffsl Jul 9, 2024
bab8da7
Postfix home by use
ruffsl Jul 9, 2024
7af09f7
Add bind volume to user home
ruffsl Jul 9, 2024
922b91d
Comment out home bind mount by default
ruffsl Jul 9, 2024
ea6f72d
Add mark comments for readability
ruffsl Jul 9, 2024
93d7a92
Sort ENVs
ruffsl Jul 9, 2024
0de7df3
Explicitly set SSH_AUTH_SOCK for devcontainer CLI
ruffsl Jul 9, 2024
94faf5b
Alway return 1
ruffsl Jul 9, 2024
00fdbc8
Formatting
ruffsl Jul 9, 2024
39ca865
Install GUI tools into dever stage
ruffsl Jul 9, 2024
6126f1f
Use always() in if condition
ruffsl Jul 10, 2024
75b849b
Add shim to build dever stage from debugger image
ruffsl Jul 10, 2024
090be7f
Fix src folder to use full repo name
ruffsl Jul 10, 2024
fd84091
Remove --symlink-install from default script
ruffsl Jul 10, 2024
23fe6ec
Simplify by removing unnecessary unset
ruffsl Jul 10, 2024
17b9464
Correct comment
ruffsl Jul 10, 2024
3fd241a
Add default gitconfig to recurse over submodules
ruffsl Jul 10, 2024
e3501fe
Add readme with quick start guide
ruffsl Jul 10, 2024
1081d47
Add alias to source underlay workspace
ruffsl Jul 10, 2024
c695baa
Extend docs on build locally or pulling remotely
ruffsl Jul 10, 2024
0314c34
Fix typo
ruffsl Jul 10, 2024
73e8ed8
Update .devcontainer/README.md
ruffsl Dec 19, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor Dev Container using Buildkit and bake files
ruffsl committed Jun 8, 2024
commit cdbe1654c5cb67553d583d09aadab20b121d643a
49 changes: 36 additions & 13 deletions .devcontainer/devcontainer.json
Original file line number Diff line number Diff line change
@@ -1,44 +1,65 @@
{
"name": "Nav2",
"build": {
"dockerfile": "../Dockerfile",
"context": "..",
"target": "dever",
"cacheFrom": "ghcr.io/ros-navigation/navigation2:main"
},
"initializeCommand": ".devcontainer/initialize-command.sh dever", // Bakes to tag nav2:devcontainer
"image": "nav2:devcontainer",
"runArgs": [
"--name=nav2"
// "--cap-add=SYS_PTRACE", // enable debugging, e.g. gdb
// "--ipc=host", // shared memory transport with host, e.g. rviz GUIs
// "--network=host", // network access to host interfaces, e.g. eth0
// "--pid=host", // DDS discovery with host, without --network=host
// "--privileged", // device access to host peripherals, e.g. USB
// "--security-opt=seccomp=unconfined", // enable debugging, e.g. gdb
// "--device=/dev/dri", // enable Intel integrated graphics
// "--ulimit", "nofile=1024:4096", // increase file descriptor limit for valgrind
//
"--runtime=nvidia", // enable NVIDIA Container Toolkit
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if no NV GPU exists?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then the user should comment out this device option and use the appropriate command for their local hardware, like --device=/dev/dri for Intel integrated graphics. Nvidia is just enabled by default as it's so common in robotics and AI development on linux (my own bias). We could leave all hardware acceleration options commented out by default instead, just a minor inconveniences to me.

"--env=NVIDIA_VISIBLE_DEVICES=all", // enable GPUs with env as --gpus doesn't parse nicely
"--env=NVIDIA_DRIVER_CAPABILITIES=all", // enable all capabilities, including `graphics`
],
"workspaceFolder": "/opt/overlay_ws/src/navigation2",
"workspaceFolder": "/opt/nav2_ws/src/navigation2",
"workspaceMount": "source=${localWorkspaceFolder},target=${containerWorkspaceFolder},type=bind",
"onCreateCommand": ".devcontainer/on-create-command.sh",
"updateContentCommand": ".devcontainer/update-content-command.sh",
"postCreateCommand": ".devcontainer/post-create-command.sh",
"remoteEnv": {
"OVERLAY_MIXINS": "release ccache lld",
"CCACHE_DIR": "/tmp/.ccache"
"CCACHE_DIR": "/opt/nav2_ws/.ccache",
// Explicitly set DISPLAY for NVIDIA Container Toolkit
"DISPLAY": "${localEnv:DISPLAY}",
},
"remoteUser": "ubuntu",
"mounts": [
{
"source": "ccache-${devcontainerId}",
"target": "/tmp/.ccache",
// Explicitly mount X11 socket for NVIDIA Container Toolkit
// as setting NVIDIA_DRIVER_CAPABILITIES to include `graphics`
// interferes with VSCode's default X11 forwarding behavior
"source": "/tmp/.X11-unix",
"target": "/tmp/.X11-unix",
"type": "bind"
},
{
// Cache apt downloads
"source": "apt-cache",
"target": "/var/cache/apt",
"type": "volume"
},
{
// Cache ccache caches
"source": "ccache",
"target": "/opt/nav2_ws/.ccache",
"type": "volume"
},
{
// Cache colcon workspace
"source": "overlay-${devcontainerId}",
"target": "/opt/overlay_ws",
"target": "/opt/nav2_ws",
"type": "volume"
}
],
"features": {
// "ghcr.io/devcontainers/features/desktop-lite:1": {},
"ghcr.io/devcontainers/features/github-cli:1": {}
// "ghcr.io/devcontainers/features/docker-outside-of-docker:1": {},
// "ghcr.io/devcontainers/features/github-cli:1": {}
},
"customizations": {
"codespaces": {
@@ -53,6 +74,8 @@
"eamodio.gitlens",
"esbenp.prettier-vscode",
"GitHub.copilot",
"hashicorp.hcl",
// "ms-azuretools.vscode-docker",
"ms-iot.vscode-ros",
"streetsidesoftware.code-spell-checker",
"twxs.cmake"
17 changes: 17 additions & 0 deletions .devcontainer/initialize-command.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/bin/bash

# Immediately catch all errors
set -eo pipefail

# Uncomment for debugging
# set -x
# env

# Use first argument as target name
target=$1

# Bake the target and export locally to static tag
docker buildx bake --load \
--file docker-bake.hcl \
--set $target.tags=nav2:devcontainer \
$target
22 changes: 21 additions & 1 deletion .devcontainer/on-create-command.sh
Original file line number Diff line number Diff line change
@@ -7,6 +7,26 @@ set -eo pipefail
# set -x
# env

# Set git config for submodules
git config --local include.path ../.gitconfig

# Set git config for colcon cache
git config --global --add safe.directory "*"

.devcontainer/update-content-command.sh
# NOTE: This is slow if not using a mounted volumes,
# i.e. using workspace from the docker image directly,
# presumably due to docker overlayfs driver overhead.
# If needing to use workspace pre-baked into base image,
# consider using a new volume to be auto populated with
# the workspace pre-baked in image via devcontainer tools.
# Either by deleting old volume from the docker engine
# Or simpler changing volume name in devcontainer.json
sudo chown -R :ubuntu $OVERLAY_WS
# Recursively update group permissions for workspace
# to allow write access via dev users current group
sudo chmod -R g+rwx $OVERLAY_WS

# Recursively update group permissions for ros home
# to allow write access such as ros node logs
sudo chown -R :ubuntu /opt/.ros
sudo chmod -R g+rwx /opt/.ros
11 changes: 5 additions & 6 deletions .devcontainer/post-create-command.sh
Original file line number Diff line number Diff line change
@@ -10,9 +10,8 @@ set -eo pipefail
# Enable autocomplete for user
cp /etc/skel/.bashrc ~/

# Check if srv folder exists
if [ -d "$ROOT_SRV" ]; then
# Setup Nav2 web app
for dir in $OVERLAY_WS/src/navigation2/.devcontainer/caddy/srv/*; \
do if [ -d "$dir" ]; then ln -s "$dir" $ROOT_SRV; fi done
fi
# Enable autocomplete using colcon
echo "source /usr/share/colcon_argcomplete/hook/colcon-argcomplete.bash" >> ~/.bashrc

# Add aliases to .bashrc
echo "alias sows='source $OVERLAY_WS/install/setup.bash'" >> ~/.bashrc
6 changes: 4 additions & 2 deletions .devcontainer/update-content-command.sh
Original file line number Diff line number Diff line change
@@ -49,11 +49,13 @@ then
fi
echo BUILD_PACKAGES: $BUILD_PACKAGES

# DEBUG: Uncoment for more sterile but slower builds
# colcon clean packages --yes \
# --packages-select ${BUILD_PACKAGES} \
# --base-select install
# --base-select build install

. $UNDERLAY_WS/install/setup.sh
# OPTOINAL: Uncomment to build packages upon update
. /opt/ros/$ROS_DISTRO/setup.sh
colcon build \
--symlink-install \
--mixin $OVERLAY_MIXINS \