Use jenkins user inside docker jobs instead of root #494

scpeters · 2021-08-07T08:45:11Z

I have noticed some times when Jenkins fails to clear a workspace for a job. After investigating the machines over ssh, I have noticed the remaining folders contain files owned by root. This is caused by our use of the root user in the docker-based Ubuntu CI jobs. To avoid using the root user, this pull request creates a user in the Dockerfile that matches the current user, and switch to this user before using commands that touch the workspace folders.

I have tested with the following job:

https://build.osrfoundation.org/job/sdformat-ci-main-bionic-i386/14/

In order for this job to pass, I had to manually call sudo chown -R jenkins:jenkins on both the build machine's workspace folder and on the shared /srv/ccache folder. To merge this; I recommend taking all the Linux nodes of build.osrfoundation.org offline, then sanitizing the workspace and /srv/ccache folders of each Linux node, then bringing them back online.

edit
commands to run on migration:

sudo find /var/lib/jenkins/workspace -user root -exec chown jenkins {} \;
# chown symbolic links too
sudo find /var/lib/jenkins/workspace -user root -exec chown -h jenkins {} \;
sudo chown jenkins -R /srv/ccache

Create a user in the Dockerfile that matches the current user, and switch to this user before using commands that touch the workspace folders. Signed-off-by: Steve Peters <[email protected]>

j-rivero · 2021-08-09T20:41:27Z

I think that the PR is a good step forward to reduce the use of root in the CI pipeline. It is probably the easiest one (more complicate would probably be the execution of docker without sudo) but this one is important. Thank Steve.

Code looks good to me. I would like to check a couple of things more: GPU build with this branch and debbuilder with this branch. My goal is to execute the merge and do the manual operations in the next two days.

scpeters · 2021-08-09T21:15:27Z

I think that the PR is a good step forward to reduce the use of root in the CI pipeline. It is probably the easiest one (more complicate would probably be the execution of docker without sudo) but this one is important. Thank Steve.

Code looks good to me. I would like to check a couple of things more: GPU build with this branch and debbuilder with this branch. My goal is to execute the merge and do the manual operations in the next two days.

sounds good. It might be useful to schedule a meeting with you, me and the current build farmer @mjcarroll to coordinate the merge and manual operations

j-rivero · 2021-08-09T21:28:23Z

I made some changes to this PR:

Remove workarounds in place inside Jenkins DSL 61f11d6
Remove workarounds inside the docker_run script
Add some missing entries for sudo when using make install b864fef

j-rivero · 2021-08-09T21:32:24Z

sounds good. It might be useful to schedule a meeting with you, me and the current build farmer @mjcarroll to coordinate the merge and manual operations

I was thinking on performing all the operations during Wednesday Europe morning if we have a clear plan with the steps ready. I'm happy to take any help.

scpeters · 2021-08-10T00:11:26Z

have you run any test builds with these latest changes?

…ith root" This reverts commit 61f11d6.

Signed-off-by: Jose Luis Rivero <[email protected]>

j-rivero · 2021-08-11T14:13:52Z

Made some modifications to this PR (sorry I should have used a different branch) and run a battery of tests:

debbuilder job:
ci job with GPU:
abichecker job:
install job:
ros2 ci job:
gazebo job:

I think we are ready to re-review @scpeters

scpeters · 2021-08-11T16:36:43Z

jenkins-scripts/docker/lib/_install_nvidia_docker.sh

 echo '# END SECTION'

 echo '# BEGIN SECTION: install nvidia-docker1 (in docker)'
-apt-get install -y wget nvidia-340 nvidia-modprobe
+sudo apt-get install -y wget nvidia-340 nvidia-modprobe
 wget -P /tmp https://github.com/NVIDIA/nvidia-docker/releases/download/v1.0.1/nvidia-docker_1.0.1-1_amd64.deb
 dpkg -i /tmp/nvidia-docker*.deb && rm /tmp/nvidia-docker*.deb


I expect this line and the apt-key and add-apt-repository commands may need sudo as well

you are right 079a093

scpeters · 2021-08-11T16:39:14Z

thanks for the additional testing. I think we can merge soon and catch further issues when they arise

j-rivero · 2021-08-11T17:31:35Z

thanks for the additional testing. I think we can merge soon and catch further issues when they arise

+1. I'll plan the migration via slack.

tfoote · 2021-08-12T20:24:51Z

jenkins-scripts/docker/lib/docker_generate_dockerfile.bash

+# Create a user with passwordless sudo
+ARG USERID
+ARG USER
+RUN adduser --uid \$USERID --gecos "Developer" --disabled-password \$USER


I would recommend a slightly more complete arguments to the user. In particular set the group ID as well.

I have a good example at: https://github.com/osrf/rocker/blob/acbbff59d6aef6a9654eabc0ff7e3f823936d6e3/src/rocker/templates/user_snippet.Dockerfile.em#L9

Thanks! I've included GID handlng in ef59662

j-rivero · 2021-08-13T10:32:17Z

Another short round of tests:

ign-rendering
ign-msgs8-debbuilder:

j-rivero · 2021-08-13T11:42:01Z

Changed machines:

drogon
maria
optimus
r2d2

Merging-

Run docker jobs without root

94b28c3

Create a user in the Dockerfile that matches the current user, and switch to this user before using commands that touch the workspace folders. Signed-off-by: Steve Peters <[email protected]>

scpeters requested a review from j-rivero August 7, 2021 08:45

j-rivero changed the title ~~Run docker jobs without root~~ Use jenkins user inside docker jobs instead of root Aug 9, 2021

j-rivero added 3 commits August 9, 2021 23:24

Remove workaround to setup jenkins permissions after docker with root

61f11d6

Remove workarounds to restore jenkins permissions

05f01ca

Use sudo for make install

b864fef

j-rivero added 3 commits August 10, 2021 21:24

Revert "Remove workaround to setup jenkins permissions after docker w…

eb169da

…ith root" This reverts commit 61f11d6.

Only remove chown on repo checkouts

ee174fb

Signed-off-by: Jose Luis Rivero <[email protected]>

Added sudo to BASE_DEPENDENCIES

ec40caa

j-rivero force-pushed the scpeters/docker_rootless branch from 12517b6 to ec40caa Compare August 10, 2021 20:09

chapulina mentioned this pull request Aug 10, 2021

Use statically-typed views for better performance gazebosim/gz-sim#856

Merged

8 tasks

j-rivero added 6 commits August 11, 2021 13:46

debbuilder needs sudo por apt and system operations

2e9fceb

Use sudo for system operations

4bea0b1

Missing sudo commands

5928bda

Use sudo for modifying /usr/bin

5ab6a9e

Signed-off-by: Jose Luis Rivero <[email protected]>

Use sudo for installing abi-checker

ad34a76

Signed-off-by: Jose Luis Rivero <[email protected]>

Do not run rosdep as root

b456b6c

Port changes to catkin_make script

1544888

scpeters commented Aug 11, 2021

View reviewed changes

More sudo commands for nvidia installation

079a093

j-rivero closed this Aug 11, 2021

j-rivero reopened this Aug 11, 2021

tfoote reviewed Aug 12, 2021

View reviewed changes

Add GID to docker user creation

ef59662

j-rivero merged commit 6727911 into master Aug 13, 2021

j-rivero deleted the scpeters/docker_rootless branch August 13, 2021 11:42

j-rivero mentioned this pull request Aug 13, 2021

Set USER var from PPID if USER is not present #498

Merged

chapulina mentioned this pull request Aug 13, 2021

Upgrade ign-sensors and support custom sensors gazebosim/gz-sim#617

Merged

8 tasks

scpeters mentioned this pull request Aug 14, 2021

docker script: chown home folder #499

Merged

j-rivero mentioned this pull request Aug 16, 2021

Right frontend debian #500

Merged

j-rivero mentioned this pull request Aug 25, 2021

Fix ratt script after removing root #502

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use jenkins user inside docker jobs instead of root #494

Use jenkins user inside docker jobs instead of root #494

scpeters commented Aug 7, 2021 •

edited by j-rivero

Loading

j-rivero commented Aug 9, 2021

scpeters commented Aug 9, 2021

j-rivero commented Aug 9, 2021

j-rivero commented Aug 9, 2021

scpeters commented Aug 10, 2021

j-rivero commented Aug 11, 2021

scpeters Aug 11, 2021

j-rivero Aug 11, 2021

scpeters commented Aug 11, 2021

j-rivero commented Aug 11, 2021 •

edited

Loading

tfoote Aug 12, 2021

j-rivero Aug 12, 2021

j-rivero commented Aug 13, 2021

j-rivero commented Aug 13, 2021

Use jenkins user inside docker jobs instead of root #494

Use jenkins user inside docker jobs instead of root #494

Conversation

scpeters commented Aug 7, 2021 • edited by j-rivero Loading

j-rivero commented Aug 9, 2021

scpeters commented Aug 9, 2021

j-rivero commented Aug 9, 2021

j-rivero commented Aug 9, 2021

scpeters commented Aug 10, 2021

j-rivero commented Aug 11, 2021

scpeters Aug 11, 2021

Choose a reason for hiding this comment

j-rivero Aug 11, 2021

Choose a reason for hiding this comment

scpeters commented Aug 11, 2021

j-rivero commented Aug 11, 2021 • edited Loading

tfoote Aug 12, 2021

Choose a reason for hiding this comment

j-rivero Aug 12, 2021

Choose a reason for hiding this comment

j-rivero commented Aug 13, 2021

j-rivero commented Aug 13, 2021

scpeters commented Aug 7, 2021 •

edited by j-rivero

Loading

j-rivero commented Aug 11, 2021 •

edited

Loading