set tracing socket path to runtime dir #4078

crazy-max · 2023-07-29T01:04:33Z

cmd/buildkitd/main_unix.go

AkihiroSuda · 2023-07-31T14:28:25Z

cmd/buildkitd/main_unix.go

@@ -47,8 +48,14 @@ func listenFD(addr string, tlsConfig *tls.Config) (net.Listener, error) {
 	return nil, errors.New("not supported yet")
 }

-func traceSocketPath(root string) string {
-	return filepath.Join(root, "otel-grpc.sock")
+func traceSocketPath(inUserNS bool) string {


This should be moved to https://github.com/moby/buildkit/blob/master/util/appdefaults/appdefaults_unix.go

crazy-max · 2023-07-31T15:03:04Z

In rootless mode in our integration tests that sounds legit:

2023/07/31 13:09:36 > startCmd 2023-07-31 13:09:36.150436901 +0000 UTC m=+0.639178596 /usr/bin/sudo -u #1000 -i -- exec nsenter -U --preserve-credentials -m -t 15717 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd3936068494/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --containerd-worker-snapshotter=native --containerd-worker-snapshotter=native --config=/tmp/bktest_config1926999853/buildkitd.toml --root /tmp/bktest_buildkitd2346048497 --addr unix:///tmp/bktest_buildkitd2346048497/buildkitd.sock --debug
2023/07/31 13:09:36 buildkitd: listen unix /run/user/1000/buildkit/otel-grpc.sock: bind: address already in use

https://github.com/moby/buildkit/actions/runs/5714997638/job/15483511608#step:7:1258

    sandbox.go:286: time="2023-07-31T13:15:57Z" level=error msg="/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = process \"/bin/sh -c mkdir -p /usr/bin && echo -n foo > /usr/bin/go\" did not complete successfully: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting \"/run/buildkit/otel-grpc.sock\" to rootfs at \"/dev/otel-grpc.sock\": possibly malicious path detected -- refusing to operate on /tmp/bktest_containerd2293416339/state/io.containerd.runtime.v2.task/buildkit/yjdfcd29mfseoazkv7uopr7ne/rootfs/dev/otel-grpc.sock (deleted): unknown"
    sandbox.go:286: process "/bin/sh -c mkdir -p /usr/bin && echo -n foo > /usr/bin/go" did not complete successfully: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/run/buildkit/otel-grpc.sock" to rootfs at "/dev/otel-grpc.sock": possibly malicious path detected -- refusing to operate on /tmp/bktest_containerd2293416339/state/io.containerd.runtime.v2.task/buildkit/yjdfcd29mfseoazkv7uopr7ne/rootfs/dev/otel-grpc.sock (deleted): unknown
    sandbox.go:286: 35741 7fd9470 buildkitd --oci-worker=false --containerd-worker-gc=false --containerd-worker=true --containerd-worker-addr /tmp/bktest_containerd2293416339/containerd.sock --containerd-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config3965535412/buildkitd.toml --root /tmp/bktest_buildkitd1187620594 --addr unix:///tmp/bktest_buildkitd1187620594/buildkitd.sock --debug

https://github.com/moby/buildkit/actions/runs/5714997636/job/15483510501#step:7:2316

Added BUILDKIT_TRACE_SOCKET env var so we can set the trace socket path to sandbox temp dir. Let me know if that sounds good.

Signed-off-by: CrazyMax <[email protected]>

crazy-max · 2023-07-31T17:24:02Z

Hum it still tries to write to xdg runtime dir in rootless mode for frontend integration tests, I might miss smth: https://github.com/moby/buildkit/actions/runs/5716370920/job/15487868441?pr=4078#step:7:1143

=== CONT  TestIntegration/TestOnBuildHeredoc/worker=oci-rootless/frontend=builtin
2023/07/31 15:15:27 stdout: /usr/bin/sudo -u #1000 -i -- exec rootlesskit buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config2668879534/buildkitd.toml --root /tmp/bktest_buildkitd3159267947 --addr unix:///tmp/bktest_buildkitd3159267947/buildkitd.sock --debug
2023/07/31 15:15:27 stderr: /usr/bin/sudo -u #1000 -i -- exec rootlesskit buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config2668879534/buildkitd.toml --root /tmp/bktest_buildkitd3159267947 --addr unix:///tmp/bktest_buildkitd3159267947/buildkitd.sock --debug
2023/07/31 15:15:27 > startCmd 2023-07-31 15:15:27.049972746 +0000 UTC m=+17.524187589 /usr/bin/sudo -u #1000 -i -- exec rootlesskit buildkitd --oci-worker=true --containerd-worker=false --oci-worker-gc=false --oci-worker-labels=org.mobyproject.buildkit.worker.sandbox=true --config=/tmp/bktest_config2668879534/buildkitd.toml --root /tmp/bktest_buildkitd3159267947 --addr unix:///tmp/bktest_buildkitd3159267947/buildkitd.sock --debug
2023/07/31 15:15:27 buildkitd: listen unix /run/user/1000/buildkit/otel-grpc.sock: bind: address already in use

Edit: I think we are losing env context with sudo, I will take a look.

Signed-off-by: CrazyMax <[email protected]>

tonistiigi

I'm not sure if this is completely safe to pick to v0.12 . Maybe there are some weird perm errors for /run/buildkit in some systems.

How about we merge this for master and pick https://github.com/moby/buildkit/pull/3483/files for v0.12?

crazy-max · 2023-08-02T10:05:52Z

How about we merge this for master and pick https://github.com/moby/buildkit/pull/3483/files for v0.12?

Fine to me

jedevc · 2023-08-03T11:27:50Z

util/testutil/integration/oci.go

@@ -65,7 +65,7 @@ func (s *OCI) New(ctx context.Context, cfg *BackendConfig) (Backend, func() erro
 			return nil, nil, errors.Errorf("unsupported id pair: uid=%d, gid=%d", s.UID, s.GID)
 		}
 		// TODO: make sure the user exists and subuid/subgid are configured.
-		buildkitdArgs = append([]string{"sudo", "-u", fmt.Sprintf("#%d", s.UID), "-i", "--", "exec", "rootlesskit"}, buildkitdArgs...)
+		buildkitdArgs = append([]string{"sudo", "-E", "-u", fmt.Sprintf("#%d", s.UID), "-i", "--", "exec", "rootlesskit"}, buildkitdArgs...)


@crazy-max looks like this actually disables all the rootless tests 👀

See https://github.com/moby/buildkit/actions/runs/5749062460/job/15583230313:

time="2023-08-03T09:40:23Z" level=warning msg="rootless mode is not supported on this host: exit status 1 (sudo: you may not specify both the -i and -E options\nusage: sudo -h | -K | -k | -V\nusage: sudo -v [-ABkNnS] [-g group] [-h host] [-p prompt] [-u user]\nusage: sudo -l [-ABkNnS] [-g group] [-h host] [-p prompt] [-U user] [-u user]\n [command [arg ...]]\nusage: sudo [-ABbEHkNnPS] [-C num] [-D directory] [-g group] [-h host] [-p\n prompt] [-R directory] [-T timeout] [-u user] [VAR=value] [-i | -s]\n [command [arg ...]]\nusage: sudo -e [-ABkNnS] [-C num] [-D directory] [-g group] [-h host] [-p\n prompt] [-R directory] [-T timeout] [-u user] file ...\n)"

yikes test step didn't failed 😕

Opened an issue

Rootless tests are not running since 2023-08-02 (sudo: you may not specify both the -i and -E options) #4100

jedevc · 2023-08-03T11:37:10Z

Removing from the v0.12.1 milestone, see #4078 (comment).

AkihiroSuda · 2024-02-03T07:45:53Z

Regression

[v0.13.0-beta1 regression] /run/buildkit is inaccessible for socket users #4618

AkihiroSuda reviewed Jul 30, 2023

View reviewed changes

cmd/buildkitd/main_unix.go Outdated Show resolved Hide resolved

AkihiroSuda reviewed Jul 30, 2023

View reviewed changes

cmd/buildkitd/main_unix.go Outdated Show resolved Hide resolved

tonistiigi added this to the v0.12.1 milestone Jul 31, 2023

crazy-max force-pushed the fix-otel-sock branch from 7af2b3e to 27688e6 Compare July 31, 2023 12:47

crazy-max changed the title ~~fallback to runtime dir if tracing socket path too long~~ set tracing socket path to runtime dir Jul 31, 2023

crazy-max force-pushed the fix-otel-sock branch from 27688e6 to 32b3bcb Compare July 31, 2023 12:54

crazy-max requested review from AkihiroSuda and tonistiigi July 31, 2023 12:55

crazy-max marked this pull request as ready for review July 31, 2023 12:55

crazy-max added the needs-cherry-pick/v0.12 label Jul 31, 2023

crazy-max force-pushed the fix-otel-sock branch from 32b3bcb to 0e83310 Compare July 31, 2023 13:00

AkihiroSuda reviewed Jul 31, 2023

View reviewed changes

crazy-max force-pushed the fix-otel-sock branch from 0e83310 to 44dc4d7 Compare July 31, 2023 14:57

AkihiroSuda approved these changes Jul 31, 2023

View reviewed changes

set tracing socket path to runtime dir

a7690c5

Signed-off-by: CrazyMax <[email protected]>

crazy-max force-pushed the fix-otel-sock branch from 44dc4d7 to a7690c5 Compare July 31, 2023 15:07

integration: preserve existing environment variables when using sudo

b57eeb2

Signed-off-by: CrazyMax <[email protected]>

tonistiigi approved these changes Aug 2, 2023

View reviewed changes

tonistiigi mentioned this pull request Aug 2, 2023

[v0.12.1] cherry-picks #4088

Merged

tonistiigi merged commit c7318df into moby:master Aug 2, 2023

crazy-max deleted the fix-otel-sock branch August 2, 2023 10:07

jedevc reviewed Aug 3, 2023

View reviewed changes

jedevc removed the needs-cherry-pick/v0.12 label Aug 3, 2023

jedevc removed this from the v0.12.1 milestone Aug 3, 2023

AkihiroSuda mentioned this pull request Aug 3, 2023

Rootless tests are not running since 2023-08-02 (sudo: you may not specify both the -i and -E options) #4100

Closed

crazy-max mentioned this pull request Aug 3, 2023

integration: fix rootless tests #4101

Merged

AkihiroSuda mentioned this pull request Feb 3, 2024

[v0.13.0-beta1 regression] /run/buildkit is inaccessible for socket users #4618

Closed

georgethebeatle mentioned this pull request Mar 15, 2024

[v0.13] OTEL socket mount requires new volumes when containerd is run in separate container #4764

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set tracing socket path to runtime dir #4078

set tracing socket path to runtime dir #4078

crazy-max commented Jul 29, 2023

AkihiroSuda Jul 31, 2023

crazy-max commented Jul 31, 2023

crazy-max commented Jul 31, 2023 •

edited

Loading

tonistiigi left a comment

crazy-max commented Aug 2, 2023

jedevc Aug 3, 2023

crazy-max Aug 3, 2023

AkihiroSuda Aug 3, 2023

jedevc commented Aug 3, 2023

AkihiroSuda commented Feb 3, 2024

set tracing socket path to runtime dir #4078

set tracing socket path to runtime dir #4078

Conversation

crazy-max commented Jul 29, 2023

AkihiroSuda Jul 31, 2023

Choose a reason for hiding this comment

crazy-max commented Jul 31, 2023

crazy-max commented Jul 31, 2023 • edited Loading

tonistiigi left a comment

Choose a reason for hiding this comment

crazy-max commented Aug 2, 2023

jedevc Aug 3, 2023

Choose a reason for hiding this comment

crazy-max Aug 3, 2023

Choose a reason for hiding this comment

AkihiroSuda Aug 3, 2023

Choose a reason for hiding this comment

jedevc commented Aug 3, 2023

AkihiroSuda commented Feb 3, 2024

crazy-max commented Jul 31, 2023 •

edited

Loading