Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go tools now resolve the real path of the input file #1943

Closed
steeve opened this issue Feb 8, 2019 · 25 comments
Closed

go tools now resolve the real path of the input file #1943

steeve opened this issue Feb 8, 2019 · 25 comments

Comments

@steeve
Copy link
Contributor

steeve commented Feb 8, 2019

It seems go install and go tool compile now resolve the real path of the input file, causing sandbox and real paths to leak into the binary.

See this dwarfdump sample:

                 AT_decl_file( "/Users/steeve/go/src/github.com/steeve/test/main.go" )
                 AT_decl_file( "/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo/gcc_android.c" )
                 AT_decl_file( "/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo/libcgo.h" )

Notice the first file is not in the sandbox, while the other is in the external bazel folder, not in the sandbox, which was:

/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__

I'm not sure how to fix that properly....

@steeve
Copy link
Contributor Author

steeve commented Feb 8, 2019

Looking at whole dwarfdump, it seems only .s and .c files are impacted for stdlib.

Which is weird since we pass trimpath to go tool asm

@steeve
Copy link
Contributor Author

steeve commented Feb 8, 2019

After doing some bash-fu, I could narrow down the following packages for stdlib that seem impacted:

internal/bytealg
internal/cpu
reflect
runtime
runtime/cgo
runtime/internal/atomic
runtime/internal/sys
sync/atomic
syscall

They all have assembly files, which I'm guessing is the reason.

@steeve
Copy link
Contributor Author

steeve commented Feb 8, 2019

Ok so the reason is that the trimpath is:

-asmflags=all=-trimpath=/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__

Which is not the path that ends up in the dwarf.

@steeve
Copy link
Contributor Author

steeve commented Feb 8, 2019

Alright so, for stdlib asmflags, it seems we need to strip the output path instead of the sandbox path.
Now only .c and .h files remain.

@steeve
Copy link
Contributor Author

steeve commented Feb 8, 2019

I managed to fix runtime/cgo by adding the output path to the -fdebug-prefix-map in CGO_CFLAGS.

Now only the main.go path leaks in the binary.

@steeve
Copy link
Contributor Author

steeve commented Feb 8, 2019

I've opened a PR for the stdlib part, however, for the file part, I'm not convinced this is the right approach, even though it works on macOS:

diff --git a/go/tools/builders/compile.go b/go/tools/builders/compile.go
index 2b6d050..d0003bf 100644
--- a/go/tools/builders/compile.go
+++ b/go/tools/builders/compile.go
@@ -38,6 +38,23 @@ type archive struct {
 	importPath, importMap, file string
 }
 
+func findExecrootPath(sandboxPath string) string {
+	realPath, _ := filepath.EvalSymlinks(sandboxPath)
+	realPath = abs(realPath)
+	if realPath == sandboxPath {
+		return sandboxPath
+	}
+
+	sandboxPathComponents := strings.Split(sandboxPath, string(filepath.Separator))
+	realPathComponents := strings.Split(realPath, string(filepath.Separator))
+	for i, part := range sandboxPathComponents {
+		if part != realPathComponents[i] {
+			return "/" + filepath.Join(realPathComponents[:i]...)
+		}
+	}
+	return sandboxPath
+}
+
 func run(args []string) error {
 	// Parse arguments.
 	args, err := readParamsFiles(args)
@@ -122,6 +139,8 @@ func run(args []string) error {
 		*packagePath = goFiles[0].pkg
 	}
 
+	execRootPath := findExecrootPath(goFiles[0].filename)
+
 	// Check that the filtered sources don't import anything outside of
 	// the standard library and the direct dependencies.
 	_, stdImports, err := checkDirectDeps(goFiles, archives, *packageList)
@@ -163,7 +182,15 @@ func run(args []string) error {
 		filenames = append(filenames, f.filename)
 	}
 	goargs = append(goargs, filenames...)
-	absArgs(goargs, []string{"-I", "-o", "-trimpath", "-importcfg"})
+	absArgs(goargs, []string{"-I", "-o", "-importcfg"})
+
+	for i := 0; i < len(goargs); i++ {
+		if goargs[i] == "-trimpath" {
+			goargs[i+1] = execRootPath
+			i++
+		}
+	}
+
 	cmd := exec.Command(goargs[0], goargs[1:]...)
 	cmd.Stdout = os.Stdout
 	cmd.Stderr = os.Stderr

@jayconrod
Copy link
Contributor

Thanks for digging into this issue and sending a fix.

Before I merge #1945 (which looks fine by the way), I'd like to understand exactly what's going wrong and why our tests didn't catch this.

I haven't actually been able to reproduce the problem though. What is your execution platform (you mentioned macOS). Target platform? Any other configuration changes (pure = "off" I'm guessing)? How are you invoking Bazel? Anything else out of the ordinary?

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2019

Actually, this goes further than that, as CC complete path leaks in the resulting binary, also via --sysroot as somehow .a files keep the whole flagset:

$ strings bazel-out/x86-dbg/bin//external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/pkg/android_386/runtime/cgo.a  | grep steeve
[["cgo_export_static","crosscall2"],["cgo_export_dynamic","crosscall2"],["cgo_export_static","_cgo_panic"],["cgo_export_dynamic","_cgo_panic"],["cgo_import_static","x_cgo_init"],["cgo_import_static","x_cgo_thread_start"],["cgo_import_static","x_cgo_sys_thread_create"],["cgo_import_static","x_cgo_notify_runtime_init_done"],["cgo_import_static","x_cgo_set_context_function"],["cgo_import_static","_cgo_yield"],["cgo_export_static","_cgo_topofstack"],["cgo_export_dynamic","_cgo_topofstack"],["cgo_import_static","x_cgo_callers"],["cgo_import_static","x_cgo_setenv"],["cgo_import_static","x_cgo_unsetenv"],["cgo_ldflag","-shared"],["cgo_ldflag","-gcc-toolchain"],["cgo_ldflag","/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/external/androidndk/ndk/toolchains/x86-4.9/prebuilt/darwin-x86_64"],["cgo_ldflag","-target"],["cgo_ldflag","i686-none-linux-android"],["cgo_ldflag","-no-canonical-prefixes"],["cgo_ldflag","-L/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/external/androidndk/ndk/sources/cxx-stl/llvm-libc++/libs/x86"],["cgo_ldflag","-static-libgcc"],["cgo_ldflag","--sysroot=/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/external/androidndk/ndk/platforms/android-21/arch-x86"],["cgo_ldflag","-llog"],["cgo_import_dynamic","__cxa_finalize","__cxa_finalize#LIBC","libc.so"],["cgo_import_dynamic","__cxa_atexit","__cxa_atexit#LIBC","libc.so"],["cgo_import_dynamic","__stack_chk_fail","__stack_chk_fail#LIBC","libc.so"],["cgo_import_dynamic","__android_log_vprint","__android_log_vprint",""],["cgo_import_dynamic","__sF","__sF#LIBC","libc.so"],["cgo_import_dynamic","abort","abort#LIBC","libc.so"],["cgo_import_dynamic","fprintf","fprintf#LIBC","libc.so"],["cgo_import_dynamic","vfprintf","vfprintf#LIBC","libc.so"],["cgo_import_dynamic","__stack_chk_guard","__stack_chk_guard#LIBC","libc.so"],["cgo_import_dynamic","free","free#LIBC","libc.so"],["cgo_import_dynamic","pthread_key_create","pthread_key_create#LIBC","libc.so"],["cgo_import_dynamic","pthread_key_delete","pthread_key_delete#LIBC","libc.so"],["cgo_import_dynamic","pthread_setspecific","pthread_setspecific#LIBC","libc.so"],["cgo_import_dynamic","nanosleep","nanosleep#LIBC","libc.so"],["cgo_import_dynamic","pthread_cond_broadcast","pthread_cond_broadcast#LIBC","libc.so"],["cgo_import_dynamic","pthread_cond_wait","pthread_cond_wait#LIBC","libc.so"],["cgo_import_dynamic","pthread_create","pthread_create#LIBC","libc.so"],["cgo_import_dynamic","pthread_detach","pthread_detach#LIBC","libc.so"],["cgo_import_dynamic","pthread_mutex_lock","pthread_mutex_lock#LIBC","libc.so"],["cgo_import_dynamic","pthread_mutex_unlock","pthread_mutex_unlock#LIBC","libc.so"],["cgo_import_dynamic","strerror","strerror#LIBC","libc.so"],["cgo_import_dynamic","memset","memset#LIBC","libc.so"],["cgo_import_dynamic","pthread_attr_destroy","pthread_attr_destroy#LIBC","libc.so"],["cgo_import_dynamic","pthread_attr_getstacksize","pthread_attr_getstacksize#LIBC","libc.so"],["cgo_import_dynamic","pthread_attr_init","pthread_attr_init#LIBC","libc.so"],["cgo_import_dynamic","pthread_sigmask","pthread_sigmask#LIBC","libc.so"],["cgo_import_dynamic","sigfillset","sigfillset#LIBC","libc.so"],["cgo_import_dynamic","setenv","setenv#LIBC","libc.so"],["cgo_import_dynamic","unsetenv","unsetenv#LIBC","libc.so"],["cgo_import_dynamic","malloc","malloc#LIBC","libc.so"],["cgo_import_dynamic","_","_","liblog.so"],["cgo_import_dynamic","_","_","libc.so"]]
/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo

@jayconrod
Copy link
Contributor

When I tried this earlier, I wasn't able to reproduce on darwin or linux amd64. From the messages here, it looks like you're building for android_386. Is the leak in the .o files produced by C compiler (packed into the .a file), or are they in the Go compiled files? We may be able to do some path adjustment on CGO_CFLAGS, but again, I want to understand what the problem is before proceeding.

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2019

I don't know how to fix the problem with cgo_ldflag leaking in the binary. I tried setting GOTMPDIR to the sandbox and give relative paths (CC, --sysroot etc..) to the sandbox, but go build cd's into each package before building them.

So until rules_go builds each package instead of using go build, I think we can't fix it :(

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2019

See the sample run:

WORK=/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/go-build170197096
mkdir -p $WORK/b001/
cd /private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo
CGO_LDFLAGS='"-shared" "-gcc-toolchain" "../../../../../../../../../../external/androidndk/ndk/toolchains/x86-4.9/prebuilt/darwin-x86_64" "-target" "i686-none-linux-android" "-no-canonical-prefixes" "-L../../../../../../../../../../external/androidndk/ndk/sources/cxx-stl/llvm-libc++/libs/x86" "-static-libgcc" "--sysroot=../../../../../../../../../../external/androidndk/ndk/platforms/android-21/arch-x86" "-llog"' /private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/darwin_amd64_stripped/filter_buildid /private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/pkg/tool/darwin_amd64/cgo -objdir $WORK/b001/ -importpath runtime/cgo -import_runtime_cgo=false -import_syscall=false -- -I $WORK/b001/ -gcc-toolchain ../../../../../../../../../../external/androidndk/ndk/toolchains/x86-4.9/prebuilt/darwin-x86_64 -target i686-none-linux-android -ffunction-sections -funwind-tables -fstack-protector-strong -fPIC -Wno-invalid-command-line-argument -Wno-unused-command-line-argument -no-canonical-prefixes -isystem../../../../../../../../../../external/androidndk/ndk/sysroot/usr/include/i686-linux-android -D__ANDROID_API__=21 -mstackrealign -O0 -g --sysroot=../../../../../../../../../../external/androidndk/ndk/platforms/android-21/arch-x86 -isystem ../../../../../../../../../../external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include -isystem ../../../../../../../../../../external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include -isystem ../../../../../../../../../../external/androidndk/ndk/sources/android/support/include -isystem../../../../../../../../../../external/androidndk/ndk/sysroot/usr/include -fdebug-prefix-map=/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__=. -fdebug-prefix-map=/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%=. -Wall -Werror ./cgo.go```

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2019

I traced it to packages using cgo somehow:

runtime/cgo (of course)
plugin (uses libdl apparently)
net (for getaddrinfo)

Related issue for plugin package: golang/go#19569

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2019

Ok so I'm trying wrapped cc with a proxy that will absolutise arguments then. It works well, as the sanbox paths don't leak in the sandbox then, as seen with runtime/cgo, for instance:

["cgo_ldflag","-Lexternal/androidndk/ndk/sources/cxx-stl/llvm-libc++/libs/x86"],["cgo_ldflag","-static-libgcc"],["cgo_ldflag","--sysroot=external/androidndk/ndk/platforms/android-21/arch-x86"],

I'm passing the sandbox path to the wrapper with an environment var.

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2019

Now the only remaining path for stdlib is this one, and I think it has to do with -fdebug-prefix-map for CC assemblies.

0x0006a97e: TAG_compile_unit [1] *
             AT_stmt_list( 0x000217dd )
             AT_low_pc( 0x000a1708 )
             AT_high_pc( 0x000a1718 )
             AT_name( "gcc_386.S" )
             AT_comp_dir( "/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo" )

The weird thing is also:

$ dwarfdump bazel-bin/android_386_debug_c-shared/libmain.so| grep -B10 steeve
0x00069d06:     TAG_subprogram [2] *
                 AT_name( "main.main" )
                 AT_low_pc( 0x000a06f0 )
                 AT_high_pc( 0x000a0775 )
                 AT_frame_base( call-frame-cfa )
                 AT_decl_file( "/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/execroot/__main__/bazel-out/x86-dbg/bin/android_386_debug/main%android_386%cgo_codegen%/main.cgo1.go" )
                                    ^
                                    \------ not that the path is not in the build sandbox


0x0006a97e: TAG_compile_unit [1] *
             AT_stmt_list( 0x000217dd )
             AT_low_pc( 0x000a1708 )
             AT_high_pc( 0x000a1718 )
             AT_name( "gcc_386.S" )
             AT_comp_dir( "/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo" )

But then:

$ strings bazel-bin/android_386_debug_c-shared/libmain.so | grep main.cgo1.go
bazel-out/x86-dbg/bin/android_386_debug/main%android_386%cgo_codegen%/main.cgo1.go
bazel-out/x86-dbg/bin/android_386_debug/main%android_386%cgo_codegen%/main.cgo1.go

$ strings bazel-bin/android_386_debug_c-shared/libmain.so | grep steeve
/private/var/tmp/_bazel_steeve/309ac22c92b055a3e3f78825fe63406b/sandbox/darwin-sandbox/3/execroot/__main__/bazel-out/x86-dbg/bin/external/io_bazel_rules_go/android_386_debug_c-shared/stdlib%/src/runtime/cgo

@steeve
Copy link
Contributor Author

steeve commented Feb 12, 2019

So I've found the root cause of the path leaking for gcc_386.S: https://bugs.llvm.org/show_bug.cgi?id=38050

It's been fixed in r336793 but NDK18 is:
Android (4751641 based on r328903) clang version 7.0.2

@steeve
Copy link
Contributor Author

steeve commented Feb 12, 2019

NDK19 is

Android (5058415 based on r339409) clang version 8.0.2

So r336793 < r339409 but apparently this doesn't ship with the fix :(

$ ~/Library/Android/sdk/ndk-bundle/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang -target i686-none-linux-android -g -o out.o -c gcc_386.S -fdebug-prefix-map=/Users/steeve/tmp=.
$ dwarfdump gna.o
----------------------------------------------------------------------
 File: gna.o (i386)
----------------------------------------------------------------------
.debug_info contents:

0x00000000: Compile Unit: length = 0x00000148  version = 0x0004  abbr_offset = 0x00000000  addr_size = 0x04  (next CU at 0x0000014c)

0x0000000b: TAG_compile_unit [1] *
             AT_stmt_list( 0x00000000 )
             AT_low_pc( 0x00000000 )
             AT_high_pc( 0x00000010 )
             AT_name( "gcc_386.S" )
             AT_comp_dir( "." )
             AT_producer( "Android (5058415 based on r339409) clang version 8.0.2 (https://android.googlesource.com/toolchain/clang 40173bab62ec746213857d083c0e8b0abb568790) (https://android.googlesource.com/toolchain/llvm 7a6618d69e7e8111e1d49dc9e7813767c5ca756a) (based on LLVM 8.0.2svn)" )
             AT_language( Unknown DW_LANG constant: 0x8001 )

0x0000012d:     TAG_label [2] *
                 AT_name( "crosscall_386" )
                 AT_decl_file( "/Users/steeve/tmp/gcc_386.S" )
                 AT_decl_line( 6 )
                 AT_low_pc( 0x00000000 )
                 AT_prototyped( 0x00 )

0x00000149:         TAG_unspecified_parameters [3]

0x0000014a:         NULL

0x0000014b:     NULL

@steeve
Copy link
Contributor Author

steeve commented Feb 12, 2019

It also doesn't work with apple's clang:

$ clang -v
Apple LLVM version 10.0.0 (clang-1000.11.45.5)
Target: x86_64-apple-darwin18.2.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

$ clang -g -c gcc_amd64.S -o out64.o -fdebug-prefix-map=/Users/steeve/tmp=.
$ dwarfdump out64.o
----------------------------------------------------------------------
 File: out64.o (x86_64)
----------------------------------------------------------------------
.debug_info contents:

0x00000000: Compile Unit: length = 0x000000b9  version = 0x0004  abbr_offset = 0x00000000  addr_size = 0x08  (next CU at 0x000000bd)

0x0000000b: TAG_compile_unit [1] *
             AT_stmt_list( 0x00000000 )
             AT_low_pc( 0x0000000000000000 )
             AT_high_pc( 0x0000000000000017 )
             AT_name( "gcc_amd64.S" )
             AT_comp_dir( "/Users/steeve/tmp" )
             AT_producer( "Apple LLVM version 10.0.0 (clang-1000.11.45.5)" )
             AT_language( Unknown DW_LANG constant: 0x8001 )

0x00000098:     TAG_label [2] *
                 AT_name( "crosscall_amd64" )
                 AT_decl_file( "/Users/steeve/tmp/gcc_amd64.S" )
                 AT_decl_line( 6 )
                 AT_low_pc( 0x0000000000000000 )
                 AT_prototyped( 0x00 )

0x000000ba:         TAG_unspecified_parameters [3]

0x000000bb:         NULL

0x000000bc:     NULL

So appart from post-processing the file (which could be an option), I'm not sure what to do :(

@steeve
Copy link
Contributor Author

steeve commented Mar 6, 2019

One way I could do this would be to post process the file's DWARF string table and remove all mentions of the sandbox path.

@jayconrod
Copy link
Contributor

I'd prefer not to post-process files in GoStdLib. That could be a big source of bugs and complexity.

If I understand correctly, the bug is in the Android NDK toolchain. Is it possible to use a wrapper inside the C toolchain that invokes clang differently or post-processes the .o file at that point?

@steeve
Copy link
Contributor Author

steeve commented Mar 6, 2019

Indeed the bug is in all clang versions prior to r336793.
Android NDK18 has it, Apple Clang has it.

The thing is: it's kind of easy to post-process in Go thanks to debug/dwarf.

What I was thinking was wrapping cc with a custom Go binary what would do the postprocessing. I want to go that way to eradicate sandbox paths too, actually.

@jayconrod
Copy link
Contributor

That sounds fine. So that wrapped cc would be configured as the compiler CROSSTOOL, right? So even if you build a cc_binary without any Go, that would be fixed up.

@steeve
Copy link
Contributor Author

steeve commented Mar 6, 2019

I'm not sure configuring it as CROSSTOOL is handy, it requires modifying bazel itself to change the crosstool templates :(

@jayconrod
Copy link
Contributor

You may want to write your own CROSSTOOL and point to it with --crosstool_top / --host_crosstool_top in this situation instead of using the auto-configured one in @bazel_tools. If it's helpful, I wrote some instructions earlier. Sadly, it appears the resources I used to put that together have been taken down.

I have no idea how a custom CROSSTOOL would interact with Android and iOS though.

@steeve
Copy link
Contributor Author

steeve commented Feb 11, 2020

On Apple, it has been fixed on Xcode 10.2.1.
Apple LLVM version 10.0.1 (clang-1001.0.46.4).

Will try Android.

@jayconrod
Copy link
Contributor

Closing old issues. Sounds like this is fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants