cmd/link: 386 and amd64 hosts produce different 386 binaries

[starius]

What version of Go are you using (go version)?

1.9.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Host: Linux amd64 and 386, Darwin amd64.
Target: {amd64, 386} {Linux, Darwin, Windows}.

What did you do?

I built the following program (located in $GOPATH/src/test/main.go):

package main

import "os"
import "os/signal"

func main() {
    sig := make(chan os.Signal, 10)
    signal.Stop(sig)
}

with the following command:

GOOS=linux GOARCH=386 go install -a -tags netgo -ldflags='-s -w' test

on two host machines: Linux amd64 and Linux 386.
The produced binaries were different.

I wrote a script that installs Go 1.4 which installs Go 1.9.1 which installs another Go 1.9.1 (as suggested on the IRC channel) which builds the program.

Results:

host=linux_amd64
* 6764b1c25110b5050739ea420d21707fe7f60c8f0f60e5aa0edc16a9ba1ed958  /tmp/gopath/bin/darwin_386/test
  099d1ea0f08c12961871062175ed1eb0f08a9c3711a04ae7b065ed464d0f7c4e  /tmp/gopath/bin/darwin_amd64/test
* 85b7da6f6627c34e50ba1801380ad28dad07cb8409c13afef78f02c0c4f3cc7d  /tmp/gopath/bin/linux_386/test
  0f56ffbe56fb47d76e34a174ca9a16e948ed31d0f1966d3ad3a9e4413bc21288  /tmp/gopath/bin/linux_amd64/test
* cccfac9dc10b2fe184581904bae8ae32a593f2d7369689ea8de92d42d82e4f8e  /tmp/gopath/bin/windows_386/test.exe
  0b9fd63afa1b5212a71912a5fa7b92d016fa53fdab7a6bd3368ff92cfeb87def  /tmp/gopath/bin/windows_amd64/test.exe

host=linux_386
* 4b356f6d7e5f8f236aa2340223bb4b6e96cd955f3615cd84d06f1fbeafc71bf2  /tmp/gopath/bin/darwin_386/test
  099d1ea0f08c12961871062175ed1eb0f08a9c3711a04ae7b065ed464d0f7c4e  /tmp/gopath/bin/darwin_amd64/test
* 9a0522c00d1b989650342efba689bd9d332fc3290d085a16949c626ce0822f02  /tmp/gopath/bin/linux_386/test
  0f56ffbe56fb47d76e34a174ca9a16e948ed31d0f1966d3ad3a9e4413bc21288  /tmp/gopath/bin/linux_amd64/test
* 35d36ded359656d9ae5dfa7f68a30585a0c327c622720bb34fbdf34dda912c0e  /tmp/gopath/bin/windows_386/test.exe
  0b9fd63afa1b5212a71912a5fa7b92d016fa53fdab7a6bd3368ff92cfeb87def  /tmp/gopath/bin/windows_amd64/test.exe

386 binaries for all 3 OS (Linux, Windows, Darwin) are different when building on Linux amd64 and Linux 386. However both hosts produce amd64 binaries that are exactly the same on any OS.

Darwin amd64 host produces the same binaries as Linux amd64.

[davecheney]

I think this is expected. Compiling from linux/amd64 -> linux/amd64 will produce a cgo enabled dynamically linked binary. All other operations will produce a statically linked static binary.

…

On Tue, Oct 10, 2017 at 3:04 AM, Boris Nagaev ***@***.***> wrote: What version of Go are you using (go version)? 1.9.1 Does this issue reproduce with the latest release? Yes What operating system and processor architecture are you using (go env)? Host: Linux amd64 and 386, Darwin amd64. Target: {amd64, 386} {Linux, Darwin, Windows}. What did you do? I built the following program (located in $GOPATH/src/test/main.go): package main import "os"import "os/signal" func main() { sig := make(chan os.Signal, 10) signal.Stop(sig) } with the following command: GOOS=linux GOARCH=386 go install -a -tags netgo -ldflags='-s -w' test on two host machines: Linux amd64 and Linux 386. The produced binaries were different. I wrote a script <https://raw.githubusercontent.com/starius/gohere/ac9a50e686b2341314c40947b670a73a9a571d64/build-demo.sh> that installs Go 1.4 which installs Go 1.9.1 which installs another Go 1.9.1 (as suggested on the IRC channel) which builds the program. Results: host=linux_amd64 * 6764b1c25110b5050739ea420d21707fe7f60c8f0f60e5aa0edc16a9ba1ed958 /tmp/gopath/bin/darwin_386/test 099d1ea0f08c12961871062175ed1eb0f08a9c3711a04ae7b065ed464d0f7c4e /tmp/gopath/bin/darwin_amd64/test * 85b7da6f6627c34e50ba1801380ad28dad07cb8409c13afef78f02c0c4f3cc7d /tmp/gopath/bin/linux_386/test 0f56ffbe56fb47d76e34a174ca9a16e948ed31d0f1966d3ad3a9e4413bc21288 /tmp/gopath/bin/linux_amd64/test * cccfac9dc10b2fe184581904bae8ae32a593f2d7369689ea8de92d42d82e4f8e /tmp/gopath/bin/windows_386/test.exe 0b9fd63afa1b5212a71912a5fa7b92d016fa53fdab7a6bd3368ff92cfeb87def /tmp/gopath/bin/windows_amd64/test.exe host=linux_386 * 4b356f6d7e5f8f236aa2340223bb4b6e96cd955f3615cd84d06f1fbeafc71bf2 /tmp/gopath/bin/darwin_386/test 099d1ea0f08c12961871062175ed1eb0f08a9c3711a04ae7b065ed464d0f7c4e /tmp/gopath/bin/darwin_amd64/test * 9a0522c00d1b989650342efba689bd9d332fc3290d085a16949c626ce0822f02 /tmp/gopath/bin/linux_386/test 0f56ffbe56fb47d76e34a174ca9a16e948ed31d0f1966d3ad3a9e4413bc21288 /tmp/gopath/bin/linux_amd64/test * 35d36ded359656d9ae5dfa7f68a30585a0c327c622720bb34fbdf34dda912c0e /tmp/gopath/bin/windows_386/test.exe 0b9fd63afa1b5212a71912a5fa7b92d016fa53fdab7a6bd3368ff92cfeb87def /tmp/gopath/bin/windows_amd64/test.exe 386 binaries for all 3 OS (Linux, Windows, Darwin) are different when building on Linux amd64 and Linux 386. However both hosts produce amd64 binaries that are exactly the same on any OS. Darwin amd64 host produces the same binaries as Linux amd64. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#22193>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAcA4ZjInfsMw1TrQyehZ_uQb97YQmGks5sqt7IgaJpZM4PzT-P> .

[ianlancetaylor]

Yes, please try your tests setting CGO_ENABLED=1 (or CGO_ENABLED=0) in both cases.

[starius]

I tried CGO_ENABLED=1 and CGO_ENABLED=0 but it didn't affect the resulting files.

[ianlancetaylor]

Can you show us the output of GOARCH=386 CGO_COMPILED=0 go build -x for your test case?

[ianlancetaylor]

Can you look into exactly how the different files are different? Perhaps by using the readelf or objdump programs?

[starius]

Compiling from linux/amd64 -> linux/amd64 will produce a cgo enabled dynamically linked binary. All other operations will produce a statically linked static binary.

Disabling cgo didn't help. Also, changing host from linux_amd64 to linux_386 affected all 386 binaries (including darwin and windows), while changing host from linux_amd64 to darwin_amd64 didn't affect any binary. So it doesn't look like it is as simple as special treatment of cases when host=target.

[starius]

Can you show us the output of GOARCH=386 CGO_COMPILED=0 go build -x for your test case?

https://gist.github.com/starius/495ff3943ac338b8472cce771d2d136b (go install instead of go build but should not be a big deal. Also for GOOS=linux only.)

[starius]

Can you look into exactly how the different files are different? Perhaps by using the readelf or objdump programs?

diff of objdumps: https://pastebin.com/HATEiHdk

Original files (xxd of):
built on linux_386: http://rgho.st/922ZdvC25
built on linux_amd64: http://rgho.st/66cJ89MjQ

BTW gist doesn't eat these files for some reason. I upload them and in one second they are already 404. So I uploaded to more friendly sites.

[bradfitz]

/cc @FiloSottile (for reproducibility issues)

[ianlancetaylor]

I tried to repeat this problem but I was unable to. Can you see whether this is still a problem for you today? Thanks.

[starius]

My sample that I posted above produces as of Go 1.11 the same amd64 Linux binaries on i386 and amd64 Linux hosts. However the real binary I am interested in (go get gitlab.com/NebulousLabs/Sia/cmd/siad) still produces different binaries. Probably I need to make another minimal example.

[bcmills]

Is this still reproducible? Go 1.21 is supposed to support fully reproducible builds.

[starius]

I checked it using Go 1.21.4. I don't have a 386 host, so I used amd64 host and 386 toolchain. (amd64 host runs 386 binaries.)

$ go version
go version go1.21.4 linux/386

$ CGO_ENABLED=0 GOOS=linux GOARCH=386 go install -trimpath -ldflags=" -s -w -buildid=" go.sia.tech/siad/cmd/[email protected]

$ sha256sum go/bin/linux_386/siad 
1c39f618f9004ab841337b66c81b7f5f8a84690ef904e6b1fc25e0386a1a5396  go/bin/linux_386/siad

I got the same binary on 386 toolchain running on amd64 host and on amd64 toolchain running on amd64 host and on arm64 toolchain and host.

To be 100% sure, it should be tested on real 386 host, but I don't have one. If you have such a host, can you run the above command and compare the binary, please?

[starius]

I ran whole Linux i386 machine using qemu and checked which binary does it produce. It matches to the binary produced on Linux amd64 machine (cross-compiling).

Also checked the same on the old release of Sia. Hashes also match!

I think the issue is solved now.