Write a script for bit-perfect deterministic build

[starius]

Following on from #2388 (comment) I think we should provide a shell script producing the released binary from scratch: downloads and installs Go, downloads Sia and all its dependencies and buildes them. Then everybody can run the script and compare results bit-by-bit.

The script can use the following commands: standard (cd grep cp ...) and build (make gcc wget git sha256sum).

I wrote a script to download and build Go of a particular version: https://github.com/starius/gohere
It is in Python and I am not sure if we want to have Python as a dependency. On the other hand Python is less fragile than shell (that is why I used it in my script).

[petabytestorage]

i agree we need reproducible builds. Here is one perhaps simple solution something like "brew doctor" on os/x. so instead of a single script that "does everything" I suggest a single "doctor" script that "checks everything we need to" in order to be sure e.g. the go version is correct, dependencies are correct, etc. it can print the first error and exit if not. and we can offer a variety of solutions for each part to devs such as: have documentation or link to go install page; bash script to download go; or python support scripts that can be used optionally to install certain parts such as go but are not required. the 'siadoctor' script would let people know their environment was considered to meet all known preconditions required for a reproducible build and would let us separate the detection of environment anomalies from the remedy.

[starius]

@petabytestorage
Why to make people do things that can be done by machines?

I made a draft of such a script depending only on bash coreutils wget tar sed gcc make (this will allow to reproduce it on wide range of machines). Currently it is working but producing different binaries on some host-target combinations.

[petabytestorage]

@starius i would not want to "make" people do anything extra manually of course. my point was rather if i were given the choice of only one of two options: a) a "doctor" script or b) a "download everything and build" script, i would choose a). because a) is more usable in a wider variety of circumstances and contains checks that can make a higher quality script of type b). that is, a script of type b) can proceed "blindly" (which is sometimes destructive and not really very flexible with other build choices) or it can run the same checks that a type a) "doctor" script would and in so doing can offer the user a more intelligent and friendly menu: 1. install automatically 2. install manually yourself. so the best case to me is two separate scripts. one doctor and another auto installer that happen to both use the same acceptance checks for each environmental dependency. it can be one script in two modes. the thing i want to avoid is an "install everything" script that is hard to maintain because it doesn't try to find out what is wrong since it always installs everything.

[starius]

The idea of the script is to have minimal environmental dependencies (bash coreutils wget tar sed gcc make) and still build everything from source. It allows everybody who has these tools installed to run the script and compare the resulting binaries with the released ones. If anything goes wrong (e.g. checksum of a downloaded file mismatches), it stops (thanks to set -xue), no blind operation should follow. Also, the script will use only two inputs: Go and Sia sources. No other dependencies. (Sia's dependencies will be vendored by that time: #2388.) The script downloads and checks against checksums these two inputs.

There's no advantage of using prebuilt Go binaries or user-provided Sia sources.

Can we update the script draft to be more doctor-like e.g. printing user friendly messages in case of download error?

[petabytestorage]

the script is long and i thought it was doing much more. it was the many lines of special cases that confused me in there for patches. now that you describe the minimum set of tools it requires (all ancient) and mentioned the strict shell option setting i think it is great already and good enough to be helpful. i don't see any need for a separate doctor type script now that i see yours does not cut corners like i imagined before.

[starius]

My script works, but produces different binaries on different host platforms :-)
I narrowed this down to the speakeasy library which is used to enter passwords, and to this code inside speakeasy: golang/go#22193

Possible workaround: Sia can be switched to x/crypto/ssh/terminal #2414

[starius]

add "Feature Request" Label

[starius]

Now after speakeasy was replaced with x/crypto/ssh/terminal it looks like the binaries are the same on Linux 386, Linux amd64.
The script: https://raw.githubusercontent.com/starius/gohere/7cbc3b08631e0c3dd21874a9af6023323e7906d9/build-sia.sh

The checksums:

bed34887c580ef5a1fb741cf48acfa97675ac1d061dd2c11a6dc177307bdefb5  /tmp/sia-build/gopath/bin/darwin_386/siac
53c2fc590ae3100a6316ae2b6f4c08de8fcd7da9ccf84c769b250468725a4952  /tmp/sia-build/gopath/bin/darwin_amd64/siac
1eca1b49f32fdc65fa84952131b5678d9887fdb2ceef0ee4bb789b864b4533e4  /tmp/sia-build/gopath/bin/linux_386/siac
ffabeb7289bd2371feff926037c8aa5bc827342672b7222f42a06b983946d7b7  /tmp/sia-build/gopath/bin/linux_amd64/siac
49ab8513dc695ca93bcc5062edd2e0379068031f26aa9d97340865154451009c  /tmp/sia-build/gopath/bin/windows_386/siac.exe
abec607d814c935a0fbb96afd7245aeb6cb9b4020ba6be2136642e97e129a373  /tmp/sia-build/gopath/bin/windows_amd64/siac.exe
cb46e27e0b0dd8edb8632dae6001e1ab9a719923e3ef22809bd152f06c3d2d6d  /tmp/sia-build/gopath/bin/darwin_386/siad
792fab62933522284f6d6c8cc3f700d19bf58abc347338e4bf6a9612cc4229f1  /tmp/sia-build/gopath/bin/darwin_amd64/siad
65ada99f59fac2f39ff2b51e59f4a5afa556d7bf933d2af3dcc9b30727ff8b30  /tmp/sia-build/gopath/bin/linux_386/siad
6b6b2819f1165729032b968f1d0ad0220b6f4f128094aaeb45f7596ccb19e6f0  /tmp/sia-build/gopath/bin/linux_amd64/siad
93ceb051dc45ec5a59731f2573d546337d0173b268ba0f56dc8a099400e9cce3  /tmp/sia-build/gopath/bin/windows_386/siad.exe
84d9e5013534ebd92f38dfb92e0c74b79186c63c3c6542496b1cb4d9d4df326e  /tmp/sia-build/gopath/bin/windows_amd64/siad.exe

Those who have ARM could verify that you have the same binaries.

[tbenz9]

I ran your script on my RaspberryPi 3. Many but not all have the same hash. What's the next step?

bc88a972c4cbacd2658b52981a734ecb4b37a73854c3d8fea5bbcfc00c8e7a06  /tmp/sia-build/gopath/bin/darwin_386/siac
53c2fc590ae3100a6316ae2b6f4c08de8fcd7da9ccf84c769b250468725a4952  /tmp/sia-build/gopath/bin/darwin_amd64/siac
1286229d18ba5388a078871324ddb78c4d62ab66b8affd11fddb2f4cede0e7e6  /tmp/sia-build/gopath/bin/linux_386/siac
ffabeb7289bd2371feff926037c8aa5bc827342672b7222f42a06b983946d7b7  /tmp/sia-build/gopath/bin/linux_amd64/siac
542495a72f3aa7d35e51777a9652b6e2a779406e949b08ed898ca9ae1005ffa4  /tmp/sia-build/gopath/bin/windows_386/siac.exe
abec607d814c935a0fbb96afd7245aeb6cb9b4020ba6be2136642e97e129a373  /tmp/sia-build/gopath/bin/windows_amd64/siac.exe
519e31bf35df227950f50f4ca518db16b1f691478417fc7c5d5b625b8cafc8d9  /tmp/sia-build/gopath/bin/darwin_386/siad
792fab62933522284f6d6c8cc3f700d19bf58abc347338e4bf6a9612cc4229f1  /tmp/sia-build/gopath/bin/darwin_amd64/siad
2e1b97bbceb8de008ae68408fc2e4182f12c7a63ed7339d6edc6a44b1c7c0551  /tmp/sia-build/gopath/bin/linux_386/siad
6b6b2819f1165729032b968f1d0ad0220b6f4f128094aaeb45f7596ccb19e6f0  /tmp/sia-build/gopath/bin/linux_amd64/siad
d963be5c448c733e7889a446cba0fed4505b7da60aaffdda10472bd4b20e61cc  /tmp/sia-build/gopath/bin/windows_386/siad.exe
84d9e5013534ebd92f38dfb92e0c74b79186c63c3c6542496b1cb4d9d4df326e  /tmp/sia-build/gopath/bin/windows_amd64/siad.exe
tbenz9@raspberrypi:~ $ uname -a
Linux raspberrypi 4.9.35-v7+ #1014 SMP Fri Jun 30 14:47:43 BST 2017 armv7l GNU/Linux
tbenz9@raspberrypi:~ $

[starius]

All files that are different belong to *_386 platforms.
May be another issue like golang/go#22193