cmd/go: only rebuild dependent packages when export data has changed

[josharian]

@mwhudson suggested in #15734:

If the export data hasn't changed, can you skip the compilation of dependent packages?

Moving this idea here to be tracked separately.

My 2c: Interesting idea, seems worth pursuing. Not sure how often it happens that code changes don't impact export data (or how cheap that is to detect), but when it does happen, that could save a bunch of computation.

[dr2chase]

It's not trivial. Years a go I worked on an ahead-of-time compiler for Java bytecodes that did just this, and it was a good thing (back in the days of 200Mhz CPUs), but it required substantial amounts of dependence information (stored on disk in gzip format) and it also required that every reference to information from another package log an entry in the internal dependence database for later storage -- in particular, every field reference or interface method reference, because those involve offsets, and offsets might change as interface declarations or field declarations are changed (granularity matters, too -- do you record the actual dependence that field F is at offset 8, or do you just record that the SHA of the struct had a particular value?

[josharian]

IIUC, the plan would be to do something like check the SHA of the export data. If that hasn't changed, then the downstream compilation won't change.

[mwhudson]

Yeah, I was just thinking that you'd store the SHA1 of the export data of
all imported packages in the .a file and only recompile if any of them had
changed. No sense trying to be too sophisticated.

I don't know often this would be useful. Certainly sometimes (especially in
a "add more and more debug prints" cycle), but it will obviously also have
a small cost all the time. Seems worth trying out, I'll try to hack
something up between now and 1.8 early.

On 20 May 2016 at 06:56, Josh Bleecher Snyder [email protected]
wrote:

IIUC, the plan would be to do something like check the SHA of the export
data. If that hasn't changed, then the downstream compilation won't change.

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#15752 (comment)

[griesemer]

Régis Crelier (currently at Google) has investigated this issue for Oberon in very much detail - I believe much of his work should be directly applicable to Go. For details see:
http://e-collection.library.ethz.ch/eserv/eth:39386/eth-39386-02.pdf

[josharian]

@rsc will this be a part of your cmd/go re-building improvements?

[rsc]

Yes the plan is that we'll be set up well to take advantage of this.
We probably won't do it initially, but as a followup.

[josharian]

Note that if we do much with #25999, the set of code changes that leave export data unchanged might shrink significantly.

[josharian]

cc @bcmills @jayconrod

Is this now just (“just”) a matter of only hashing the export data when determining whether object files are stale?

[bcmills]

I think so, yes.

[mwhudson]

I think it's a little bit more than that because you still need to re-link any executables?