cmd/compile: inlining after devirtualization

[FiloSottile]

In #33160 we talked about an escape analysis pass after devirtualization, to prevent arguments of devirtualized interface method calls from escaping to the heap. This made it possible to use crypto/sha256.New256().Write(buf) without leaking buf. ✨

Separately, I've been using the mid-stack inliner to outline allocations into the caller, enabling allocation-free APIs that ergonomically return a []byte. 🎉

I am now designing the crypto/ecdh API, and I'd like to combine these two techniques, because the cleanest API we could come up with @rsc so far is something like this

type Curve interface {
	GenerateKey(rand io.Reader) (*PrivateKey, error)
	NewPublicKey(key []byte) (*PublicKey, error)
	ECDH(local *PrivateKey, remote *PublicKey) ([]byte, error)
	// unexported methods so can't be externally implemented
}

func P256() Curve
func P384() Curve
func P521() Curve

which you'd use like this

p256 := ecdh.P256()

ourKey, err := p256.GenerateKey(rand.Reader)
if err != nil { t.Fatal(err) }

peerPublic, err := p256.NewPublicKey(peerShare)
if err != nil { t.Fatal(err) }

sharedSecret, err := p256.ECDH(ourKey, peerPublic)
if err != nil { t.Fatal(err) }

Unfortunately, there is no inlining pass after devirtualization, so although p256.GenerateKey et al get devirtualized, they don't get inlined, and their allocations don't end up on the caller's stack.

How doable would it be to have an inlining pass after devirtualization and before escape analysis? (It doesn't have to come before crypto/ecdh, but how likely it is to come later will influence whether we go with this or with more verbose concrete return values.)

/cc @mdempsky @randall77

[mvdan]

cc @mundaym @josharian as I seem to recall some threads years ago about repeating some of the compiler optimization passes where the improvement versus compile time tradeoff would still make sense.

[egonelbre]

Maybe it would be possible to restrict the extra inlining pass to the funcs that contain devirtualizations to reduce the extra compile time?

[CAFxX]

Related: #38992

[gopherbot]

Change https://go.dev/cl/528321 mentions this issue: cmd/compile: interleave inlining and devirtualization