encoding/gob: Decoder cannot be reused if a composite types are involved

[myitcv]

What version of Go are you using (go version)?

$ go version
go version go1.12beta2 linux/amd64

Does this issue reproduce with the latest release?

Testing with beta pre the 1.12 release.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/myitcv/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/myitcv/gostuff"
GOPROXY=""
GORACE=""
GOROOT="/home/myitcv/gos"
GOTMPDIR=""
GOTOOLDIR="/home/myitcv/gos/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/myitcv/gostuff/src/github.com/myitcv/playground/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build266146048=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Reusing a gob.Decoder with a non-composite type works just fine:

func TestGobDecodeInt(t *testing.T) {
	var v, v2 int
	buf := bytes.NewBuffer(nil)
	e := gob.NewEncoder(buf)
	if err := e.Encode(v); err != nil {
		panic(err)
	}
	r := bytes.NewReader(buf.Bytes())
	d := gob.NewDecoder(r)
	fmt.Println("TestGobDecodeInt")
	for i := 0; i < 5; i++ {
		if err := d.Decode(&v2); err != nil {
			panic(err)
		}
		fmt.Printf("  Decoded: %v\n", v2)
		r.Reset(buf.Bytes())
	}
}

gives:

TestGobDecodeInt
  Decoded: 0
  Decoded: 0
  Decoded: 0
  Decoded: 0
  Decoded: 0

However where a composite type is involved, things start to break down:

func TestGobDecodeStruct(t *testing.T) {
	var v, v2 struct{}
	buf := bytes.NewBuffer(nil)
	e := gob.NewEncoder(buf)
	if err := e.Encode(v); err != nil {
		panic(err)
	}
	r := bytes.NewReader(buf.Bytes())
	d := gob.NewDecoder(r)
	fmt.Println("TestGobDecodeStruct")
	for i := 0; i < 5; i++ {
		if err := d.Decode(&v2); err != nil {
			panic(err)
		}
		fmt.Printf("  Decoded: %v\n", v2)
		r.Reset(buf.Bytes())
	}
}

gives:

TestGobDecodeStruct
  Decoded: {}
--- FAIL: TestGobDecodeStruct (0.00s)
panic: extra data in buffer [recovered]
        panic: extra data in buffer

goroutine 8 [running]:
testing.tRunner.func1(0xc000104300)
        /home/myitcv/gos/src/testing/testing.go:827 +0x388
panic(0x567dc0, 0xc000011ac0)
        /home/myitcv/gos/src/runtime/panic.go:522 +0x1b5
github.com/myitcv/playground.TestGobDecodeStruct(0xc000104300)
        /home/myitcv/gostuff/src/github.com/myitcv/playground/main_test.go:49 +0x3be
testing.tRunner(0xc000104300, 0x5a3f48)
        /home/myitcv/gos/src/testing/testing.go:862 +0xc0
created by testing.(*T).Run
        /home/myitcv/gos/src/testing/testing.go:913 +0x357

Same behaviour observed if a slice or map are involved.

What did you expect to see?

I'm unclear whether this is to be expected or not. I don't understand encoding/gob particularly well, so there might be reasons why reusing a Decoder in this way might break.

If this is expected, then I'd suggest documenting the restriction in some way.

Otherwise, I guess this is a bug?

cc @robpike @mvdan

[mvdan]

One interesting observation is that gob.Encoder can be reused just fine. Other encoding types, such as json.Encoder and json.Decoder, can be reused too. So I'd expect the gob decoder to be the same, even though this isn't explicitly documented.

[robpike]

This is working as intended. The output from the Encoder is a stream, and is meant to be read, once, as a stream. You're using it wrong.

Mechanically, if the stream contains type information, as it will if there is a non-base type, then as the Decoder runs it will change state to absorb that type. That means the second time you run it with the same data, it will be inconsistent and break.

[myitcv]

Thanks for confirming, @robpike.

[mvdan]

Thanks for the quick reply, Rob. I wonder if the decoder could give a better error message if it is misused like this. Alternatively, if this happened often enough, perhaps we could add a warning on the Decoder type.