Extra bytes at the end of encoded buffer (zstd)

[Feresey]

I have found that current implementation of zstd (v1.10.10) adds three extra bytes to the end of encoded byte stream (01 00 00). These bytes makes unable to uncompress archive with standard GNU zstd utility.

Example code:

package main

import (
	"bytes"
	"fmt"
	"strings"

	"github.com/klauspost/compress/zstd"
)

var src = `0`

func main() {
	buffer := bytes.NewBuffer(nil)
	encoder, err := zstd.NewWriter(buffer)
	if err != nil {
		panic(err)
	}
	if _, err := encoder.ReadFrom(strings.NewReader(src)); err != nil {
		panic(err)
	}
	if err := encoder.Close(); err != nil {
		panic(err)
	}
	fmt.Printf("% x\n", buffer.Bytes())
}

Prints: 28 b5 2f fd 04 00 09 00 00 30 ec af 44 12 01 00 00

At the same time with GNU zstd:

$ echo -n 0 | zstd | hexdump -C
00000000  28 b5 2f fd 04 58 09 00  00 30 ec af 44 12        |(./..X...0..D.|
0000000e

Note last 3 extra bytes.

Also, version 1.9.0 works like the GNU zstd.

[klauspost]

Thanks for the report, I will take a look!

[klauspost]

@Feresey It is a raw block size 0 but with a "final" block marker. Technically it should be fine, while it could be simplified.

What is the error you are seeing?

[Feresey]

What is the error you are seeing?

$  zstd -d test.zst
zstd: test already exists; overwrite (y/N) ? y
test.zst             : 0 MB...     zstd: test.zst: unknown header

[klauspost]

Ah, it could be that it is after the final block. 👍

[klauspost]

Ah, yes

2020/08/12 18:26:35 Adding RAW block, length 1 last: true
2020/08/12 18:26:35 Adding RAW block, length 0 last: true

[klauspost]

I will fix and also add this check to the decoder.

[Feresey]

Great thanks for such rapid response! Happy to see that this problem can be resolved!

[klauspost]

Fix in #277 - I will probably make a new release soon.

[klauspost]

@Feresey v1.10.11 has been released with the fix.

[zachcheung]

Sorry to reply in closed issue, but I want to say this unknown header error takes me hours to realize it's related to this package (used v1.10.5), then dig changelog to find here. 😭

[klauspost]

@zachcheung Sorry, do you have a point? What would you expect me to do?

[zachcheung]

@klauspost Nope, bumped compress version fixed the problem.

Thanks for writing such great package. :)