compress/gzip: writer spend too long time

[sunhuaibo]

Go version

go version go1.23.1 linux/amd64

Output of go env in your module/workspace:

...

What did you do?

Trying to compress a file (75M) with gzip, but it took too long.

import (
	"bufio"
	"compress/gzip"
	"os"
	// gzip "github.com/klauspost/pgzip"
)

func main() {

	// R22015519_R1.fastq: size 76M
	f, err := os.Open("R22015519_R1.fastq")
	if err != nil {
		panic(err)
	}
	defer f.Close()

	fw, err := os.Create("R22015519_R1.fastq.gz")
	if err != nil {
		panic(err)
	}
	defer fw.Close()

	w := gzip.NewWriter(fw)
	defer w.Close()
	defer w.Flush()

	sc := bufio.NewScanner(f)
	for sc.Scan() {
		w.Write(sc.Bytes())
		w.Write([]byte("\n"))
	}
}

What did you see happen?

The time spent by the compress/gzip package is dozens of times that of the pgzip (github.com/klauspost/pgzip) package.

Using compress/gzip:

real	2m39.768s
user	0m20.957s
sys	0m1.569s

Using github.com/klauspost/pgzip:

real	0m1.222s
user	0m1.161s
sys	0m0.098s

What did you expect to see?

NULL

[gabyhelp]

Related Issues and Documentation

• compress/gzip: use bufio to improve compress speed #56449 (closed)
• compress/gzip: memory leak using gzip.Writer #58455 (closed)
• archive/zip: compression performance #20031 (closed)
• compress/flate: decompression performance #23154
• compress/bzip2: Slow performance #6754 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[seankhliao]

As the documentation for pgzip mentions, it starts multiple go routines to run compression in parallel. This is something we are unlikely to do in the standard compress/gzip.