Handle truncated compressed response (Z_BUF_ERROR)

[gajus]

Z_BUF_ERROR can occur if a response is truncated.

I am seeing this relatively a lot when making requests through proxies (mishandled stream or proxy node disconnecting mid response).

If this happens, then gunzip will throw Z_BUF_ERROR error.

Once you have Z_BUF_ERROR then you can run gunzip again with {finishFlush: zlib.constants.Z_SYNC_FLUSH} option. This will allow to flush all data before the error. (The reason for not using Z_SYNC_FLUSH on the first go is because you couldn't tell apart corrupt from partial output).

See: https://github.com/nodejs/node/blob/master/doc/api/zlib.md#compressing-http-requests-and-responses

In my case, I have been seeing the "unexpected end of file" error a lot in our logs and dug into the origins. It turned out that one of the websites that we are accessing is terminating connections sockets seemingly at random (presumably when it is under high load) – therefore producing truncated output. As the output was gziped, it wasn't obvious what is going on. I have disabled decompression in Got and used zlib.constants.Z_SYNC_FLUSH and saw that half of the response is missing.

It would be a nice DX improvement if this was detected and a more descriptive error would be thrown instead.

---

IssueHunt Summary

szmarczak has been rewarded.

Backers (Total: $40.00)

• issuehunt ($40.00)

Submitted pull Requests

• #20 Throw more descriptive error when handling truncated responses
• #27 Node.js compatibility fixes

---

Tips

• Checkout the Issuehunt explorer to discover more funded issues.
• Need some help from other developers? Add your repositories on IssueHunt to raise funds.

[gajus]

I think I have used enough keywords to help others find this issue through Google in relation to Node.js and gzip. Hopefully this will save someone a productive day!

[issuehunt-oss]

@issuehunt has funded $40.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.

[hicom150]

I've been looking into this issue and although I understand your explanation I not able to create a test case where I can see any difference between using zlib.constants.Z_SYNC_FLUSH or default zlib.constants.Z_FINISH

I created this small test code to tweak the truncated input but the result is the same for me 😅

Do you have any example or more details so we can create a better test case?

Thank you!

const zlib = require('zlib');
const {Readable: ReadableStream, PassThrough: PassThroughStream} = require('stream');

const decompress = zlib.createUnzip();
// const decompress = zlib.createUnzip({finishFlush: zlib.constants.Z_SYNC_FLUSH});

const input = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.';
const buffer = Buffer.from(input, 'utf-8');
const compressed = zlib.deflateSync(buffer);

const response = new ReadableStream();
response.push(compressed.slice(0, -10));
response.push(null);

const stream = new PassThroughStream();

decompress.on('error', error => {
	// Ignore empty response
	if (error.code === 'Z_BUF_ERROR') {
		stream.end();
		return;
	}

	stream.emit('error', error);
});

const output = response.pipe(decompress).pipe(stream);

output
	.on('data', data => console.log(data.toString()))
	.on('error', error => console.log(error))
	.on('end', () => console.log('end'));

[gajus]

compressed.slice(0, -10).push(Buffer.from('foo', 'utf-8'))

should do the trick.

[hicom150]

I've been re-reading the issue and there is something that I don't fully understand 😅

As Node.js Zlib Documentation points out, we have two ways to decompress data.

One is piping the source stream data through a zlib stream into a destination stream:

const decompress = zlib.createUnzip();
const output = response.pipe(decompress).pipe(stream);

The other one is to decompress in a single step:

const output = zlib.unzipSync(response);

Changing the flushing method to zlib.constants.Z_SYNC_FLUSH to recover the last chunk of input data only works when decompressing data in a single step.

Here is a little example

const zlib = require('zlib');

const input = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.';
const buffer = Buffer.from(input, 'utf-8');
const compressed = zlib.deflateSync(buffer);

// zlib.unzipSync(compressed.slice(0, -10)); // default value Z_FINISH
// -> Error: unexpected end of file

zlib.unzipSync(compressed.slice(0, -10), {finishFlush: zlib.constants.Z_SYNC_FLUSH});
// -> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id e

When we decompress a stream with pipes, all the data decompressed before the error is recovered by default.

const zlib = require('zlib');
const {Readable: ReadableStream, PassThrough: PassThroughStream} = require('stream');

const decompress = zlib.createUnzip();

const input = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.';
const buffer = Buffer.from(input, 'utf-8');
const compressed = zlib.deflateSync(buffer);

const response = new ReadableStream();
response.push(compressed.slice(0, -10));
response.push(null);

const stream = new PassThroughStream();

decompress.on('error', error => {
	// Ignore empty response
	if (error.code === 'Z_BUF_ERROR') {
		stream.end();
		return;
	}

	stream.emit('error', error);
});

const output = response.pipe(decompress).pipe(stream);

output
	.on('data', data => console.log(data.toString()))
	.on('error', error => console.log(error))

// -> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id e

As a conclusion, I think that we already handle correctly truncated compressed responses 💪

[gajus]

Define correctly?

The current implementation returns truncated response without any warning of it being truncated.

The current issue proposes to detect when this happens and throw an error:

It would be a nice DX improvement if this was detected and a more descriptive error would be thrown instead.

[hicom150]

You are absolutely right 😅 I've focus on zlib.constants.Z_SYNC_FLUSH and completely forgot about throwing a more descriptive error 😇

I hope #20 fixes this issue, where an error is thrown with error.code: 'Z_BUF_ERROR' and error.message: 'unexpected end of file' when handling truncated responses 😉

[issuehunt-oss]

@sindresorhus has rewarded $36.00 to @szmarczak. See it on IssueHunt

• 💰 Total deposit: $40.00
• 🎉 Repository reward(0%): $0.00
• 🔧 Service fee(10%): $4.00