Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory #244

Open
RouR opened this issue May 23, 2022 · 3 comments
Open

Out of memory #244

RouR opened this issue May 23, 2022 · 3 comments

Comments

@RouR
Copy link

RouR commented May 23, 2022

Try unpack 6Mb file with 32Kb dictionary
CompressionLevel = 18
"zstd-codec": "^0.1.4"

OOM
zstd-codec-binding-wasm.js:8 
abort	@	zstd-codec-binding-wasm.js:8
abortOnCannotGrowMemory	@	zstd-codec-binding-wasm.js:8
_emscripten_resize_heap	@	zstd-codec-binding-wasm.js:8
$_sbrk	@	0025c056:0x9130a
$_malloc	@	0025c056:0x81a42
$func418	@	0025c056:0x83c70
$func76	@	0025c056:0x212f
$func73	@	0025c056:0x2021
$func82	@	0025c056:0x22c1
$dynCall_viiii	@	0025c056:0x913f1
Module.dynCall_viiii	@	zstd-codec-binding-wasm.js:8
dynCall_viiii_4	@	VM14:4
VectorU8$resize	@	VM81:10
(anonymous)	@	zstd-codec.js:147
withBindingInstance	@	zstd-codec.js:9
withCppVector	@	zstd-codec.js:18
(anonymous)	@	zstd-codec.js:141
withBindingInstance	@	zstd-codec.js:9
withCppVector	@	zstd-codec.js:18
decompressUsingDict	@	zstd-codec.js:140
@RouR
Copy link
Author

RouR commented May 24, 2022

 Available Emscripten's heap size is 16MiB
 (input.length + output.length) should be less than 12MiB

@0Ams
Copy link

0Ams commented Nov 15, 2022

I am commenting because a similar problem occurred

async function zstdCodecStreamingTest(level) {
  const zstdCodecStreaming = new Promise(async (resolve) =>
    ZstdCodec.run((binding) => resolve(new binding.Streaming()))
  );
  const dst = path.join(__dirname, 'samples', `${testFileName}.tar.zstd.codec`);
  let cTime;
  let dTime;

  // 5-1) compress
  await new Promise(async (resolve, reject) => {
    const start = Date.now();
    const compressed = (await zstdCodecStreaming).compress(Buffer.from(JSON.stringify(COMPRESS_DATA)), level);
    fs.writeFileSync(dst, compressed);
    cTime = Date.now() - start;
    resolve();
  });

  // 5-2) decompress
  await new Promise(async (resolve, reject) => {
    const start = Date.now();
    const compressed_bytes = fs.readFileSync(dst);
    (await zstdCodecStreaming).decompress(compressed_bytes);
    dTime = Date.now() - start;
    resolve();
  });
}

for (let i = 1; i <= 20; i += 1) {
   zstdCodecStreamingTest(i);
}

OUTPUT

# Original Size: 0.5219583511352539 MB
lvl: 1,  data size: 547313.0000 after: 66734.0000
lvl: 2,  data size: 547313.0000 after: 65423.0000
lvl: 3,  data size: 547313.0000 after: 60736.0000
lvl: 4,  data size: 547313.0000 after: 60434.0000
lvl: 5,  data size: 547313.0000 after: 58441.0000
lvl: 6,  data size: 547313.0000 after: 55838.0000
lvl: 7,  data size: 547313.0000 after: 51970.0000
lvl: 8,  data size: 547313.0000 after: 51943.0000
lvl: 9,  data size: 547313.0000 after: 50548.0000
OOM
OOM
(node:3350) UnhandledPromiseRejectionWarning: abort(OOM). Build with -s ASSERTIONS=1 for more info.

data size: 0.5mb

conclusion

  • Normal operation using simple method
  • OOM occurs when using the streaming method. Case with the same structure, only changing the method

@0Ams
Copy link

0Ams commented Nov 22, 2022

I found that OOM occurs when testing while adjusting the level.
But I haven't found the cause, so is there anything else I need to check?

@joscha joscha mentioned this issue Sep 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants