Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add zstd decompression support #648

Merged
merged 88 commits into from
Sep 29, 2020
Merged
Show file tree
Hide file tree
Changes from 74 commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
7ff4651
Attach zstd decompressor
Jaifroid Apr 10, 2020
68961ac
Attempt to integrate decompressor
Jaifroid Apr 10, 2020
fcd61ca
Eliminate memory error
Jaifroid Apr 11, 2020
15f61c5
Redo decompressor
Jaifroid Apr 11, 2020
d9801bf
Update to dev bundle.js (confirmed working in browser)
Jaifroid Apr 11, 2020
8a563a2
Add working example distribution
Jaifroid Apr 11, 2020
0d96660
Correct streaming code (but still bugs out)
Jaifroid Apr 12, 2020
8c3a183
Add c source code for compiling zstddeclib with Emscripten
Jaifroid Apr 12, 2020
9a34fa3
Update zstandard single file
Jaifroid Jul 6, 2020
0236bb3
First attempt to compile zstdec.js with asm
Jaifroid Jul 6, 2020
2061b6d
Add foo-zstd.zim
Jaifroid Jul 13, 2020
d03fc1e
Create wikipedia_bm_all_2020-07.zstd.zim
Jaifroid Jul 21, 2020
8654e98
Update compile.sh
Jaifroid Jul 21, 2020
b30482e
Initial testing of init process
Jaifroid Jul 21, 2020
dc8e78e
Update compile.sh
Jaifroid Jul 21, 2020
8ab4947
Added _ZSTD_isError function
Jaifroid Jul 21, 2020
68c28fa
Replace zstdec.js
Jaifroid Jul 21, 2020
5bf0d39
Update zstdec_wrapper.js
Jaifroid Jul 21, 2020
fd8a373
Update compile options and test compile
Jaifroid Jul 22, 2020
80bf87d
Compile with modularization
Jaifroid Jul 23, 2020
ba4f025
Load xzdec and zstdec via requireJS instead of index.html
Jaifroid Jul 23, 2020
d23e564
Load zstdec within Promise
Jaifroid Jul 23, 2020
419f88e
Save interim work (non-functional)
Jaifroid Jul 26, 2020
544e168
Add mallocOrDie function
Jaifroid Jul 26, 2020
4a52a47
Some progress but decompressStream returns error code
Jaifroid Jul 26, 2020
c29c609
decompressStream no longer returns error!
Jaifroid Jul 26, 2020
8e8ca93
Experiments with binding
Jaifroid Aug 5, 2020
2703293
Revert embindings that don't compile
Jaifroid Aug 16, 2020
a422de3
Add decompressStreamSimpleArgs
Jaifroid Aug 16, 2020
9fe5afd
Fix stupid coding error grrr
Jaifroid Aug 27, 2020
658dff3
Update zstdec_wrapper.js
Jaifroid Aug 28, 2020
cbe4ebb
Update zstdec_wrapper.js
Jaifroid Aug 28, 2020
54c65af
Increase memory to avoid out-of-memory errors
Jaifroid Aug 28, 2020
426c27e
Simplify some code
Jaifroid Aug 29, 2020
9335b4c
Better streaming logic (but not finished)
Jaifroid Aug 30, 2020
3ee378a
Recompile with functions to give recommended buffer sizes
Jaifroid Aug 30, 2020
cdf5e65
Set TOTAL_MEMORY to 140247040
Jaifroid Aug 30, 2020
bb309b5
Update zstdec_wrapper.js
Jaifroid Aug 30, 2020
2120de7
Update zstdec_wrapper.js
Jaifroid Aug 31, 2020
95223be
Enable proper error reporting
Jaifroid Aug 31, 2020
c9c6ad4
Code working with zstd-foo ZIM
Jaifroid Aug 31, 2020
777add9
Set up re-usable decHandle to avoid memory issues
Jaifroid Aug 31, 2020
c58a051
Fix counters
Jaifroid Aug 31, 2020
7c29d9f
Address minor codefactor issues
Jaifroid Sep 1, 2020
84618f5
Document zstdec_wrapper.js
Jaifroid Sep 1, 2020
dd60f79
Compile zstdec with O1 optimizations
Jaifroid Sep 1, 2020
6d468c5
Compile with closure 1
Jaifroid Sep 1, 2020
b354ce4
Implement memory sparing idea: re-use all allocated memory buffers
Jaifroid Sep 2, 2020
707ea40
Add documentation revisions from self review
Jaifroid Sep 2, 2020
9629408
Delete redundant distribution but retain readme.md
Jaifroid Sep 3, 2020
594dad7
Clearer documentation
Jaifroid Sep 3, 2020
4e78ab7
Prevent reading beyond end of cluster?
Jaifroid Sep 3, 2020
c3d927a
Fix buffer fill error and test lower memory usage
Jaifroid Sep 3, 2020
f81b43d
Test with max buffer size
Jaifroid Sep 20, 2020
c0f59d8
Remove experimental code
Jaifroid Sep 20, 2020
76b9b6f
Compile with -O2 optimizations
Jaifroid Sep 23, 2020
4232c06
Compile with O3 optimizations and add _malloc function
Jaifroid Sep 23, 2020
053d632
Remove unused exported functions for production
Jaifroid Sep 24, 2020
e920bab
Remove debug console logging
Jaifroid Sep 24, 2020
6c43f3b
Clearer variable name
Jaifroid Sep 24, 2020
8fd7b4a
Compile for web and add wasm binary
Jaifroid Sep 25, 2020
841c524
Interim cleanup
Jaifroid Sep 26, 2020
df8e791
More logical use of loop
Jaifroid Sep 27, 2020
970a8b9
Always consume buffer
Jaifroid Sep 27, 2020
7cb27a4
Use 100% (but sometimes doesn't)....
Jaifroid Sep 27, 2020
5468dfc
Better test
Jaifroid Sep 27, 2020
0feb07c
Revert "Better test"
Jaifroid Sep 27, 2020
8ddf3a4
Revert "Use 100% (but sometimes doesn't)...."
Jaifroid Sep 27, 2020
1ebaaf5
Revert "Always consume buffer"
Jaifroid Sep 27, 2020
7ce5a63
Revert "More logical use of loop"
Jaifroid Sep 27, 2020
81211d6
Replace fillInBufferIfNeeded with fillInBuffer
Jaifroid Sep 27, 2020
874bebc
Increase chunk size to max recommended
Jaifroid Sep 27, 2020
5b4a385
Rename zstdec zstddec throughout
Jaifroid Sep 27, 2020
b4b1d9e
Changes from code review
Jaifroid Sep 27, 2020
ed96113
Add EMSInstanceExt type
Jaifroid Sep 27, 2020
804f45b
Remove test files
Jaifroid Sep 27, 2020
79a7989
Remove wasm version
Jaifroid Sep 27, 2020
24c22fa
Remove wikipedia_bm_all test file
Jaifroid Sep 27, 2020
2ff76a7
Delete zstddec_full.js
Jaifroid Sep 29, 2020
e592030
Clearer comment
Jaifroid Sep 29, 2020
65fa09f
Clean up whitespace
Jaifroid Sep 29, 2020
dadd7dd
Clean up comment
Jaifroid Sep 29, 2020
e74dab2
Add zstandard licence info and correct spelling of license->licence
Jaifroid Sep 29, 2020
c47af66
One more glaring License -> Licence
Jaifroid Sep 29, 2020
de414e0
Two more!
Jaifroid Sep 29, 2020
5d0c12c
Cleanup spaces
Jaifroid Sep 29, 2020
81c55b2
Typo
Jaifroid Sep 29, 2020
5959001
Final cleanup
Jaifroid Sep 29, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions emscripten/zstandard/compile.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
emcc --memory-init-file 0 -O3 --closure 1 -s ENVIRONMENT="web" -s WASM=0 -s MALLOC="emmalloc" -s TOTAL_MEMORY=140247040 -s FILESYSTEM=0 -s DOUBLE_MODE=0 -s DYNAMIC_EXECUTION=0 -s MIN_IE_VERSION=11 -s EXPORT_NAME="ZD" -s MODULARIZE=1 -s EXPORTED_FUNCTIONS="['_malloc', '_free', '_ZSTD_createDStream', '_ZSTD_initDStream', '_ZSTD_decompressStream', '_ZSTD_isError', '_ZSTD_getErrorName', '_ZSTD_freeDStream', '_ZSTD_DStreamInSize', '_ZSTD_DStreamOutSize']" -s EXPORTED_RUNTIME_METHODS="['cwrap']" *.c -o zstddec.js
8 changes: 8 additions & 0 deletions emscripten/zstandard/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Source code and compilations of zstandard decompression library

The file `zstddeclib.c` is the single-file source code for the zstandard decompressor (compressor is not included).
It has been combined by following the instructions at https://github.com/facebook/zstd/tree/dev/contrib/single_file_libs. It can be used as the source file of an Emscripten webassembly or asm compilation.

The source code can be compiled with Emscripten by setting up the Emscripten SDK and running the Linux script `./compile.sh` in this directory. This will produce `zstddec.js`. Further optimization and minification of this file may be possible by adjusting the commandline options in `compile.sh`.

For a fuller Emscripten distribution of the full zstandard library for `node.js`, see https://github.com/yoshihitoh/zstd-codec/tree/develop/js/example.
43 changes: 43 additions & 0 deletions emscripten/zstandard/wasm/zstddec.js

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Binary file added emscripten/zstandard/wasm/zstddec.wasm
Binary file not shown.
Loading