[Feature] HTTP Gzip payload decryption

[windhamwong]

[ADD] Library Pako (https://github.com/nodeca/pako)
[ADD] Operation: HTTP Gzip Decrypt - Decrypts gzip payload in HTTP request/response and keeps the HTTP header in the result

As the original library CyberChef using (zlib.js - https://github.com/imaya/zlib.js) has different bugs that cause failing decrypting gzip. I worked out that Pako can provide better/faster/stable way to decrypt gzip. The operation I added utilize this library for gzip payload. Although there is still a minor bug when the payload is too long, the overall result is pretty good.

I do not own/participate on this Pako library.

[n1474335]

Hi @windhamwong, could you give some examples of bugs in zlib.js that cause incorrect decompression of gzipped data?

[windhamwong]

I cannot provide a good sample for you to test out, but when the payload is too large or in a particular situation (I didn't look into it), it won't decrypt and prompt you saying "Input buffer is broken". I looked up the issues in its github, it has loads of issues but no one has an idea to solve it.

Try to grab some HTTP response with gzip, you should see more than 50% of them are not working.

Probably it happens in HTTP only. I tried all the way to decrypt the payload (including ways by modifying codes), but none of the current operation can do the job.

[windhamwong]

Sample:
HTTP response from www.bbc.co.uk

485454502f312e3120323030204f4b0d0a782d616d7a2d69642d323a20597a71593739544856305a724939684c6c7a7374595055384d43576d55664b344645447a532b6e4659752f344a4e6b782f744f516d63394a374e55315438683734302f33513562515731513d0d0a782d616d7a2d726571756573742d69643a20343334443245343239384432314145380d0a4c6173742d4d6f6469666965643a205468752c203036204a756c20323031372030393a35303a333820474d540d0a455461673a20226234383436656331373732353338633532393264326538353563376139623162220d0a43616368652d436f6e74726f6c3a207075626c69632c206d61782d6167653d33313632323430300d0a4163636570742d52616e6765733a2062797465730d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a6176617363726970740d0a5365727665723a20416d617a6f6e53330d0a566172793a204163636570742d456e636f64696e670d0a436f6e74656e742d456e636f64696e673a20677a69700d0a446174653a205475652c203131204a756c20323031372030393a32383a343520474d540d0a5472616e736665722d456e636f64696e673a20206368756e6b65640d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a436f6e6e656374696f6e3a205472616e736665722d456e636f64696e670d0a4163636573732d436f6e74726f6c2d4d61782d4167653a203330300d0a4163636573732d436f6e74726f6c2d416c6c6f772d43726564656e7469616c733a2066616c73650d0a4163636573732d436f6e74726f6c2d416c6c6f772d486561646572733a202a0d0a4163636573732d436f6e74726f6c2d416c6c6f772d4d6574686f64733a20484541442c4745540d0a4163636573732d436f6e74726f6c2d416c6c6f772d4f726967696e3a202a0d0a0d0a30303030363030300d0a1f8b0800000000000000dcbdeb76db489230f87f9e82c2d72303a51445fa52d5050ad6e78baaec6edfc672754f1f959a07225322ca20c00f00256b249eb3afb1afb74fb21191176402099272b9667ab74fb545007989cc8c8c5b464678cb92f7caaa48269537ba8a8b1eff52f16c1a5d2cb34995e499cf5915dc5ee4858f1fb35e92f5aaa0eacfe2f2fd75f6a1c817bca86efc2cd8ddf5f969761655f04f302a78b52cb21e5fb1e4a48aabf2553ee78bf8921bcd06b7d8601589fefcdb15e301cba28ae59157c6e3343ee769e9b122f292129bd8cff2aa88279f3d9644de58bc1b7b7b6fe36ad62fe26c9acffd6054f5c749069fb2097f3d8d121a50cccae876b6f850f08ffcff2c7959853b83159b44b7d5cd82879ec796450a7f562c8d864f0603b68c86fc315b44f87b1a9d9e61a3936559e5f3e32b9e55e5477e09bd1737d1949abfb0a68a65f564e5d1f4949f26677d4ef5feca6fce4eab3318d180c551de4f797659cd46c5613c2af6f682e4c2f7544b5e142170f9452f3f2dce7677f15f98e3288a76868198db9da19c641ccc4c0ca60c6f3d9aa324bbbc8893b42f66c9830133fda15c4e26bc2cad6f059f26059f54d6cb499e7f4ef8a4e071c5a7c697158395482e6ec28b159bb706cffbf1744a13f506aaf08c1747ed573e14653094103e55553c99d1d723ebc9f7601ef6b0cd909f8adf6751b66257160e29448bf8dddd750258702d669bf17e151797bcbabbe3fdb2981ca77c0eaf57ec32ba159d8473069f3f51a1f06ac56eea76056af2e87635526b5921e2cffa34cb81fcdbdc0495d80400e6a9b505c61d10f70bbe48e309f70f7edd3bb864debf3f1c78019bf2493ee5bf7c7cfd229f2ff20c6782072b76ded891aa99fe248dcbf25d3ce7bbbbc643bf5ca409cc61cf0bfa302dfccbfb0b806f278af6872b766d0e553594c96e7d398d79710e005ef0a2802574bc0ba7f9648973aadf00942f1a5056112c80b72b284b869b893767cdcb005c0fa62eeb2f96e5ccd780d4e39745f6bcc8db737ce67dfc1e047ab1725c2c1eb4bacaa1172f3fff0d30dddb513b8c9fe6b0c37c2f4db2cf6ff2492cf75f941fad01080b8f5355ba1b32683b0842013f4c7e0efd67e5584d987cd5dd4fbea1e51acbdab36a74b36172cd926be6b82e1600a1eeff962719a0d48a1d37d65c2c351f4df35ba469cf7032b37e062d22624af2d5cb46f031834f952668abeb5992727809c0c605f4fa0eea40075facfda37724300b8ee002c3d0a828689642fe116c27a4b689496d13a2b6d44c1521611dc1264138fa93595c3cabfc41300a006ffbe5f21cf96276e90f5925eb0708f5804aab5d95056a488e9d6bb692c936eac682959a8c659aaed8e71631256466319bb014d8d2b17fd9d7240ba69e55b3a40c46cbdddd657f066b03583c89fc8c5ff75e02c90ea86c32e77e009c6d290938be801ff3c5eeee64bff5f270c1d2bbbb737fc98ae0eececf76774b60e7f82bfae48b4e60edf3e85d1fb97691c52970326271b8213e2151fc846c864fa11a83ea80dd3180d5ea289a30c56c22f5e3ee0e64807730afa265988325a013abfb926fe07f2b76e29e2b24d96a4bb8c0ea9b7200104c217be46c736118763fa60ec74838226f9226288ce8b7b8c7415691d0efebcf53a84ce40286ea5c9c1cfa195309104460367096e125ec37d1d12c994e79e689576a0b46d75013666e0ab36bf4914c713aa701d3237b09bc56c28a4bd737c91c2cacfd22f260c74cfad6005ffb19f3cc99f0d80b204c8da670a5dee344d262b07cc53ed62b84dcbe102b141b2b546dbf4231db5c38e828a396646363aa60c062fc4d137a51e4735a17c90115cd17ab5497eb5adc4c2c515d10562833572886158209c66fb840717381e2f602fd16bdf0636800c45212ce5e10d183c9ff8d2d0336eb0bf18cd6dd25c2c152b018765a743b4dca455c8128f499a93d169e30350de1c7158abe2437f2a28cde01bed183b5b00112790e6d1f67f179caa77e406f764a908302cd9da81e151f0179a79d9e81dc7029c6a87e46a79a909fe7d39bb380794ba0b21749c6a7b548ac4a134783b9ad6bab5f67862860f18e4b296002cd50e8ad3fc348def5d58c400144e915509a22d21d2a1eb2bf3f52d8acbe211ff900ab4282ce248a01c146933655441196463fe9cb797d4600f129ee455cd5d687686710ac562b6af893351a64a0a0ef94c79a3e02d1a637af350d55ccc9530b0c7aceeb06ed6439ee4eaee83137e9b17ad04389aac63b01ed2fb04532fb8bc6d9dc7e2f45f1a858b1970d54122b96ed0965c50311a20783ac50d621594270d21a15e88b46b2316cbb37a436fe04bbf64470ddb1a84e8c5bc97ecefaf85b4a20c8fd0d

This image shows that the basic decode of HEX to Plaintext.

So, the hex converts the hex to plaintext. We can see the gzip encoded data in the response. When we try to do "Strip HTTP Headers" and "GunZip" to decode Gzip payload, it appears failed because there are few numbers before the header of Gzip payload. This is common in HTTP traffic and it stops the Gunzip operation to decode due to the incorrect data in the first few bytes.

The operation I made supports searching for the gzip header, i.e. 1f8b0800000000000000. It starts decoding the gzip in the payload and still remain the HTTP header in the result. It helps analysts analyzing gzip HTTP response in a quicker way without modifying any input data.

[windhamwong]

@n1474335 Looking into imaya/zlib.js, it has multiple issues on non-Ascii language, i.e. Chinese, and large content decoding. I encountered the same issue when I was developing my operation as well. It often throws Input Buffer issue when the content is too large (Actually not large in practical usage, but zlib.js just can't handle).
Please see https://github.com/imaya/zlib.js/issues

[n1474335]

Hi @windhamwong, thanks for the extra information.

Rather than creating a new operation, it would make more sense to modify the current Gunzip operation as this effectively does the same thing. I would add an option that allows the user to specify whether there is an HTTP header or not and if there is, carry out the decompression in place as you have done in this one. If pako is a better option than zlib.js then we should be replacing it across the board rather than using both. zlib.js is currently used in several operations (all in src/core/operations/Compress.js) and it looks like pako could replace it in each case.

[windhamwong]

I am currently using my own recipe. I found out that sometimes zlib.js is better than pako, likely 30% of the time, however, I am still looking for a better solution to solve the issue of failing GZIP decoding.

[n1474335]

Closing due to inactivity. We can reopen this if a clear decision is reached over the future of zlib operations.