Upgrade Apache Commons Compress to 1.27.1

[wladimirleite]

Processing a 7z of ~7 GB, another user here got several errors.
Testing the archive (with the 7-zip program), it seems fine.
After investigating other possibilities, I tried to use the latest version of Apache Commons Compress (1.24.0), and the processing finished successfully (currently, version 1.21 is used).

@lfcnassif, I know that upgrading this kind of dependency is critical, so I will leave this issue as a suggestion to be evaluated.

[lfcnassif]

Hi @tc-wleite! I think we can upgrade this and other dependencies on master, targeting 4.2.0. I already plan to upgrade Tika for 4.2.0 and I should run a large regression test before releasing it.

[wladimirleite]

I noticed that #2091 pointed out a possible vulnerability in 1.21, recommending the upgrade to 1.26.

[lfcnassif]

There are 2 fixed CVEs in 1.26, CVE-2024-26308 (OOME) is worse to me, let's upgrade. Sorry @wladimirleite, I missed this in the queue.

[lfcnassif]

Reopening, while testing #2093, I detected in a small test a lot of *.ar (application/x-archive) files stopped to be expanded. I'll test Compress 1.24 and 1.25 to see if that *.ar regression disappears, but I think we should do a large test before this upgrade.

[lfcnassif]

CommonsCompress-1.25 doesn't have the *.ar regression I detected. I'll downgrade to it on master, but I'll leave this open until we run a larger test to see if there are other regressions.

[wladimirleite]

Reopening, while testing #2093, I detected in a small test a lot of *.ar (application/x-archive) files stopped to be expanded.

That is bad news.

CommonsCompress-1.25 doesn't have the *.ar regression I detected. I'll downgrade to it on master, but I'll leave this open until we run a larger test to see if there are other regressions.

Are the vulnerabilities solved in 1.25?

[lfcnassif]

Are the vulnerabilities solved in 1.25?

I think no. Although non ideal, we have timeouts to protect against infinite loops (DoS) and external parsing to isolate OOMEs. Fortunately there is no (known) Remote Code Execution vulnerability.

[lfcnassif]

Hopefully

I meant Fortunately.

[wladimirleite]

If the only problem with 1.26 is with AR, maybe one option would be using SevenZipParser instead of PackageParser.

[lfcnassif]

If the only problem with 1.26 is with AR, maybe one option would be using SevenZipParser instead of PackageParser.

That worked!

[wladimirleite]

@lfcnassif, working on a real case, very important valid ZIP files were not expanded. Fortunately, I noticed the parser exceptions and tried to figure out what was going on.
Using the latest version of common-compress (1.27.1) solved the issue.
I know that there are a lot of things to deal with before releasing 4.2, but upgrading this library seems very important.

[lfcnassif]

Hi @wladimirleite, thank you for reporting that. ZIP is much more important to handle correctly than AR. I'll upgrade when I have some time. But could you test the original 7zip sample referenced by the first post here with commons compress 1.27.1?

[wladimirleite]

But could you test the original 7zip sample referenced by the first post here with commons compress 1.27.1?

Sure! I will try with that 7z.

[lfcnassif]

One question: does 4.1.6 version handle those valid zip files correctly?

[wladimirleite]

One question: does 4.1.6 version handle those valid zip files correctly?

No, it doesn't.
I tested the 7z I mentioned when I opened this issue using common compress 1.27.1, and it worked fine.
However, I found other ZIPs (from the some source I mentioned yesterday), which still fail using 1.27.1 (they also fail using previous versions). Not sure what is causing this, as they are opened with other tools. I am investing it a bit further, and will update here.

[wladimirleite]

One question: does 4.1.6 version handle those valid zip files correctly?

I am investing it a bit further, and will update here.

I found out that SevenZipParser does handle the ZIPs I mentioned (PackageParser doesn't).

@lfcnassif, as compressed files are quite important (may contain a lot of files), have you ever considered using a MultipleParser?At least for some more common types of archives, and only using the second parser if the first fails.
I noticed MultipleParser already has a stopAfterSomeParserWorks, although it seems currently not used.

[lfcnassif]

I found out that SevenZipParser does handle the ZIPs I mentioned (PackageParser doesn't).

Great!

@lfcnassif, as compressed files are quite important (may contain a lot of files), have you ever considered using a MultipleParser?

Not for generic containers, but for specialized ones, yes, e.g. for DOCX files run PackageParser if OOXMLParser fails (running RawStringParser over ZIP files doesn't extract too much info), for DOC files run GenericOLEParser if OfficeParser fails, etc. A possible bad side effect would be extracting duplicated subitems if first parser fails after extracting a subset of them...

I noticed MultipleParser already has a stopAfterSomeParserWorks, although it seems currently not used.

Yes, I put it there thinking about use cases like above, but I never tested it. Taking a quick look at the code, at least the tikaException = null at line 124 could be moved to before the break...

[lfcnassif]

Running a small to medium size regression test between compress 1.25.0 and 1.27.1...

[lfcnassif]

Tests seem fine, there are very minor differences for good and for bad, but the critical *.ar regression I detected with compress 1.26.0 is gone with 1.27.1.

[lfcnassif]

@wladimirleite, please feel free to open another issue to tackle the problem of some ZIPs still not being handled by compress-1.27.1, maybe using a MultipleParser.

[wladimirleite]

@wladimirleite, please feel free to open another issue to tackle the issue of ZIPs still not being handled by compress-1.27.1, maybe using a MultipleParser.

Great! For now upgrading to 1.27.1 will already solve a lot of cases.