Compress each file in a ThreadPool

[rik]

fix #148

---

I've tested this on a 2020 M1 MacBook Air (4 big + 4 small cores). In milliseconds.

Executable	Files	Brotli	Sequential	Parallel	Win
CLI	4656	No	3190	1050	3.0x
CLI	4656	Yes	78800	17630	4.2x
CLI	783	No	674	299	2.2x
CLI	783	Yes	10740	3630	2.9x
collectstatic	4656	No	8860	6790	1.3x
collectstatic	4656	Yes	82410	25110	3.3x
collectstatic	783	No	1510	1370	1.1x
collectstatic	783	Yes	11370	5120	2.2x

Commands to test yourself

• CLI without Brotli

  time python -m whitenoise.compress --quiet --no-brotli staticfiles

• CLI with Brotli

  time python -m whitenoise.compress --quiet staticfiles

• collectstatic without Brotli

  rm -rf staticfiles/; pip uninstall brotli --yes; time ./manage.py collectstatic --no-input

• collectstatic with Brotli

  rm -rf staticfiles/; pip install brotli; time ./manage.py collectstatic --no-input

[rik]

A previous version of this patch used ProcessPoolExecutor but that resulted in a slight slowdown for some non-Brotli cases.

[rik]

@evansd @adamchainz Apologies for pinging you. May I get some feedback on this approach?

[stumpylog]

It would be great to have this merged. I've patched in a Docker build, where it was made the build much faster, but it would be best to have it built in instead.

[petrprikryl]

Some bench for my project. Tested on i9-11900 (8 cores, 2 threads per core).

Without ThreadPool:

1037 static files copied to '/app/staticfiles', 4137 post-processed.

real    1m37.004s
user    1m35.530s
sys     0m1.356s

With ThreadPool:

1037 static files copied to '/app/staticfiles', 4137 post-processed.

real    0m28.889s
user    3m4.132s
sys     0m2.164s

In CI it goes from 213s to 38s 🚀

[adamchainz]

Thank you very much @rik . I eventually got round to checking this.

I did some refactoring, particularly to avoid adding extra methods that are only needed for this compression step. Using inner functions avoids bloating the classes :)

It would be nice to avoid the duplication between the different sites, but I don't have the energy to figure that out right now.

I checked that current tests would fail if any of the compression didn't work. I don't think there's an easy way to check the executor is used or works correctly, but at least the basics seem to pass.

Merging and releasing now.

[Archmonger]

@adamchainz A lot of cleanup for this PR occurred on ServeStatic. Could be worth a follow up PR.

EDIT: Looks like similar solutions were taken in your clean up, so can be skipped.

[petrprikryl]

Strange, after yesterday's:

force-pushed the multiprocess branch from 32e2611 to 511b57e

I have it 6x slower on my project: 1211 static files copied to '/app/staticfiles', 4971 post-processed.

I've tested also the newest release (6.8.1) and older 6.7.0 (before merge) with same results. No gains.

[Archmonger]

@petrprikryl I'm curious if it's related to the way the executor is written.

It's written a bit differently on ServeStatic. Would you mind testing ServeStatic==2.1.1?

If so then I can PR the fixes upstream.

Additionally, what operating system are you executing on?

[adamchainz]

I was also wondering if the default of up to 32 threads could lead to detrimental performance. Can you try with 4 or 8 threads on your project? Add max_workers= to the ThreadPoolExecutor creation.

[Archmonger]

Good thought - probably makes more sense for it to be CPU_CORES * 2

[Archmonger]

Just double checked standard lib, the default isn't 32.

Default is min(32, (os.cpu_count() or 1) + 4)

[graingert]

I wonder if the problem is materializing , 4971 futures all in one go and waiting on them all.

[Archmonger]

Creating the future itself should be an O(1) operation since it just enqueues it within the executor.

Would be a bit strange if doing the equivalent to appending objects to a list had a significant performance impact, but I've been proven wrong before 😅

[petrprikryl]

After my testing the number of threads isn't related to this. Looks like problem are generators:

This is fast:

    def _compress_one(self, name: str) -> list[tuple[str, str]]:
        compressed: list[tuple[str, str]] = []
        path = self.path(name)
        prefix_len = len(path) - len(name)
        for compressed_path in self.compressor.compress(path):
            compressed_name = compressed_path[prefix_len:]
            compressed.append((name, compressed_name))
        return compressed

This is 6x slower:

        def _compress_path(path: str) -> Generator[tuple[str, str]]:
            full_path = self.path(path)
            prefix_len = len(full_path) - len(path)
            for compressed_path in self.compressor.compress(full_path):
                compressed_name = compressed_path[prefix_len:]
                yield (path, compressed_name)

Tested on i9-11900 (8 cores, 2 threads per core).
Ubuntu 24.04.1 LTS
python:3.13-slim-bookworm Docker image

[adamchainz]

Just double checked standard lib, the default isn't 32.

Default is min(32, (os.cpu_count() or 1) + 4)

Yes, sorry, was taking a shortcut. That works out as 32 on most machines (for now!).

[akx]

Yes, sorry, was taking a shortcut. That works out as 32 on most machines (for now!).

That's min(), not max() 😄 On my Macbook (M2 Max),

>>> import os
>>> min(32, (os.cpu_count() or 1) + 4)
16