[vpdq] Issues with Continuous Integration - Unable to Fetch mesa-vdpau-drivers

[Dcallies]

Running into multiple problems trying to work with vpdq:

CI Failures

e.g. https://github.com/facebook/ThreatExchange/actions/runs/12297680927/job/34319368069

It's failing during apt-get time for some packages:

rr:93 http://security.ubuntu.com/ubuntu jammy-updates/universe amd64 mesa-va-drivers amd64 23.2.1-1ubuntu3.1~22.04.2
  404  Not Found [IP: 40.81.13.82 80]
Err:94 http://security.ubuntu.com/ubuntu jammy-updates/main amd64 mesa-vdpau-drivers amd64 23.2.1-1ubuntu3.1~22.04.2
  404  Not Found [IP: 40.81.13.82 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/universe/m/mesa/mesa-va-drivers_23.2.1-1ubuntu3.1%7e22.04.2_amd64.deb  404  Not Found [IP: 40.81.13.82 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mesa/mesa-vdpau-drivers_23.2.1-1ubuntu3.1%7e22.04.2_amd64.deb  404  Not Found [IP: 40.81.13.82 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

[Solved] Local Install Failure

To repo:

1. Fresh devcontainer
2. cd to vpdq
3. pip install .

This one is strange - looks like the wrong version of cmake is being used when cmake is called by subprocess:

Building wheels for collected packages: vpdq
  Building wheel for vpdq (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for vpdq (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [14 lines of output]
      INFO:root:running bdist_wheel
      INFO:root:running build
      INFO:root:running build_ext
      INFO:setup.py:Removing CPP build directory /workspaces/devcontainer-ThreatExchange/vpdq/cpp/build...
      INFO:setup.py:Removing compiled Cython files /workspaces/devcontainer-ThreatExchange/vpdq/python/vpdq.cpp...
      INFO:setup.py:Removing libraries-dir.txt file /workspaces/devcontainer-ThreatExchange/vpdq/cpp/libraries-dirs.txt...
      INFO:setup.py:Creating CPP build directory...
      INFO:setup.py:Generating CMake project...
      CRITICAL:setup.py:Traceback (most recent call last):
        File "/home/vscode/.local/bin/cmake", line 5, in <module>
          from cmake import cmake
      ModuleNotFoundError: No module named 'cmake'
      
      CRITICAL:setup.py:Failed to compile vpdq library.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for vpdq
Failed to build vpdq
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (vpdq)

The setup.py is incredibly suspect - we shouldn't be calling to subprocess at all imo.

Manually running the call in the right directory seems to succeed.

Solved: Fixed by adding cmake as a build dependency.

[Dcallies]

@ianwal - you appear to understand vpdq better than I do, any ideas?

[ianwal]

I see that you fixed the missing cmake dependency issue.

I'll look at the CI python lint install stage issue. Seems like running apt-get update before trying to apt-get install will fix it (see Github action documentation).

---

The setup.py is definitely hackish, but it's needed to link the generated Cython code with the libav libraries, if I recall.

An alternative is moving the build backend to something like scikit-build at some point, which is designed specifically for building Python bindings from other languages. This shouldn't be that hard if someone wants to do it. Maybe another option is out there too, but I've only used scikit-build. If I have some time off work soon I'll probably take a stab at it, since I don't want to leave this in a bad state.

Additionally, could also switch to pybind11 instead of Cython, which is way less finicky if you are familiar with C++, because pybind11 can expose C++ APIs to Python directly using C++ super easily.

Gotta love the C++ tooling ecosystem!