[python3] cmake FindPython3 Development component

[heydojo]

---

vcpkg does not present it's python3 port well to other ports and or there may be missing development files. cmake has built in support for detecting python3 and it is struggling to find the development component when python3 is installed via vcpkg.

For context, if you install python3 for Windows via the Visual Studio installer and a port you are building uses cmake and in the cmake files there is:

find_package(Python3 COMPONENTS Development)

https://cmake.org/cmake/help/v3.12/module/FindPython3.html

When the port is then built by vcpkg, cmake will pickup the python3 development component and the build will succeed.

If instead, you do not install python3 for Windows via the Visual Studio installer and simply add the dependency python3 to the CONTROL file of the port and attempt to build, if you ensure that:

vcpkg_find_acquire_program(PYTHON3)
get_filename_component(PYTHON3_DIR "${PYTHON3}" DIRECTORY)
set(ENV{PATH} "$ENV{PATH};${PYTHON3_DIR}")

is inside the portfile, then the python exe is picked up but the development component is not found at all by cmake and the build will fail.

If you then go on to look at the python3 port's installed files and compare them to the python for Windows installed files, it appears that there may be a large number of files useful for developing python applications missing from vcpkg's python3 port's installation.

Proposed solution
vcpkg_find_acquire_program(PYTHON3) might benefit from special logic to determine if the python3 port has been built and installed so that it can present what cmake expects to find.

The expected development files are revealed to cmake via vcpkg's python3 port's installation or a python3-dev port is created to ensure that those who only want the lib get only what they need but the development files can be had separately.

Describe alternatives you've considered
I tried the MS Visual Studio Installer Python for Windows and it's system integration was much better than vcpkg's. cmake had absolutely no problems. Obviously, that kind of support would be nice in vcpkg.

Additional context
Python's pretty big now. It's become the common language at large and it's popularity continues to grow. Python 2 has been officially announced as being depreciated at the start of next year, so there will be lots of work going on now where developers are porting to Python 3.

Although vcpkg is primarily a C++ package manager, the growing number of C++ Python 3 bindings and the way in which vcpkg is especially useful for reproducible builds makes Python 3 integration on par with the integration found in the Visual Studio Installer an especially interesting goal.

Whilst it could be argued to just use Visual Studio's Python 3 installation, it appears more suitable to me, to use the Python 3 port instead.

[heydojo]

I was attempting to build a vcpkg port of PySide2 and the advice is that when building, the same C++ library versions used to compile Qt, Python, llvm and PySide2 must all be the same, which is when I observed the python integration and what I think is something that might be improved in vcpkg. I wanted to avoid downloading pre-compiled binaries from here there and everywhere and just use vcpkg instead.

One downside about the Visual Studio for Windows Python installation I noticed is that pip doesn't work out of the box on Windows 10 and you need to use a command line switch to switch to using a user folder for installs due to permissions. Having to do so doesn't really make easy system wide integration of modules downloaded with pip.
That's maybe somewhere vcpkg could move ahead of Visual Studio's python3 integration. A working pip; right off the bat, simply by installing the python3 port could make a lot of python devs very happy (I would like to think.)

[penumbra23]

Any news on this?

I'm trying to use pybind11 through vcpkg and it install the python3 dependency. But after generating via CMake and building I get errors that Python3.h is missing.

After adding the header directory manually in VS2019, next is the linking error (python37_d.lib). Is this port not linking development files through vcpkg or am I missing something big here?

[heydojo]

@penumbra23

Is this port not linking development files through vcpkg or am I missing something big here?

vcpkg does not present it's python3 port well to other ports and or there may be missing development files.

The development files aren't being presented at all as far as I could tell. You are (honestly in my opinion) going to get much better results with python3 from the Visual Studio 2019 installer (select individual components.)

My report here is basically saying that python3 development using vcpkg is pretty much in a sorry state, whereas to the point that we can currently forget about any C++/python interopt libs building. The way it currently is I couldn't get PySide2 to build at all, although I will confess it was for an open source project I was forking and I personally prefer to keep scripting languages away from my C++ (however trite that my or may not seem.)

I know it's not the news you likely want to hear but that's the current status unfortunately.
I'd take a further look myself and try and fix it if I had the time and python wasn't completely foreign to me. I wouldn't know where to start honestly and python as a language is unfortunately not interesting to me at all.

Hope this helps; at least in some way.

[penumbra23]

Wow, that's sad to hear 😢

Nevermind, I can link it to my local installation of python through CMake but I wanted to automate our build pipelines to use ONLY vcpkg. I found a solution by linking to the python interpreter in vcpkg/downloads/tools directory, but seems to me it's just temporary and unstable in the long run...

Thnx @heydojo anyway!

[andreabedini]

Sad indeed. We had started with a build solution centered around vcpkg but might be pivoting away from it.