PySide2 issues due to compiler / SDK difference between Qt and Python in conda-forge

[rggjan]

Reopening #45 as I found out in the meantime that this is an environment / compilation issue and not an upstream Qt issue.

See also ContinuumIO/anaconda-issues#11297 and https://bugreports.qt.io/browse/PYSIDE-1083

PySide2 in conda-forge doesn't support dark mode and behaves very buggy with regards to repainting / updating widget. The reason seems to be that Qt enables some features (like dark mode) depending on the version of the OSX SDK it is linked against.

Qt in conda-forge / PySide2 seems to be linked against a different version than Python, thus disabling some features (like dark mode) and severely breaking other features (like updating the text of a QLabel when calling setText()).

Here my latest observation which led me to reopen #45:

I tried rebuilding different python versions from

https://github.com/conda-forge/python-feedstock

I rebuilt python 3.6.7 with this basic conda_build_config.yaml:

CONDA_BUILD_SYSROOT:
- optMacOSX10.9.sdk
DEBUG_C:
- yes

And had (as expected) the same issues as with the conda or conda-forge built python. However, adding:

c_compiler_version:
 - 9

(and making sure conda-forge was in the channels, as conda-forge provides clang-9)

fixes the original issue with the updates, although the dark mode is still not working. I suspect changing the OSX SDK or minimum required version would fix this as well...

So there seems to be some compatibility issue regarding the clang version Qt / PySide2 and Python is compiled with...

[patricksnape]

@jakirkham I'm a novice here - does updating the version of clang mean updating the whole libc/libc++ stack? Normally we would use xcode for these builds so shipping a specific version of clang falls outside my experience.

[jschueller]

I just rerendered in #47: the bot did add c_compiler_version=9, maybe updating will solve the issue.

[isuruf]

Can you try updating pyside2 and libc++ to the latest versions and see if this issue is still there?

[patricksnape]

Seems (at least from pypi) that 5.13.1 is the latest version and that is built here. @jschueller thanks for the rerender though it doesn't seem to fix the signalling issue we saw previously. What version of clang is Python 3.6 built with at the moment?

[jschueller]

oh its clang 4.0.1, I'll try to rerender the 3.6 branch

[rggjan]

Cool, thanks!

[jschueller]

@rggjan could you try with python 3.8 ? that version uses the newest compiler

[rggjan]

Just tried it. Unfortunately, there seems to be an issue with python 3.8 and Pyside2. I get the following error message when trying to run the Pyside2 example:

Traceback (most recent call last):
  File "example.py", line 23, in <module>
    class Test(QWidget):
TypeError: 'Shiboken.ObjectType' object is not iterable

which seems to be this issue:
https://bugzilla.redhat.com/show_bug.cgi?id=1750107

Any chance python 3.7 can be rebuilt with the newest compiler? 3.6 still seems to have issues with clang 9, AFAIK...

[isuruf]

See #46 (comment)

[rggjan]

See #46 (comment)

I currently have these packages installed (which I think are the latest), and the issue is definitely not solved:

python                    3.6.7             h8dc6b48_1004    conda-forge
pyside2                   5.13.1           py36ha8f7116_6    conda-forge
libcxx                    9.0.0                h89e68fa_1    conda-forge

[rggjan]

Since conda-forge/python-feedstock#275 was merged, I tried to rerun the test example with the rerendered python and the newest pyside:

python                    3.6.7             h4285619_1006    conda-forge
pyside2                   5.13.1           py36ha8f7116_6    conda-forge

Unfortunately, the issue still persists. I wonder why going to clang-9 solved the issue for me, but not for the python version built on the CI...?

[rggjan]

I couldn't reproduce building a working python 3.6.7 with just using clang-9, not sure why.

In any case, building with OSX sdk 10.14 or 10.15 fixes the issue. Buildig with anything lower than that (including the currently used 10.9) makes pySide2 unusable.

Is there any possibility of building python using a newer OSX SDK?

[rggjan]

I guess this is in line with what's written here:
https://doc.qt.io/qt-5/macos.html

which basically says Qt needs OSX 10.14 or 10.15 for building and at least 10.12 for running...

[jschueller]

I see qt feedstock requires 10.12, so that's should be raised ? Does pyside2 feedstock need also 10.14 ?

[rggjan]

From https://doc.qt.io/qt-5/macos.html, I understand that building qt requires 10.14, building an application that uses qt (which would probably be pyqt and python in our case?) requires 10.14 (or optionally 10.13) and running all of this stuff requires 10.12.

So if the qt feedstock MACOSX_DEPLOYMENT_TARGET is 10.12, this should probably be fine if it is meant for just running...

However, for building qt, pyside2 and python we would need to link against the 10.14 SDK to make qt work properly, I guess...

[rggjan]

I guess this is also why all of these ugly patches were added, like:

https://github.com/conda-forge/qt-feedstock/blob/9b93510ac3db6e7d62811f08683b1bb47c4878e1/recipe/patches/0008-qtbase-Define-kIOSurfaceSuccess-if-it-is-not-defined.patch

which try to make it compile with the 10.12 SDK. I don't see why this is needed, though, as compiling with the 10.14 SDK will still make it run on 10.12...

[jschueller]

do you know how we would require a feedstock to be built on a minimal osx version ?

I just know how to require a minimal runtime requirement:
https://github.com/conda-forge/qt-feedstock/blob/master/recipe/conda_build_config.yaml

[rggjan]

The OSX version doesn't matter... only the OSX SDK version. I have no idea how to specify the SDK version on the CI, though...

Locally, this can be changed by setting CONDA_BUILD_SYSROOT in conda_build_config.yaml and pointing it to the corresponding folder downloaded from https://github.com/phracker/MacOSX-SDKs, as described here: https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html

[isuruf]

as compiling with the 10.14 SDK will still make it run on 10.12...

Where did you read this? That's not how how SDKs work. You build with 10.12 and it will run on 10.14, not the other way around.

[rggjan]

Where did you read this? That's not how how SDKs work. You build with 10.12 and it will run on 10.14, not the other way around.

See on developer.apple.com:

It works both ways on OSX.

In the example of the link I posted, they build with base SDK 10.6 and run on 10.4 and newer. If the code doesn't use any of the newer API's (not existing in 10.6) this works out of the box. If the code wants to use new 10.5 or 10.6 API's on supported operating systems, the code must first check the OS version before using the API's. Which I guess Qt does, when they say on their official docs that compiling against the 10.14 SDK and then running on 10.12 works fine...

So Apple recommends always using the newest SDK, and then setting the Deployment target to the lowest OSX version you want to support...

[rggjan]

Which is why I would recommend building python against the 10.15 SDK and then (possibly) setting the Deployment target to 10.9 or whatever you still want to support...

[isuruf]

I stand corrected. We should rethink how we build OSX software. Can you open an issue in https://github.com/conda-forge/conda-forge-ci-setup-feedstock ?

[rggjan]

@isuruf Thanks, I just did...

[jakirkham]

If Qt requires a newer SDK (presumably due to newer features), telling it to build for an older OS (without those features) seems unlikely to work.

[jakirkham]

Furthermore I’m not sure what to make of that text. The OS versions that they single out are a bit atypical. For instance 10.6 is the last OS that used GCC exclusively (before Apple began moving to Clang). I don’t follow if they mean this will generally work for an arbitrary OS version or if these particular versions are the only ones we can use this way.

[rggjan]

If Qt requires a newer SDK (presumably due to newer features), telling it to build for an older OS (without those features) seems unlikely to work.

I'm not sure I agree. Assume you have feature X in SDK 10.15. When compiling for 10.15, they can do a:

If (X available)
 use X;
else
 fallback;

However, compiling against the older SDK where the symbol for X doesn't exist would create a build error.

The same thing happens with symbols used by newer OSX versions, like the one backported NSEventTypeScrollWheel in the patch here:
https://github.com/conda-forge/qt-feedstock/blob/9b93510ac3db6e7d62811f08683b1bb47c4878e1/recipe/patches/0007-qtwebengine-macOS-10.11-support-Define-NSEventTypeScrollWheel-as.patch

Not having this symbol on older OSX versions is no issue, as the OS will just never hand this event to the Qt code. But compiling it with an old SDK will not work, as the symbol doesn't exist (yet).

[rggjan]

Furthermore I’m not sure what to make of that text. The OS versions that they single out are a bit atypical. For instance 10.6 is the last OS that used GCC exclusively (before Apple began moving to Clang). I don’t follow if they mean this will generally work for an arbitrary OS version or if these particular versions are the only ones we can use this way.

I'm sure it applies in general. Setting a lower target might disable some features, see for example:
https://forum.juce.com/t/fix-for-using-c-11-features-with-osx-10-7-deployment-target/25247

Also, Qt clearly supports building with 10.14 and running on 10.12.

What might happen sometimes is that API's are deprecated for security reasons and removed in newer SDK's. This could then lead to build errors. Since I expect most projects (including Qt) to use a new (or the newest) SDK for compiling anyway (as this is what you get when you install a recent Xcode), I think this should not be an issue...

[jschueller]

is this still an issue @rggjan ?

[rggjan]

It is, still waiting for conda-forge/python-feedstock#371

[jschueller]

still happening @rggjan ?
looks super old

[rggjan]

Yes, seems to work as intended for newest PySide2 on python 3.8 and python 3.9, thanks!