second install of watchdog on non-macos incorrectly requires darwin dependencies #617

reversefold · 2020-01-27T18:28:23Z

If I run pip install watchdog on a clean python environment it works fine and doesn't install the pyobjc dependencies, although pip does output what look like warnings or errors:

watchdog 0.10.0 requires pyobjc-framework-Cocoa>=4.2.2, which is not installed.
watchdog 0.10.0 requires pyobjc-framework-FSEvents>=4.2.2, which is not installed.

If I run pip install watchdog again pip does see it as already installed but then tries to install the pyobjc dependencies and fails. Here's the full command output:

(_venv) vagrant@ubuntu-bionic:~/py3$ python --version
Python 3.6.9
(_venv) vagrant@ubuntu-bionic:~/py3$ pip install watchdog
Collecting watchdog
  Using cached https://files.pythonhosted.org/packages/01/01/cc9b2fd111e19687dd42155f066718bccb414c8a728eb7df0b229742caf7/watchdog-0.10.0.tar.gz
  Ignoring pyobjc-framework-Cocoa: markers 'sys_platform == "darwin"' don't match your environment
  Ignoring pyobjc-framework-FSEvents: markers 'sys_platform == "darwin"' don't match your environment
Collecting pathtools>=0.1.1 (from watchdog)
  Using cached https://files.pythonhosted.org/packages/e7/7f/470d6fcdf23f9f3518f6b0b76be9df16dcc8630ad409947f8be2eb0ed13a/pathtools-0.1.2.tar.gz
Building wheels for collected packages: watchdog, pathtools
  Running setup.py bdist_wheel for watchdog ... done
  Stored in directory: /home/vagrant/.cache/pip/wheels/d4/74/90/4e0d4d5050776ce9a03e8a2c873e523b0149bf6412546dfc3f
  Running setup.py bdist_wheel for pathtools ... done
  Stored in directory: /home/vagrant/.cache/pip/wheels/0b/04/79/c3b0c3a0266a3cb4376da31e5bfe8bba0c489246968a68e843
Successfully built watchdog pathtools
watchdog 0.10.0 requires pyobjc-framework-Cocoa>=4.2.2, which is not installed.
watchdog 0.10.0 requires pyobjc-framework-FSEvents>=4.2.2, which is not installed.
Installing collected packages: pathtools, watchdog
Successfully installed pathtools-0.1.2 watchdog-0.10.0
You are using pip version 18.0, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(_venv) vagrant@ubuntu-bionic:~/py3$ pip install watchdog
Requirement already satisfied: watchdog in ./_venv/lib/python3.6/site-packages (0.10.0)
Requirement already satisfied: pathtools>=0.1.1 in ./_venv/lib/python3.6/site-packages (from watchdog) (0.1.2)
Collecting pyobjc-framework-Cocoa>=4.2.2 (from watchdog)
  Downloading https://files.pythonhosted.org/packages/7a/99/bfac679ee346f344bca72ee315249e56f4f34bc974345f66d7810380ce7a/pyobjc-framework-Cocoa-6.1.tar.gz (3.9MB)
    100% |████████████████████████████████| 3.9MB 4.1MB/s
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-vnyl8c9t/pyobjc-framework-Cocoa/setup.py", line 25, in <module>
        extra_link_args=["-framework", "CoreFoundation"],
      File "/tmp/pip-install-vnyl8c9t/pyobjc-framework-Cocoa/pyobjc_setup.py", line 408, in Extension
        os_level = get_os_level()
      File "/tmp/pip-install-vnyl8c9t/pyobjc-framework-Cocoa/pyobjc_setup.py", line 218, in get_os_level
        pl = plistlib.readPlist("/System/Library/CoreServices/SystemVersion.plist")
      File "/usr/lib/python3.6/plistlib.py", line 162, in readPlist
        with _maybe_open(pathOrFile, 'rb') as fp:
      File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
        return next(self.gen)
      File "/usr/lib/python3.6/plistlib.py", line 120, in _maybe_open
        with open(pathOrFile, mode) as fp:
    FileNotFoundError: [Errno 2] No such file or directory: '/System/Library/CoreServices/SystemVersion.plist'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-vnyl8c9t/pyobjc-framework-Cocoa/
You are using pip version 18.0, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I have tested this with python 2.7 and 3.6 and the behavior is the same.

Additionally, if you try installing while the wheel version is cached, you also get the same error, even if watchdog is not installed:

(_venv) vagrant@ubuntu-bionic:~/py3$ pip install watchdog
Processing /home/vagrant/.cache/pip/wheels/d4/74/90/4e0d4d5050776ce9a03e8a2c873e523b0149bf6412546dfc3f/watchdog-0.10.0-cp36-none-any.whl
Collecting pyobjc-framework-FSEvents>=4.2.2
  Downloading pyobjc-framework-FSEvents-6.1.tar.gz (24 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/vagrant/py3/_venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/setup.py'"'"'; __file__='"'"'/tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/pip-egg-info
         cwd: /tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/
    Complete output (15 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/setup.py", line 20, in <module>
        ext_modules=[Extension("FSEvents._callbacks", ["Modules/_callbacks.m"])],
      File "/tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/pyobjc_setup.py", line 408, in Extension
        os_level = get_os_level()
      File "/tmp/pip-install-2slm3k7e/pyobjc-framework-FSEvents/pyobjc_setup.py", line 218, in get_os_level
        pl = plistlib.readPlist("/System/Library/CoreServices/SystemVersion.plist")
      File "/usr/lib/python3.6/plistlib.py", line 162, in readPlist
        with _maybe_open(pathOrFile, 'rb') as fp:
      File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
        return next(self.gen)
      File "/usr/lib/python3.6/plistlib.py", line 120, in _maybe_open
        with open(pathOrFile, mode) as fp:
    FileNotFoundError: [Errno 2] No such file or directory: '/System/Library/CoreServices/SystemVersion.plist'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Upgrading to the latest pip (20.0.2) exhibits the same error.

The text was updated successfully, but these errors were encountered:

BoboTiG · 2020-01-27T21:25:17Z

At first glance, it seems a pip issue.

This is weird. Do you reproduce the same behavior with watchdog 0.9.0?

gorakhargosh/watchdog#617

BoboTiG · 2020-01-29T07:26:24Z

From inside the same venv, could you tell me what returns:

import os
import sys
from distutils.util import get_platform

print(sys.platform)
print(os.getenv("_PYTHON_HOST_PLATFORM"))
print(get_platform())
print(os.name)
print(os.uname())

BoboTiG · 2020-01-29T11:14:12Z

Could you try to install the version from that branch: https://github.com/gorakhargosh/watchdog/tree/fix-use-sys-platform?

But I still do not understand what happended. This may not be fixed right now :/

reversefold · 2020-01-29T22:20:21Z

The same does not happen with watchdog-0.9.0, the problem only showed up when watchdog-0.10.0 was released and our builds started picking it up. I worked around the problem temporarily by making the requirement watchdog<0.10.0.

I agree, it does seem like a pip issue, but it only started with 0.10.0, so if we can figure out what change caused the issue we can submit an upstream bug.

BoboTiG · 2020-01-29T22:24:40Z

Could you try the same actions with the https://github.com/gorakhargosh/watchdog/tree/fix-use-sys-platform branch? Just to let me know it is fixed before merging.

reversefold · 2020-01-29T22:27:16Z

Te output you asked for:

(_venv) vagrant@ubuntu-bionic:~/py3$ python
Python 3.6.9 (default, Nov  7 2019, 10:44:02)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import sys
>>> from distutils.util import get_platform
>>>
>>> print(sys.platform)
linux
>>> print(os.getenv("_PYTHON_HOST_PLATFORM"))
None
>>> print(get_platform())
linux-x86_64
>>> print(os.name)
posix
>>> print(os.uname())
posix.uname_result(sysname='Linux', nodename='ubuntu-bionic', release='4.15.0-74-generic', version='#84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019', machine='x86_64')
>>>

Will try the branch next.

reversefold · 2020-01-29T22:36:51Z

Here's what I see if I checkout that branch, run python setup.py sdist, install from the sdist, then just ask for watchdog again:

(_venv) vagrant@ubuntu-bionic:~/py3/watchdog$ pip install dist/watchdog-0.10.1.tar.gz
Processing ./dist/watchdog-0.10.1.tar.gz
  Ignoring pyobjc-framework-Cocoa: markers 'sys_platform == "darwin"' don't match your environment
  Ignoring pyobjc-framework-FSEvents: markers 'sys_platform == "darwin"' don't match your environment
Requirement already satisfied: pathtools>=0.1.1 in /home/vagrant/py3/_venv/lib/python3.6/site-packages (from watchdog==0.10.1) (0.1.2)
Building wheels for collected packages: watchdog
  Building wheel for watchdog (setup.py) ... done
  Created wheel for watchdog: filename=watchdog-0.10.1-py3-none-any.whl size=74374 sha256=a261901569ff26cfeedb51be3ae0082c0b40fba859ca6dcb5d3fd7c753475053
  Stored in directory: /home/vagrant/.cache/pip/wheels/8e/05/d2/72562aa8a5f024f4b9172ada430aaae9a1d9c2782f14d7d20e
Successfully built watchdog
ERROR: watchdog 0.10.1 requires pyobjc-framework-Cocoa>=4.2.2, which is not installed.
ERROR: watchdog 0.10.1 requires pyobjc-framework-FSEvents>=4.2.2, which is not installed.
Installing collected packages: watchdog
Successfully installed watchdog-0.10.1
(_venv) vagrant@ubuntu-bionic:~/py3/watchdog$ pip install watchdog
Requirement already satisfied: watchdog in /home/vagrant/py3/_venv/lib/python3.6/site-packages (0.10.1)
Requirement already satisfied: pathtools>=0.1.1 in /home/vagrant/py3/_venv/lib/python3.6/site-packages (from watchdog) (0.1.2)
Collecting pyobjc-framework-Cocoa>=4.2.2
  Using cached pyobjc-framework-Cocoa-6.1.tar.gz (3.9 MB)
    ERROR: Command errored out with exit status 1:
     command: /home/vagrant/py3/_venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/pip-egg-info
         cwd: /tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/
    Complete output (15 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/setup.py", line 25, in <module>
        extra_link_args=["-framework", "CoreFoundation"],
      File "/tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/pyobjc_setup.py", line 408, in Extension
        os_level = get_os_level()
      File "/tmp/pip-install-s1xg2j3g/pyobjc-framework-Cocoa/pyobjc_setup.py", line 218, in get_os_level
        pl = plistlib.readPlist("/System/Library/CoreServices/SystemVersion.plist")
      File "/usr/lib/python3.6/plistlib.py", line 162, in readPlist
        with _maybe_open(pathOrFile, 'rb') as fp:
      File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
        return next(self.gen)
      File "/usr/lib/python3.6/plistlib.py", line 120, in _maybe_open
        with open(pathOrFile, mode) as fp:
    FileNotFoundError: [Errno 2] No such file or directory: '/System/Library/CoreServices/SystemVersion.plist'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Same error. If I install from the tarball a second time it does not do this, but the problem only seems to appear when the watchdog requirement is already satisfied.

I'm running this on an Ubuntu 18.04 vagrant box with a simple virtualenv.

reversefold · 2020-01-29T22:38:49Z

If I do the same from the v0.9.0 tag everything is fine:

(_venv) vagrant@ubuntu-bionic:~/py3/watchdog$ pip install dist/watchdog-0.9.0.tar.gz
Processing ./dist/watchdog-0.9.0.tar.gz
Collecting PyYAML>=3.10
  Downloading PyYAML-5.3.tar.gz (268 kB)
     |████████████████████████████████| 268 kB 1.5 MB/s
Collecting argh>=0.24.1
  Downloading argh-0.26.2-py2.py3-none-any.whl (30 kB)
Requirement already satisfied: pathtools>=0.1.1 in /home/vagrant/py3/_venv/lib/python3.6/site-packages (from watchdog==0.9.0) (0.1.2)
Building wheels for collected packages: watchdog, PyYAML
  Building wheel for watchdog (setup.py) ... done
  Created wheel for watchdog: filename=watchdog-0.9.0-py3-none-any.whl size=72168 sha256=61d339694654397a0284a9fd1306efb2ff2655ea89fbaaed5c2ac676733a3a88
  Stored in directory: /home/vagrant/.cache/pip/wheels/6e/d0/84/b7ef2971a913a2deb77848a835d8aae5080af52a35c0a29493
  Building wheel for PyYAML (setup.py) ... done
  Created wheel for PyYAML: filename=PyYAML-5.3-cp36-cp36m-linux_x86_64.whl size=45522 sha256=53be4d72e432d2133fdf2c65748095e3bd48ae05a26176ea26a62dda3f7c91a2
  Stored in directory: /home/vagrant/.cache/pip/wheels/b1/86/0d/10e6c39d3a2b85ba807d7657ee80f08cc16c03f2aa2adf8e46
Successfully built watchdog PyYAML
Installing collected packages: PyYAML, argh, watchdog
Successfully installed PyYAML-5.3 argh-0.26.2 watchdog-0.9.0
(_venv) vagrant@ubuntu-bionic:~/py3/watchdog$ pip install watchdog
Requirement already satisfied: watchdog in /home/vagrant/py3/_venv/lib/python3.6/site-packages (0.9.0)
Requirement already satisfied: argh>=0.24.1 in /home/vagrant/py3/_venv/lib/python3.6/site-packages (from watchdog) (0.26.2)
Requirement already satisfied: PyYAML>=3.10 in /home/vagrant/py3/_venv/lib/python3.6/site-packages (from watchdog) (5.3)
Requirement already satisfied: pathtools>=0.1.1 in /home/vagrant/py3/_venv/lib/python3.6/site-packages (from watchdog) (0.1.2)

Note the lack of the ERROR output on the first install of 0.10.x as well.

reversefold · 2020-01-29T22:47:43Z

From the fix-use-sys-platform branch if I just comment out the pyobjc requirements in install_requires it fixes the problem for me.

These seem like perfectly valid conditional requirement constructs so I still think there's a pip bug to file here.

BoboTiG · 2020-01-29T22:53:05Z

I was thinking of moving the requirements to a requierements.txt file. But the thing is that those requirements were not defined in 0.9.0. So perhaps are they not needed now as they were never hardly requested.

BoboTiG · 2020-01-29T23:56:16Z

Do you mind retrying with the same branch? I force pushed on it.

breid1313 · 2020-01-30T16:14:35Z

I am seeing the same issue here building RHEL containers. The interesting piece is that on RHEL7 I get the installation error that @reversefold posted above. However, on RHEL8 watchdog appears to install just fine, but my code breaks further down the line.

Everything behaves as expected if I pin watchdog to 0.9.0

BoboTiG · 2020-01-30T16:43:52Z

Same question: could you tell me if the version from the fix-use-sys-platform branch works for you please?

breid1313 · 2020-01-30T16:57:03Z

That worked for me

Step 6/19 : COPY ./dist /tmp/packages
 ---> 514c8d5fa099
Step 7/19 : RUN pip install /tmp/packages/watchdog-0.10.1.tar.gz
 ---> Running in f41910d2b163
Processing /tmp/packages/watchdog-0.10.1.tar.gz
Building wheels for collected packages: watchdog
  Building wheel for watchdog (setup.py): started
  Building wheel for watchdog (setup.py): finished with status 'done'
  Created wheel for watchdog: filename=watchdog-0.10.1-py3-none-any.whl size=74335 sha256=4d79f96474d38cd114926837b838dcf4090fddd73df6cb63c7b25eb5b8cfaf07
  Stored in directory: /tmp/pip-ephem-wheel-cache-39p2a13c/wheels/fb/62/b0/3032ed076bd75e0ae5187a1e0cafb4ff3fdef28f3e5ea21c5c
Successfully built watchdog
Installing collected packages: watchdog
Successfully installed watchdog-0.10.1
Removing intermediate container f41910d2b163
 ---> 3178af77fdc6

BoboTiG · 2020-01-30T17:01:27Z

Excellent. I merge the fix and plan the new release. Thanks everyone 💪

jontwo · 2020-01-31T09:11:22Z

I reported this as a pip issue #7673 yesterday, as it worked the first time, but failed when installing from cache. Good to see it has been resolved though. Thanks!

BoboTiG · 2020-01-31T09:31:46Z

Thanks for the report on pip @jontwo 👍

reversefold · 2020-01-31T22:01:38Z

I've had odd errors with dependencies in the past with python version markers when they're in the install_requires list. When I moved those to extras_require it fixed the problem but that may or may not fix this issue. I would recommend opening a pip ticket if you haven't already.

Example of using extras_require:

    extras_require={
        ':python_version < "3"': [
            'reversefold.util>=1.15.4,<3.0.0',
        ],
        ':python_version >= "3"': [
            'reversefold.util>=3.0.0',
        ]
    },

reversefold · 2020-01-31T22:02:34Z

Ah, the page wasn't showing me the latest comments. Glad there's a pip ticket.

reversefold · 2020-02-04T17:23:01Z

I also tried updating setuptools and it fixed the problem in my python3 test environment but our builds are still python 2.7 and we had to peg earlier versions of setuptools and pip to keep things working until we can upgrade.

greenape added a commit to Flowminder/FlowKit that referenced this issue Jan 28, 2020

Pin watchdog

d87814c

gorakhargosh/watchdog#617

This was referenced Jan 29, 2020

Installation fails on Linux (watchdog latest version 0.10.0, python 3.6) #619

Closed

Fix the build_ext command on macOS #622

Merged

BoboTiG added this to the 0.10.1 milestone Jan 30, 2020

BoboTiG added the bug label Jan 30, 2020

BoboTiG closed this as completed in #622 Jan 30, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

second install of watchdog on non-macos incorrectly requires darwin dependencies #617

second install of watchdog on non-macos incorrectly requires darwin dependencies #617

reversefold commented Jan 27, 2020

BoboTiG commented Jan 27, 2020 •

edited

Loading

BoboTiG commented Jan 29, 2020 •

edited

Loading

BoboTiG commented Jan 29, 2020

reversefold commented Jan 29, 2020

BoboTiG commented Jan 29, 2020

reversefold commented Jan 29, 2020

reversefold commented Jan 29, 2020

reversefold commented Jan 29, 2020

reversefold commented Jan 29, 2020

BoboTiG commented Jan 29, 2020

BoboTiG commented Jan 29, 2020

breid1313 commented Jan 30, 2020 •

edited

Loading

BoboTiG commented Jan 30, 2020

breid1313 commented Jan 30, 2020 •

edited

Loading

BoboTiG commented Jan 30, 2020

jontwo commented Jan 31, 2020

BoboTiG commented Jan 31, 2020

reversefold commented Jan 31, 2020

reversefold commented Jan 31, 2020

reversefold commented Feb 4, 2020

second install of watchdog on non-macos incorrectly requires darwin dependencies #617

second install of watchdog on non-macos incorrectly requires darwin dependencies #617

Comments

reversefold commented Jan 27, 2020

BoboTiG commented Jan 27, 2020 • edited Loading

BoboTiG commented Jan 29, 2020 • edited Loading

BoboTiG commented Jan 29, 2020

reversefold commented Jan 29, 2020

BoboTiG commented Jan 29, 2020

reversefold commented Jan 29, 2020

reversefold commented Jan 29, 2020

reversefold commented Jan 29, 2020

reversefold commented Jan 29, 2020

BoboTiG commented Jan 29, 2020

BoboTiG commented Jan 29, 2020

breid1313 commented Jan 30, 2020 • edited Loading

BoboTiG commented Jan 30, 2020

breid1313 commented Jan 30, 2020 • edited Loading

BoboTiG commented Jan 30, 2020

jontwo commented Jan 31, 2020

BoboTiG commented Jan 31, 2020

reversefold commented Jan 31, 2020

reversefold commented Jan 31, 2020

reversefold commented Feb 4, 2020

BoboTiG commented Jan 27, 2020 •

edited

Loading

BoboTiG commented Jan 29, 2020 •

edited

Loading

breid1313 commented Jan 30, 2020 •

edited

Loading

breid1313 commented Jan 30, 2020 •

edited

Loading