Creation of a new virtuelenv with Python 3.3 fails

[michaelklishin]

When trying to provision 3.3 RC1 [upd.: and later RC2] for travis-ci.org, I am hitting the following (Chef output):
https://gist.github.com/c47e2107af831c83391b

I am not familiar with the Python ecosystem enough to understand if this qualifies as a virtualenv problem but decided to file this issue. If there are workarounds, I think we can tweak things just for 3.3.

[jezdez]

Is that the latest version of virtualenv/pip?

[pnasrat]

This looks similar to the issues I addressed on OS X for 3.3, which should be fixed in latest virtualenv/pip.

You probably need the virtualenv on the base image to be 1.8.1 - if you need a PPA or whatever for that, then I'm not sure where to look but it should be easy to package.

ISTR It's possible to create a travis vagrant instance from your OSS repos - can you point me at a scriptable reproducer and I can poke more.

[pnasrat]

I'm going to hop on IRC to see if we can get a quicker turnaround

[pnasrat]

Full logs: https://gist.github.com/3dd90923365081565e9e

Looks like possibly something odd in the new ubuntu packaging with the minimal package

PKG source https://launchpad.net/~python-dev/+archive/ppa

  Traceback (most recent call last):
    File "setup.py", line 220, in <module>
      scripts = scripts,
    File "/usr/lib/python3.3/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/lib/python3.3/distutils/dist.py", line 917, in run_commands
      self.run_command(cmd)
    File "/usr/lib/python3.3/distutils/dist.py", line 935, in run_command
      cmd_obj.ensure_finalized()
    File "/usr/lib/python3.3/distutils/cmd.py", line 107, in ensure_finalized
      self.finalize_options()
    File "build/src/setuptools/command/install.py", line 29, in finalize_options
      _install.finalize_options(self)
    File "/usr/lib/python3.3/distutils/command/install.py", line 334, in finalize_options
      (prefix, exec_prefix) = get_config_vars('prefix', 'exec_prefix')
    File "/home/travis/virtualenv/python3.3/lib/python3.3/distutils/__init__.py", line 92, in sysconfig_get_config_vars
      real_vars = old_get_config_vars(*args)
    File "/usr/lib/python3.3/distutils/sysconfig.py", line 505, in get_config_vars
      func()
    File "/usr/lib/python3.3/distutils/sysconfig.py", line 449, in _init_posix
      from _sysconfigdata import build_time_vars
    File "/usr/lib/python3.3/_sysconfigdata.py", line 6, in <module>
      from _sysconfigdata_m import *
  ImportError: No module named '_sysconfigdata_m'

[michaelklishin]

The package we use wasn't very heavily tested but the maintainer is very kind and I think we can ask him to rebuild it if we identify what is missing/wrong.

[hukka]

You are hitting virtualenv bug #301 in combination of either http://bugs.python.org/issue15822 (3.3rc1) or http://bugs.python.org/issue15890 (rc2).

[carljm]

Thanks @tpievila for the additional pointers. #301 is an error-reporting/debugging improvement, not something that will cause a failure itself; so if that's correct this is not something we can fix in virtualenv. Fortunately it looks like http://bugs.python.org/issue15822 is fixed in 3.3rc2, and http://bugs.python.org/issue15890 is not hard to work around if needed. Closing for now, unless more information is provided indicating this is a virtualenv bug.

[michaelklishin]

@carljm does it mean that using RC2 will solve the issue? We are not Python experts and working around Python issues is not necessarily something we can do. We would also prefer to not modify virtualenv in any way.

[michaelklishin]

The issue is still present with Python 3.3 RC2. Log with RC2. What our next steps should be? We cannot provide Python 3.3 on travis because of this :( Maybe some Python hackers would be interested in helping us a little bit?

[michaelklishin]

We use a different PPA for 3.3 this time, by the way.

[carljm]

http://bugs.python.org/issue15822 should be fixed in rc2, and on further review it seems unlikely to me that you are hitting http://bugs.python.org/issue15890, since the consensus on that bug seems to be that it's not a regression, so if your umask was set too strictly you'd have seen that problem on earlier Pythons as well (also, not likely that a PPA build would be broken in that way). So it seems likely that @tpievila 's analysis was incorrect.

Reviewing this thread, I don't see confirmation anywhere of the version of virtualenv/pip you are using, @michaelklishin ?

[michaelklishin]

pip 1.2.1, virtualenv 1.8.2.

[carljm]

@michaelklishin I notice in your log that you're running virtualenv over top of an existing virtualenv directory; can you try rm -rf /home/travis/virtualenv/python3.3 and then rerun the virtualenv-creation? Also, can you run with -vvv? Thanks.

(If you're working on this now, I'm also carljm on Freenode in the #pip IRC channel.)

[michaelklishin]

Log with -vvv

[pnasrat]

http://packages.ubuntu.com/quantal/i386/libpython3.3-minimal/filelist

Gives /usr/lib/python3.3/plat-i386-linux-gnu/_sysconfigdata_m.py I'm betting that's not in sys.path

https://wiki.ubuntu.com/MultiarchSpec

[michaelklishin]

@pnasrat here's what sys.path returns: https://gist.github.com/521958c163fa9626dfed

[michaelklishin]

travis@precise32:~$ ls -lh /usr/lib/python3.3/plat-i386-linux-gnu/
total 32K
-rw-r--r-- 1 root root 5.9K Sep 17 14:57 IN.py
drwxr-xr-x 2 root root 4.0K Sep 17 17:31 __pycache__
-rw-r--r-- 1 root root  19K Sep 17 14:58 _sysconfigdata_m.py

[pnasrat]

Note for self/others - http://bazaar.launchpad.net/~doko/python/pkg3.3-debian/files has the patches, etc.

[michaelklishin]

@Mezzle suggests there is a patch in Quantal that our package did not include. We are waiting for another build to finish to try it out with supposedly full patch set.

[pnasrat]

oops fat fingered.

[carljm]

@michaelklishin What is the value of sys.path inside the virtualenv? (i.e. /home/travis/virtualenv/python3.3/bin/python -c "import sys; print(sys.path)")

[michaelklishin]

travis@precise32:~$ /home/travis/virtualenv/python3.3/bin/python -c "import sys; print(sys.path)"
['', '/home/travis/virtualenv/python3.3/lib/python3.3', '/home/travis/virtualenv/python3.3/lib/python3.3/plat-i386-linux-gnu', '/home/travis/virtualenv/python3.3/lib/python3.3/lib-dynload', '/usr/lib/python3.3', '/home/travis/virtualenv/python3.3/lib/python3.3/site-packages']

[carljm]

Yes, @pnasrat 's hunch was correct; we need to add a new special case in virtualenv's site.py to check if we're on a system using this multi-arch spec and add the system multi-arch-specific plat dir to sys.path. Thanks @michaelklishin for the debugging help.

[pnasrat]

I'm guessing this is still occuring - @carljm do you have time for this, else I'lll try take a poke on the weekend

[carljm]

@pnasrat If I have time it won't be until mid-next-week at the earliest; if you have time this weekend that'd be great.

[pnasrat]

Sure - I should have time then, I'm at conference during the week.

[michaelklishin]

Any progress on this? can someone who is not a virtualenv core contributor submit a fix? travis-ci.org users are already asking for 3.3.

[g2p]

@carljm I think it's more a question of symlinking the plat-$(MULTIARCH) dir when building the virtualenv.
The path is built correctly where I am (from Quantal's python3.3, which travis will get backported in due course), but the directory it refers to doesn't exist. It's part of the platform-dependent sys.path default, which exists before site.py files are read (confirmed with python3.3 -ESsc 'import sys; print(sys.path)' inside and outside the virtualenv).

[michaelklishin]

Were there any Virtualenv releases that include this fix? We still cannot provide Python 3.3 on travis-ci.org because of this issue.

[carljm]

I don't think any releases have included this fix yet. It could certainly be worth a bugfix release (@jezdez?) but in the meantime you should be able to use master virtualenv without negative consequences.

[pnasrat]

+! for a release.