Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow passing an interpreter to pex.build_pex #277

Closed
wants to merge 1 commit into from
Closed

Allow passing an interpreter to pex.build_pex #277

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions pex/bin/pex.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -457,9 +457,10 @@ def interpreter_from_options(options):
return interpreter


def build_pex(args, options, resolver_option_builder):
with TRACER.timed('Resolving interpreter', V=2):
interpreter = interpreter_from_options(options)
def build_pex(args, options, resolver_option_builder, interpreter=None):
Copy link
Contributor

@kwlzn kwlzn Jun 8, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little worried about the implicit lack of interpreter setup with this proposed change - seems to me you'd either have to do that manually or call interpreter_from_options anyway.

so.. why not just pass options.python in build_pex(options, ...) with a fully qualified path to the preferred interpreter to achieve the same behavior?

this would provide interpreter setup for ~free and works in today's API.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The short answer is that I wasn't able to get that to work and I'm unsure why. :) I'm far from a pex or even python expert and the previous author of the code in question did a lot of setup work to properly initialize the interpreter. See https://github.com/twitter/heron/blob/master/3rdparty/pex/_pex.py#L135-L152

I would love to remove as much of that code as possible and simplify the integration, but I wasn't able to do so and have the build work both locally (darwin) and on CI (centos). I spent a few days trying to rip parts of that code out but was met with multiple failures that were had to diagnose. Passing the interpreter was the only way I could get the build to work.

If you're curious you can reproduce the heron build by running the command below. Or if you're up for pairing a bit on this I can revisit and we can figure out where the friction is. Or maybe it's just an easy fix that I was overlooking.

$ heron --config=darwin build heron/...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

apologies - this fell off my radar due to EOQ activity, but circling back now.

where might I find the heron command referenced in your comment above?

Copy link
Contributor

@kwlzn kwlzn Jul 7, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah, assuming you meant bazel, I seem to get a failure just trying to get to an initial baseline run:

[illuminati heron (master)]$ git pull
Already up-to-date.
[illuminati heron (master)]$ git rev-parse HEAD
4a61bae0ac030f75c24e7377ef01ffbe9e495ca2
[illuminati heron (master)]$ bazel build --config=darwin heron/...
INFO: Found 335 targets...
ERROR: /Users/kwilson/dev/heron/heron/shell/src/python/BUILD:17:1: null failed: _pex failed: error executing command bazel-out/local_darwin-fastbuild/bin/third_party/pex/_pex --entry-point heron.shell.src.python.main bazel-out/local_darwin-fastbuild/bin/heron/shell/src/python/heron-shell.pex ... (remaining 1 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
pex options: {'use_wheel': False, 'inherit_path': False, 'script': None, 'python': '/usr/bin/python2.7', 'repos': [PyPIFetcher('https://pypi.python.org/simple/')], 'pex_name': None, 'pypi': True, 'requirement_files': [], 'platform': 'macosx-10.11-intel', 'cache_dir': '/Users/kwilson/.pex/build', 'entry_point': 'heron.shell.src.python.main', 'verbosity': 0, 'zip_safe': True, 'interpreter_cache_dir': '/Users/kwilson/.pex/interpreters', 'ignore_errors': False, 'cache_ttl': None, 'python_shebang': None, 'always_write_cache': False, 'find_links': ''}
pex requirements: ['requests==2.3.0', 'tornado==4.0.2']
Traceback (most recent call last):
  File "/private/var/tmp/_bazel_kwilson/248c4d7c0e2d8586f9fa08f1b261a0ae/heron/bazel-out/local_darwin-fastbuild/bin/third_party/pex/_pex.runfiles/third_party/pex/_pex.py", line 208, in <module>
    sys.exit(main())
  File "/private/var/tmp/_bazel_kwilson/248c4d7c0e2d8586f9fa08f1b261a0ae/heron/bazel-out/local_darwin-fastbuild/bin/third_party/pex/_pex.runfiles/third_party/pex/_pex.py", line 172, in main
    os.path.join(pex_builder.BOOTSTRAP_DIR, 'pkg_resources.py'))
  File "/Users/kwilson/dev/heron/third_party/pex/pex/pex_builder.py", line 138, in add_source
    self._copy_or_link(filename, env_filename, "source")
  File "/Users/kwilson/dev/heron/third_party/pex/pex/pex_builder.py", line 340, in _copy_or_link
    self._chroot.link(src, dst, label)
  File "/Users/kwilson/dev/heron/third_party/pex/pex/common.py", line 267, in link
    os.link(abs_src, abs_dst)
OSError: [Errno 2] No such file or directory
INFO: Elapsed time: 5.105s, Critical Path: 4.79s

fwiw, I did run the bazel_configure.py script beforehand. any ideas?

if interpreter is None:
with TRACER.timed('Resolving interpreter', V=2):
interpreter = interpreter_from_options(options)

if interpreter is None:
die('Could not find compatible interpreter', CANNOT_SETUP_INTERPRETER)
Expand Down