Wickman's proposal for pants reorg

[jsirois]

This was originally filed here

When pants moves to its own repository, I'd like to take the opportunity to do a code reorg in order to make it easier for other projects / repositories / organizations to use pants.

current layout

Backends (java, scala, python, etc) consist of tasks and targets, and basic wiring into phases. Right now the code org is something like this:

• pants.bin
• pants.base
• pants.commands
• pants.doc
• pants.goal
• pants.java
• pants.python
• pants.pants_doc
• pants.python
• pants.targets
• pants.tasks

Bolded is what I consider to be "pants core" – as in pants simply can't operate without code in there. Specifically this is the execution engine and ancillary stuff like base classes for everything else (Task, Goal, Target.) Currently the Task/TaskError base classes are in pants.tasks.init but should be moved into pants.base.

While pants/init.py contains some code necessary for pants to run (get_buildroot), most of it should be moved out to the leaves (is_* e.g. is_java, is_scala) and 'from twitter.pants.targets import *' should go away entirely.

new backend layout

I propose that everything in pants.java, pants.python, pants.targets and pants.tasks be reorganized (temporarily) into a new toplevel directory:

• pants.backend

With the following layout:

• pants.backend.{language}.init
• pants.backend.{language}.base
• pants.backend.{language}.targets
• pants.backend.{language}.tasks

Each backend is treated as a namespace package so that they can more easily be developed by third parties. This model works naturally with the python.new backend, and for the most part with the Java and Scala backends.

One case where this is more challenging is IDL backends. My proposal for handling this is by making the CodeGen(Task) base class in pants.base the canonical base class for code generation, and specific pants.backend.thrift and pants.backend.protobuf IDL backends that other backends can then depend upon. For example, pants.backend.python would depend upon pants.backend.thrift, and subclass the ThriftGen base from pants.backend.thrift.base to create a Python-specific codegen target. (I've already done the refactor in the python.new branch to make this more straightforward through language-specific createtarget abstract base classes.) So rather than the 'gen' phase with 'gen:thrift' and 'gen:protobuf' goals, you'd have the 'gen' phase with 'gen:thrift-python', 'gen:thrift-java', 'gen:protobuf-go' etc.

bootstrapping

To bootstrap a repository to use pants, you would do the following:

1. copy the bootstrap_pants.py script into your repository and name it 'pants', chmod +x
2. add a root BUILD file which sets up your source root and defines which plugins you'd like

The pants bootstrap script currently bootstraps itself from PyPI, though some of the dependent source dists have not yet been published (specifically twitter.pants, twitter.common-core, twitter.common.python) but we can do that when we're ready – especially since twitter.pants could be 'pb' or 'pantsbuild' or something else. If you don't have external network access, you'd add a field in pants.ini that specifies where the bootstrapper should be looking for artifacts and you'd cache them locally. Similarly, by default the bootstrap script bootstraps the latest version of pants, but you could address specific versions in the pants.ini using the existing Requirement format (e.g. 'pb==0.3.2' or 'pb>0.3,<1').

The bootstrapper script would write pants.pex and on subsequent runs 1) check the pants.pex version to make sure it's compatible with the one defined in pants.ini and if so 2) os.execv.

repository BUILD

The root BUILD would look like:

require('pants.backend.java==0.2.3')
require('pants.backend.python')
require_path('plugins/foursquare')
# [any extra project specific methods or variables defined here]

Each backend would be a PythonLibrary with a corresponding SetupPy artifact, which can have install_requires in order to dictate dependencies (e.g. pystache, boto, pants.backend.thrift, whatever.) The upside being that 1) this is all already implemented and 2) they could write a backend as a standard setup.py project outside of pants.

require_path would simply look for an init.py in that directory and install everything there into the root namespace.

Currently for BUILD files to be usable, we autoinject 'from pants.targets import *' and 'from twitter.common.quantity import Amount, Time' in every single one. Instead, we'd deposit the union of the namespaces created by the buildroot BUILD, and the init.py of each registered backend into all down-tree BUILDs. The backend _init_s would also be responsible for wiring tasks into phases.

[jinfeng]

@jsirois What's our plan here? I'm writing because I'm hitting an issue to bootstrap pants in twitter pants_internal [1].

Seems we have 3 different type of pants related repos:

1. pantsbuild/pants:

   The originating open source pants source repo. In this repo, its bootstrap script (pants at root) either checks a pants.pex exists or not, and execute it directly if found, if not then build it and execute it; or runs live against source code if PANTS_DEV=1 is specified. When run live or build a pants.pex, lots of external world accesses are needed, like getting virtualenv from pypi, and subsequently installing requirements from pypi during bootstrap phase. I’m not sure if pants.pex exists, and directly invoked, would virtualenv still be prep’ed and external pypi download still take place? Doesn’t seem so but want to confirm.

   In this repo’s pants.ini there is a section [python_repo], but this is used after pants is up and running and when we need a new python_chroot. So the section doesn’t help during bootstrap phase unless we add code there to parse it and tell pip what repos to look into.

2. Twitter's pants_internal:

   This is Twitter extension to OS pants. In this repo, its bootstrap script clones OS pants from github (or from an internal mirror of it due to strict outside access control), adds some extra python requirements, and then directly invoking the locally cloned pantsbuild/pants bootstrap script. However, this would only work if run from a machine where outside world access is granted even if OS_PANTS_ORIGIN points to an internal mirror, because pantsbuild/pants's bootstrap script will always try to get virtualenv or other requirements from pypi directly. Also it seems in pants_internal, the bootstrap pants script doesn't check the existence of pants.pex at built root or not. So in short, running pants_internal directly inside an access restricted environment even with OS_PANTS_ORIGIN pointing to internal git mirror doesn’t work.

3. Twitter's other repo like science/birdcage:

   These are the end repos that consume pants, i.e. using pants to build other products. This repo's root bootstrap pants (not a bash script but a python script), is this the one you mentioned earlier as bootstrap_pants.py? I can't find it anywhere pantsbuild/pants. Anyway, this script simply checks local pants.pex; if yes run it, no, look into pants.ini to find pants PEX caching URL and download the PEX locally and then execute it. Again, not sure if/how virtualenv is activated in the PEX context.

My problem came from 2. above. When we want to execute tests for pants_internal repo, meaning we want to run pants goal test against test code we written in that repo, the bootstrap script would try to clone OS pants and then execute, which isn't allowed in our CI environment.

So I tried with using OS_PANTS_ORIGIN to point to our internal mirror of OS pants. Still not working, because the OS pants bootstrap will get virtualenv and other requirements from pypi. Two options, 1) I could've amend OS pants bootstrap scripts somehow to let them be aware of caching locations, but seems a bit more involved so I opted for 2) I simply using a pre-built pants.pex to run goal tests which bypass all these external accesses.

However, this only goes so for running simple tests. When I run a test case derived from pants_run_integration_test.py which shells out and call pants_internal repo root's pants bootstrap script, since it doesn't check for local pants.pex, it again starts to clone OS pants.

So my questions are (sorry for typing so much, it took me a while to unravel these different code paths during bootstrapping):

1. To solve my immediate problem: I'm thinking about adding check pants.pex existence into pants_intenral pants script. So it's bit similar to OS pants bootstrap and pants consuming repo bootstrap scripts. Does that sound right?
2. Looks like you've mentioned above, any concrete plan on making OS pants bootstrap be internal cache aware so we can directly execute OS pants inside firewall?
3. It would be nice to either doc or having code somewhere to clear layout how to setup repos like 2 (a pants plugin/backend development repo) and 3 (a pants consuming end product repo), what are the templatized bootstrap scripts to use, how additional requirements can be added, how caches should be working, etc?

[1] For non current/former twitter pants devs: this pants_internal repo isn't an end product git repo that uses pants to build, but it's a repo inside twitter where we develop pants plugins/backends and wrap the open source pants in (or rather more precisely, insert our plugins/backends into open sources pants).

[ericzundel]

I think this is complete. Close?

[lahosken]

We haven't achieved the stuff in the "bootstrapping" section. But I ?think? we have achieved the code-layout parts of working towards that "boostrapping" section.

[jsirois]

We do need a bootsrap script much like gradlew that can bootsrap a virtualenv for pantsbuild.pants to be pip installed into. But we also are still short in the 'repository BUILD' section. Namely we cannot bootstrap 3rdparty deps - including plugins - for you today and we need to.

[stuhood]

All issues mentioned in John's most recent comment have been addressed, and the packaging concerns have all been invalidated (if not fixed).