Enable drake to be consumed as a bazel external (as a local and remote repository)

[EricCousineau-TRI]

As part of #3129, we should be able to use Daniels' shambhala example for making a bazel project that consumes drake as an external. There are different ways that drake could be consumed:

1. Source-only: All needed externals are directly ported into the active WORKSPACE, such that the user can tune (and be responsible for) dependencies and their versions.
   • This is working in 8419617.
2. Use all of drake's dependencies.
   • Per this comment and workaround in bazelbuild, this can be done for the most part by separating all rule-dependent externals (which must be copy/pasta'd) from other externals, which can be placed in a *.bzl file
   • This is working as a prototype in my own offshoot
     • Caveats:
       * Nesting workspaces in Bazel is not supported (Recursive WORKSPACE file loading bazelbuild/bazel#1943)
       * We have to symlink tools/CROSSTOOL, tools/bazel.rc, and scrape some stuff from tools/BUILD to make sure compiler / config settings are passed through properly.
3. Use a portion of drake's dependencies, and substitute others.

Implementation stage suggestions to support this (not necessarily in order):

• Ensure drake's label references use @drake// instead of @// when possible, and ensure that drake files are referred via @drake//* rather than relative via *
  • To be completed via Teach drake bazel rules to play nicely when drake is an external bazel project #6190.
• Separate as much of the dependencies in WORKSPACE into something like @drake//tools:externals.bzl, which can be incorporated into a macro such as drake_external_repositories()
  • The above prototype completes this via externals.bzl
• (May be external to this issue) Determine what packages to export in drake in both CMake and Bazel.
  • See prior discussion
• (Related to Add unit tests for installation helpers #6117) Incorporate shambhala into CI, with whatever configurations we think are most relevant.
• Permit selection of what dependencies drake uses

This is a draft; please let me know what other thoughts y'all have.

\cc @stonier @jwnimmer-tri @jamiesnape @mwoehlke-kitware

[jwnimmer-tri]

I guess I'm not entirely sure how much of a priority this is. For example, I've never needed to do any of this, and I use Drake regularly from a TRI-internal project that pulls in Drake as a Bazel dependency / repository. Most users will want to stick with CMake, so it seems like making the CMake conops work first would be more important. Sure, Bazel is fun and this should be part of our roadmap, but why now? There are lots of other build system problems that are ripe for improvement.

[EricCousineau-TRI]

To complete every last item, I would say it is much lower priority than CMake.

However, for the second item (and possibly the last), to just move stuff from @drake//:WORKSPACE into something else, I would say that the benefits easily outweigh the costs, and it is low-hanging fruit since we already have a working prototype.

I believe that I've checked the repo that you mentioned, and I believed you had most of drakes dependencies already present (and I assume you would want concrete control, and not drift along with Drake's dependency versions).
However, for me, I'd like to be able to make quick, small test downstream projects, and just use what drake already has to offer with bazel's hermitic nature - possibly even symlink'ing my working version of drake into a Bazel repo (or using git-new-workdir) - no need for copy-pasta, mega-symlinks, etc.

I have updated the prototype just a tad more to try and encapsulate as much of the dependencies as possible, such that all dependency versions are incorporated directly into drake:

• drake_external/README

The external WORKSPACE boils down to something to the effect of:

workspace(name = "drake_external")
drake_relative_path = "externals/drake-distro"
drake_cmake_install_path = "{}/{}/{}".format(__workspace_dir__,
    drake_relative_path, "build/install")  # #5621
local_repository(
    name = "drake",
    path = drake_relative_path,
)
load("@drake//tools:externals_rules.bzl", "drake_external_rule_repositories")
drake_external_rule_repositories(
    drake_relative_path = drake_relative_path,
)
load("@drake//tools:externals.bzl", "drake_external_repositories")
drake_external_repositories(
    cmake_install_path = drake_cmake_install_path,
)

The diff between the downstream WORKSPACE and @drake//:WORKSPACE is as follows:

• drake_external/scripts/WORKSPACE.patch

This then allows me to use the entirety of drake from the outside, with relatively low cost, in bazel.

NOTE: I'm gonna go ahead and submit a PR on the second item, just to move the discussion of that stuff there and see if I can sneak that in :P

[jwnimmer-tri]

However, for me, I'd like to be able to make quick, small test downstream projects ...

Please explain this more. In my workflow, for example, I cut a dev branch of Drake (which could involve adding a new, temporary subdir), and then prototype code within Drake's WORKSPACE directly. What is the advantage of having a separate project, where Bazel things like crosstool and --config memcheck and everything don't work as well? If the end code is to push code into Drake anyway, why not prototype it there in the first place?

Not to say there aren't reasons. But even if the 6195 change is already written (sunk cost), it's still going to be something we have to maintain (future cost) and I'm not yet convinced on the development-spike use case. I am more convinced on "example for users", but that is a different beast.

[jamiesnape]

Given that Bazel 0.5 is going to change a few things about WORKSPACEs, maybe we should wait to see what that brings in any case.

[EricCousineau-TRI]

Please explain this more. [...]

I would like to test integration with other external / downstream projects (in my case, research code that was built on an older version of drake) for a working baseline, and have a more concrete set of boundaries as I migrate external code internal to drake. I would like to have the Git boundary of a submodule, as this permits me to more easily track what extra, non-hermitic prototype dependencies I have in my downstream module (opencv, pcl, etc.) - stuff I would prefer to keep away from drake, even in pure dev branches. (Right now, my dev branch is pretty heavily polluted with test dependency injections, which I would much rather prefer to keep outside of drake.)

This also permits me to more easily track and make smaller PR chunks, such that I don't have as large of discrepancies between rebased PR branches under review and what I am actually using in my dev branch.

[...] where Bazel things like crosstool and --config memcheck and everything don't work as well?

This setup actually does use drakes crosstool and configuration setup (as a very minimal external consumer project), using @drake//tools:CROSSTOOL, @drake//tools:bazel.rc, so things like --config memcheck are exposed through this setup.
(The caveat is, if they want to add things, thing you need make a physical copy, but to me, that is quite a simple change, and could even be solved by using some sort inline text boundary / #include directive, since bazel itself does not seem to provide that sort of mechansim (I assume to simplify its parsing job.) However, I would believe this to be a caveat in any build system, CMake, etc.)

[...] it's still going to be something we have to maintain (future cost) [...]

At least from my perspective, the future maintenance cost seems minimal / negligible.
The only constraint that it truly places is that we do not have load statements heavily depend on external repository rules (at least to anything more than first-degree).

Given that Bazel 0.5 is going to change a few things about WORKSPACEs, maybe we should wait to see what that brings in any case.

I saw the mention of this in the slack convo (posting a snippet from the release log for context):

Remote repositories must define any remote repositories they themselves use (e.g., if @x//:foo depends on @y//:bar, @y must be defined in @x's WORKSPACE file).

To follow up on @soonho-tri's test with pure drake, I had tested out bazel 0.5.0 with this setup, and there does not appear to be any change in behavior:

$ {
    cur="${HOME}/tmp/$(date -Im | sed 's#:#-#g')"
    mkdir -p $cur && cd $cur
    git clone https://github.com/EricCousineau-TRI/drake_external.git
    cd drake_external
    git submodule update --init
}
$ git log -n1 --oneline
6450519 Update branch
$ which bazel && bazel version
/home/eacousineau/.local/bazel/0.5.0/usr/bin/bazel
Build label: 0.5.0
...
$ bazel run //example
...
Trivial Solution: 0

[EricCousineau-TRI]

Looks like recursive WORKSPACE loading may be coming in for bazel 0.6: bazelbuild/bazel#1943.

[jwnimmer-tri]

All issues must have owners. I'm assigning one arbitrarily. (Fix it up if I'm wrong.)

[jwnimmer-tri]

Now that #7259 has made some progress, I wonder if this issue still has any remaining fixups or requests?

[sammy-tri]

This feels done to me...

[EricCousineau-TRI]

Nope, no requests atm, will close. Thanks!