#816 compile heron with bazel 0.2.3 #869

caofangkun · 2016-06-06T08:56:58Z

Since Bazel Release 0.1.5 (2016-02-05)

"Repository rules must use names that are valid workspace names.", see: bazelbuild/bazel#874

➜  heron git:(master) ✗ bazel version 
Build label: 0.2.3
Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue May 17 14:21:13 2016 (1463494873)
Build timestamp: 1463494873
Build timestamp as int: 1463494873

…y/java/BUILD

kramasamy · 2016-06-14T06:54:02Z

3rdparty/java/BUILD

@@ -6,90 +6,90 @@ java_library(
    name = "aws-java-sdk",


Thanks for the fix. In order for the travis ci to compile correctly, you need to download bazel 0.2.3 in the .travis.yml
https://github.com/twitter/heron/blob/master/.travis.yml

Can you make the changes and update the PR?

Hi @kramasamy I have fixed this bug.

The Travis CI still not pass yet, I will get into the root of the matter to find out why.

… should not be null

kramasamy · 2016-06-14T21:24:03Z

@caofangkun - When I looked the log of travis ci, it seems to failing in pants. I was wondering if you could enable some logs so that we can identify why the error message is being issued.

Based on @billonahill comments - I recommend add some debugging to see what file isn't being copied, or to set should_log to return True in tracer.py

… should not be null

taishi8117 · 2016-06-16T01:46:18Z

@caofangkun We've figured out that Bazel made some internal changes with handling Python scripts, which seem to have caused the travis failure. The following diff should fix this issue.

diff --git a/3rdparty/pex/_pex.py b/3rdparty/pex/_pex.py
index 14e1ffe..a0c4f3b 100644
--- a/3rdparty/pex/_pex.py
+++ b/3rdparty/pex/_pex.py
@@ -20,7 +20,7 @@ if not zipfile.is_zipfile(sys.argv[0]):
     sys.modules.pop('twitter.common.python', None)

     root = os.path.join(
-        os.sep.join(__file__.split(os.sep)[:-6]), '3rdparty/pex/_pex.runfiles/3rdparty')
+        os.sep.join(__file__.split(os.sep)[:-6]), 'pex/_pex.runfiles/__main__/3rdparty')
     sys.path.insert(0, os.path.join(root, 'pex'))
     sys.path.insert(0, os.path.join(root, 'setuptools'))
     setuptools_py =  os.path.join(root, 'setuptools')

Additionally, the following modifications may be necessary.

diff --git a/scripts/packages/BUILD b/scripts/packages/BUILD
index a990e75..4779d77 100644
--- a/scripts/packages/BUILD
+++ b/scripts/packages/BUILD
@@ -1,7 +1,6 @@
 package(default_visibility = ["//visibility:public"])

-load("/tools/build_defs/pkg/pkg", "pkg_tar")
-load("/tools/build_defs/pkg/pkg", "pkg_deb")
+load("@bazel_tools//tools/build_defs/pkg:pkg.bzl", "pkg_tar", "pkg_deb")

diff --git a/tools/java/BUILD b/tools/java/BUILD
index 3a099b8..74a39b4 100644
--- a/tools/java/BUILD
+++ b/tools/java/BUILD
@@ -9,7 +9,7 @@ java_toolchain(
     target_version = "7",
     bootclasspath = ["@bazel_tools//tools/jdk:bootclasspath"],
     extclasspath = ["@bazel_tools//tools/jdk:extdir"],
-    javac = ["@bazel_tools//tools/jdk:javac"],
+    javac = ["@bazel_tools//third_party/java/jdk/langtools:javac_jar"],
     javabuilder = ["@bazel_tools//tools/jdk:JavaBuilder_deploy.jar"],
     singlejar = ["@bazel_tools//tools/jdk:SingleJar_deploy.jar"],
     genclass = ["@bazel_tools//tools/jdk:GenClass_deploy.jar"],

kramasamy · 2016-06-16T04:25:39Z

I am also getting this warnings -

bazel build --config=ubuntu heron/...
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/zookeeper/BUILD:82:16: in includes attribute of cc_library rule //3rdparty/zookeeper:zookeeper-cxx: 'include' resolves to '3rdparty/zookeeper/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/libevent/BUILD:100:16: in includes attribute of cc_library rule //3rdparty/libevent:libevent-cxx: 'include' resolves to '3rdparty/libevent/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/gflags/BUILD:49:16: in includes attribute of cc_library rule //3rdparty/gflags:gflags-cxx: 'include' resolves to '3rdparty/gflags/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/gtest/BUILD:94:16: in includes attribute of cc_library rule //3rdparty/gtest:gtest-cxx: 'include' resolves to '3rdparty/gtest/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/gperftools/BUILD:71:16: in includes attribute of cc_library rule //3rdparty/gperftools:tcmalloc-cxx: 'include' resolves to '3rdparty/gperftools/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/jansson/BUILD:46:16: in includes attribute of cc_library rule //3rdparty/jansson:jansson-cxx: 'include' resolves to '3rdparty/jansson/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/gmock/BUILD:83:16: in includes attribute of cc_library rule //3rdparty/gmock:gmock-cxx: 'include' resolves to '3rdparty/gmock/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/glog/BUILD:54:16: in includes attribute of cc_library rule //3rdparty/glog:glog-cxx: 'include' resolves to '3rdparty/glog/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/protobuf/BUILD:141:16: in includes attribute of cc_library rule //3rdparty/protobuf:protobuf-cxx: 'include' resolves to '3rdparty/protobuf/include' not in 'third_party'. This will be an error in the future.
WARNING: /Users/kramasamy/workspace/heron24/3rdparty/yaml-cpp/BUILD:127:16: in includes attribute of cc_library rule //3rdparty/yaml-cpp:yaml-cxx: 'include' resolves to '3rdparty/yaml-cpp/include' not in 'third_party'. This will be an error in the future.

kramasamy · 2016-06-16T07:56:52Z

When I looked at the travis-ci logs, it seems to be failing with java tests and looking at each of the logs shows the following -

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
Class not found: [rotating-map_unittest]

BazelTestRunner exiting with a return value of 2
JVM shutdown hooks (if any) will run now.
The JVM will exit once they complete.

-- JVM shutdown starting at 2016-06-16 04:33:10 --

caofangkun · 2016-06-16T08:27:00Z

Since Bazel 0.2.3, Bazel warns if a cc rule's includes attribute points out of third_party
bazelbuild/bazel@f0a1884

taishi8117 · 2016-06-16T18:35:44Z

Bazel team is working on this issue -- they might remove these warnings as well.
bazelbuild/bazel#1286

taishi8117 · 2016-06-16T18:52:02Z

For unit tests failure, it seems because of Bazel's change in java_test behavior since 0.2.1.
bazelbuild/bazel#1017

taishi8117 · 2016-06-16T22:15:05Z

@caofangkun Since Bazel's internal test structure changed, the pull from #938 (already merged to master) and the following changes are also necessary, which should resolve the unit test failures in Travis.

diff --git a/heron/common/tests/java/com/twitter/heron/common/config/Constants.java b/heron/common/tests/java/com/twitter/heron/common/config/Constants.java
index bd13b68..db1ed1e 100644
--- a/heron/common/tests/java/com/twitter/heron/common/config/Constants.java
+++ b/heron/common/tests/java/com/twitter/heron/common/config/Constants.java
@@ -24,7 +24,7 @@ public final class Constants {
   public static final String LAUNCHER_CLASS_KEY = "heron.launcher.class";

   public static final String TEST_DATA_PATH =
-      "/heron/common/tests/java/com/twitter/heron/common/config/testdata";
+      "/__main__/heron/common/tests/java/com/twitter/heron/common/config/testdata";

   private Constants() {
   }

diff --git a/heron/instance/tests/java/com/twitter/heron/resource/Constants.java b/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
index cd8b8d1..dadef82 100644
--- a/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
+++ b/heron/instance/tests/java/com/twitter/heron/resource/Constants.java
@@ -41,7 +41,7 @@ public final class Constants {
   // For bazel, we use the env var to get the path of heron internals config file
   public static final String BUILD_TEST_SRCDIR = "TEST_SRCDIR";
   public static final String BUILD_TEST_HERON_INTERNALS_CONFIG_PATH =
-      "/heron/config/src/yaml/conf/test/test_heron_internals.yaml";
+      "/__main__/heron/config/src/yaml/conf/test/test_heron_internals.yaml";

   private Constants() {
   }

diff --git a/heron/spi/tests/java/com/twitter/heron/spi/common/TestConstants.java b/heron/spi/tests/java/com/twitter/heron/spi/common/TestConstants.java
index f968363..cc2cd59 100644
--- a/heron/spi/tests/java/com/twitter/heron/spi/common/TestConstants.java
+++ b/heron/spi/tests/java/com/twitter/heron/spi/common/TestConstants.java
@@ -16,7 +16,7 @@ package com.twitter.heron.spi.common;

 final class TestConstants {
   public static final String TEST_DATA_PATH =
-      "/heron/spi/tests/java/com/twitter/heron/spi/common/testdata";
+      "/__main__/heron/spi/tests/java/com/twitter/heron/spi/common/testdata";

   private TestConstants() {
   }

diff --git a/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java b/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java
index d206f33..deb96cd 100644
--- a/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java
+++ b/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/LocalFileSystemConstantsTest.java
@@ -16,7 +16,7 @@ package com.twitter.heron.uploader.localfs;

 final class LocalFileSystemConstantsTest {
   public static final String TEST_DATA_PATH =
-      "/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/testdata";
+      "/__main__/heron/uploaders/tests/java/com/twitter/heron/uploader/localfs/testdata";

   private LocalFileSystemConstantsTest() {
   }

caofangkun · 2016-06-17T03:21:15Z

Hi @kramasamy @taishi8117
Thank you very much for your kind help. CI passed ... at last!

kramasamy · 2016-06-17T04:38:32Z

Awesome! The only issue remaining is the warnings which is not fixed even in bazel 0.3.0

caofangkun · 2016-06-17T10:20:12Z

Hi @kramasamy

According to code of bazelbuild/bazel@f0a1884#diff-b1e05be607e9de9e40f306162531dc81R431

I did the following changes, and the warnings gone.

twitter-heron git:(heron-816-1) ✗ mv 3rdparty third_party
twitter-heron git:(heron-816-1) ✗ sed -i "s/3rdparty/third_party/g" `grep 3rdparty -rl ./`

Should i fix this in this PR or file a new issue and fix it later?

billonahill · 2016-06-17T16:21:08Z

Please update bazel version for the website here too:
https://github.com/twitter/heron/blob/master/website/config.yaml#L24

And in the md files we should reference it via:

{{% bazelVersion %}}

billonahill · 2016-06-17T16:22:01Z

website/content/docs/developers/compiling/docker.md

@@ -162,11 +162,11 @@ RUN \
 ENV JAVA_HOME /usr/lib/jvm/java-8-oracle
 ```

-#### Step 5 - An installation script for [Bazel](http://bazel.io/) version 0.1.2 or above.
+#### Step 5 - An installation script for [Bazel](http://bazel.io/) version 0.2.3 or above.


Please update to use {{% bazelVersion %}} here and below.

caofangkun · 2016-06-20T07:01:55Z

Hi @billonahill I have included your feedback in d74c1dc , Could you please have a review?

kramasamy · 2016-06-20T16:38:33Z

@caofangkun - travis-ci seems to have failed. Can you please check?

taishi8117 · 2016-06-20T22:39:12Z

@kramasamy @caofangkun For the not a valid file name (cc, h, cpp, cu, cuh) warnings, the following diff should resolve it.

diff --git a/tools/java/src/com/twitter/bazel/checkstyle/CppCheckstyle.java b/tools/java/src/com/twitter/bazel/checkstyle/CppCheckstyle.java
index dd5d758..d1b6f87 100644
--- a/tools/java/src/com/twitter/bazel/checkstyle/CppCheckstyle.java
+++ b/tools/java/src/com/twitter/bazel/checkstyle/CppCheckstyle.java
@@ -134,6 +134,7 @@ public final class CppCheckstyle {
                     Predicates.not(Predicates.containsPattern("third_party/")),
                     Predicates.not(Predicates.containsPattern("config/heron-config.h")),
                     Predicates.not(Predicates.containsPattern(".*pb.h$")),
+                    Predicates.not(Predicates.containsPattern(".*cc_wrapper.sh$")),
                     Predicates.not(Predicates.containsPattern(".*pb.cc$"))
             )
     );

taishi8117 · 2016-06-20T23:11:16Z

Looking at the Travis failure message, I just realized that a similar error existed even from cc5da14, although Travis CI was successful for some reasons.

ERROR: /home/travis/build/twitter/heron/heron/proto/BUILD:109:1: error executing shell command: 'set -e
rm -rf bazel-out/local-fastbuild/genfiles/heron/proto/proto_stmgr_java_src.srcjar.srcs
mkdir bazel-out/local-fastbuild/genfiles/heron/proto/proto_stmgr_java_src.srcjar.srcs
bazel-out/local-f...' failed: namespace-sandbox failed: error executing command

[Update]
It seems that Travis started failing after merging #943 (as javadoc failure check is enabled by set -e in heron/website/scripts/javadocs.sh). Related to the issue #766.

taishi8117 · 2016-06-21T01:08:48Z

website/content/docs/developers/compiling/mac.md

-curl -O -L https://github.com/bazelbuild/bazel/releases/download/0.1.2/bazel-0.1.2-installer-darwin-x86_64.sh
-chmod +x bazel-0.1.2-installer-darwin-x86_64.sh
-./bazel-0.1.2-installer-darwin-x86_64.sh --user
+wget -O /tmp/bazel.sh https://github.com/bazelbuild/bazel/releases/download/{{% bazelVersion %}}/bazel-{{% bazelVersion %}}-installer-linux-x86_64.sh


s/linux/darwin/

caofangkun · 2016-06-21T01:31:00Z

Hi @taishi8117 , Thank you so much for your help, I have fixed it as you suggested.

kramasamy · 2016-06-21T01:37:16Z

@caofangkun - can you update your branch since yours seems to have conflicts?

caofangkun · 2016-06-21T01:47:34Z

@kramasamy I'm working on the conficts now, will fix it soon :)

taishi8117 · 2016-06-21T21:09:04Z

@caofangkun Thanks for fixing the conflicts. The Javadoc error might be resolved by merging #968 (already merged to master) --- can you try that? (and enable set -e again)

This reverts commit a01f902.

caofangkun · 2016-06-22T03:44:22Z

Thank you @taishi8117 After merge the CI passed now.

kramasamy · 2016-06-22T03:56:04Z

Looks good to me! 👍

taishi8117 · 2016-06-22T04:04:57Z

Looks great!

nlu90 · 2016-06-22T18:29:45Z

@kramasamy What are the changes need to be made for our Twitter internal building/testing environment?

ablecao added 2 commits June 6, 2016 16:53

#816 complie heron with bazel 0.2.3

4edcae2

#816 complie heron with bazel 0.2.3, add some dependencies of 3rdpart…

2c74601

…y/java/BUILD

kramasamy reviewed Jun 14, 2016
View reviewed changes

ablecao added 2 commits June 14, 2016 17:09

#816 modify .travis.yaml change bazel version from 0.1.2 to 0.2.3

27a8543

#816 modify 'tools/java/BUILD' extclasspath,genclass,ijar,javabuilder…

f2e6f14

… should not be null

kramasamy changed the title ~~#816 complie heron with bazel 0.2.3~~ #816 compile heron with bazel 0.2.3 Jun 15, 2016

ablecao added 2 commits June 15, 2016 11:16

#816 modify 'tools/java/BUILD' extclasspath,genclass,ijar,javabuilder…

eced139

… should not be null

Merge branch 'master' of https://github.com/twitter/heron

3d66eab

ablecao added 2 commits June 16, 2016 09:53

#816 fix travis failure

250ee12

#816 fix travis failure rm BUILD.orig

ef76fdb

taishi8117 mentioned this pull request Jun 16, 2016

Java unit test name changes in BUILD files #938

Merged

kramasamy mentioned this pull request Jun 17, 2016

changes for compiling in bazel 0.2.3 #838

Closed

ablecao added 3 commits June 17, 2016 09:43

#816 fix bug since Bazel's internal test structure changed ref:#938

bb9b365

#816 fix conflicts

cc5da14

#816 change all bazel version description docs

4a737f8

billonahill reviewed Jun 17, 2016
View reviewed changes

#816 include website updates for Bazel 0.2.3, ref: ffda5d7

d74c1dc

#816 revert 6527ded

393218a

taishi8117 reviewed Jun 21, 2016
View reviewed changes

#816 fix bug warnings

4549f1b

ablecao added 6 commits June 21, 2016 10:18

#816 fix conflicts

8594cd4

#816 fix conflicts rename 3rdparty to third_party

e805a47

#816 fix mac.md s/linux/darwin/

2697485

#816 modify 'website/scripts/javadocs.sh' remove 'set -e'

69a4200

#816 delete file '1'

7d758c1

#816 modify javadocs.sh add some build information

a01f902

ablecao added 4 commits June 22, 2016 09:44

Revert "#816 modify javadocs.sh add some build information"

83e840b

This reverts commit a01f902.

Merge remote-tracking branch 'origin/master' into heron-816-1

bd5912f

#815 modify 'javadocs.sh' enable 'set -e' again

0240416

#815 modify 'javadocs.sh' enable 'set -e' again

d3588e9

kramasamy merged commit ca87470 into apache:master Jun 22, 2016

kramasamy mentioned this pull request Jun 22, 2016

Move to bazel2 #816

Closed

caofangkun deleted the heron-816 branch June 23, 2016 01:21

zhangzhonglai mentioned this pull request Jul 5, 2016

remove dependency on log4j-over-slf4j #1025

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

#816 compile heron with bazel 0.2.3 #869

#816 compile heron with bazel 0.2.3 #869

caofangkun commented Jun 6, 2016

kramasamy Jun 14, 2016

caofangkun Jun 14, 2016

kramasamy commented Jun 14, 2016 •

edited

Loading

taishi8117 commented Jun 16, 2016

kramasamy commented Jun 16, 2016

kramasamy commented Jun 16, 2016 •

edited

Loading

caofangkun commented Jun 16, 2016

taishi8117 commented Jun 16, 2016

taishi8117 commented Jun 16, 2016 •

edited

Loading

taishi8117 commented Jun 16, 2016 •

edited

Loading

caofangkun commented Jun 17, 2016

kramasamy commented Jun 17, 2016

caofangkun commented Jun 17, 2016 •

edited

Loading

billonahill commented Jun 17, 2016

billonahill Jun 17, 2016

caofangkun commented Jun 20, 2016 •

edited

Loading

kramasamy commented Jun 20, 2016

taishi8117 commented Jun 20, 2016 •

edited

Loading

taishi8117 commented Jun 20, 2016 •

edited

Loading

taishi8117 Jun 21, 2016

caofangkun Jun 21, 2016

caofangkun commented Jun 21, 2016

kramasamy commented Jun 21, 2016

caofangkun commented Jun 21, 2016

taishi8117 commented Jun 21, 2016 •

edited

Loading

caofangkun commented Jun 22, 2016

kramasamy commented Jun 22, 2016

taishi8117 commented Jun 22, 2016

nlu90 commented Jun 22, 2016

#816 compile heron with bazel 0.2.3 #869

#816 compile heron with bazel 0.2.3 #869

Conversation

caofangkun commented Jun 6, 2016

kramasamy Jun 14, 2016

Choose a reason for hiding this comment

caofangkun Jun 14, 2016

Choose a reason for hiding this comment

kramasamy commented Jun 14, 2016 • edited Loading

taishi8117 commented Jun 16, 2016

kramasamy commented Jun 16, 2016

kramasamy commented Jun 16, 2016 • edited Loading

caofangkun commented Jun 16, 2016

taishi8117 commented Jun 16, 2016

taishi8117 commented Jun 16, 2016 • edited Loading

taishi8117 commented Jun 16, 2016 • edited Loading

caofangkun commented Jun 17, 2016

kramasamy commented Jun 17, 2016

caofangkun commented Jun 17, 2016 • edited Loading

billonahill commented Jun 17, 2016

billonahill Jun 17, 2016

Choose a reason for hiding this comment

caofangkun commented Jun 20, 2016 • edited Loading

kramasamy commented Jun 20, 2016

taishi8117 commented Jun 20, 2016 • edited Loading

taishi8117 commented Jun 20, 2016 • edited Loading

taishi8117 Jun 21, 2016

Choose a reason for hiding this comment

caofangkun Jun 21, 2016

Choose a reason for hiding this comment

caofangkun commented Jun 21, 2016

kramasamy commented Jun 21, 2016

caofangkun commented Jun 21, 2016

taishi8117 commented Jun 21, 2016 • edited Loading

caofangkun commented Jun 22, 2016

kramasamy commented Jun 22, 2016

taishi8117 commented Jun 22, 2016

nlu90 commented Jun 22, 2016

kramasamy commented Jun 14, 2016 •

edited

Loading

kramasamy commented Jun 16, 2016 •

edited

Loading

taishi8117 commented Jun 16, 2016 •

edited

Loading

taishi8117 commented Jun 16, 2016 •

edited

Loading

caofangkun commented Jun 17, 2016 •

edited

Loading

caofangkun commented Jun 20, 2016 •

edited

Loading

taishi8117 commented Jun 20, 2016 •

edited

Loading

taishi8117 commented Jun 20, 2016 •

edited

Loading

taishi8117 commented Jun 21, 2016 •

edited

Loading