Skip to content

Commit

Permalink
BUILD: Bump Hadoop versions in the release build.
Browse files Browse the repository at this point in the history 
Also, minor modifications to the MapR profile.
  • Loading branch information
@pwendell
pwendell committed Aug 20, 2014
1 parent f2f26c2 commit ceb1983
Show file tree
Hide file tree
Showing 2 changed files with 40 additions and 9 deletions.
10 changes: 5 additions & 5 deletions dev/create-release/create-release.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -118,11 +118,11 @@ make_binary_release() {
}

make_binary_release "hadoop1" "-Phive -Dhadoop.version=1.0.4" &
make_binary_release "cdh4" "-Phive -Dhadoop.version=2.0.0-mr1-cdh4.2.0" &
make_binary_release "hadoop2" \
"-Phive -Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -Pyarn.version=2.2.0" &
make_binary_release "hadoop2-without-hive" \
"-Pyarn -Phadoop-2.2 -Dhadoop.version=2.2.0 -Pyarn.version=2.2.0" &
make_binary_release "hadoop2.3" "-Phadoop-2.3 -Phive -Pyarn" &
make_binary_release "hadoop2.4" "-Phadoop-2.4 -Phive -Pyarn" &
make_binary_release "hadoop2.4-without-hive" "-Phadoop-2.4 -Pyarn" &
make_binary_release "mapr3" "-Pmapr3 -Pyarn -Phive" &
make_binary_release "mapr4" "-Pmapr4 -Pyarn -Phive" &

This comment has been minimized.

Sign in to view

Copy link
@srowen

srowen Aug 21, 2014

Member

I personally think it's fine to stop making cdh4 artifacts. But here there are even more MapR-specific profiles added. Are these not covered by the Hadoop releases? MapR is perfectly capable of overriding these versions in their distro, so can't be for them.

I am not clear this is OK from an ASF perspective. See this recent thread on the Flink list: http://mail-archives.eu.apache.org/mod_mbox/incubator-flink-dev/201408.mbox/%3CCANC1h_u%3DN0YKFu3pDaEVYz5ZcQtjQnXEjQA2ReKmoS%2Bye7%3Do%3DA%40mail.gmail.com%3E

The upshot seemed to be: helping people build for vendor artifacts is cool, and making release for particular upstream Hadoop/YARN combos to match a well-known vendor release is cool, but offering downloads for some vendors is a gray area as it could be perceived as endorsement.

This comment has been minimized.

Sign in to view

Copy link
@pwendell

pwendell Aug 29, 2014

Author Contributor

Unfortunately unlike CDH 5+ and HDP 2+, MapR still requires a separate Spark build since it's not API comatible with the hadoop cleint library. I think we actually can't post MapR binaries on ASF space since they include MapR's client libraries which are not even open source... it needs to be hosted by MapR themselves and identified as a third party build. However, I did add them as part of the RC process since ideally we want people testing builds against MapR as well. Sean - if you feel that even this is in violation of the ASF policies, we can change the build to not produce MapR-compatible binaries when cutting release candidates. I don't have strong feelings on this I was just trying to increase test coverage for the release.

This comment has been minimized.

Sign in to view

Copy link
@srowen

srowen Aug 29, 2014

Member

@pwendell Great, thanks for explaining the reasoning. You're saying these aren't going into the final release? I think that moots any issue surrounding distributing them then.

This is all I know of from the ASF: https://community.apache.org/projectIndependence.html I don't read it as expressly forbidding this kind of thing although you can see how it bumps up against the spirit. There's not a bright line -- what about Tomcat providing binaries compiled for Windows for example? does that favor an OS vendor?

From this technical ASF perspective only the releases matter -- do what you want with snapshots and RCs. The only issue there is maybe releasing something different than was in the RC; is that at all confusing? Just needs a note.

I think this theoretical issue doesn't exist if these binaries aren't released, so I see no reason to not proceed.

The rest is a different question about whether you want to spend time maintaining this profile and candidate. The vendor already manages their build I think and -- and I don't know -- may even prefer not to have a different special build floating around. There's also the theoretical argument that this turns off other vendors from adopting Spark if it's perceived to be too connected to other vendors. I'd like to maximize Spark's distribution and there's some argument you do this by not making vendor profiles. But as I say a different question to just think about over time...
wait

# Copy data
Expand Down
39 changes: 35 additions & 4 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1115,18 +1115,49 @@
</profile>

<profile>
<id>mapr</id>
<id>mapr3</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hadoop.version>1.0.3-mapr-3.0.3</hadoop.version>
<yarn.version>2.3.0-mapr-4.0.0-beta</yarn.version>
<hbase.version>0.94.17-mapr-1403</hbase.version>
<zookeeper.version>3.4.5-mapr-1401</zookeeper.version>
<yarn.version>2.3.0-mapr-4.0.0-FCS</yarn.version>
<hbase.version>0.94.17-mapr-1405</hbase.version>
<zookeeper.version>3.4.5-mapr-1406</zookeeper.version>
</properties>
</profile>

<profile>
<id>mapr4</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<properties>
<hadoop.version>2.3.0-mapr-4.0.0-FCS</hadoop.version>
<yarn.version>2.3.0-mapr-4.0.0-FCS</yarn.version>
<hbase.version>0.94.17-mapr-1405-4.0.0-FCS</hbase.version>
<zookeeper.version>3.4.5-mapr-1406</zookeeper.version>
</properties>
<dependencies>
<dependency>
<groupId>org.apache.curator</groupId>
<artifactId>curator-recipes</artifactId>
<version>2.4.0</version>
<exclusions>
<exclusion>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5-mapr-1406</version>
</dependency>
</dependencies>
</profile>

<!-- Build without Hadoop dependencies that are included in some runtime environments. -->
<profile>
<id>hadoop-provided</id>
Expand Down

0 comments on commit ceb1983

Please sign in to comment.