Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEZ-4227 Introduce convenient methods in TezID subclasses #166

Merged
merged 1 commit into from
Feb 10, 2022

Conversation

ghanko
Copy link
Contributor

@ghanko ghanko commented Nov 23, 2021

Change-Id: I6cabfa75e9b6b62e41ba8c2cc5e3d2d1a8a49102

@tez-yetus

This comment has been minimized.

@tez-yetus

This comment was marked as outdated.

@@ -154,8 +154,7 @@ private void processDownstreamVertices(Vertex vertex) {
List<Vertex> newlyScheduledVertices = Lists.newLinkedList();
Map<Vertex, Edge> outputVertexEdgeMap = vertex.getOutputVertices();
for (Vertex destVertex : outputVertexEdgeMap.keySet()) {
if (vertexAlreadyScheduled(destVertex)) { // Nothing to do if already scheduled.
} else {
if (!vertexAlreadyScheduled(destVertex)) { // Nothing to do if already scheduled.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can delete the comment "Nothing to do" from here as it corresponds to the deleted branch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

import org.apache.hadoop.yarn.api.records.ApplicationId;

public interface DAGIDAware {
TezDAGID getDAGId();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

even if it looks strange (too many capitals), this might be changed to

getDAGID();

to stay consistent with others, so it should suggest that a TezID Object is returned here which is "ID", not an integer, which is id

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@tez-yetus

This comment was marked as outdated.

@tez-yetus

This comment was marked as outdated.

Change-Id: I6cabfa75e9b6b62e41ba8c2cc5e3d2d1a8a49102
@tez-yetus

This comment was marked as outdated.

@tez-yetus
Copy link

🎊 +1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 13m 2s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 1s No case conflicting files found.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
+1 💚 test4tests 0m 0s The patch appears to include 20 new or modified test files.
_ master Compile Tests _
+0 🆗 mvndep 5m 1s Maven dependency ordering for branch
+1 💚 mvninstall 9m 12s master passed
+1 💚 compile 6m 19s master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 compile 6m 4s master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 checkstyle 6m 1s master passed
+1 💚 javadoc 5m 48s master passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 5m 18s master passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+0 🆗 spotbugs 0m 45s Used deprecated FindBugs config; considering switching to SpotBugs.
+1 💚 findbugs 9m 12s master passed
_ Patch Compile Tests _
+0 🆗 mvndep 0m 9s Maven dependency ordering for patch
+1 💚 mvninstall 4m 12s the patch passed
+1 💚 compile 3m 43s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javac 3m 43s the patch passed
+1 💚 compile 3m 25s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 javac 3m 25s the patch passed
-0 ⚠️ checkstyle 0m 10s tez-common: The patch generated 16 new + 15 unchanged - 0 fixed = 31 total (was 15)
+1 💚 checkstyle 0m 12s tez-runtime-internals: The patch generated 0 new + 33 unchanged - 1 fixed = 33 total (was 34)
+1 💚 checkstyle 0m 13s tez-mapreduce: The patch generated 0 new + 55 unchanged - 1 fixed = 55 total (was 56)
-0 ⚠️ checkstyle 1m 7s tez-dag: The patch generated 20 new + 1783 unchanged - 20 fixed = 1803 total (was 1803)
+1 💚 checkstyle 0m 13s tez-tests: The patch generated 0 new + 34 unchanged - 2 fixed = 34 total (was 36)
+1 💚 checkstyle 0m 10s The patch passed checkstyle in tez-ext-service-tests
+1 💚 checkstyle 0m 10s The patch passed checkstyle in tez-protobuf-history-plugin
-0 ⚠️ checkstyle 0m 11s tez-plugins/tez-yarn-timeline-history: The patch generated 1 new + 58 unchanged - 2 fixed = 59 total (was 60)
+1 💚 checkstyle 0m 10s The patch passed checkstyle in tez-yarn-timeline-history-with-acls
+1 💚 checkstyle 0m 9s The patch passed checkstyle in tez-yarn-timeline-cache-plugin
-0 ⚠️ checkstyle 0m 10s tez-plugins/tez-yarn-timeline-history-with-fs: The patch generated 1 new + 35 unchanged - 1 fixed = 36 total (was 36)
+1 💚 checkstyle 0m 10s The patch passed checkstyle in tez-history-parser
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 2m 59s the patch passed with JDK Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04
+1 💚 javadoc 2m 42s the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
+1 💚 findbugs 7m 55s the patch passed
_ Other Tests _
+1 💚 unit 0m 34s tez-common in the patch passed.
+1 💚 unit 0m 34s tez-runtime-internals in the patch passed.
+1 💚 unit 1m 16s tez-mapreduce in the patch passed.
+1 💚 unit 4m 25s tez-dag in the patch passed.
+1 💚 unit 38m 50s tez-tests in the patch passed.
+1 💚 unit 4m 4s tez-ext-service-tests in the patch passed.
+1 💚 unit 0m 27s tez-protobuf-history-plugin in the patch passed.
+1 💚 unit 1m 36s tez-yarn-timeline-history in the patch passed.
+1 💚 unit 1m 50s tez-yarn-timeline-history-with-acls in the patch passed.
+1 💚 unit 0m 18s tez-yarn-timeline-cache-plugin in the patch passed.
+1 💚 unit 1m 35s tez-yarn-timeline-history-with-fs in the patch passed.
+1 💚 unit 2m 29s tez-history-parser in the patch passed.
+1 💚 asflicense 2m 22s The patch does not generate ASF License warnings.
159m 2s
Subsystem Report/Notes
Docker ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/artifact/out/Dockerfile
GITHUB PR #166
Optional Tests dupname asflicense javac javadoc unit spotbugs findbugs checkstyle compile
uname Linux 11e5938b2063 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality personality/tez.sh
git revision master / 6d7ef20
Default Java Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
Multi-JDK versions /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.13+8-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/artifact/out/diff-checkstyle-tez-common.txt
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/artifact/out/diff-checkstyle-tez-dag.txt
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/artifact/out/diff-checkstyle-tez-plugins_tez-yarn-timeline-history.txt
checkstyle https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/artifact/out/diff-checkstyle-tez-plugins_tez-yarn-timeline-history-with-fs.txt
Test Results https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/testReport/
Max. process+thread count 1412 (vs. ulimit of 5500)
modules C: tez-common tez-runtime-internals tez-mapreduce tez-dag tez-tests tez-ext-service-tests tez-plugins/tez-protobuf-history-plugin tez-plugins/tez-yarn-timeline-history tez-plugins/tez-yarn-timeline-history-with-acls tez-plugins/tez-yarn-timeline-cache-plugin tez-plugins/tez-yarn-timeline-history-with-fs tez-plugins/tez-history-parser U: .
Console output https://ci-hadoop.apache.org/job/tez-multibranch/job/PR-166/6/console
versions git=2.25.1 maven=3.6.3 findbugs=3.0.1
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Contributor

@abstractdog abstractdog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 after addressing comments, this patch looks good to me, approving
I like the introduction of *IDAware interfaces, also having a consistent letter case: "ID" for TezID objects, "Id" for the integer inside
as the patch touches lots of code, let me give another 24-48h for others to tell their opinion (cc: @rbalamohan, @jteagles)

@ghanko: could you please check how the patch applies to branch-0.9? I'm pretty sure it has a bunch of conflicts, but I'm hoping all of them are obvious...if we could have the patch on branch-0.9, it helps to reduce the divergence between master/branch-0.9...only if it doesn't take too much time

@abstractdog abstractdog merged commit 7b66d3d into apache:master Feb 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants