Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

xds: Build third-party protos in separate build step #9269

Merged
merged 7 commits into from
Jul 7, 2022
Merged

xds: Build third-party protos in separate build step #9269

merged 7 commits into from
Jul 7, 2022

Conversation

ejona86
Copy link
Member

@ejona86 ejona86 commented Jun 13, 2022

This dramatically shortens build time, even for full builds. A full
assemble of xds on my laptop goes from 1m 46s to 33s at least because
errorprone is disabled for the protos.

CC @sergiitk

@ejona86 ejona86 requested a review from YifeiZhuang June 13, 2022 19:03
@ejona86 ejona86 mentioned this pull request Jun 13, 2022
Merged
@sergiitk
Copy link
Member

Looks like the build still fails

> Task :extractIncludeProto FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':extractIncludeProto'.
> Could not resolve all files for configuration ':compileProtoPath'.
   > Could not find :unspecified:.
     Required by:
         project : > io.grpc:grpc-xds:1.48.0-SNAPSHOT

@ejona86
Copy link
Member Author

ejona86 commented Jun 13, 2022

Ugh. This is going to take some more to deal with the dependencies used by thirdparty build but not by xds itself. I could workaround that by using compileOnly dependencies in thirdparty and re-declaring them for xds, but...

Part of this trouble is made worse because the dependencies are sort of wonky to begin with to deal with shadow.

ejona86 added 6 commits June 29, 2022 08:01
@ejona86
xds: Build third-party protos in separate build step
39dcc1e 
This dramatically shortens build time, even for full builds. A full
assemble of xds on my laptop goes from 1m 46s to 33s at least because
errorprone is disabled for the protos.
@ejona86
Fix Java 9+ compilation
5199cf4
@ejona86
Commit generated code
0782481
@ejona86
Don't include broken 'unspecified' dep in pom
85079fd
@ejona86
Fix -PskipCodegen=true for sourceSets other than main/test
e04a352
@ejona86
Fix deps generation in pom
e6d2c90
@ejona86 ejona86 force-pushed the separate-xds-proto-compile branch from bcee635 to e6d2c90 Compare June 29, 2022 15:05
@ejona86
Copy link
Member Author

ejona86 commented Jun 29, 2022

@YifeiZhuang, this should be working now.

@ejona86
Copy link
Member Author

ejona86 commented Jun 29, 2022

Ugh. OOMing.

@ejona86
Copy link
Member Author

ejona86 commented Jun 30, 2022

@YifeiZhuang, the build OOMing is now resolved.

YifeiZhuang
YifeiZhuang approved these changes Jul 7, 2022
View reviewed changes
Copy link
Contributor

@YifeiZhuang YifeiZhuang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the optimization and simplification, it looks great!

@ejona86 ejona86 merged commit 3de7e74 into grpc:master Jul 7, 2022
@ejona86 ejona86 deleted the separate-xds-proto-compile branch July 7, 2022 14:26
ejona86 added a commit to ejona86/grpc that referenced this pull request Jul 8, 2022
@ejona86
Allow Gradle to use more memory when building Java interop
db12332 
Should fix "Expiring Daemon because JVM heap space is exhausted".

grpc/grpc-java#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238438006
@ejona86 ejona86 mentioned this pull request Jul 8, 2022
Merged
ejona86 added a commit to ejona86/grpc-java that referenced this pull request Jul 8, 2022
@ejona86
xds: Allow Gradle to use more memory when building interop
af5aebb 
Should fix "Expiring Daemon because JVM heap space is exhausted".

grpc#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.
ejona86 added a commit to ejona86/grpc-java that referenced this pull request Jul 8, 2022
@ejona86
xds: Allow Gradle to use more memory when building interop
8bc897a 
Should fix "Expiring Daemon because JVM heap space is exhausted".

grpc#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
@ejona86 ejona86 mentioned this pull request Jul 8, 2022
Merged
ejona86 added a commit that referenced this pull request Jul 8, 2022
@ejona86
xds: Allow Gradle to use more memory when building interop
5f9ef98 
Should fix "Expiring Daemon because JVM heap space is exhausted".

#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
ejona86 added a commit to grpc/grpc that referenced this pull request Jul 8, 2022
@ejona86
Allow Gradle to use more memory when building Java interop (#30249)
f4c162f 
Should fix "Expiring Daemon because JVM heap space is exhausted".

grpc/grpc-java#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238438006
drfloob pushed a commit to drfloob/grpc that referenced this pull request Jul 8, 2022
@ejona86 @drfloob
Allow Gradle to use more memory when building Java interop (grpc#30249)
96020cd 
Should fix "Expiring Daemon because JVM heap space is exhausted".

grpc/grpc-java#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238438006
sergiitk added a commit to sergiitk/grpc-java that referenced this pull request Jul 9, 2022
@sergiitk
xds: Allow Gradle to use more memory when building interop - GCE
094efe8 
Same as grpc#9347, but for GCE framework too (xds and xds_v3 jobs).

Should fix "Expiring Daemon because JVM heap space is exhausted".

PR grpc#9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
@sergiitk sergiitk mentioned this pull request Jul 9, 2022
Merged
sergiitk added a commit that referenced this pull request Jul 9, 2022
@sergiitk
xds: Allow Gradle to use more memory when building interop - GCE (#9354)
d7a6c1e 
Same as #9347, but for GCE framework too (xds and xds_v3 jobs).

Should fix "Expiring Daemon because JVM heap space is exhausted".

PR #9269 probably pushed the build
over the edge, but there's been evidence via flakes for a good while
that we've been reaching the limit.

b/238334438
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 6, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@YifeiZhuang YifeiZhuang YifeiZhuang approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ejona86 @sergiitk @YifeiZhuang