-
Notifications
You must be signed in to change notification settings - Fork 56
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: generate_library.sh
with postprocessing
#1951
feat: generate_library.sh
with postprocessing
#1951
Conversation
This reverts commit 3d612f8.
exit 0 | ||
fi | ||
if [ -z "${versions_file}" ];then | ||
echo "no versions.txt argument provided. Please provide one in order to enable post-processing" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's OK to make it required for now since it's only for existing libraries. For new client libraries, I'm assuming we have to generate versions.txt first before calling this script? Is it generated by new client library script currently? cc @JoeWang1127
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blakeli0 the version of a new library defaults to 0.0.1-SNAPSHOT
.
The new library script applies current versions but this is for other referenced artifacts such as google-cloud-java
(example)
I think we still need a versions file for referenced artifacts other than the library being generated when dealing with a new monorepo library
# owlbot | ||
# 4 - scripts_root: location of the generation scripts | ||
# 5 - destination_path: used to transfer the raw grpc, proto and gapic libraries | ||
# 6 - repository_path: path from output_folder to the location of the source of |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I still don't fully understand the usefulness of repository_path
? If it's for the poms, they should be already in the output folders, am I missing something?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@blakeli0 the pre-existing poms with their folder structure are indeed necessary for the postprocessor.
We need google-cloud-java
to be present beforehand in output_folder
as well, but the script does not know which folder in the monorepo should obtain the poms, owlbot.py
and .OwlBot.yaml
from. I don't think there is a good heuristic to infer the monorepo path from proto_path
or destination_path
. For example looking for .OwlBot.yaml
files that have a deep-copy
containing destination_path
may work but we have cases where the owlbot files do not deal with a single gapic library.
To clarify, the purpose of repository_path
is to give the script a way to find the monorepo folder associated with the library being generated. This folder contains the necessary files for owlbot copy and postprocessor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed, I removed usage of destination_path
. The integration test, however, will pass repository_path
(e.g. java-asset) as destination_path
.
NOTE: The usage of owlbot copy works with a yaml file expecting a certain input folder structure. For example, that yaml has a deep-copy
section with wildcards in order to build the owl-bot-staging
folder (i.e. owlbot copy sends the files stored in deep-copy
to owlbot-staging
and the owlbot postprocessor runs on them)
I'm taking advantage of the wildcards for owlbot copy to work. The path in this deep-copy section is referring a path in googleapis-gen (example) which is could be something like google-cloud-asset-v1-java
but I just named it generated-java
(which will be ok for the wildcard)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm yet to update the readme
edit: done
library_generation/test/resources/google-cloud-java/java-asset/.repo-metadata.json
Outdated
Show resolved
Hide resolved
[gapic-generator-java-root] Kudos, SonarCloud Quality Gate passed! |
[java_showcase_integration_tests] Kudos, SonarCloud Quality Gate passed! |
This PR adds post-processing to
generate_library.sh
The changes are intended for pure GAPICs in
google-cloud-java
Remarks:
main
branch. We are skipping testing against HW librariesjava-bigtable
anddialogflow
, we have owlbot.py files referencing libraries or versions that have not been generated by the current run ofgenerate_library.sh
(doc). For this reason, HW libraries are excluded as well asdialogflow
, which is left as a commented row for future inspectiongoogleapis-gen
tests as a separate option in the integration tests, which can be activated by using--enable_postprocessing=false
. This is a convenience for local developmentpackage-info.java
files with incorrect order of variables, although logically equivalent. This is solved by usingLC_COLLATE=C
for byte-level sorting of proto files when passing as input to the gapic generator.pom.xml
level, whitespace includedIntegration test
google-cloud-java
and the specifiedrepository_path
(e.g.google-cloud-java/java-asset
)generate_library.sh
.workspace
folder every time it's calledrepository_path
ingoogle-cloud-java
git diff
compare-poms.py
to comparepom.xml
files ignoring element order and whitespacegit diff
orcompare-poms.py
found differences, exit with non-zero and stop the test suitePost-processing workflow:
new-library.py
. For example,java-asset
would be prepared in a folder tree "google/cloud/asset/v2
"owlbot-cli copy-code
.OwlBot.yaml
. The pre-existing necessary files are specified withrepository_path
(which can be, for example,google-cloud-java/java-asset
). The files are then sent to anowl-bot-staging
folderowl-bot-staging
folder is sent to aworkspace
folder (root directory the owlbot postprocessor works with)pom.xml
files andowlbot.py
to theworkspace
folderworkspace
folderWe runapply_current_versions.sh
(a modified file borrowed from google-cloud-java)