Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Protocol test clients generation #3247

Merged
merged 3 commits into from
Jan 24, 2025
Merged

Protocol test clients generation #3247

merged 3 commits into from
Jan 24, 2025

Conversation

SergeyRyabinin
Copy link
Contributor

@SergeyRyabinin SergeyRyabinin commented Jan 14, 2025 •
edited
Loading

Issue #, if available:
WIP for protocol tests for this SDK
Description of changes:

  1. Refactor python wrappers over code generators present on this SDK by extracting the logic into separate submodules for re-use and read-ability (I hope so).
  2. Add "protocol test clients" (sample clients for a fictional services used for test purposes) generation to the SDK.
    These clients should look, act, and behave just like any regular generated SDK client but are used for SDK testing only.

Tested by generating all and observing that no unexpected difference is generated.
The only difference is

$ python3 tools/scripts/run_code_generation.py --generate_protocol_tests
...
$ ls -a generated/protocol-tests/test-clients
aws-cpp-sdk-ec2-protocol
aws-cpp-sdk-json-protocol
aws-cpp-sdk-json-rpc-10
aws-cpp-sdk-query-protocol
aws-cpp-sdk-rest-json-protocol
aws-cpp-sdk-rest-xml-protocol
aws-cpp-sdk-rest-xml-protocol-namespace

Check all that applies:

  • Did a review by yourself.
  • Added proper tests to cover this PR. (If tests are not applicable, explain.)
  • Checked if this PR is a breaking (APIs have been changed) change.
  • Checked if this PR will not introduce cross-platform inconsistent behavior.
  • Checked if this PR would require a ReadMe/Wiki update.

Check which platforms you have built SDK on to verify the correctness of this PR.

  • Linux
  • Windows
  • Android
  • MacOS
  • IOS
  • Other Platforms

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@SergeyRyabinin SergeyRyabinin changed the title [draft] Protocol test clients generation Protocol test clients generation Jan 15, 2025
@SergeyRyabinin SergeyRyabinin marked this pull request as ready for review January 15, 2025 18:48
@SergeyRyabinin SergeyRyabinin force-pushed the sr/protoTestClients branch 3 times, most recently from d47996e to 4596bfd Compare January 21, 2025 18:58
@SergeyRyabinin SergeyRyabinin force-pushed the sr/protoTestClients branch 5 times, most recently from 86bad5d to 90f1bf6 Compare January 22, 2025 17:16
sbera87
sbera87 reviewed Jan 22, 2025
View reviewed changes
tools/code-generation/smithy/codegen/settings.gradle.kts Outdated
@@ -43,4 +43,4 @@ dependencyResolutionManagement {
library("junit-jupiter-engine", "org.junit.jupiter", "junit-jupiter-engine").versionRef("junit")
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this file can be excluded

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice catch! thank you for noticing this.

sbera87
sbera87 reviewed Jan 22, 2025
View reviewed changes
tools/scripts/codegen/__init__.py
@@ -0,0 +1,4 @@
#!/usr/bin/env python3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this file needed? Usually this is for python packages (though not needed for 3.3 onwards)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is a good question.
I'm just used to create a python init.py file when I want to create a python sub-package.
according to the spec and stackoverflow:

Keep on adding empty init.py to your directories because 99% of the time you just want to create regular packages. Also, Python tools out there such as mypy and pytest require empty init.py files to interpret the code structure accordingly. This can lead to weird errors if not done with care.

I'm not sure if empty means literally empty, or shebang + license is still okay.
I only checked on py 3.7 and 3.9 and on our CI that it works.

sbera87
sbera87 reviewed Jan 22, 2025
View reviewed changes
tools/scripts/codegen/protocol_tests_gen.py
:return:
"""
return self._generate_test_clients(executor, max_workers)
# TODO: self._generate_tests()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this still to do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is work in progress, I will submit a following PR soon, just splitting the work into sub-parts.

sbera87
sbera87 reviewed Jan 22, 2025
View reviewed changes
tools/scripts/codegen/protocol_tests_gen.py
pathlib.Path(f"{sdk_root_dir}/{PROTOCOL_TESTS_BASE_DIR}").resolve())
self.c2j_client_generator.output_location = PROTOCOL_TESTS_GENERATED_CLIENTS_DIR

def generate(self, executor: ProcessPoolExecutor, max_workers: int):
Copy link
Contributor

@sbera87 sbera87 Jan 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Food for thought: This is great and we can add to this more abstraction to simplify the main runner. We can perhaps use a generator base class with generate() and other relevant methods abstracted (can use builder design patter for example to build the generator class) and implement the different methods and from the main script just instantiate the appropriate class based upon args and invoke generate of one or more codegen objects (client, smoke, protocol)

sbera87
sbera87 approved these changes Jan 22, 2025
View reviewed changes
Copy link
Contributor

@sbera87 sbera87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with minor and good to have comments.

@SergeyRyabinin SergeyRyabinin force-pushed the sr/protoTestClients branch 2 times, most recently from 4beec94 to 99c034e Compare January 23, 2025 18:39
@sbera87 sbera87 self-requested a review January 24, 2025 18:22
@SergeyRyabinin SergeyRyabinin merged commit 1deaebf into main Jan 24, 2025
4 of 5 checks passed
@SergeyRyabinin SergeyRyabinin deleted the sr/protoTestClients branch January 24, 2025 18:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sbera87 sbera87 sbera87 approved these changes

@sbiscigl sbiscigl sbiscigl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SergeyRyabinin @sbiscigl @sbera87