Update smithy-build to be streaming #211
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
smithy-build used to load every model from every projection into memory at once. This was not ideal since some use cases utilize hundreds of projections. For example, in AWS SDKs, we create a projection for every AWS service, which is hundreds of them with tens-of-thousands of shapes and operations. This commit updates smithy-build to support two modes: in-memory and streaming. The in-memory mode is useful for testing or one-off integrations. The streaming mode is useful for actually executing smithy-build from the CLI so that results are outputted to stdout as they happen and not all models are loaded into memory. The CLI was updated to significantly change its output to show more information about things that go wrong and print more statistics. Every projection and plugin is attempted to be executed -- the CLI previously failed on the first exception. Finally, executing tasks is now done with an ExecutorService rather than parallel streams to have more control over task execution.
kstich
approved these changes
Nov 21, 2019
| } | ||
| } | ||
|
|
||
| private void executeParallelProjections( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
smithy-build used to load every model from every projection into memory
at once. This was not ideal since some use cases utilize hundreds of
projections. For example, in AWS SDKs, we create a projection for every
AWS service, which is hundreds of them with tens-of-thousands of shapes
and operations.
This commit updates smithy-build to support two modes: in-memory and
streaming. The in-memory mode is useful for testing or one-off
integrations. The streaming mode is useful for actually executing
smithy-build from the CLI so that results are outputted to stdout as
they happen and not all models are loaded into memory.
The CLI was updated to significantly change its output to show more
information about things that go wrong and print more statistics. Every
projection and plugin is attempted to be executed -- the CLI previously
failed on the first exception.
Finally, executing tasks is now done with an ExecutorService rather than
parallel streams to have more control over task execution.
Issue #, if available:
Description of changes:
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.