Add DatumReader Cache to improve de-serialization performance #65
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hhkkxxx133
reviewed
Jul 30, 2021
.../main/java/com/amazonaws/services/schemaregistry/deserializers/avro/DatumReaderInstance.java
Outdated
Show resolved
Hide resolved
blacktooth
added a commit
that referenced
this pull request
Aug 23, 2021
* Add DatumReader Cache to improve de-serialization performance * Fix exception message Co-authored-by: Ravindranath Kakarla <[email protected]>
blacktooth
added a commit
that referenced
this pull request
Feb 11, 2022
* Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Protobuf support (#60) * Add message indices parser for Protobuf schemas. * Encode Protobuf messages with message index Co-authored-by: Ravindranath Kakarla <[email protected]> * Fix checkstyle build error and reduce tests for canaries * Update README.md * Update README.md * Pull everit from maven central instead of jitpack (#54) Co-authored-by: mohit <[email protected]> * Bump up version to v1.1.1 * Update ReadMe to v1.1.1 * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Protobuf support (#60) * Add message indices parser for Protobuf schemas. * Encode Protobuf messages with message index Co-authored-by: Ravindranath Kakarla <[email protected]> * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Update dependencies for Protobuf and create class structure cr: https://code.amazon.com/reviews/CR-52122396 * Add ProtobufMessageType to aid Protobuf deserialization cr: https://code.amazon.com/reviews/CR-54151956 * Protobuf DynamicMessage deserialization cr: https://code.amazon.com/reviews/CR-54424261 * Fix serializer build issue * Fix serializer build issue * Pull everit from maven central instead of jitpack (#54) Co-authored-by: mohit <[email protected]> * Fix resource clean up in Kafka integration test * Add documentation for configuring a specific GSR resource with the Kafka Connect converter (#58) * Reduce logging (#64) * Add .gitignore To avoid having files accidentally added to the project * Reduce and update logging It looks like these changes were lost in a merge at some point. Created based on the diff on GitHub. * Add DatumReader Cache to improve de-serialization performance (#65) * Add DatumReader Cache to improve de-serialization performance * Fix exception message Co-authored-by: Ravindranath Kakarla <[email protected]> * Introduce cache to improve serialization performance (#67) Co-authored-by: Ravindranath Kakarla <[email protected]> * Bump up version to v1.1.2 (#70) Co-authored-by: mohit <[email protected]> * Bump up v1.1.2 in README.md * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object (#72) Co-authored-by: Ravindranath Kakarla <[email protected]> * Add a utility to derive class name from message descriptor (#74) * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object * Add a utility to derive Protobuf compiled Java class name from FileDescriptor. Co-authored-by: Ravindranath Kakarla <[email protected]> * Add support to deserialize to Protobuf POJOs. (#76) Co-authored-by: Ravindranath Kakarla <[email protected]> * Handle ProtobufMessageType not set. (#85) Co-authored-by: Ravindranath Kakarla <[email protected]> * Introduce cache to improve Protobuf serializer/de-serializer performance. (#90) Co-authored-by: Ravindranath Kakarla <[email protected]> * Merging latest approved changes to protobuf-support branch. (#98) * Modify UserAgent to emit usage metrics (#77) * Add tests to include key and value schemas both (#79) Co-authored-by: Mohit Paliwal <[email protected]> * Bump to 1.1.3 (#78) * Modify UserAgent to emit usage metrics * Bump version to 1.1.3 Co-authored-by: Ravindranath Kakarla <[email protected]> * Update README.md * Upgrade Kafka version to 2.8.1 * Bump up version to 1.1.4 * Upgrade transitive dependencies to latest versions. Co-authored-by: Ravindranath Kakarla <[email protected]> * Remove configuration logging information (#89) * Bump up version to 1.1.5 (#91) * Specify the avro-maven-plugin version explicitly (#97) Co-authored-by: Ravindranath Kakarla <[email protected]> * Bump up version to v1.1.1 * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Ravindranath Kakarla <[email protected]> Co-authored-by: Kexin Hui <[email protected]> Co-authored-by: Linyu Yao <[email protected]> * Add integration tests for Protobuf support (#99) * Protobuf integration tests for Kinesis use-cases * Protobuf integration test cases for Kafka use-cases. Co-authored-by: Ravindranath Kakarla <[email protected]> * Upgrade to latest versions of Protobuf dependencies. (#103) Co-authored-by: Ravindranath Kakarla <[email protected]> * Protobuf support (#117) * Modify UserAgent to emit usage metrics (#77) * Add tests to include key and value schemas both (#79) Co-authored-by: Mohit Paliwal <[email protected]> * Bump to 1.1.3 (#78) * Modify UserAgent to emit usage metrics * Bump version to 1.1.3 Co-authored-by: Ravindranath Kakarla <[email protected]> * Update README.md * Upgrade Kafka version to 2.8.1 * Bump up version to 1.1.4 * Upgrade transitive dependencies to latest versions. Co-authored-by: Ravindranath Kakarla <[email protected]> * Remove configuration logging information (#89) * Bump up version to 1.1.5 (#91) * Specify the avro-maven-plugin version explicitly (#97) Co-authored-by: Ravindranath Kakarla <[email protected]> * Fix bug with canary tests running the entire suite. (#102) Co-authored-by: Ravindranath Kakarla <[email protected]> * Upgrade to latest versions of Protobuf dependencies. * Bump log4j-core from 2.13.2 to 2.15.0 (#107) * Bump log4j-api from 2.13.2 to 2.15.0 (#106) Bumps log4j-api from 2.13.2 to 2.15.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.5 to 1.1.6 (#108) Co-authored-by: Mohit Paliwal <[email protected]> * Bump log4j-api from 2.15.0 to 2.16.0 (#109) Bumps log4j-api from 2.15.0 to 2.16.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.6 to 1.1.7 (#111) Co-authored-by: Mohit Paliwal <[email protected]> * Pull current region for integration tests (#112) Co-authored-by: Mohit Paliwal <[email protected]> * Bump log4j-api from 2.16.0 to 2.17.0 (#113) Bumps log4j-api from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump log4j-core from 2.16.0 to 2.17.0 (#114) Bumps log4j-core from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.7 to 1.1.8 (#115) Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Ravindranath Kakarla <[email protected]> Co-authored-by: Kexin Hui <[email protected]> Co-authored-by: Linyu Yao <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove extra text from Protobuf schema generation (#132) * Remove extra text from Protobuf schema generation Co-authored-by: Kexin Hui <[email protected]> * Add documentation for Protobuf support (#134) * Add documentation for Protobuf support Co-authored-by: Kexin Hui <[email protected]> * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Protobuf support (#60) * Add message indices parser for Protobuf schemas. * Encode Protobuf messages with message index Co-authored-by: Ravindranath Kakarla <[email protected]> * Bump up version to v1.1.1 * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Pull everit from maven central instead of jitpack (#54) Co-authored-by: mohit <[email protected]> * Fix resource clean up in Kafka integration test * Add message indices parser for Protobuf schemas. (#52) Co-authored-by: Ravindranath Kakarla <[email protected]> * Update dependencies for Protobuf and create class structure cr: https://code.amazon.com/reviews/CR-52122396 * Add ProtobufMessageType to aid Protobuf deserialization cr: https://code.amazon.com/reviews/CR-54151956 * Protobuf DynamicMessage deserialization cr: https://code.amazon.com/reviews/CR-54424261 * Fix serializer build issue * Fix serializer build issue * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object (#72) Co-authored-by: Ravindranath Kakarla <[email protected]> * Add a utility to derive class name from message descriptor (#74) * Refactor GlueSchemaRegistryDataFormatDeserializer to accept Schema object * Add a utility to derive Protobuf compiled Java class name from FileDescriptor. Co-authored-by: Ravindranath Kakarla <[email protected]> * Add support to deserialize to Protobuf POJOs. (#76) Co-authored-by: Ravindranath Kakarla <[email protected]> * Handle ProtobufMessageType not set. (#85) Co-authored-by: Ravindranath Kakarla <[email protected]> * Introduce cache to improve Protobuf serializer/de-serializer performance. (#90) Co-authored-by: Ravindranath Kakarla <[email protected]> * Add integration tests for Protobuf support (#99) * Protobuf integration tests for Kinesis use-cases * Protobuf integration test cases for Kafka use-cases. Co-authored-by: Ravindranath Kakarla <[email protected]> * Upgrade to latest versions of Protobuf dependencies. * Protobuf support (#117) * Modify UserAgent to emit usage metrics (#77) * Add tests to include key and value schemas both (#79) Co-authored-by: Mohit Paliwal <[email protected]> * Bump to 1.1.3 (#78) * Modify UserAgent to emit usage metrics * Bump version to 1.1.3 Co-authored-by: Ravindranath Kakarla <[email protected]> * Update README.md * Upgrade Kafka version to 2.8.1 * Bump up version to 1.1.4 * Upgrade transitive dependencies to latest versions. Co-authored-by: Ravindranath Kakarla <[email protected]> * Remove configuration logging information (#89) * Bump up version to 1.1.5 (#91) * Specify the avro-maven-plugin version explicitly (#97) Co-authored-by: Ravindranath Kakarla <[email protected]> * Fix bug with canary tests running the entire suite. (#102) Co-authored-by: Ravindranath Kakarla <[email protected]> * Upgrade to latest versions of Protobuf dependencies. * Bump log4j-core from 2.13.2 to 2.15.0 (#107) * Bump log4j-api from 2.13.2 to 2.15.0 (#106) Bumps log4j-api from 2.13.2 to 2.15.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.5 to 1.1.6 (#108) Co-authored-by: Mohit Paliwal <[email protected]> * Bump log4j-api from 2.15.0 to 2.16.0 (#109) Bumps log4j-api from 2.15.0 to 2.16.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.6 to 1.1.7 (#111) Co-authored-by: Mohit Paliwal <[email protected]> * Pull current region for integration tests (#112) Co-authored-by: Mohit Paliwal <[email protected]> * Bump log4j-api from 2.16.0 to 2.17.0 (#113) Bumps log4j-api from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-api dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump log4j-core from 2.16.0 to 2.17.0 (#114) Bumps log4j-core from 2.16.0 to 2.17.0. --- updated-dependencies: - dependency-name: org.apache.logging.log4j:log4j-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump up version from 1.1.7 to 1.1.8 (#115) Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Ravindranath Kakarla <[email protected]> Co-authored-by: Kexin Hui <[email protected]> Co-authored-by: Linyu Yao <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Remove extra text from Protobuf schema generation (#132) * Remove extra text from Protobuf schema generation Co-authored-by: Kexin Hui <[email protected]> * Add documentation for Protobuf support (#134) * Add documentation for Protobuf support Co-authored-by: Kexin Hui <[email protected]> Co-authored-by: Ravindranath Kakarla <[email protected]> Co-authored-by: mohit <[email protected]> Co-authored-by: Mohit Paliwal <[email protected]> Co-authored-by: Kexin Hui <[email protected]> Co-authored-by: Jake Baker <[email protected]> Co-authored-by: Mark Lambert <[email protected]> Co-authored-by: Thomas Vestergaard Trolle <[email protected]> Co-authored-by: Kexin <[email protected]> Co-authored-by: Linyu Yao <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of changes:
Adding a cache to improve the performance of de-serializer by not parsing the schema and not re-creating the
DatumReaderfor every message.Testing
Profiled the code using AWS Code Guru and don't see
createDatumReaderandSchema.Parsemethod consuming CPU.Integration tests are successful locally.