-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Alter OncePerSparkSession
to be instantiatable even without provided SparkSession
#82
Labels
Comments
Bonus |
TebaleloS
added a commit
that referenced
this issue
Mar 9, 2023
TebaleloS
added a commit
that referenced
this issue
Mar 9, 2023
TebaleloS
added a commit
that referenced
this issue
Mar 15, 2023
TebaleloS
added a commit
that referenced
this issue
Mar 16, 2023
TebaleloS
added a commit
that referenced
this issue
Mar 16, 2023
TebaleloS
added a commit
that referenced
this issue
Mar 16, 2023
TebaleloS
added a commit
that referenced
this issue
Mar 28, 2023
TebaleloS
added a commit
that referenced
this issue
May 8, 2023
#93 (#94) * #83: Create a Spike for error handling * new functions `null_col` and `call_udf` * `ErrorMessage` refactoring * `ErrorHandling` trait designed to serve as the interface for different implementations * Implement error handling by putting the info into column of `ErrorMessage` array * numerous support classes * * UT fix * headers fix * * Work in progress * * Relatively big overwrite to use `map` instead of errCol and sequence of raw values * `ErrorMessageSubmitJustErrorValue` class created to offer the ability to submit errors without source column but with error value * * Forgotten `register` function call * * line ends improved * * ErrorMessageSubmits moved to submits sub-package * `ErrorMessageSubmitWithoutColumn` changed from `case class` to `class` to allow inheritance * some PR comments addressed * * Added UTs for `ColumnOrValue` * Fixed few minor things discovered by the UTs * Fixes #82 - Added logic to filter out rows with errors * Fixes #82 * Fixes #82 - created class * Fixes #93 * Fixes #93 * Fixes #93 * Fixes #93 - Changed the logic for evaluate method * Fixes #93 - Added scala documentation for ErrorHandlingFilterRowsWithErrors * Fixes #93 - added ErrorHandlingFilterRowsWithErrors test file * Fixes #93 * Fixes #93 - Object test progress. * Fixes #93 - Object test progress * Fixes #93 - Object test progress * Fixes #93 - Object test progress * * Forgotten `register` function call * * line ends improved * * ErrorMessageSubmits moved to submits sub-package * `ErrorMessageSubmitWithoutColumn` changed from `case class` to `class` to allow inheritance * some PR comments addressed * * Added UTs for `ColumnOrValue` * Fixed few minor things discovered by the UTs * * Work in progress * Fixes #93 - merged with #83 * Fixes #93 - merged with #83 * Merge branch 'feature/83-create-a-spike-for-error-handling' into feature/93-Implement-error-handling-that-will-filter-the-rows-that-have-any-error * Unit test, work in progress * Unit test, work in progress * Fixes #93 - Refactored `doTheColumnAggregation` method * #93 - Unit test work in progress * * changed `ErrorMessageArrayTest` to actual test suite * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * Pull changes * #93 - Unit test work in progress * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * #93 - Unit test work in progress * #93 - Unit test work in progress * #93 - Unit test work in progress * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala Co-authored-by: David Benedeki <[email protected]> * #93 - Unit test work in progress * #93 - Unit test work in progress * Fixes #93 conflicts * Fixes #93 * Fixes #93 * Fixes #93 * Closes #93 --------- Co-authored-by: David Benedeki <[email protected]> Co-authored-by: David Benedeki <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
Currently
OnlyOncePerSparkSession
can be instantiated only when provided withSparkSession
. This turns out to be limiting for classes that needs to be created before Spark is available.Feature
register
method public, while keeping the feature that actual registration happens only once perSparkSession
Up to consideration if to make the
SparkSession
parameter of the old constructor explicit, to avoid confusion when the (auto-) registration happened and when not.On the other hand that would create a breaking change for all current users.
The text was updated successfully, but these errors were encountered: