Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a Spike for error handling #83

Closed
9 tasks done
benedeki opened this issue Mar 3, 2023 · 1 comment
Closed
9 tasks done

Create a Spike for error handling #83

benedeki opened this issue Mar 3, 2023 · 1 comment
Assignees
@benedeki
Labels
enhancement New feature or request epic A bigger feature that needs more deliverable subtasks to finish spike Proof of concept, research and investigation tasks

Comments

@benedeki
Copy link
Contributor

benedeki commented Mar 3, 2023 •
edited
Loading

Background

Many of our Spark extending libraries need a way to report errors that emerged during data processing.
They can be either re-implementing the preferred method of the application their main usage is expected in, or having a possible different way of error reporting than the application or simply ignore the error.

This Spike is to investigate, if it would be possible to create a generic trait to report the errors. This traits different implementation then would be provided to the library upon initialization by the higher level app therefore decoupling the style of error handling from the library that needs it.

Questions To Answer

  1. Can such a generic trait be effectively created
  2. Would that give enough flexibility and anchor-points at the same time

Desired Outcome

Have a main trait and system of supporting classes for general error handling.

Tasks
Preview Give feedback
  4. Implement error handling by putting the info into __Dev Null_ (silently ignoring it) #88
    enhancement good first issue
    TebaleloS
  5. Add implicits for easier usage of the trait #89
    enhancement good first issue
    TebaleloS
  6. Add methods to the trait ErrorHandling that will return the type/schema of the generated ErrorColumn and their aggregation. #91
    enhancement good first issue
    TebaleloS
  7. ErrorHandling documentation and fields renames #101
    4 of 4
    documentation refactoring
    benedeki
  8. Implement error handling that will filter the rows that have any error #93
    enhancement good first issue
    TebaleloS
  9. Create ErrorHandling implementation that throws an exception on error detected #111
    enhancement good first issue
    benedeki

Depends on #82
@benedeki benedeki added the enhancement New feature or request label Mar 3, 2023
@benedeki benedeki changed the title Create a Spike for errro handling Create a Spike for error handling Mar 3, 2023
@benedeki benedeki added spike Proof of concept, research and investigation tasks epic A bigger feature that needs more deliverable subtasks to finish labels Mar 3, 2023
@benedeki benedeki self-assigned this Mar 3, 2023
benedeki added a commit that referenced this issue Mar 7, 2023
@benedeki
#83: Create a Spike for error handling
7178f32 
* new functions `null_col` and `call_udf`
* `ErrorMessage` refactoring
* `ErrorHandling` trait designed to serve as the interface for different implementations
* Implement error handling by putting the info into column of `ErrorMessage` array
* numerous support classes
TebaleloS added a commit that referenced this issue Apr 19, 2023
@TebaleloS
Fixes #93 - merged with #83
bebc7b3
TebaleloS added a commit that referenced this issue Apr 19, 2023
@TebaleloS
Fixes #93 - merged with #83
fa8a0ec
@benedeki benedeki moved this from 🆕 To groom to 🏗 In progress in CPS small repos project Apr 28, 2023
benedeki added a commit that referenced this issue May 4, 2023
@benedeki @lsulak
#83: Create a Spike for error handling (#84)
c657364 
* new functions `null_col` and `call_udf`
* `ErrorMessage` refactoring
* `ErrorHandling` trait designed to serve as the interface for different implementations
* Implement error handling by putting the info into column of `ErrorMessage` array
* numerous support classes
* Added Jacoco exclusion for adapters

Co-authored-by: Ladislav Sulak <[email protected]>
TebaleloS added a commit that referenced this issue May 8, 2023
@TebaleloS @benedeki
implement error handling that will filter the rows that have any error:
279b4e5 
#93 (#94)

* #83: Create a Spike for error handling
* new functions `null_col` and `call_udf`
* `ErrorMessage` refactoring
* `ErrorHandling` trait designed to serve as the interface for different implementations
* Implement error handling by putting the info into column of `ErrorMessage` array
* numerous support classes

* * UT fix
* headers fix

* * Work in progress

* * Relatively big overwrite to use `map` instead of errCol and sequence of raw values
* `ErrorMessageSubmitJustErrorValue` class created to offer the ability to submit errors without source column but with error value

* * Forgotten `register` function call

* * line ends improved

* * ErrorMessageSubmits moved to submits sub-package
* `ErrorMessageSubmitWithoutColumn` changed from `case class` to `class` to allow inheritance
* some PR comments addressed

* * Added UTs for `ColumnOrValue`
* Fixed few minor things discovered by the UTs

* Fixes #82 - Added logic to filter out rows with errors

* Fixes #82

* Fixes #82 - created class

* Fixes #93

* Fixes #93

* Fixes #93

* Fixes #93 - Changed the logic for evaluate method

* Fixes #93 - Added scala documentation for ErrorHandlingFilterRowsWithErrors

* Fixes #93 - added ErrorHandlingFilterRowsWithErrors test file

* Fixes #93

* Fixes #93 - Object test progress.

* Fixes #93 - Object test progress

* Fixes #93 - Object test progress

* Fixes #93 - Object test progress

* * Forgotten `register` function call

* * line ends improved

* * ErrorMessageSubmits moved to submits sub-package
* `ErrorMessageSubmitWithoutColumn` changed from `case class` to `class` to allow inheritance
* some PR comments addressed

* * Added UTs for `ColumnOrValue`
* Fixed few minor things discovered by the UTs

* * Work in progress

* Fixes #93 - merged with #83

* Fixes #93 - merged with #83

* Merge branch 'feature/83-create-a-spike-for-error-handling' into feature/93-Implement-error-handling-that-will-filter-the-rows-that-have-any-error

* Unit test, work in progress

* Unit test, work in progress

* Fixes #93 - Refactored `doTheColumnAggregation` method

* #93 - Unit test work in progress

* * changed `ErrorMessageArrayTest` to actual test suite

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* Pull changes

* #93 - Unit test work in progress

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* #93 - Unit test work in progress

* #93 - Unit test work in progress

* #93 - Unit test work in progress

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* Update spark-commons/src/main/scala/za/co/absa/spark/commons/errorhandling/implementations/ErrorHandlingFilterRowsWithErrors.scala

Co-authored-by: David Benedeki <[email protected]>

* #93 - Unit test work in progress

* #93 - Unit test work in progress

* Fixes #93 conflicts

* Fixes #93

* Fixes #93

* Fixes #93

* Closes #93

---------

Co-authored-by: David Benedeki <[email protected]>
Co-authored-by: David Benedeki <[email protected]>
@benedeki
Copy link
Contributor Author

This item depends on:

@github-project-automation github-project-automation bot moved this from 🏗 In progress to ✅ Done in CPS small repos project Jul 25, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benedeki benedeki

Labels
enhancement New feature or request epic A bigger feature that needs more deliverable subtasks to finish spike Proof of concept, research and investigation tasks
Projects
CPS small repos project
Archived in project
Milestone
No milestone
Development

No branches or pull requests
1 participant
@benedeki