Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrow CSV writer should not fail when cannot cast the value #3547

Closed
comphead opened this issue Jan 17, 2023 · 3 comments · Fixed by #3570
Closed

Arrow CSV writer should not fail when cannot cast the value #3547

comphead opened this issue Jan 17, 2023 · 3 comments · Fixed by #3570
Assignees
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog

Comments

@comphead
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Original problem reported in apache/datafusion#4947
Arrow writer uses .unwrap() on options and fails when cast is not succeeded

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/jackfan.us.kg-1ecc6299db9ec823/arrow-csv-30.0.1/src/writer.rs:262:18

Describe the solution you'd like
Investigate on better options

  • fail with meaningful error
  • return and display the error as part of output

Describe alternatives you've considered

Additional context

@comphead comphead added the enhancement Any new improvement worthy of a entry in the changelog label Jan 17, 2023
@comphead comphead changed the title Arrow CSV writer should not fail when cannot cast the valu Arrow CSV writer should not fail when cannot cast the value Jan 17, 2023
@comphead
Copy link
Contributor Author

in old spark versions, if timestamp cast cannot be done then null returned that was really confusing, we should consider a runtime error if such invalid cast happened

@comphead
Copy link
Contributor Author

@tustvold @alamb please assign this to me

@tustvold
Copy link
Contributor

label_issue.py automatically added labels {'arrow'} from #3570

@tustvold tustvold added the arrow Changes to the arrow crate label Jan 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arrow Changes to the arrow crate enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants