Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review use of DataFusion::Internal and ensure they are returned for bugs in DataFusion #6108

Closed
alamb opened this issue Apr 24, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@alamb
Copy link
Contributor

alamb commented Apr 24, 2023

Is your feature request related to a problem or challenge?

There are a variety of places where internal errors are raised in DataFusion:
https://github.com/search?q=repo%3Aapache%2Farrow-datafusion+DataFusionError%3A%3AInternal&type=code

End applications often translate internal errors into something different (aka hide details from the user as they can't do anything about it) rather than a Plan or Execution error (which signals an issue in the query)

Describe the solution you'd like

  1. Audit all the uses of DataFusionError::Internal errors and change any to DataFusion::Plan or DataFusion::Execution as appropriate
  2. Update the doc strings on DataFusionError to make it clear DataFusionError::Internal should be used only when there is a bug in DataFusion (rather than an error in the query)

Describe alternatives you've considered

No response

Additional context

https://github.com/influxdata/influxdb_iox/issues/7606

@alamb alamb added the enhancement New feature or request label Apr 24, 2023
@wolffcm
Copy link
Contributor

wolffcm commented Apr 24, 2023

I think one such place where Internal is returned when it should not be is here:
https://github.com/apache/arrow-datafusion/blob/729586258fe6371e394b8b2caa4e1b55eccbf6c5/datafusion/optimizer/src/optimizer.rs#L307-L311

Logical optimizer rules should be able to fail in a way that advises the user on how to correct their query, rather than having the optimizer assume that any error is internal. This is relevant to https://github.com/influxdata/influxdb_iox/issues/7330

@comphead
Copy link
Contributor

comphead commented Jul 9, 2024

I'm closing it @alamb feel free to reopen if needed.

@comphead comphead closed this as completed Jul 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants