Improve DataFusion error handling

[comphead]

Is your feature request related to a problem or challenge?

I'm trying to collect known issues related to error handling in datafusion.

So far I can see different categories which can be improved

1. DataFusion heavily relies on Rust Result<T, E> and sometimes its hard to identify context where the actual error is happening, the situation becomes even worse for downstream projects.

• Add Backtraces to DataFusionError #5283
• Enhance TaskContext and add task failure root cause #3410 (this related to upstream arrow-rs)

2. Overuse panics

• Review use of panic in datafusion-common crate #3313
• Review use of panics in datafusion-row crate #3317
• Review use of panic in datafusion-physical-expr crate #3316

3. Overuse internal error type in situations where more specific error kind can be used

• Review use of DataFusion::Internal and ensure they are returned for bugs in DataFusion #6108
• Internal_err! is over used in core code and should be replaced with exec_err! or plan_err! in many places #9164

4. General

• Improve optimizer performance by not using Errors in the happy path #7552

Describe the solution you'd like

We can try to address some of the issues with implementing a generic error macros that will be used for handling DF errors. Having done that allows to process errors in standard unified way, experiment with stacktraces and other details needed to be exposed for the end user.

• Add plan_err! error macro #7115
• Add internal_err! error macro #7293
• Add not_impl_err error macro #7340
• Add exec_err! error macro #7361
• Add arrow_err! macros, optional backtrace to ArrowError #8586
• SQL Error macros
• Parquet Error macros
• ObjectStore Error macros
• IOError macros
• Add schema_err! error macros with optional backtrace #8620
• Introduce resources_err! error macro #11374
• External Error macros
• Context Error macros
• Substrait Error macros
• Add backtrace to error messages #7434

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I think we have made significant progress on this issue. I wonder if we still have more planned work?