Consolidate SessionConfig and ConfigOptions

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

While messing with thishttps://github.com//pull/3885 I found it quite confusing that SessionConfig is different than ConfigOptions and they had different ways of serializing / deserializing their values.

The new ConfigOptions was added by @andygrove #2754 and we can now see and set them via environment variables etc and are documented 🎉

https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/config.rs#L235-L239

But it seems like SessionConfig is the old way and is still used (e.g. a field was just added by @Dandandan in #3846)

Describe the solution you'd like
It would be nice to unify the two (by moving all SessionConfig values into ConfigOptions)

https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/execution/context.rs#L1128-L1156

Which is what I believe this comment in to_props is talking about:
https://github.com/apache/arrow-datafusion/blob/b654fdea697b9aec1cb487e292dd288e5c9d09e3/datafusion/core/src/execution/context.rs#L1282-L1289

Describe alternatives you've considered
Leave split brained as is

Additional context
#3821

[waitingkuo]

@alamb thank you! while i was changing ConfigOption I was thinking about this as well.

some of the setter (in SessionConfig) contains validator

e.g. n should be greater than 0 for batch_size

https://github.com/apache/arrow-datafusion/blob/feff5dc805449afa4a7ecb5ca88b979ad5739cce/datafusion/core/src/execution/context.rs#L1209-L1213

Do you think we should leave them in SessionConfig or move into ConfigOptions?

[alamb]

Do you think we should leave them in SessionConfig or move into ConfigOptions?

I think if there are specialized setters they can do the validation -- so if left the setter in SessionConfig we would leave the validation there too.

I guess in the long term it isn't so clear to me why we need two structures (SessionConfig and ConfigOptions) 🤔

[alamb]

@thinkharderdev inspired me to propose making the config options strongly typed -- would love to get other opinions on it #4517 (comment)