Add new ballista-cli crate

[andygrove]

Which issue does this PR close?

Closes #2493 and #2433 and #2517

Rationale for this change

• Allows us to release DataFusion and Ballista separately. This was not practical before because the datafusion-cli depended on ballista which in turn depended on datafusion, leading to issues such as datafusion-cli crate has circular dependency #2433)
• Allows DataFusion and Ballista CLIs to evolve separately without Ballista adding development overhead to the DataFusion CLI

Here is the new crate dependency diagram:

What changes are included in this PR?

• Create new ballista-cli crate by copying all of datafusion-cli crate
• Remove ballista features from datafusion-cli crate
• Update dependency diagram
• Refactor to have ballista-cli re-use some of the code from datafusion-cli instead of duplicating all of the code
• Update CI scripts
• Update user guide

Are there any user-facing changes?

Yes

[alamb]

cc @liukun4515 and team

[alamb]

this can probably be simplified over time as this cli is always for ballista

[andygrove]

Agreed. I think ballista still needs a local option but it should be using a local (in-process) ballista scheduler and executor and not using datafusion directly.

[alamb]

Thanks @andygrove looks much better to me

[andygrove]

Thanks for the review @alamb. I pushed another commit to fix the datafusion-cli lock file that had conflicts after merging from master and also added the ballista-cli lock file that was previously ignored. I also updated the CI integration test to build the ballista-cli.

[andygrove]

AMD64 tests failed with No space left on device ... trying again

[andygrove]

AMD64 tests failed with No space left on device 3 times in a row now.

[andygrove]

@alamb All PR builds are failing on AMD64 with no space left on device. Any idea what we need to do to resolve?

[alamb]

@alamb All PR builds are failing on AMD64 with no space left on device. Any idea what we need to do to resolve?

Perhaps we can try to update the cache-key in https://github.com/apache/arrow-datafusion/blob/master/.github/workflows/rust.yml#L55 to see if something about the old caches is too large?

[andygrove]

@alamb All PR builds are failing on AMD64 with no space left on device. Any idea what we need to do to resolve?

Perhaps we can try to update the cache-key in https://github.com/apache/arrow-datafusion/blob/master/.github/workflows/rust.yml#L55 to see if something about the old caches is too large?

Thanks. I suppose this cache just continues to grow forever and doesn't clear out old versions of deps that we no longer use? Perhaps we should include the month name in the cache so it resets periodically?

[andygrove]

@alamb Moving the ballista tests to a separate job fixes this. Ok to merge?

[andygrove]

We were previously only running cargo test in the datafusion-examples crate in this step

[andygrove]

We test datafusion-cli in the integration tests and we no longer support running this with ballista

[andygrove]

I'm going to go ahead and merge to unblock other PRs and so that I can create the release candidate PR.