feat: provide direct TableProvider integration in datafusion-python

[timsaucer]

Description

This change adds in a single method on the Delta Table object to expose a PyCapsule that provides a DataFusion FFI Table Provider. With this change, you can register a delta table in datafusion-python without the need to export a pyarrow_dataset. This enables full push down filter support, greatly improving performance in some cases.

Related Issue(s)

Closes #2536

Documentation

This is a follow on from apache/datafusion#12920 and apache/datafusion-python#823

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[timsaucer]

FYI @ion-elgreco

[ion-elgreco]

@timsaucer super excited with this change!

[ion-elgreco]

Can you add the attribute in here as well:

https://github.com/delta-io/delta-rs/blob/main/python/deltalake/_internal.pyi

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.

Project coverage is 72.43%. Comparing base (c091a82) to head (2d7fdb8).
Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
python/src/lib.rs	0.00%	10 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3012      +/-   ##
==========================================
- Coverage   72.47%   72.43%   -0.04%     
==========================================
  Files         128      128              
  Lines       40831    40841      +10     
  Branches    40831    40841      +10     
==========================================
- Hits        29592    29584       -8     
- Misses       9356     9366      +10     
- Partials     1883     1891       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles

[timsaucer]

I'll try to address that ci failure tomorrow morning

[adriangb]

Amazing!