feat: add pre-commit hooks and format existing code

[muyihao]

Description

This PR introduces pre-commit hooks to ensure code adheres to style guidelines and quality standards. The following hooks have been added:

General Hooks:

• Remove trailing whitespace
• Ensure files end with a newline
• Validate YAML and JSON files
• Prevent large files from being added
• Detect merge conflict markers

Python:

• Format code with Black (max line length 120)
• Lint code with Flake8 and flake8-bugbear (ignoring F401, max line length 120)
• Perform static type checking with Mypy

Rust:

• Format code with cargo fmt

Related Issue:
#5
#73

How are the changes test-covered

• N/A
• Automated tests (unit and/or integration tests)
• Manual tests
  • Details are described below

[xushiyan]

@muyihao nice change! i'd suggest the following

1. change pyproject.toml by adding sections like [tool.mypy] and [tool.ruff] and configure whatever needed for python format. We need to let IDE to also respect the settings, the pyproject.toml should be the source of truth for everything about the python project including style
2. split into 2 PRs: first PR titled "style: ..." to include pyproject.toml and python style changes. it should also make the python lintting part of the GH actions workflow compliance
3. Move everything about pre commit to another PR titled chore: ...

[xushiyan]

can all the python tools settings be put in pyproject.toml ? we can use section like [tool.mypy], [tool.ruff] ? we can use ruff instead of flake8+black, much faster

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 87.24%. Comparing base (36316e0) to head (9dbd569).
Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #100   +/-   ##
=======================================
  Coverage   87.24%   87.24%           
=======================================
  Files          14       14           
  Lines         690      690           
=======================================
  Hits          602      602           
  Misses         88       88           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[muyihao]

@xushiyan Thanks for your suggestions! I appreciate your feedback. I've made the changes as per your recommendations.

I've split the changes into two PRs:

1. PR1
2. PR2

However, I noticed that the Compliance check failed upon submission. It doesn't seem related to my changes. The error message indicates an issue with a Rust dependency: arrow_schema::Schema::all_fields: Use flattened_fields instead.

Should I address this issue within my PR, or would it be better to handle it separately?

[xushiyan]

Rust dependency: arrow_schema::Schema::all_fields: Use flattened_fields instead.

@muyihao looks like this is reported by clippy - yes feel free to raise another PR to fix the clippy warning, then we can land the python style change PR without touching rust code

[xushiyan]

Rust dependency: arrow_schema::Schema::all_fields: Use flattened_fields instead.

@muyihao looks like this is reported by clippy - yes feel free to raise another PR to fix the clippy warning, then we can land the python style change PR without touching rust code

@muyihao interestingly clippy is checking against arrow 52.2.0 despite that cargo workspace is setting it to 52.0.0. do you know if we can make clippy to respect the exact dep version for checking? we may not want to upgrade arrow just because clippy complains - the rust code fix can be done when we manually decide to upgrade arrow.

[muyihao]

Rust dependency: arrow_schema::Schema::all_fields: Use flattened_fields instead.

@muyihao looks like this is reported by clippy - yes feel free to raise another PR to fix the clippy warning, then we can land the python style change PR without touching rust code

@muyihao interestingly clippy is checking against arrow 52.2.0 despite that cargo workspace is setting it to 52.0.0. do you know if we can make clippy to respect the exact dep version for checking? we may not want to upgrade arrow just because clippy complains - the rust code fix can be done when we manually decide to upgrade arrow.

To ensure the project runs consistently across different environments, we should include the Cargo.lock file.

[xushiyan]

Rust dependency: arrow_schema::Schema::all_fields: Use flattened_fields instead.

@muyihao looks like this is reported by clippy - yes feel free to raise another PR to fix the clippy warning, then we can land the python style change PR without touching rust code

@muyihao interestingly clippy is checking against arrow 52.2.0 despite that cargo workspace is setting it to 52.0.0. do you know if we can make clippy to respect the exact dep version for checking? we may not want to upgrade arrow just because clippy complains - the rust code fix can be done when we manually decide to upgrade arrow.

To ensure the project runs consistently across different environments, we should include the Cargo.lock file.

hmmm that'd actually be an overkill, as lock file is generally not recommended for lib repo. the root fix is on the dep version itself (7566337). I've updated your style change PR - will make some setting tweaks and then merge it.

We can now close this one.