GH-14932: [Python] Add python bindings for JSON streaming reader

[pan-x-c]

Rationale for this change

The C++ arrow has a JSON streaming reader which is not exposed on the Python interface.

What changes are included in this PR?

This PR is based on #33761. It adds the open_json method to open a streaming reader for a JSON file.

Are these changes tested?

Yes

Are there any user-facing changes?

Yes. A new open_json method has been added to the Python interface, located at pyarrow.json.open_json, and its parameters are the same as the pyarrow.json.read_json

• GitHub Issue: [Python] Expose streaming JSON reader #14932

[github-actions]

⚠️ GitHub issue #14932 has been automatically assigned in GitHub to PR creator.

[pitrou]

This looks mostly excellent @pan-x-c , thank you.

Here are a couple minor comments.

Also, could you add the new API to the docs in https://github.com/apache/arrow/blob/main/docs/source/python/api/formats.rst#json-files and perhaps mention it in https://github.com/apache/arrow/blob/main/docs/source/python/json.rst ?

[pan-x-c]

I have fixed the above comments and some similar issues in the csv module. I also added a brief introduction to open_json based on the doc of open_csv.

[pitrou]

Thanks a lot for the update, one comment remaining below.

[pitrou]

Thanks a lot for your contribution @pan-x-c ! This looks good to me now, I will merge if CI is green.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 16c7f1a.

There were 8 benchmark results with an error:

• Commit Run on arm64-t4g-2xlarge-linux at 2025-02-06 16:20:42Z
  • tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-03, scale_factor=1
  • tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-03, scale_factor=1
• and 6 more (see the report linked below)

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.