GH-45457: [Python] Add `pyarrow.ArrayStatistics` #45550

kou · 2025-02-17T06:06:13Z

Rationale for this change

Apache Arrow C++ can attach statistics read from Apache Parquet data to arrow::Array. If we have the bindings of the feature in Python, Python users can also use attached statistics.

What changes are included in this PR?

Add pyarrow.ArrayStatistics
Add pyarrow.Array.statistics().

Are these changes tested?

Yes.

Are there any user-facing changes?

Yes.

GitHub Issue: [Python] Add arrow::ArrayStatistics bindings #45457

kou · 2025-02-17T06:06:37Z

@github-actions crossbow submit -g python

github-actions · 2025-02-17T06:06:38Z

⚠️ GitHub issue #45457 has been automatically assigned in GitHub to PR creator.

kou · 2025-02-20T20:32:41Z

@pitrou @jorisvandenbossche Could you take a look at this?

kou · 2025-02-24T06:10:36Z

I'll merge this in a few days if nobody objects it.

pitrou

Thanks a lot @kou ! Some minor comments below, but LGTM in general.

python/pyarrow/array.pxi

pitrou · 2025-02-24T14:07:39Z

python/pyarrow/array.pxi

+        if null_count.has_value():
+            return null_count.value()
+        else:
+            return None


For the record, I've opened a Cython feature request to make this more automatic.

Thanks! I've added a comment that refers the issue.

pitrou · 2025-02-24T14:09:05Z

python/pyarrow/array.pxi

+        std::optional<arrow::ArrayStatistics::ValueType>> data.
+
+        arrow::ArrayStatistics::ValueType is
+        std::variant<bool, int64_t, uint64_t, double, std::string>.


uint64_t isn't handled below, should the docstring or the code be fixed?

Oh... The code was wrong... I've added the uint64_t case.

pitrou · 2025-02-24T14:10:26Z

python/pyarrow/array.pxi

+        raise TypeError("Do not call {}'s constructor directly"
+                        .format(self.__class__.__name__))
+
+    cdef void init(self, const shared_ptr[CArrayStatistics]& sp_statistics) except *:


except * means it could raise Python exceptions, but it doesn't here, so perhaps you can remove that annotation (though it's not really a problem either).

Thanks! I didn't know much about except in Cython...

python/pyarrow/array.pxi

It's the bindings of `arrow::ArrayStatistics`. You can get it by `pyarrow.Array.statistics()`.

Co-authored-by: Antoine Pitrou <[email protected]>

kou · 2025-02-25T05:16:50Z

@github-actions crossbow submit -g python

kou · 2025-02-25T06:57:12Z

@github-actions crossbow submit -g python

pitrou · 2025-02-25T08:58:20Z

python/pyarrow/tests/parquet/test_parquet_file.py

+    assert statistics.min == -1
+    assert statistics.is_min_exact
+    assert statistics.max == 3
+    assert statistics.is_max_exact


Can we have a test for repr(statistics) to make sure that the string representation works?

It's a good idea. I've added it.

kou · 2025-02-25T11:10:08Z

@github-actions crossbow submit -g python

github-actions · 2025-02-25T11:12:50Z

Revision: e3a20b5

Submitted crossbow builds: ursacomputing/crossbow @ actions-747dbaddf2

Task	Status
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-conda-python-3.10
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.13
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-python-3
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-python-3

conbench-apache-arrow · 2025-02-25T20:07:58Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 631fa0a.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 12 possible false positives for unstable benchmarks that are known to sometimes produce them.

### Rationale for this change Apache Arrow C++ can attach statistics read from Apache Parquet data to `arrow::Array`. If we have the bindings of the feature in Python, Python users can also use attached statistics. ### What changes are included in this PR? * Add `pyarrow.ArrayStatistics` * Add `pyarrow.Array.statistics()`. ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * GitHub Issue: apache#45457 Lead-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Sutou Kouhei <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Sutou Kouhei <[email protected]>

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Feb 17, 2025

This comment was marked as outdated.

Sign in to view

kou requested review from pitrou and jorisvandenbossche and removed request for pitrou February 18, 2025 00:45

pitrou reviewed Feb 24, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Feb 25, 2025

kou requested review from assignUser, jonkeane, raulcd, thisisnic, wgtmac and westonpace as code owners February 25, 2025 05:14

github-actions bot added Component: R Component: Parquet Component: C++ Component: Documentation awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 25, 2025

kou and others added 4 commits February 25, 2025 14:15

apacheGH-45457: [Python] Add pyarrow.ArrayStatistics

cd2aa85

It's the bindings of `arrow::ArrayStatistics`. You can get it by `pyarrow.Array.statistics()`.

Use f-string

49bf5ef

Co-authored-by: Antoine Pitrou <[email protected]>

Use f-string

5837501

Co-authored-by: Antoine Pitrou <[email protected]>

Format

d5a15f5

kou removed the Component: Documentation label Feb 25, 2025

kou removed request for assignUser, jonkeane, raulcd, thisisnic, wgtmac and westonpace February 25, 2025 05:16

Fix wrong change

bca14f5

This comment was marked as outdated.

Sign in to view

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 25, 2025

Fix English

fd6d18e

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Feb 25, 2025

This comment was marked as outdated.

Sign in to view

pitrou reviewed Feb 25, 2025

View reviewed changes

Assert repr() too

e3a20b5

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Feb 25, 2025

pitrou approved these changes Feb 25, 2025

View reviewed changes

kou merged commit 631fa0a into apache:main Feb 25, 2025
15 checks passed

kou removed the awaiting changes Awaiting changes label Feb 25, 2025

kou deleted the python-array-statistics branch February 25, 2025 13:25

kou mentioned this pull request Feb 25, 2025

[Python] Add arrow::ArrayStatistics bindings #45457

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-45457: [Python] Add `pyarrow.ArrayStatistics` #45550

GH-45457: [Python] Add `pyarrow.ArrayStatistics` #45550

kou commented Feb 17, 2025 •

edited by github-actions bot

Loading

kou commented Feb 17, 2025

github-actions bot commented Feb 17, 2025

This comment was marked as outdated.

kou commented Feb 20, 2025

kou commented Feb 24, 2025

pitrou left a comment

pitrou Feb 24, 2025

kou Feb 25, 2025

pitrou Feb 24, 2025

kou Feb 25, 2025

pitrou Feb 24, 2025

kou Feb 25, 2025

kou commented Feb 25, 2025

This comment was marked as outdated.

kou commented Feb 25, 2025

This comment was marked as outdated.

pitrou Feb 25, 2025

kou Feb 25, 2025

kou commented Feb 25, 2025

github-actions bot commented Feb 25, 2025

conbench-apache-arrow bot commented Feb 25, 2025

GH-45457: [Python] Add pyarrow.ArrayStatistics #45550

GH-45457: [Python] Add pyarrow.ArrayStatistics #45550

Conversation

kou commented Feb 17, 2025 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

kou commented Feb 17, 2025

github-actions bot commented Feb 17, 2025

This comment was marked as outdated.

kou commented Feb 20, 2025

kou commented Feb 24, 2025

pitrou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou commented Feb 25, 2025

This comment was marked as outdated.

kou commented Feb 25, 2025

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou commented Feb 25, 2025

github-actions bot commented Feb 25, 2025

conbench-apache-arrow bot commented Feb 25, 2025

GH-45457: [Python] Add `pyarrow.ArrayStatistics` #45550

GH-45457: [Python] Add `pyarrow.ArrayStatistics` #45550

kou commented Feb 17, 2025 •

edited by github-actions bot

Loading