feat(python,rust): build_info() provides detailed information how polars was built #5423

slonik-az · 2022-11-04T00:16:15Z

(resolves #5422)

[new functionality is behind build_info feature gate]

polars.build_info() returns dictionary with hierarchically organized information about how this polars was built -- features enabled, dependencies and their versions, compilation details, optimization/debug flags, rustc version, git commit information, and more.

This information is useful when troubleshooting polars deployments.

zundertj

Left a couple of code-specific questions and suggestions (Python only).

On a high level, the most important question: why not integrate this information in pl.show_versions?

And could you add an example output here (and in the docstring), so we know what we are talking about?

zundertj · 2022-11-05T07:21:29Z

py-polars/polars/show_versions.py

+
+
+build_info = _closure_()
+del _closure_


Why would the method need to be deleted?

_closure_ should be called only once. After it is deleted it cannot be called at all. Otherwise a user can call it again as

from polars.show_versions import _closure_ _closure_()

Python unfortunately does not have proper private visibility. Closures is the only way to emulate private.

I decided to refactor the code. _closure_ is gone altogether and code is simplified.

def build_info(): moved from polars/show_versions.py into polars/build_info.py to avoid any confusion with show_versions(). Pushed PR update.

zundertj · 2022-11-05T07:23:04Z

py-polars/tests/unit/test_show_versions.py

+def test_build_info() -> None:
+    build_info = pl.build_info()
+    features = build_info.get("features", {})
+    if features:


It would be nice if we can test both branches here somehow.

I think the code already tests both branches:

When build_info feature gate in enabled features field is populated and tested for presence of BUILD_INFO feature

Otherwise without build_info feature the only element present in dictionary is version and its presence is tested in the else part of the code (line 18, not shown in the snippet).

Moved testing code into test_build_info.py file.

zundertj · 2022-11-05T07:23:43Z

py-polars/polars/show_versions.py

+    return build_info
+
+
+build_info = _closure_()


Is there a specific reason to run this on import polars? Why not leave as a method?

It is a function at the same level as show_versions and version. If changing to method, of what class?

zundertj · 2022-11-05T07:24:29Z

py-polars/polars/show_versions.py

@@ -69,3 +70,36 @@ def _get_dep_version(dep_name: str) -> str:

    # all our dependencies (as of 2022-08-11) implement __version__
    return module_version
+
+
+def _closure_() -> _typing_.Callable[[], _typing_.Any]:


The method name could be more descriptive. I propose to call this build_info, and import this method directly in __init__.py

The function is already called build_info. Closure _closure_ is here to create a private scope and initiate private variables that build_info() returns. Closure executed once and then deleted. This is standard way to overcome python's limitation of not having proper private or static variables.

_closure_() is gone now. See the updated PR.

zundertj · 2022-11-05T07:24:48Z

py-polars/polars/show_versions.py

@@ -3,6 +3,7 @@
 import importlib
 import platform
 import sys
+import typing as _typing_


Why not do from typing import Callable, Any? The underscores are not needed either?

Good call. I will change it and import only specific elements of typing.

slonik-az · 2022-11-05T10:02:55Z

On a high level, the most important question: why not integrate this information in pl.show_versions?

pl.show_versions() is already quite heavy and does not return anything (None). pl.build_info() returns a large dictionary with lots of build information (for example, 200+ rust dependencies with their versions). It should be up to a user what parts of this tree to process and how. Also pl.show_versions() indiscriminately prints to stdout in a fixed format without possibility to redirect output -- an anti-pattern if used for logging. Among other things pl.build_info() is aimed at use in logging and deployment troubleshooting.

To avoid confusion with show_versions(), I can move build_info() into its own file build_info.py. The same for the tests.

slonik-az · 2022-11-05T10:10:17Z

And could you add an example output here (and in the docstring), so we know what we are talking about?

Good call. I will show structure of the output in the docstring. The returned dictionary is large (it lists all 200+ polars rust dependencies and their versions). For brevity, I will include partial output.

…ars was built (implements pola-rs#5422) [new functionality is behind `build_info` feature gate] `polars.build_info()` returns dictionary with hierarchically organized information about how this polars was built -- features enabled, dependencies and their versions, compilation details, optimization/debug flags, rustc version, git commit information, and more. This information is useful when troubleshooting polars deployments.

@zundertj

Many thanks to @zundertj for insightful comments/suggestions.

slonik-az · 2022-11-05T14:49:46Z

Refactored python code per code review. Many thanks to @zundertj for insightful comments/suggestions. Also rebased the PR onto the latest master.

zundertj · 2022-11-06T09:34:21Z

@slonik-az : one thing I should have mentioned: could you also add an entry to this function in the Python API reference? I know this one is merged, but in a new PR.

ritchie46 · 2022-11-06T09:36:34Z

Thanks @slonik-az for the PR and thank you @zundertj for the review.

slonik-az · 2022-11-06T23:03:03Z

could you also add an entry to this function in the Python API reference? I know this one is merged, but in a new PR.

Will do.

…la-rs#5423)

slonik-az · 2022-11-07T17:17:35Z

could you also add an entry to this function in the Python API reference? I know this one is merged, but in a new PR.

Done in PR #5442

github-actions bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars rust Related to Rust Polars labels Nov 4, 2022

slonik-az mentioned this pull request Nov 4, 2022

Provide detailed build information in the installed polars #5422

Closed

slonik-az force-pushed the build-info branch from 4f15a90 to fcf92eb Compare November 4, 2022 01:59

zundertj suggested changes Nov 5, 2022

View reviewed changes

slonik-az added 3 commits November 5, 2022 10:23

Fixing mypy parsing error

de14a55

Refactor build_info() python code per code review pola-rs#5423

e5cceb4

Many thanks to @zundertj for insightful comments/suggestions.

slonik-az force-pushed the build-info branch from fcf92eb to e5cceb4 Compare November 5, 2022 14:44

ritchie46 merged commit 1f45cb4 into pola-rs:master Nov 6, 2022

slonik-az added a commit to slonik-az/polars that referenced this pull request Nov 7, 2022

chore(python,doc): add build_info() to the API doc (followup to PR po…

ed8d742

…la-rs#5423)

slonik-az mentioned this pull request Nov 7, 2022

docs(python): add build_info() to the API doc #5442

Merged

stinodego mentioned this pull request Nov 24, 2022

build_info feature causes caching problems during Python Polars development #5612

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(python,rust): build_info() provides detailed information how polars was built #5423

feat(python,rust): build_info() provides detailed information how polars was built #5423

slonik-az commented Nov 4, 2022 •

edited

Loading

zundertj left a comment

zundertj Nov 5, 2022

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az Nov 5, 2022 •

edited

Loading

zundertj Nov 5, 2022

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az Nov 5, 2022

zundertj Nov 5, 2022

slonik-az Nov 5, 2022

zundertj Nov 5, 2022

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az Nov 5, 2022

zundertj Nov 5, 2022

slonik-az Nov 5, 2022

slonik-az commented Nov 5, 2022 •

edited

Loading

slonik-az commented Nov 5, 2022

slonik-az commented Nov 5, 2022

zundertj commented Nov 6, 2022

ritchie46 commented Nov 6, 2022

slonik-az commented Nov 6, 2022

slonik-az commented Nov 7, 2022

feat(python,rust): build_info() provides detailed information how polars was built #5423

feat(python,rust): build_info() provides detailed information how polars was built #5423

Conversation

slonik-az commented Nov 4, 2022 • edited Loading

zundertj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slonik-az Nov 5, 2022 • edited Loading

Choose a reason for hiding this comment

slonik-az Nov 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slonik-az Nov 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slonik-az Nov 5, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

slonik-az commented Nov 5, 2022 • edited Loading

slonik-az commented Nov 5, 2022

slonik-az commented Nov 5, 2022

zundertj commented Nov 6, 2022

ritchie46 commented Nov 6, 2022

slonik-az commented Nov 6, 2022

slonik-az commented Nov 7, 2022

slonik-az commented Nov 4, 2022 •

edited

Loading

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az Nov 5, 2022 •

edited

Loading

slonik-az commented Nov 5, 2022 •

edited

Loading