sum in window function deals with NULL's differently after upgrade to latest polars version

[MatthiasRoels]

Checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

It is not really a bug, but more a change in behaviour when running the code below:

import polars as pl

df = pl.DataFrame(
    {
        "id": [1, 2, 2, 3, 3],
        "val": [None, 1, None, 2, 2]
    }
)

result = df.with_columns(pl.col("val").sum().over("id").alias("windowed_sum"))
result
>> shape: (5, 3)
┌─────┬──────┬──────────────┐
│ id  ┆ val  ┆ windowed_sum │
│ --- ┆ ---  ┆ ---          │
│ i64 ┆ i64  ┆ i64          │
╞═════╪══════╪══════════════╡
│ 1   ┆ null ┆ 0            │
│ 2   ┆ 1    ┆ 1            │
│ 2   ┆ null ┆ 1            │
│ 3   ┆ 2    ┆ 4            │
│ 3   ┆ 2    ┆ 4            │
└─────┴──────┴──────────────┘

Issue description

I noticed a change in behaviour of how NULL values are handled in a sum over a partition after I upgraded from polars 0.18.4 to the latest version. The new behaviour is different from e.g. pyspark which has the same behaviour as we had in the old version. I was wondering if this was intentional or if this is a regression introduced in one of the later versions (after 0.18.4)?

Expected behavior

When running the snippet above, I was expecting to see the following result (note the difference in the last column of the first row in the result)

import polars as pl

df = pl.DataFrame(
    {
        "id": [1, 2, 2, 3, 3],
        "val": [None, 1, None, 2, 2]
    }
)

result = df.with_columns(pl.col("val").sum().over("id").alias("windowed_sum"))
result
>> shape: (5, 3)
┌─────┬──────┬──────────────┐
│ id  ┆ val  ┆ windowed_sum │
│ --- ┆ ---  ┆ ---          │
│ i64 ┆ i64  ┆ i64          │
╞═════╪══════╪══════════════╡
│ 1   ┆ null ┆ null         │
│ 2   ┆ 1    ┆ 1            │
│ 2   ┆ null ┆ 1            │
│ 3   ┆ 2    ┆ 4            │
│ 3   ┆ 2    ┆ 4            │
└─────┴──────┴──────────────┘

Installed versions

New version:

>>> pl.show_versions()
--------Version info---------
Polars:              0.18.13
Index type:          UInt32
Platform:            Linux-5.10.184-175.749.amzn2.x86_64-x86_64-with-glibc2.31
Python:              3.10.12 (main, Jul 28 2023, 05:51:21) [GCC 10.2.1 20210110]

----Optional dependencies----
adbc_driver_sqlite:  <not installed>
cloudpickle:         2.2.1
connectorx:          <not installed>
deltalake:           <not installed>
fsspec:              2023.6.0
matplotlib:          3.7.1
numpy:               1.23.5
pandas:              1.5.3
pyarrow:             11.0.0
pydantic:            1.10.12
sqlalchemy:          2.0.19
xlsx2csv:            <not installed>
xlsxwriter:          <not installed>

Old version:

>>> pl.show_versions()
--------Version info---------
Polars:      0.18.4
Index type:  UInt32
Platform:    Linux-5.10.184-175.749.amzn2.x86_64-x86_64-with-glibc2.31
Python:      3.10.12 (main, Jul  4 2023, 06:15:20) [GCC 10.2.1 20210110]

----Optional dependencies----
numpy:       1.23.5
pandas:      1.5.3
pyarrow:     11.0.0
connectorx:  <not installed>
deltalake:   <not installed>
fsspec:      2023.6.0
matplotlib:  3.7.1
xlsx2csv:    <not installed>
xlsxwriter:  <not installed>
>>> 

[jpteb]

This seems to be intended behavior, see: #5773 and #5604.

I spoke too soon and dug a little deeper, but it still seems to be intended behavior: #9558 and #9576.
It got released with 0.18.5.

[orlp]

I think this is intended behavior. Regardless of what other systems do, sum() over a column of numbers returning null is nonsense. null indicates missing values, but a sum of a collection can never be 'missing', if the collection is empty or consists of only missing values, the sum is simply zero.

[MatthiasRoels]

Thanks for the quick reply and confirming it is indeed intentional! Do you then perhaps have a suggestion on how to modify the snippet above so that I get the same result as with v0.18.4?

You can close the issue btw 😉

[orlp]

@MatthiasRoels

Do you then perhaps have a suggestion on how to modify the snippet above so that I get the same result as with v0.18.4?

def sum_empty_null(col):
    return pl.when((~col.is_null()).any()).then(col.sum())

result = df.with_columns(sum_empty_null(pl.col("val")).over("id"))

[MatthiasRoels]

Thanks a lot for the help!