Left/right join with single row dataframe with explicit types replaces nulls with default values.

[DavideCanton]

Checks

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of Polars.

Reproducible example

>>> df = pl.DataFrame({'k': [1,2]})

>>> df
shape: (2, 1)
┌─────┐
│ k   │
│ --- │
│ i64 │
╞═════╡
│ 1   │
│ 2   │
└─────┘

>>> df2 = pl.DataFrame({'k': [1], 'int': [None], 'float':[None], 'str': [None]}, schema_overrides={'int': pl.Int64, 'float': pl.Float64, 'str': pl.String})
shape: (1, 4)
┌─────┬──────┬───────┬──────┐
│ k   ┆ int  ┆ float ┆ str  │
│ --- ┆ ---  ┆ ---   ┆ ---  │
│ i64 ┆ i64  ┆ f64   ┆ str  │
╞═════╪══════╪═══════╪══════╡
│ 1   ┆ null ┆ null  ┆ null │
└─────┴──────┴───────┴──────┘

>>> df.join(df2, on='k', how='left')
shape: (2, 4)
┌─────┬──────┬───────┬──────┐
│ k   ┆ int  ┆ float ┆ str  │
│ --- ┆ ---  ┆ ---   ┆ ---  │
│ i64 ┆ i64  ┆ f64   ┆ str  │
╞═════╪══════╪═══════╪══════╡
│ 1   ┆ 0    ┆ 0.0   ┆      │
│ 2   ┆ null ┆ null  ┆ null │
└─────┴──────┴───────┴──────┘

>>> df2_casted = pl.DataFrame({'k': [1], 'int': [None], 'float':[None], 'str': [None]}).with_columns(int=pl.col.int.cast(pl.Int64), float=pl.col.float.cast(pl.Float64), str=pl.col.str.cast(pl.String))

>>> df2_casted
shape: (1, 4)
┌─────┬──────┬───────┬──────┐
│ k   ┆ int  ┆ float ┆ str  │
│ --- ┆ ---  ┆ ---   ┆ ---  │
│ i64 ┆ i64  ┆ f64   ┆ str  │
╞═════╪══════╪═══════╪══════╡
│ 1   ┆ null ┆ null  ┆ null │
└─────┴──────┴───────┴──────┘

>>> df.join(df2_casted, on='k', how='left')
shape: (2, 4)
┌─────┬──────┬───────┬──────┐
│ k   ┆ int  ┆ float ┆ str  │
│ --- ┆ ---  ┆ ---   ┆ ---  │
│ i64 ┆ i64  ┆ f64   ┆ str  │
╞═════╪══════╪═══════╪══════╡
│ 1   ┆ 0    ┆ 0.0   ┆      │
│ 2   ┆ null ┆ null  ┆ null │
└─────┴──────┴───────┴──────┘

Log output

join parallel: true
LEFT join dataframes finished

Issue description

It seems that left joining a dataframe with another one does not yield correct results if the right dataframe has none in rows that have a match and an explicit type is specified via schema_overrides or casting. Not casting the columns yields the correct result.

It's not shown in the example, but reversing the join by doing df2_casted.join(df, on='k', how='right') behaves the same.

Expected behavior

I would expect the two joins to be equivalent, since by specification in a left join the rows matched should be kept as they are in the original dataframe.

Installed versions

--------Version info---------
Polars:              1.13.1
Index type:          UInt32
Platform:            Windows-10-10.0.19045-SP0
Python:              3.12.7 (tags/v3.12.7:0b05ead, Oct  1 2024, 03:06:41) [MSC v.1941 64 bit (AMD64)]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
great_tables         <not installed>
matplotlib           3.9.2
nest_asyncio         1.6.0
numpy                2.1.2
openpyxl             <not installed>
pandas               2.2.3
pyarrow              17.0.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>

[DavideCanton]

It seems related to df2 having a single row. If it has many rows, the issue does not show up:

>>> df = pl.DataFrame({'k': [1,2]})

>>> df
shape: (2, 1)
┌─────┐
│ k   │
│ --- │
│ i64 │
╞═════╡
│ 1   │
│ 2   │
└─────┘

>>> df2 = pl.DataFrame({'k': [1,4], 'int': [None,None], 'float':[None,None], 'str': [None,None]}, schema_overrides={'int': pl.Int64, 'float': pl.Float64, 'str': pl.String})

>>> df2
shape: (2, 4)
┌─────┬──────┬───────┬──────┐
│ k   ┆ int  ┆ float ┆ str  │
│ --- ┆ ---  ┆ ---   ┆ ---  │
│ i64 ┆ i64  ┆ f64   ┆ str  │
╞═════╪══════╪═══════╪══════╡
│ 1   ┆ null ┆ null  ┆ null │
│ 4   ┆ null ┆ null  ┆ null │
└─────┴──────┴───────┴──────┘

>>> df.join(df2, on='k', how="left")
shape: (2, 4)
┌─────┬──────┬───────┬──────┐
│ k   ┆ int  ┆ float ┆ str  │
│ --- ┆ ---  ┆ ---   ┆ ---  │
│ i64 ┆ i64  ┆ f64   ┆ str  │
╞═════╪══════╪═══════╪══════╡
│ 1   ┆ null ┆ null  ┆ null │
│ 2   ┆ null ┆ null  ┆ null │
└─────┴──────┴───────┴──────┘

[DavideCanton]

It seems that the bug shows up since version 1.13.0, the 1.12.0 it's working fine.

[ritchie46]

fixed by #19823. Will issue a release tomorrow.

[DavideCanton]

Thanks!