BUG: Columns resulting from joining multiIndex dataFrame are incorrect in python 3.12.1

[yuji38kwmt]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas

df1=pandas.DataFrame({("A",""):[1,2,3]},index=["x","y","z"])
df2=pandas.DataFrame({("B",""):[11,12,13]},index=["x","y","z"])

df3 = df1.join(df2)

print(f"{df3.columns=}")

Issue Description

In Python 3.12.1, the print(f"{df3=}") statement output the following string.

df3.columns=Index([slice(None, None, None), slice(None, None, None)], dtype='object')

And df3[("A","")] occurs KeyError.

But in python 3.11.7, df3[("A","")] does not occur KeyError.

Expected Behavior

The print(f"{df3=}") statement output the following string.

df3.columns=MultiIndex([('A', ''),('B', '')])

Installed Versions

pd.show_versions()

INSTALLED VERSIONS

commit : a671b5a
python : 3.12.1.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.133.1-microsoft-standard-WSL2
Version : #1 SMP Thu Oct 5 21:02:42 UTC 2023
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : C.UTF-8

pandas : 2.1.4
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.8.2
setuptools : None
pip : 24.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.1.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.21.0
pandas_datareader : None
bs4 : None
bottleneck : None
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.2
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.4
qtpy : None
pyqt5 : None

[rajnee28]

@yuji38kwmt - can you pls provide folder path for this issue.

[yuji38kwmt]

@rajnee28

can you pls provide folder path for this issue.

The bug does not depend on folder path.

For example, you can reproduce the bug in IPython.

In [59]: import pandas
    ...:
    ...: df1=pandas.DataFrame({("A",""):[1,2,3]},index=["x","y","z"])
    ...: df2=pandas.DataFrame({("B",""):[11,12,13]},index=["x","y","z"])
    ...:
    ...: df3 = df1.join(df2)
    ...:
    ...: print(f"{df3.columns=}")
df3.columns=Index([slice(None, None, None), slice(None, None, None)], dtype='object')

[Groni3000]

I don't know what causes the problem, but concat is a little workaround:

df3 = pd.concat([df1, df2], axis=1) #there is join options too

[yuji38kwmt]

The following code behaved expectedly.

In [89]: import pandas
    ...:
    ...: df1=pandas.DataFrame({("A","X"):[1,2,3]},index=["x","y","z"])
    ...: df2=pandas.DataFrame({("B","Y"):[11,12,13]},index=["x","y","z"])
    ...:
    ...: df3 = df1.join(df2)

In [90]: print(f"{df3.columns=}")
df3.columns=MultiIndex([('A', 'X'),
            ('B', 'Y')],
           )

Apparently the problem occurs when the key with label 1 is an empty string.

[rhshadrach]

Confirmed that this still exists on 2.2.1, but not yet on main. Further investigations and PRs to fix are welcome!

[yuji38kwmt]

I did a litter investigated the bug.

The following code creates a pd.Series containing index which of name is slice(None, None, None).
https://github.com/pandas-dev/pandas/blob/v2.2.1/pandas/core/reshape/merge.py#L837-L838

Trial1

• Python 3.12.1
• pandas 2.2.1

In [21]: df=pandas.DataFrame({"A":[1,2,3]},index=["x","y","z"])

In [22]: df[:]
Out[22]:
   A
x  1
y  2
z  3

In [24]: df[:].columns
Out[24]: Index(['A'], dtype='object')



In [25]: df11=pandas.DataFrame({("A","x"):[1,2,3]},index=["x","y","z"])

In [26]: df11[:]
Out[26]:
   x
x  1
y  2
z  3

In [27]: df11[:].columns
Out[27]: Index(['x'], dtype='object')

In [28]: df12=pandas.DataFrame({("A",""):[1,2,3]},index=["x","y","z"])

# Look at the code !
In [29]: df12[:]
Out[29]:
x    1
y    2
z    3
Name: slice(None, None, None), dtype: int64

In [30]: type(df12[:])
Out[30]: pandas.core.series.Series

Trial2

• Python 3.11.7
• pandas 2.2.1

In [3]: df11=pandas.DataFrame({("A","x"):[1,2,3]},index=["x","y","z"])

In [4]: df11[:]
Out[4]:
   A
   x
x  1
y  2
z  3

In [5]: df11[:].columns
Out[5]:
MultiIndex([('A', 'x')],
           )

In [6]: df12=pandas.DataFrame({("A",""):[1,2,3]},index=["x","y","z"])

In [7]: df12[:]
Out[7]:
   A

x  1
y  2
z  3

In [8]: df12[:].columns
Out[8]:
MultiIndex([('A', '')],
           )

[tehunter]

Also encountering this issue. I'll see if I can come up with a bug fix.

It seems related to Python 3.12 changes, not Pandas changes. I'm failing all of my tests for py3.12 against Pandas 1.5, 2.0, 2.1, and 2.2.

[tehunter]

As @yuji38kwmt noted, the error is due to the way in which Pandas handles the [:] indexer in Python 3.12. This comes from a change in Python 3.12 which makes slice objects hashable. This is leading to a different control flow in the DataFrame.__getitem__().

Also note that slice(None) in df.columns returns True.

I'll be submitting a PR. I think it should be as easy as switching the position of the conditions in __getitem__, since slice and is_hashable were mutually exclusive pre-3.12.

[yuji38kwmt]

@tehunter
I ran it in the following environment, but the result was the same.
What has changed with #58043 ?

• Python 3.12.4
• pandas 2.2.3

[tehunter]

@tehunter
I ran it in the following environment, but the result was the same.
What has changed with #58043 ?

• Python 3.12.4
• pandas 2.2.3

It appears the fix was placed in the planned changes for the pandas 3.0 release. I'm not on the dev team so I don't know when that will be released.

[yuji38kwmt]

Thank you! Got it.