[BUG] cuDF inner join and iloc seemingly broken

[danieldemarch]

Assume ticks is a list of cuDF DataFrames. If it matters, these cuDF DataFrames were made by converting pandas DataFrames to cuDF.

dfa= ticks[i].copy()
dfb= ticks[j].copy()

dfc= dfa.join(dfb, how='inner', rsuffix="2").copy()

indy = int(len(dfc)*0.65)

dummy = dfc.iloc[:indy].copy()

print(dummy.tail(5))

The .copy() can be removed optionally, I get the same behavior with/without them.

This behavior produces two errors. First, the dataframes are all sorted in chronological order before the inner join. Post-inner join, the sort is wrong. It's like halfway sorted but with random insertions - e.g. a bunch of 04/04/2023 timestamps in a row and then a random o4/03/2023, before back to 04/04/2023.

Second, the "dummy = dfc.iloc[:indy].copy()" produces inconsistent results when running the same block of code twice. It just doesn't work. It seems to cycle between 5-6 different outcomes if I repeatedly run the cell.

Sticking with vanilla pandas does not produce either of the above errors. It's purely the act of converting to cuDF DataFrames that produces this bug.

Very disappointed. This is the sort of thing unit testing is supposed to catch. If it's some bizarre setting on the cuDF DataFrame - no setting should produce this outcome.

[shwina]

Thanks for reporting, @danieldemarch. Is this reproducible with any data or specific to a certain dataset?

[shwina]

It's worth noting that by default, cuDF's merge/join operations do not guarantee output ordering or sortedness (https://docs.rapids.ai/api/cudf/legacy/api_docs/api/cudf.dataframe.merge).

Does using sort=True in the call to join help?

[danieldemarch]

Pardon - so by default, arrays will be essentially randomly reordered after a merge?

[shwina]

Correct. Please see this section of our documentation that provides more details about result ordering: https://docs.rapids.ai/api/cudf/stable/user_guide/pandas-comparison/#result-ordering

[vyasr]

There has also been a lot of discussion on this in #1781 (and #5286, although that's referring to a different function but the same sorting-related behavior).

[danieldemarch]

Thanks much!

[wence-]

I'll close this in favour of the broader discussions, but just to summarise that this is "working as expected", it's just the expectations are a bit different.

If there are cases where you might expect to be able to recreate the original order from the index but can't please do open issues since those would be actual bugs.