[FEA] Support various outcomes for NaN equality comparison in equality comparator

[ttnghia]

Currently, cudf::equality_compare specialized for floating point numbers considers NaNs as equal. That result may not be desirable in some cases. Depending on situations, sometimes we want to have NaNs considered as equal, but some other time we want the opposite.

We should reimplement cudf::equality_compare and the corresponding comparators chain (element comparator, row comparator etc), adding an optional parameter:

enum class nan_equality /*unspecified*/ {
  ALL_EQUAL,  ///< All NaNs compare equal, regardless of sign
  UNEQUAL     ///< All NaNs compare unequal (IEEE754 behavior)
};

Currently, there are many APIs that support this parameter (such as lists::drop_list_duplicates) but these APIs implement such support locally. As more and more APIs adopt it, NaN handling should be supported in a more structural way.

[jrhemstad]

See also #4760

I think this change to the row comparators is fine so long as it is proven that it does not impact performance for all existing cases.

However, I still maintain that it is a bad idea to make any promises that we will support the full extent of NaN nonsense that Spark does.

[ttnghia]

I'm thinking about a way that allows having different NaN outcomes while that has minimal impact on cudf build and its downstream usage. One way for this is to have template internal APIs (similar to #10870) but we only instantiate one NaN config in cudf (which is the default, current config). The remaining config (which Spark wants) will be instantiated somewhere else (rapids-spark-jni).

[jrhemstad]

I'm thinking about a way that allows having different NaN outcomes while that has minimal impact on cudf build and its downstream usage. One way for this is to have template internal APIs (similar to #10870) but we only instantiate one NaN config in cudf (which is the default, current config). The remaining config (which Spark wants) will be instantiated somewhere else (rapids-spark-jni).

That is the solution I described here: #4760 (comment)

It's possible for some things, but it would be a good bit of work to make it robust.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[ttnghia]

Now NaN handling should be good with the current experimental row comparator. Close this.