MAINT: Support NumPy 2 and build Python 3.12 wheels

[seberg]

This adds NumPy support, and for good sport also 3.12 wheels. I didn't bother with supporting 3.8 (which NumPy 2 doesn't support) for wheels, since scikit-learn doesn't either anymore and you can use old wheels.

Tests pass locally, and will run with NumPy 2 now anyway, I presume (not that there are many).

[seberg]

There seems to be a further test error:

================================================================================== FAILURES ===================================================================================
________________________________________________________________________ test_hdbscan_distance_matrix _________________________________________________________________________

    def test_hdbscan_distance_matrix():
        D = distance.squareform(distance.pdist(X))
        D /= np.max(D)
    
        labels, p, persist, ctree, ltree, mtree = hdbscan(D, metric="precomputed")
        # number of clusters, ignoring noise if present
        n_clusters_1 = len(set(labels)) - int(-1 in labels)  # ignore noise
        assert n_clusters_1 == n_clusters
    
        labels = HDBSCAN(metric="precomputed").fit(D).labels_
        n_clusters_2 = len(set(labels)) - int(-1 in labels)
        assert n_clusters_2 == n_clusters
    
        validity = validity_index(D, labels, metric="precomputed", d=2)
>       assert validity >= 0.6
E       assert 0.5441246857483076 >= 0.6

hdbscan/tests/test_hdbscan.py:165: AssertionError
=========================================================================== short test summary info ===========================================================================
FAILED hdbscan/tests/test_hdbscan.py::test_hdbscan_distance_matrix - assert 0.5441246857483076 >= 0.6
============================================================================== 1 failed in 0.73s ==============================================================================

That test error seems unrelated to NumPy 2? (I downgraded NumPy again) but I am not sure what it is related to, since downgrading scipy/sklearn didn't seem to do much. Some round-about issue with Cython?

[seberg]

Ah, the:

E   TypeError: 'numpy.float64' object cannot be interpreted as an integer

is specific to certain versions of Cython I think.

node_list = sorted(stability.keys(), reverse=True)

in get_clusters contains floats for the nodes. Which seems strange, but if that needs to work, the easiest is maybe to just add a:

node_list = [int(n) for n in node_list]

since I suspect that was always a requirement.

[seberg]

I don't like it, but it works. Cython and NumPy have a weird interaction here with allowing conversion from float or not depending on both Cython and NumPy versions.

If you want to convert from float, you could also just cdef it to long or int, IIRC that should always work, and I don't think you care about mroe than 2**31 clusters....

[lmcinnes]

I do agree that there are some weird interactions here. It is down to the fact that at some point the values are contained in a single numpy array of type float64 (since other values in the array are floats) and this was "fine", but it is causing catches now in the cython/numpy interactions. I think this workaround looks fine under the circumstances.

[seberg]

Seems correct, but may warrent a look. Anything else doesn't make sense to me anyway, the order="F" was breaking things when switching to isin, which doesn't flatten the result (keeps shape of vertices).

The other places want to work with 1-D arrays as far as I can tell, so this distinction doesn't matter.

[lmcinnes]

Yes, this should be correct. isin is a much nicer way to do this, but didn't exist back when this was first written.

[lmcinnes]

Thanks for taking a loom at this. There are definitely some difficulties. There are good chunks of the codebase that are pretty old at this point and fail to use modern idioms that would make things easier. If you can change the testing grid to drop python 3.7 and 3.8, and add 3.12 (so all the tests come out green) that would be appreciated.

[seberg]

Done, I guess it'll probably pass. The fact that floats are passed around as cluster IDs probably slip some things.

It might be that the integer conversion doesn't matter anymore so long you build with NumPy 2, but then I am not sure that it builds right with 1.x and cython 2, the integer conversion problem may be cropping up in other places as well.

Not sure I can help with auditing that if that is the case. The hacky way it could be solved is to use steal soething like this from the NumPy definitions (or just decide that int is sufficient anyway):

cdef extern from "numpy/arrayobject.h":
    # It would be nice to use size_t and ssize_t, but ssize_t has special
    # implicit conversion rules, so just use "long".
    # Note: The actual type only matters for Cython promotion, so long
    #       is closer than int, but could lead to incorrect promotion.
    #       (Not to worrying, and always the status-quo.)
    ctypedef signed long npy_intp

(the good way is probably a proper audit to actually use integers)

[lmcinnes]

Thanks for this it is honestly greatly appreciated!

[seberg]

Thanks, would be cool if you can do a release soon (and remember that if there is an issue with integers and Cython, that is plausible for Cython 3 + old NumPy 1.x).
Although, it sounds like I may want to suggest looking at fast_hdbscan also if that is more maintained.