findKNN on "DelayedMatrix" or "ReloadedMatrix"

[alikhuseynov]

Hi, is findKNN compatible with "DelayedMatrix" or "ReloadedMatrix"?

out <- BiocNeighbors::findKNN(counts(milo_test), k=10)
Error: unable to find an inherited method for function ‘findKNN’ for signature ‘X = "ReloadedMatrix"’

See this issue when using in miloR -> MarioniLab/miloR#360

[LTLA]

As of BiocNeighbors version 2.0.0, you'll have to convert it to an ordinary matrix - see the NEWS. Note that the older versions did this under the hood as there's no support for anything other than a regular matrix in the C++ code. So the need for an explicit coercion in the latest version isn't any less efficient than the implicit coercion in the old version, but it does at least require the user to recognize that a coercion is being performed.

[alikhuseynov]

I see, Thanks. I think coercion to sparse matrix under the hood makes more sense, and one wouldn't get into such errors, when warning or a message is issued saying that some coercion is happening.

[DarioS]

I am working with a data set of 2.7 million cells from 975 donors, so memory efficiency is definitely important.

[LTLA]

Just to be clear: at some point, a collection of dense vectors will inevitably be made by all supported NN search algorithms. This is how implementations like Annoy and HNSW store the data in their internal search indices, and I don't control that.

For this issue, the relevant question is whether we can afford to create a dense matrix in R before feeding it to the BiocNeighbors functions. It is theoretically possible to avoid this conversion, but at best, this will cut memory usage by 2/3. (We avoid creating a double-precision matrix, but every approximate method still creates a single-precision dense matrix.)

I have not considered this optimization as a priority, given that (i) the memory savings aren't spectacular, and (ii) most single-cell NN searches are already operating on a dense matrix of low-dimensional embeddings.