Operating on Tabular Data

[vishakh]

Greetings,

We need hep with figuring out how to easily perform nontrivial operations with our data.

Users provide genomic data in FASTQ format files such as https://raw.githubusercontent.com/Cryptonomic/MonadicDNA/e1f71f4f7d28b5dae183a43f94b7f3bf4fa59453/services/nillion-interactor/testdata/hu278AF5_20210124151934.txt.

So far, we have been int-encoding the data row by row and storing each row as an individual secret on Nillion: https://github.com/Cryptonomic/MonadicDNA/blob/e1f71f4f7d28b5dae183a43f94b7f3bf4fa59453/services/nillion-interactor/app.py#L72C11-L100

This lets us perform trivial (but valuable) computations relating to disease risk, e.g. https://github.com/Cryptonomic/MonadicDNA/blob/e1f71f4f7d28b5dae183a43f94b7f3bf4fa59453/services/nillion-interactor/programs/thrombosis.py.

Ideally, we would like to store large chunks of the input data and process them within Nada programs as this unlocks a lot of additional use cases related to ancestry, data analysis, machine learning, etc. The docs indicate this is not possible at the moment and even blobs are purely for storage and retrieval.

We haven't yet looked at the nada-numpy and nada-ai libraries in detail but their presence indicates some intention to operate on tabular or arrayed data.

Is there any way to accomplish what we want to do?

Thank you.

EDIT: I just had another quick look at nada-numpy. It seems to create abstractions over individual secrets to allow array operations. However, it's still not clear to me whether we can store and fetch secrets as arrays just as we do integers and bools in way that is compatible with our use case with everything like permissioning working the same way. It's also unclear what the performance tradeoffs and maximum data sizes are.

EDIT2: I see on https://github.com/NillionNetwork/nada-numpy/blob/main/examples/linear_regression/tests/linear_regression_256_2.yaml that we still need to store secrets cell by cell. I wonder whether this can be abstracted over using some syntactic sugar. The tests also look like they don't actually load a lot of data so I also wonder what the eventual goal is for maximum data size.

[mathias-nillion]

Hi @vishakh,
Looks like an exciting project!

I think you are indeed looking for nada-numpy here.
E.g. the example for matrix multiplication should give a decent feel for how it works.

You're spot on in the sense that we do try to make array storing & fetching as similar to the equivalent operations with integers, bools, etc. as possible.
In essence, you can store a NadaArray by giving it a name and then just fetch it via that same name. The only small difference is that you need to define a fixed datatype for the array (e.g. SecretInteger or na.SecretRational if you're working with floating point numbers).
As you rightfully spotted in the test YAML, behind the scenes each element in the array will still be stored individually but nada-numpy handles that for you.

Permissioning etc. still works in the same way. Migrating to nada-numpy should not lead to any issues here - let us know if it does though!
NadaArrays supports many of the same data wrangling operations you'd find in numpy arrays. These operations are highly optimized by our data/AI team. I'd highly recommend using it.

As for maximum data size, this is highly dependent on the specific operations you want to use, so difficult to put a single number on it.
That said, in the both the nada-numpy examples and the nada-ai examples, we provide information regarding the maximum dataset sizes the examples have been tested on. Hopefully this gives some intuition.

Hope this helped!

[mathias-nillion]

Hi @vishakh, funny you mention this as we are actually currently updating our testing logic to work a bit more nicely with data/AI use cases.
Sounds like this would be a welcome improvement for you! Happy to keep you in the loop here.

As for how to store NadaArrays, follow the Python SDK example.
In the example, the sample array is indeed all ones but you can drop in your own dataset instead.
Hope this helps!

[vishakh]

Excellent, do keep me in the loop!

Regarding NadaArrays, just to confirm, you are saying the best way is to construct a NadaArray in an ad hoc manner and then store in on the network, right? Utilities for doing this out of the box from various data sources will be coming later, I imagine?

[mathias-nillion]

Will do!
That is currently the way I'd recommend approaching it yes.
By these utilities, you mean e.g. a way to connect a Nada program to a .csv file that recurrently syncs?

[vishakh]

I meant adding functionality to the Python SDK which loads data from, let's say, a CSV file and returns a NadaArray. Otherwise, everyone will have to roll their own solution.

[vishakh]

@mathias-nillion I did a little test to see how much tabular data I could store using nada-numpy.

https://github.com/Cryptonomic/MonadicDNA/blob/2d86c291233b0457283a8c28b0ced2d5bc0e4d28/services/nillion-interactor/large_dataset-test.ipynb

The good news: Loading int-encoded gene data works nicely with nada-numpy.

The bad news: We seem to reach a limit at the 10,000 record mark (RuntimeError: requesting price quote: price quote query failed: price quote query failed: payload size (11171406) exceeds maximum (6291456)). We need to be able to store 500k-1m records per user to make our app work smoothly.

Is this some kind of fundamental Nillion limit? If so, are there any plans to extend it?

P.S. Would be good to promote the functions in common/utils to package-level members.