fix: role.fromIndex should be +ve integer

[gmaclennan]

Fairly minor fix - fromIndex should be a positive integer, not just any number

[EvanHahn]

I wonder if we should update the proto to use uint32 instead...

[gmaclennan]

Yeah, makes sense. It increases the max supported number of records in a hypercore. Is 2^32 records per hypercore per device enough? since this currently refers to the index of the auth cores, that seems enough, but I've also been thinking of adding a fromIndex for other cores, so that peers could choose not to index a particular peer beyond fromIndex. I can see it's feasible that a blob core could have > 2^32 records, because blobs are chunked into 64kb. So maybe just make them all uint64?

[EvanHahn]

I don't know enough to make a decision here.

If each chunk is exactly 64 kilobytes, then I think uint32 is fine. 64 bytes × 1000 × 2³² = 274877906944000 bytes = = ~275 terabytes.¹

If chunks could be 1 byte, then that only allows about 4.3 gigabytes, which feels a little restrictive, and we might want to use uint64.

We should keep in mind that 64-bit integers don't fit in JavaScript numbers. It looks like ts-proto uses a custom Long class by default, but we can make it use bigints with --ts_proto_opt=forceLong=bigint. Not sure if that affects anything else, such as JSON serialization.

In any case, I think this is best done in a followup.

Footnotes

1. This assumes a kilobyte is 1000 bytes instead of 1024, which is more conservative in this case. ↩

[gmaclennan]

ok also sanity-checking number of records, OSM currently has about 8 billion nodes and 1 billion ways. With a uint32 we would be limited to about 4 billion records in a single hypercore, so it wouldn't be possible to import the entire OSM db into a single hypercore, but that would probably be a bad idea anyway. Anyway this is just a limit for evaluating capabilities, only really relevant with self-created data, so I think the limits are fine.