[Bounty] Add raw storage operations

[willemneal]

1. 1. NEAR Bounty Terms

Before beginning work on the bounty, you must submit a proposal. Only if your proposal is accepted will you be able to claim the reward of the bounty.

1. 1. Description

Currently when writing to and reading from storage data is serialized and deserialized. This is useful when the the contract needs to understand or process the data. However, in many use cases the data doesn't need to be altered, just written and read by the caller of the contract.

For example, consider a contract registry, which allows users to store contract binaries that can be use in a generic factory for deploying contracts.

1. 1. Context

The two host functions for reading/writing storage are:

 function storage_write(
 key_len: u64,
 key_ptr: u64,
 value_len: u64,
 value_ptr: u64,
 register_id: u64
 ): u64;

function storage_read(
 key_len: u64,
 key_ptr: u64,
 register_id: u64
 ): u64;

This C-like interface informs the host of the size and location of the bytes to be stored and read. In the case of storage_read, the corresponding value read is placed into a register. The host provides a hash map of registers, which maps id's to byte arrays. There are two more functions needed to interact with the registers:

 function register_len(register_id: u64): u64;
 function read_register(register_id: u64, ptr: u64): void;

The first function returns the number of bytes that should be allocated and the second writes those bytes to ptr.

Thus typical interaction with storage requires both serialization and memory. However, in any function that takes a length and a pointer, if a length of zero is passed, the pointer is treated as a register_id. The register is read instead of the contracts memory. For example,

export function setFoo()

 { input(1); // Reads input into register 1\. storage_write(key_len, key_ptr, 0, 1, 1); // Uses contents of register 1 and if an old value was present it is written into register1 value_return(0, 1); // Returns the old value of Foo // value_return(value_len: 64, value_ptr: u64): void } 

This can be very useful for large values as you can skip the expensive copying and serializing.

1. 1. Relevant repos or issues
2. 1. API

class Storage 

 { ...... /** * Write value of key into register. Return value is errorcode. `1` means sucess. * Key will need to be encoded into bytes see `sdk/util`. */ static read_raw(key: string, register_id: u64): u64; static write_raw(key: string, register_id: u64): u64; /** * Write input into register and then write that into storage at key. */ static write_input(key: string, register_id: u64 = 0): void; } 

namespace env 

 { ... export function return_raw(register_id: u64): void; //Hint in above example. } 

The same is true for every collection class. For example,

class PersistentMap<K, V>

 { ................................ // Returns the register used, default is 0\. // Show throw an error if key is not present. get_raw(key: string, register_id: u64 = 0): u64; // Write contents of register to corresponding key set_raw(key: string, register_id: u64): u64; } 

1. 1. Acceptance Criteria

Each of the follow also includes tests

• Storage static methods
• env function
• PersistentMap
• PersistentVector
• PersistentDeque
• PersistentSet
• PersistentUnorderedMap
• AVL Tree

1. 1. Bounty

300 DAI

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

Work has been started.

These users each claimed they can complete the work by 266 years, 3 months from now.
Please review their action plans below:

1) kichjang has been approved to start work.

These functions look like a very straightforward API change. As I understand it, it is a matter of adding host functions to the contract registry, and making new APIs for the collection classes to utilize them. The real difficulty here is ensuring that all the collection classes are ensuring adequate test coverage and that they all pass correctly.
2) monder022 has applied to start work (Funders only: approve worker | reject worker).

I'll do my best anything! But please let me guide me
3) pathleader has applied to start work (Funders only: approve worker | reject worker).

Get a team, and work with due diligence in the discharge of my duty

Learn more on the Gitcoin Issue Details page.

[willemneal]

@KiChjang left you a comment on gitcoin.

[KiChjang]

@willemneal I'm trying to understand env.ts, how are the functions there being hooked up by the host?

[willemneal]

They are indirect calls to global functions. This way the declared imports are compiled only when it's called.

I would check out storage.ts to start.

[KiChjang]

@willemneal Ok, what I am trying to understand here is if I can just add export function return_raw(register_id: u64): void; to env.ts and expect it to work, and I don't think I can. I think the problem here is that I don't know where and how the global functions are defined.

[willemneal]

Sorry if it wasn't clear but you add a function in the name space and it is not a new import from the host. You will write it; it's in env because it's low level and a helper function.

You don't need to worry about adding any new imports.

[gitcoinbot]

@KiChjang Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

[KiChjang]

@gitcoinbot Yes, currently working on writing tests for the collection classes.

[willemneal]

@KiChjang I'd recommend in the future immediately creating a PR that is a work in progress and then pushing to that often. At the end of the PR I'll squash your commits anyway.

[KiChjang]

@willemneal Per your request, I made PR that doesn't contain any tests yet. It'd be great if you can also take a look at it and see if I'm on the right track.

[gitcoinbot]

@KiChjang Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

• reminder (3 days)
• escalation to mods (6 days)

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days