accessing memory within wasm module

[noot]

Hey, I was wondering if it's possible to access the wasm memory in go code that'll be compiled to wasm. eg something like:

//export my_func
func my_func(loc, size int32) int32 {
	data := memory[loc:loc+size]
        // do some operations using the memory that was read, write something to the memory for usage by the native code
       copy(memory[out:out+size], result)
       return out
}

If this is supported are there any docs on this? If not, is there some workaround you might suggest? Thanks!

[aykevl]

Why do you need to do this?
In general you can't just access raw memory. You would overwrite existing Go data structures.
If you want to use raw memory outside of WebAssembly, you could allocate a buffer in Go (for example, make([]byte, length)), and use that from JavaScript for example. However you'd need to be careful that it doesn't get collected by the garbage collector.

[noot]

I mean accessing the wasm linear memory in particular, for example on the native side I want to write some data to the linear memory and read it on the wasm side, or vice versa. I don't mean raw memory, only the wasm linear memory.

Looking at this issue #411, the solution seems to be similar to what I'd like. Would buf end up being placed somewhere in the linear memory? If I pass a byte array to/from wasm, how does it decide where to place it?

[noot]

alright, I think I almost answered my own question. Compiling this code with tinygo build -o wasm.wasm -target wasm ./wasm/wasm.go:

package main

var buf [1024]byte

func main() {}

//go:export my_func
func my_func(ptr, len int32) int32 {
	data := buf[ptr:ptr+len]
        // just return first byte for now, in future would like to do other things
	return int32(data[0]) 
} 

//go:export bufAddr
func bufAddr() *byte {
	return &buf[0]
}

then running this wasmer code:

package main

import (
	"fmt"
	"io/ioutil"
	"github.com/wasmerio/wasmer-go/wasmer"
)

func main() {
    wasmBytes, err := ioutil.ReadFile("wasm.wasm")
    if err != nil {
    	panic(err)
    }

    engine := wasmer.NewEngine()
    store := wasmer.NewStore(engine)

    module, err := wasmer.NewModule(store, wasmBytes)
    if err != nil {
    	panic(err)
    }

    imports := map[string]wasmer.IntoExtern{}
    wasi_unstable_imports := map[string]wasmer.IntoExtern{}
    wasi_unstable_imports["fd_write"] = wasmer.NewFunction(
		store,
		wasmer.NewFunctionType(
			wasmer.NewValueTypes(wasmer.I32, wasmer.I32, wasmer.I32, wasmer.I32), 
			wasmer.NewValueTypes(wasmer.I32), 
		),
		func(args []wasmer.Value) ([]wasmer.Value, error) {
			fmt.Println("fd_write")
			fmt.Println(args[0].I32())
			fmt.Println(args[1].I32())
			fmt.Println(args[2].I32())
			fmt.Println(args[3].I32())
			return []wasmer.Value{wasmer.NewI32(0)}, nil
		},
	)

    importObject := wasmer.NewImportObject()
    importObject.Register("env", imports)
    importObject.Register("wasi_unstable", wasi_unstable_imports)

    instance, err := wasmer.NewInstance(module, importObject)
    if err != nil {
    	panic(err)
    }

    f, err := instance.Exports.GetFunction("bufAddr")
    if err != nil {
    	panic(err)
    }

    result, err := f()
    if err != nil {
    	panic(err)
    }

    fmt.Println("&buf[0]", result)
    offset := result.(int32)

    mem, err := instance.Exports.GetMemory("memory")
    if err != nil {
    	panic(err)
    }

    memory := mem.Data()
    copy(memory[offset:offset+1], []byte{0xff})

    f, err = instance.Exports.GetFunction("my_func")
    if err != nil {
    	panic(err)
    }

    result, err = f(offset, 1)
    if err != nil {
    	panic(err)
    }

    fmt.Println("my_func", result)
}

shows that the buffer starts at offset 65808 in linear memory, but then when I call my_func I see that it calls into wasi_unstable.fd_write , then panics. The full output is:

$ go run main.go 
&buf[0] 65808
fd_write
1
65656
1
65800
panic: unreachable

goroutine 1 [running]:
main.main()
	/home/elizabeth/go/src/github.com/noot/tinygo-test/main.go:95 +0x74f
exit status 2

Any idea what's going wrong here? What is fd_write supposed to do?

[EclesioMeloJunior]

hey! I've made a similar code where I could manipulate the data within wasm module and outside without errors:

This is my wasm module, it has 3 functions read (get a slice from the byte array), starts_at (get the start mem location form byte array), mul (just get 2 first items from byte array, multiply them and insert the result in the next position on byte array)

package main

var buff [1024]byte

func main() {}

//go:export read
func read(start int32, til int32) []byte {
	return buff[start : start+til]
}

//go:export starts_at
func starts_at() *byte {
	return &buff[0]
}

//go:export mul
func mul() {
	buff[2] = byte(buff[0] * buff[1])
}

Using the generated wasm from the code above in the code bello:

// ... instantiating wasmer engine, store, setup wasi_unstable_imports 

startsAt, err := inst.Exports.GetFunction("starts_at")
check(err)

s, err := startsAt()
check(err)

fmt.Println("wasm buff[0]", s)
offset := s.(int32)

mem, err := inst.Exports.GetMemory("memory")
check(err)

memory := mem.Data()
copy(memory[offset:offset+1], []byte{0x02}) // add 2 to wasm buff at position 0
copy(memory[offset+1:offset+2], []byte{0x03}) // add 3 to wasm buff at position 1

fmt.Println(memory[offset:offset+3])

mulfn, err := inst.Exports.GetFunction("mul")
check(err)

// calling wasm mul exported function that will access
_, err = mulfn()
check(err)

fmt.Println(memory[offset:offset+3])

The outuput:

wasm buff[0] 65741
[2 3 0] 131072 131072
[2 3 6] 131072 131072

The wasm memory could be manipulated from inside and outside the wasm module.

[EclesioMeloJunior]

@noot I think the error you reported:

panic: unreachable

goroutine 1 [running]:
main.main()
	/home/elizabeth/go/src/github.com/noot/tinygo-test/main.go:95 +0x74f
exit status 2

is a kind of index out of range error because:

• In the wasm you start a var buff [1024]byte
• the wasm exported function my_func try to get the range buff[ptr:ptr+len]
• in the wasmer code you get the offset beign the value 65808
• so you pass the following arguments to your wasm exported func my_func(65808, 1)
• So it will try to access the buff[65808: 65808 + 1] range resulting in the problem

[noot]

@EclesioMeloJunior thanks for the info, revisited my code and got it working now with printing a string via an imported go function that's passed via the memory:

package main

var buf [1024]byte

func main() {
}

func ext_print(ptr, len int32)

//go:export call_ext_print
func call_ext_print(ptr, len int32) {
	ext_print(ptr, len)
}

//go:export bufAddr
func bufAddr() *byte {
	return &buf[0]
}

package main

import (
	"fmt"
	"io/ioutil"
	"github.com/wasmerio/wasmer-go/wasmer"
)

func main() {
    wasmBytes, err := ioutil.ReadFile("wasm.wasm")
    if err != nil {
    	panic(err)
    }

    engine := wasmer.NewEngine()
    store := wasmer.NewStore(engine)

    module, err := wasmer.NewModule(store, wasmBytes)
    if err != nil {
    	panic(err)
    }

    imports := map[string]wasmer.IntoExtern{}

    var memory []byte

    imports["main.ext_print"] = wasmer.NewFunction(
     store,
     wasmer.NewFunctionType(
         wasmer.NewValueTypes(wasmer.I32, wasmer.I32, wasmer.I32, wasmer.I32), 
         wasmer.NewValueTypes(), 
     ),
     func(args []wasmer.Value) ([]wasmer.Value, error) {
        fmt.Println("main.ext_print")
        ptr := args[0].I32()
        len := args[1].I32()
        fmt.Printf("ext_print: %s\n", memory[ptr:ptr+len])
        return []wasmer.Value{}, nil
     },
    )

    wasi_unstable_imports := map[string]wasmer.IntoExtern{}
    wasi_unstable_imports["fd_write"] = wasmer.NewFunction(
		store,
		wasmer.NewFunctionType(
			wasmer.NewValueTypes(wasmer.I32, wasmer.I32, wasmer.I32, wasmer.I32), 
			wasmer.NewValueTypes(wasmer.I32), 
		),
		func(args []wasmer.Value) ([]wasmer.Value, error) {
			fmt.Println("fd_write")
			fmt.Println(args[0].I32())
			fmt.Println(args[1].I32())
			fmt.Println(args[2].I32())
			fmt.Println(args[3].I32())
			return []wasmer.Value{wasmer.NewI32(0)}, nil
		},
	)

    importObject := wasmer.NewImportObject()
    importObject.Register("env", imports)
    importObject.Register("wasi_unstable", wasi_unstable_imports)

    instance, err := wasmer.NewInstance(module, importObject)
    if err != nil {
    	panic(err)
    }

    f, err := instance.Exports.GetFunction("bufAddr")
    if err != nil {
    	panic(err)
    }

    result, err := f()
    if err != nil {
    	panic(err)
    }

    fmt.Println("&buf[0]", result)
    offset := result.(int32)

    mem, err := instance.Exports.GetMemory("memory")
    if err != nil {
        panic(err)
    }

    memory = mem.Data()
    msg := []byte("nootwashere")
    copy(memory[offset:offset+int32(len(msg))], msg)

    f, err = instance.Exports.GetFunction("call_ext_print")
    if err != nil {
    	panic(err)
    }

    _, err = f(offset, len(msg))
    if err != nil {
    	panic(err)
    }
}

output is as expected:

&buf[0] 65808
main.ext_print
ext_print: nootwashere

I think I can go ahead and close this now, it's still unclear how tinygo determines the layout of the module memory but I'll try to figure that out separately :)