You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, if input and output tensors are not F32, both runtimes will panic.
This stems from the way we handle and store input and output tensors, particularly when using the native ONNX runtime - where both input and output tensors are Vec<Array<f32, Dim<IxDynImpl>>>.
The additional tensor types defined by WASI NN are F16, U8, and I32.
We need to either:
make both tensors generic over the data type. This would potentially make the entire State struct generic, which could introduce difficulties in how we manage it.
keep storing every tensor type as F32, introduce fields for input and output tensor types (Question: how to handle input model not containing tensor shapes? #12) and transform them just-in-time (when compute is called, for input tensors, and when get_output is called, for output tensors). This would however introduce an additional performance penalty at inference time.
The text was updated successfully, but these errors were encountered:
Currently, if input and output tensors are not F32, both runtimes will panic.
This stems from the way we handle and store input and output tensors, particularly when using the native ONNX runtime - where both input and output tensors are
Vec<Array<f32, Dim<IxDynImpl>>>
.The additional tensor types defined by WASI NN are F16, U8, and I32.
We need to either:
State
struct generic, which could introduce difficulties in how we manage it.compute
is called, for input tensors, and whenget_output
is called, for output tensors). This would however introduce an additional performance penalty at inference time.The text was updated successfully, but these errors were encountered: