Discussion: Using the C API vs. Tract

[radu-matei]

Currently, there are two very different ways in which one could run an ONNX model from Rust:

• using Rust bindings to the ONNX C API, through the https://github.com/nbigaouette/onnxruntime-rs project - this is the current approach. Specifically, we use the non-unsafe crate from the repo linked (which downloads and links the shared library for the ONNX runtime version 1.6), then use it to load models and execute inferences. The main advantage here is that linking against the original ONNX implementation, we are guaranteed to have full compatibility across all models, and could theoretically get GPU support ([C API]: Compile ONNX bindings with GPU support #9). The disadvantages are around distributing the shared library, and the fact that the Rust bindings are slightly old (1.6 vs. 1.8 the latest release), not official, and apparently unmaintained.
• using Tract - https://github.com/sonos/tract - this is a great project that natively implements (most of the?) ONNX operators. The disadvantage is that it is not a goal of the Tract project to add GPU support, so we would only running inferences on the CPU (which may not be a disadvantage for constrained devices).

Ultimately, I think we might have to implement both and expose the option as a feature flag.
For now, we are experimenting with the C API, and will update this issue as progress is made.