Are ONNX Models Necessary?

[snakers4]

In a few days we will be radically changing the models:

• Probably dropping ONNX VAD models (we have not decided yet);
• Reducing chunk size to 30ms (chunk will be flexible, but larger than 30ms);
• Removing separate 8 / 16 kHz models, now all models would work with 8 and 16 kHz;
• Most likely deprecating micro, mini and ordinary models in favor of just a mini-sized models (still running last experiments);
• New models will be compatible with mobile builds of PyTorch;
• Dropping the batched buffering approach we used because of large chunks;

i.e. we will be radically simplifying everything.

We have seen limited use of ONNX models, so therefore I am asking.

[Kai-Karren]

I think it should be possible to load PyTorch models in most languages like e.g. with DJL from Amazon for Java.

Also, I am not sure, but isn't it possible to convert load pretrained PyTorch models and to convert them to ONNX models anytime? So if someone needs them he could do the conversion by himself. I remember that I read something like that, but not sure about the details or if there were requirements.

I have not used the ONNX models though. Maybe someone will respond that has used the ONNX models and could explain why it would be good if they would remain 😄

[snakers4]

ONNX models are created from nn.Module or JIT models in torch.

The key issue for me here is not really maintaining or creating the models, but since we have been able to radically simplify everything, just leaving one VAD model for everything seems enticing (different SR, variable chunk size, much less code for iterators, etc).

The problem is that ONNX does not support if statements and new JIT models have one if statement inside.

[khursani8]

I thought we can export ONNX model if use JIT script models even though got if statement?

I prefer ONNX model because don't want to install torch in my prod env

[snakers4]

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

[khursani8]

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

Sure, https://colab.research.google.com/drive/1sl33VBqT8fy46zyrhU1gVjWJOKw0Ya1i?usp=sharing

[khursani8]

In case you guys don't want to share onnx anymore, can share jit model without quantization?
Currently I cannot export text enhancement model to onnx because of quantization operation

[snakers4]

Currently I cannot export text enhancement model to onnx because of quantization operation

Text enhancement has a lot of logic under the hood, so no ONNX planned there by design
There is a lot of python inference code tucked in

[snakers4]

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

Sure, https://colab.research.google.com/drive/1sl33VBqT8fy46zyrhU1gVjWJOKw0Ya1i?usp=sharing

Nice, many thanks
We discussed this issue with colleagues this morning
This link also helps - https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting

Tbh, I did not know that control flows are supported, but only in scripting model
We could replicate this on toy examples as well
But with real models we face a cryptic error we cannot yet solve

Probably we may opt for the following solution

vad.jit 8/16 kHz 
vad_q.jit 8/16 kHz 
vad.onnx 16 kHz only

[khursani8]

Currently I cannot export text enhancement model to onnx because of quantization operation

Text enhancement has a lot of logic under the hood, so no ONNX planned there by design There is a lot of python inference code tucked in

Yeah, saw a lot of function in it like enhance text, unitoken_into_token, process_unicode etc
I can move it into my inference code, the only issue at exporting the te_model because of quantization operation

[khursani8]

I thought we can export ONNX model if use JIT script models even though got if statement?

Can you provide a minimal working export example with if?

Sure, https://colab.research.google.com/drive/1sl33VBqT8fy46zyrhU1gVjWJOKw0Ya1i?usp=sharing

Nice, many thanks We discussed this issue with colleagues this morning This link also helps - https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting

Tbh, I did not know that control flows are supported, but only in scripting model We could replicate this on toy examples as well But with real models we face a cryptic error we cannot yet solve

Probably we may opt for the following solution

vad.jit 8/16 kHz 
vad_q.jit 8/16 kHz 
vad.onnx 16 kHz only

cool, later when you guys share vad.jit I might try export it to onnx
If I can :p

[snakers4]

Yeah, saw a lot of function in it like enhance text, unitoken_into_token, process_unicode etc

I can move it into my inference code, the only issue at exporting the te_model because of quantization operation

Tucking this in was a design decision, because we wanted to package and support as little as possible
Also a TE model is a more niche model, so we just decided not to bother with publicly sharing those

[khursani8]

Yeah, saw a lot of function in it like enhance text, unitoken_into_token, process_unicode etc

I can move it into my inference code, the only issue at exporting the te_model because of quantization operation

Tucking this in was a design decision, because we wanted to package and support as little as possible Also a TE model is a more niche model, so we just decided not to bother with publicly sharing those

Ohh ok got it
I guess my alternative choice is to train it myself like what you did.
Thanks

[snakers4]

cool, later when you guys share vad.jit I might try export it to onnx

If I can :p

There are some low level cryptic errors for some reason, basically saying that some data type becomes int64 and fails in ONNX. But we have some custom modules, that we also intentionally tucked in. So if we cannot fix it, we will just say that ONNX works only for 16k.

We want to radically simplify everything and having more than one variety model is a no-no for future. We had to be inventive to have the best quality, you will see in the PR releases after the VAD release.

[khursani8]

cool, later when you guys share vad.jit I might try export it to onnx

If I can :p

There are some low level cryptic errors for some reason, basically saying that some data type becomes int64 and fails in ONNX. But we have some custom modules, that we also intentionally tucked in. So if we cannot fix it, we will just say that ONNX works only for 16k.

We want to radically simplify everything and having more than one variety model is a no-no for future. We had to be inventive to have the best quality, you will see in the PR releases after the VAD release.

Ohhh, might be bug in onnx graph code
Ok cannot wait for the release :)

[snakers4]

In the end we could solve all of the ONNX problems that we could identify (with our custom modules), but ended up with cryptic export errors nonetheless.
We will return to this in future, when we have more time / focus.

Turns out for "if-less" models we just used tracing, which produced horribly looking, but working models.

For models with ifs (they need scripting) we could beat the ONNX limitations and polish our custom modules to be compatible, but ended up with more cryptic errors.

We will have a final module by module debugging session some time 1Q next year.

For now we decided to opt for simplicity and collect feedback.

[tmacwill]

Congratulations on the new release, looks really exciting!

We were using the Silero VAD ONNX models via the onnxruntime JavaScript package. It looks like there are a few JavaScript PyTorch/TorchScript bindings that might work (though I haven't tried yet), but they don't look nearly as well-maintained or commonly-used as the ONNX runtime, so would love to have an ONNX model for this new version. The limitations you mention (e.g., 16k only) seem totally reasonable.

Thanks!

[snakers4]

Thanks for your support.
We may just end up keeping only 16 kHz for ONNX, but that seems like a departure from core principles.
I.e. having a million models (_8, _16, _mobile, etc) seems like a chore and will likely bloat in future, which we do not want to repeat.

[snakers4]

Also it is kind of disheartening that such a fundamental thing as a VAD gains very little traction, despite the fact that basically any voice / automation / transcription service must a proper VAD at its core.

[tmacwill]

Sounds great, thanks so much! For what it's worth, we're big fans of this project :) Totally agree with you that a high-quality VAD is a really important component of any speech system. The quality and performance of the Silero VAD is significantly better than every other VAD we've tried, so we really appreciate your work!

[garymmi]

hope still provide onnx format model
because it can reduce the size of inference library when run on android device
as below table

the size of inference library (libonnxruntime.so) can be shrinked by operators of vad model

[garymmi]

if run on Linux x86_64 platform

the size of libtorch is very huge

[snakers4]

on x86 most likely you dockerize your app, so torch size is much less of an issue
most SSDs now are 128GB+
but it is large, yes

[snakers4]

The above comments make sense. Most likely we will just publish a 16 kHz ONNX model, and postpone a unified model for the time being (or just reserve it only for commercial customers, idk, I do no want more than 1 VAD at the moment, or it will turn into chaos again).

Also a /files folder in the repo needs clean-up, since we are focusing on the VAD itself, kind of hiding other networks that did not really gain traction yet, we would reduce the repo size 20x and treat as a sort of package, because the utils code is rudimentary and with backends other than PyTorch packages do not seem like a worthy investment of time for now.

But who knows, some feedback or help on properly adding a python conda package with Github actions may be appreciated.

[snakers4]

All of the ONNX runtime adopters, rejoice
https://github.com/snakers4/silero-vad/releases/tag/v3.1

[tmacwill]

Thanks so much for the fast turnaround! Really excited to try these out.

[snakers4]

Silero VAD: Support For Sampling Rates Higher Than 16 kHz

• jit model now can handle 8, 16, 32 and 48 kHz directly (change implemented within the model itself)
• onnx model as well, but only via external wrappers (we just use each n-th sample for higher sampling rates)
• This support is mostly a hack, i.e. we just use each n-th sample for higher sampling rates (instead of averaging)

You can also average over every 2 or 3 samples manually and just use 16 kHz if you want.

[emcodem]

Hm as far as i see the onnx model is still alive but reading this thread makes me worry about if new projects should use onnx or not.
Might i ask why you finally decided to continue supporting innx?

[snakers4]

Hi,

Might i ask why you finally decided to continue supporting innx?

ONNX runtime is backed up by Microsoft, which invests a lot of resources in it.