How to use the MODNet demo?

[benbro]

The MODNet demo is missing from the README.
How is the wasm model created? Is there a WebGL version?
I've tried to clone and run the demo locally but not sure what URL do load in the browser.

[w-okada]

I have not yet solved how to use MODNet.
Currently no idea.

[PINTO0309]

@benbro
There was a bug in the model committed three months ago, so it was fixed and re-committed at the end of February.

https://drive.google.com/file/d/1Y6B6l72hib9t5ammT-7ya-muBzRKZ7xq/view?usp=sharing
PINTO0309/PINTO_model_zoo#75 (comment)

[benbro]

@w-okada does the new model helps? I still don't understand how to use it with tfjs,

[w-okada]

I tried. But very slow.

[benbro]

Can you please explain how to test the code?
I'm sure the community can help with the performance and model translation to tfjs issues.

[w-okada]

yes please wait. Above comment is just a quick report.
I'll commit the source later.

[w-okada]

I commited the demo. But as I said, it is very heavy. If you have idea to make it fast, teach me.

[benbro]

@w-okada thank you for the demo.
Does the demo use WebAssembly or WebGL? Maybe WebSeembly SIMD in Chrome 91 will help?
I suggest setting onLocal to false by default in the demo because it makes the browser unresponsive.
What does the strict parameter do?
What's the difference between model 16 and 32?

@PINTO0309 do you have idea why the worker demo result doesn't look as good as the original demo? Is there something wrong in the model translation or use?
Input:

Result with the worker demo:

Result with the original demo:

[w-okada]

Does the demo use WebAssembly or WebGL? Maybe WebSeembly SIMD in Chrome 91 will help?
The demo uses WebGL Generally wasmSIMD is slower than WebGL.

What does the strict parameter do?
None. I forgot to erase it

What's the difference between model 16 and 32?
Float16 and Float32.

[w-okada]

@benbro

@PINTO0309 have created the low resolution models (256x256, 192x192, 128x128). They are much faster than the original one. But A little less quality. Please try with my demo page.

In my environment (with GTX 1660),
512x512:500ms
256x256:60ms
128x128:30ms

[benbro]

Thank you for the new models, as you said they are much faster but quality isn't good.

[PINTO0309]

@benbro
Perhaps this is not the right place to discuss this, but U^2-Net may also be promising.
https://github.com/xuebinqin/U-2-Net
https://github.com/w-okada/image-analyze-workers#u2-net-portrait-drawing
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/061_U-2-Net

[benbro]

@PINTO0309 is there a U^2-Net demo for background matting instead of portrait drawing?

[w-okada]

Maybe there is no javascript demo. I think it is also slow, because the portrait drawing model is so slow and the background matting is basically the same model as the portrait one.

Can I clonse this issue?

[benbro]

All models in the MODNet demo gives me bad result compared with the original project demo. Something is probably broken with the model translation.
Thank you for the help. You can close this issue.

[benbro]

This might be interesting:
https://github.com/Volcomix/virtual-background#ml-kit-selfie-segmentation
Volcomix/virtual-background#2
https://developers.google.com/ml-kit/vision/selfie-segmentation
https://developers.google.com/ml-kit/images/vision/selfie-segmentation/selfie-model-card.pdf

[w-okada]

Wow! Very insteresting!!

First of all, I'll download and save model-card!! :P

[w-okada]

@benbro
Thanks, I added selfie segmentation model in my zoo.

[benbro]

Thank you for adding a demo so fast.

I'm getting bad result with this video even with the 256x256 model. Any way to improve it?
https://www.shutterstock.com/video/clip-1048026787-businessman-talking-camera-making-conference-business-call

What's the difference between the original model and pinto_x models? Should I expect better quality or performance?
Does it make sense to use a process size larger then the model size? (300x300 and 512x512)
Padding has no visible effect.
What does the Strict option do?

[w-okada]

What's the difference between the original model and pinto_x models?

These model from https://github.com/PINTO0309/PINTO_model_zoo
I have no idea if we can expect improvement. But in my demo there is no effect.

Does it make sense to use a process size larger then the model size? (300x300 and 512x512)

In my demo, resizing image is done in wasm with opencv. "Interpolation" is used for the interpolation.
If you want to use opencv to resize, you set the different size from model output size. If you want to resize with HTML Canvas, you need not.

Padding has no visible effect.

Yes. Just I forgot to erase it.

What does the Strict option do?

If you enable strict, input image to model is kept and output from model is applied to it
If not, output from model is applied to current frame image.
Which is used depends on your thought about the time lag of processing model.

[benbro]

Thanks!

[PINTO0309]

In addition, I converted Selfie Segmentation to various models and committed them.
https://github.com/PINTO0309/PINTO_model_zoo/tree/main/109_Selfie_Segmentation

TFLite Float32/Float16/INT8, TFJS, TF-TRT, ONNX, CoreML, OpenVINO IR FP32/FP16, Myriad Inference Blob