tflite model used in Google meet Virtual Background

[jiangjianping]

https://ai.googleblog.com/2020/10/background-features-in-google-meet.html

We are evaluating to add virtual background support to our app based on media pipe. Could you please open sourced the mobilenetv3 small tflite model used in Google Meet?

[Aayush795]

@jiangjianping Hey, I have downloaded the tflite file for Google Meets Virtual Background.
Do you think I can just use these models for my own application for the virtual background use case just like i use TF.js body pix models?

I am currently using Tensorflow JS models, BodyPix... but the accuracy is not that good. Can I somehow improve my model?

[floe]

According to the model card, the model is licensed under Apache License 2.0, so as long as your application has a compatible license, you should be good to go.

[Aayush795]

Thanks, @floe . However is it mandatory to convert the tflite model to tfjs model or I can use tf lite model as is? I want to use the model for real-time inference(Virtual Background/Blur) in a video web chat-based application that is utilizing Twilio video. Basically, I am looking for some open-source implementation for the same. something similar to what body pix already has
here
PS: I am new to this.

[floe]

Disclaimer: I am not a lawyer. That being said, I would assume that when your application source code is Apache 2.0 licensed, there shouldn't be any issues, no matter what format the model is in.

[fransiskusyoga]

@Aayush795 where can I find the model for the background segmentation?

[Aayush795]

@fransiskusyoga here you go https://meet.google.com/_/rtcvidproc/release/336842817/segm_lite_v509.tflite

[PINTO0309]

The license of MediaPipe Meet Segmentation Model seems to have been changed from Apache 2.0 to Google Terms of Service. I don't know if it is allowed to be used as open source, modified, or used commercially.

• Model Card - MediaPipe Meet Segmentation Model
  https://drive.google.com/file/d/1lnP1bRi9CSqQQXUHa13159vLELYDgDu0/view

[floe]

Hmmm. Interesting. I wonder if they can change it retroactively, I would assume that only applies to newer releases?

[PINTO0309]

I'm not a legal expert, so I can't say for sure, but I suspect it will be applied after the new release.

The following is just one Japanese lawyer's view.

• License Changes: Can an open source license be changed once it has been applied? (Japanese article)
  http://www.homu.net/%E3%83%A9%E3%82%A4%E3%82%BB%E3%83%B3%E3%82%B9%E3%81%AE%E5%A4%89%E6%9B%B4%EF%BC%9A%E4%B8%80%E5%BA%A6%E9%81%A9%E7%94%A8%E3%81%95%E3%82%8C%E3%81%9F%E3%82%AA%E3%83%BC%E3%83%97%E3%83%B3%E3%82%BD%E3%83%BC/

==== Quote
Since the contract is concluded by accepting the terms proposed by the copyright holder at the time of licensing, if different license terms are proposed at that time, the license contract will be concluded based on the new terms. On the other hand, a contract that has already been concluded cannot be changed at the will of one party. Therefore, for software that has already started to be used, the license agreement has already been concluded under the license terms at that time, and those terms will continue to exist.
　：
　：
If you want to change the license terms to something else, all the copyright holders of the software in question must agree to change the license to that other license. Therefore, if there are countless programmers committed, as in the case of the Linux kernel, it would be very difficult to get everyone's consent, so changing the license would be virtually impossible.
==== Quote

[wangqi-cybrook]

From https://meet.google.com/,

https://meet.google.com/_/rtcvidproc/release/hashed/segm_full_sparse_v1008_0bda82336d236e21e52f2b74129b9883.dat
https://meet.google.com/_/rtcvidproc/release/hashed/segm_lite_v1082_c59fbb2b8451df2c2752e562c6523bcc.dat

Looks like the latest model is hashed and can not be downloaded anymore.

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please reopen if you'd like to work on this further.