[tfjs-react-native] cameraWithTensors shows a Black Screen after Software updates on both Android and iOS (CODE NEVER CHANGES)

[jomariya23156]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): Exactly
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Android and iOS
• Mobile device: Both Android and iOS after latest software update
• TensorFlow.js installed from (npm or script link): yarn
• TensorFlow.js version (use command below):3.8.0
• Tensorflow.js Converter Version: 3.8.0
  (package.json is provided below)

Describe the current behavior
I have done a project about pose estimation on mobile devices and successfully deploy it at Expo. However, after a few months without touching it, I came back and wanted to develop new features. So, I went to the Expo page with the QR code that I have deployed my app (link provided below) and tested it. Somehow, on the screen that is used to open the front camera and do pose detection, it's not working properly. No errors were thrown, but the camera never opens (it shows the black screen). I have tried changing from cameraWithTensors to bare Camera component from expo-camera to check where the problem could be and the Camera component can work properly. I also tried changing the models that I have loaded to check whether there are any problems, but there is none. With that said, I think the problem should be the cameraWithTensors component. Moreover, I have cloned the Github repo of this project and tried it locally (WITHOUT ANY CODE AND PACKAGES CHANGES), it also shows the same behavior. So, I believe this is the problem with the package version that occurs after the Software/OS updates on devices. How to fix this and make it work again?

Describe the expected behavior
Open the camera and models work properly

Standalone code to reproduce the issue
My minimal version should be something similar to this example: https://github.com/tensorflow/tfjs/blob/master/tfjs-react-native/integration_rn59/components/webcam/realtime_demo.tsx
But make it in pure js version, not typescript, and use hooks instead of those componentDidMount things.
Full version (it's long and some parts are quite messy): https://github.com/jomariya23156/Multi-Fit/blob/master/screens/Exercise.js OR https://drive.google.com/file/d/1LjvkqkKKKF94lttu1R-CGRZZ5T0iOLS0/view?usp=sharing
Github repo: https://github.com/jomariya23156/Multi-Fit
If you have Expo Go installed and use devices with updated software, you can scan this QR code to observe this behavior: https://expo.dev/@ariya23156/Multifit
If you scan the QR code above (or clone Github repo and start expo), this is what you should do to navigate to the page where this problem occurs:

1. You will see the orange screen with the loading progress circle. Just tap anywhere on the screen.
2. On the login page. Just type anything in both ID and Password and tab login
3. After the login page, tab the Tutorial button (second from the left side) on the bottom nav bar
4. Tab Squat.
5. Waiting for loading to finish, you will be directed to the play screen with an orange bar on top and an orange button on the bottom. On this page you don't need to tab anything, the camera should open and you should see yourself with the keypoint detected displays.

package.json

  "main": "node_modules/expo/AppEntry.js",
  "scripts": {
    "start": "expo start",
    "android": "expo start --android",
    "ios": "expo start --ios",
    "web": "expo start --web",
    "eject": "expo eject"
  },
  "dependencies": {
    "@expo/vector-icons": "^12.0.0",
    "@mediapipe/pose": "^0.4.1624666670",
    "@react-native-async-storage/async-storage": "~1.15.0",
    "@react-navigation/bottom-tabs": "^6.0.1",
    "@react-navigation/native": "^6.0.1",
    "@react-navigation/native-stack": "^6.0.1",
    "@tensorflow-models/pose-detection": "0.0.3",
    "@tensorflow/tfjs": "^3.8.0",
    "@tensorflow/tfjs-backend-wasm": "^3.8.0",
    "@tensorflow/tfjs-backend-webgl": "^3.8.0",
    "@tensorflow/tfjs-converter": "^3.8.0",
    "@tensorflow/tfjs-react-native": "^0.6.0",
    "@types/react-native-vector-icons": "^6.4.8",
    "expo": "~42.0.1",
    "expo-av": "~9.2.3",
    "expo-camera": "~11.2.2",
    "expo-font": "~9.2.1",
    "expo-gl": "~10.4.1",
    "expo-gl-cpp": "~10.4.0",
    "expo-screen-orientation": "^3.2.1",
    "expo-sensors": "^10.2.2",
    "expo-status-bar": "~1.0.4",
    "lottie-react-native": "^4.0.3",
    "native-base": "^3.0.7",
    "package.json": "^2.0.1",
    "react": "16.13.1",
    "react-dom": "16.13.1",
    "react-native": "https://github.com/expo/react-native/archive/sdk-42.0.0.tar.gz",
    "react-native-fs": "^2.18.0",
    "react-native-gesture-handler": "^1.10.3",
    "react-native-leaderboard": "^1.0.6",
    "react-native-paper": "^4.9.2",
    "react-native-reanimated": "^2.2.0",
    "react-native-safe-area-context": "3.2.0",
    "react-native-screens": "^3.4.0",
    "react-native-slideshow": "^1.0.1",
    "react-native-stopwatch-timer": "0.0.21",
    "react-native-svg": "12.1.1",
    "react-native-swipe-list-view": "^3.2.9",
    "react-native-unimodules": "^0.14.6",
    "react-native-vector-icons": "^8.1.0",
    "react-native-web": "~0.13.12",
    "react-navigation": "^4.4.4",
    "react-navigation-drawer": "^2.7.1",
    "react-navigation-stack": "^2.10.4",
    "styled-components": "^5.3.0",
    "styled-system": "^5.1.5",
    "underscore": "^1.13.1"
  },
  "devDependencies": {
    "@babel/core": "^7.9.0"
  },
  "private": true
}

[CyberScooter]

Same issue for me, works on my Android 9 device but stopped working on my Android 11 device even after updating packages

[jomariya23156]

Me either. I also tried upgrading and downgrading this package and related packages many times, but nothing works.

[edwrodrig]

Same issue

[jinjingforever]

Hi,

Could you guys try this example? I tried it on iOS 15.3 (just updated my phone) and it seems to work. Sorry I don't have android phones at the moment.

Thanks!

[jomariya23156]

@jinjingforever It doesn't work for me on Android 11. No error is thrown, but when I start the app, it just crashed and closed immediately.

[jinjingforever]

Thanks @jomariya23156 Our team is trying to get some newer android devices and I will update here when I find anything. Thanks for your patience.

[jomariya23156]

@jinjingforever Update: I sent your example to one of my friends to test it on his iOS 15. It's working properly. Only Android devices have the problem now I think.

[NWehb]

Hi,

I just ran into the same problem in a project of my own. I have tested it with both Android and iOS devices. Sometimes a camera image is displayed very briefly. After that, the element is just black. The same is true for the text that is inserted into the element.

Is there already a solution in this regard?

[JaydipMagan]

Hello,

I am experiencing the same issue with the cameraWithTensors. The preview appears for a split second before displaying a black screen. I noticed the preview appears on 60hz phones but not on high refresh screens.

Tested on Android 10,11,12 and IOS 15.

[rthadur]

@ALL did you get chance to check @jinjingforever comments here #6230 (comment) for related issue. Please let us know if this helps. Thank you

[google-ml-butler]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you.

[google-ml-butler]

Closing as stale. Please @mention us if this needs more attention.

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[jomariya23156]

@rthadur The solution works fine on my friend's iOS 15. But I have not tested it on my android phone yet. I will update as soon as I try it.