Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. #6461

matanlurey · 2024-04-04T20:28:50Z

WIP: We do not plan to land this PR until the next stable release (>= April 3rd 2024).

Details

Migrates uses of createSurfaceTexture to createSurfaceProducer, which is intended to have no change in behavior, but does change the backend rendering path, so it will require more testing (and we're also open to minor API renames or changes before it becomes stable).

Background

Android plugins previously requested a SurfaceTexture from the Android embedder, and used that to produce a Surface to render external textures on (i.e. video_player). This worked because 100% of Flutter applications on Android used OpenGLES (via our Skia backend), and SurfaceTexture is actually an (opaque) OpenGLES-texture.

Starting soon (roughly ~Q3, this is not a guarantee and just an estimate), Flutter on Android will start to use our new Impeller graphics backend, which on newer devices (>= API_VERSION_28), will default to the Vulkan, not OpenGLES. In other words, SurfaceTexture will cease to work (it is possible, but non-trivial, to map an OpenGLES texture over to Vulkan).

After consultation with the Android team, they helped us understand that vending SurfaceTexture (the consumer-side API) was never the right abstraction, and we should have been vending the producer-side API, or Surface directly. The new SurfaceProducer API is exactly that - it generates a Surface, and similar to our platform view strategy, picks the "right" consumer-side implementation details for the user/plugin packages.

The new SurfaceProducer API has 2 possible rendering types (as an implementation detail):

SurfaceTexture, for older OpenGLES devices, which works exactly as it does today.
ImageReader, for newer OpenGLES or Vulkan devices.

These are some subtle nuances in how these two APIs work differently (one example: flutter/flutter#144407), but our theory at this point is we don't expect these changes to be observed by any users, and we have other ideas if necessary.

Note

These invariants are tested on CI in flutter/engine.

Points of contact:

@matanlurey or @jonahwilliams (Flutter Engine)
@johnmccutchan or @reidbaker (Flutter on Android)

…cer-camera-android

This pull request is created by [automatic cherry pick workflow](https://github.com/flutter/flutter/wiki/Flutter-Cherrypick-Process#automatically-creates-a-cherry-pick-request) Please fill in the form below, and a flutter domain expert will evaluate this cherry pick request. ### Issue Link: What is the link to the issue this cherry-pick is addressing? Test failures in Impeller migration PR: flutter/packages#6461 ### Changelog Description: Explain this cherry pick in one line that is accessible to most Flutter developers. See [best practices](https://github.com/flutter/flutter/wiki/Hotfix-Documentation-Best-Practices) for examples Ensure that Android platform view updates are posted on correct thread when platform view is created on background handler. ### Impact Description: What is the impact (ex. visual jank on Samsung phones, app crash, cannot ship an iOS app)? Does it impact development (ex. flutter doctor crashes when Android Studio is installed), or the shipping production app (the app crashes on launch) Migration to Impeller plugin APIs is blocked. ### Workaround: Is there a workaround for this issue? Disable impeller ### Risk: What is the risk level of this cherry-pick? ### Test Coverage: Are you confident that your fix is well-tested by automated tests? However the tests are in flutter/packages and not flutter/engine. ### Validation Steps: What are the steps to validate that this fix works? Run the unit tests on the linked PR in flutter/packages

…ducer-camera-android

matanlurey · 2024-05-16T21:50:03Z

Due to a release hiccup, the stable branch (3.22) is temporarily ahead of the master branch (3.22.0-35.0.pre.34), leading to CI failures (see discord discussion). I've merged back into HEAD and updated the pubspec/CHANGELOG - the rest might be done by someone else while I'm on vacation for a couple of weeks.

stuartmorgan · 2024-05-24T17:04:18Z

...s/camera/camera_android/android/src/test/java/io/flutter/plugins/camera/CameraTest.java.orig

Is this file a local conflict resolution mistake?

probably something went wrong applying the formatting diff.

jonahwilliams · 2024-05-24T17:24:48Z

packages/camera/camera_android/android/src/test/java/io/flutter/plugins/camera/CameraTest.java

@@ -104,7 +104,8 @@ public void createCaptureSession(
  }

  @Override
-  public void close() {}
+  public void close() {


hmm, that formatting didn't work either

jonahwilliams · 2024-05-24T17:52:21Z

Thank you @stuartmorgan !

jonahwilliams · 2024-05-24T18:14:42Z

Current test failure is:



java.lang.Exception: ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞═════════════════
The following PlatformException was thrown running a test:
PlatformException(error, Attempt to invoke virtual method 'int
android.hardware.camera2.CameraCaptureSession.capture(android.hardware.camera2.CaptureRequest,
android.hardware.camera2.CameraCaptureSession$CaptureCallback,
android.os.Handler)' on a null object reference, null,
java.lang.NullPointerException: Attempt to invoke virtual method
'int
android.hardware.camera2.CameraCaptureSession.capture(android.hardware.camera2.CaptureRequest,
android.hardware.camera2.CameraCaptureSession$CaptureCallback,
android.os.Handler)' on a null object reference
at
io.flutter.plugins.camera.Camera.runPrecaptureSequence(Camera.java:660)
at io.flutter.plugins.camera.Camera.takePicture(Camera.java:645)
at
io.flutter.plugins.camera.MethodCallHandlerImpl.onMethodCall(MethodCallHandlerImpl.java:112)
at
io.flutter.plugin.common.MethodChannel$IncomingMethodCallHandler.onMessage(MethodChannel.java:267)
at
io.flutter.embedding.engine.dart.DartMessenger.invokeHandler(DartMessenger.java:292)
at
io.flutter.embedding.engine.dart.DartMessenger.lambda$dispatchMessageToQueue$0$io-flutter-embedding-engine-dart-DartMessenger(DartMessenger.java:319)
at
io.flutter.embedding.engine.dart.DartMessenger$$ExternalSyntheticLambda0.run(Unknown
Source:12)

jonahwilliams · 2024-05-28T15:38:09Z

@johnmccutchan @camsim99 PTAL

…ce-producer-camera-android

camsim99

LGTM

packages/camera/camera_android/CHANGELOG.md

Co-authored-by: Camille Simon <[email protected]>

…cer`. (flutter/packages#6461)

flutter/packages@31d3329...910fabb 2024-05-29 [email protected] Amend package:web tweaks to allow package:web roll (flutter/packages#6793) 2024-05-29 [email protected] Migrate `video_player/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6456) 2024-05-29 [email protected] [google_maps_flutter] Undeprecate BitmapDescriptor methods (flutter/packages#6832) 2024-05-29 [email protected] Migrate CameraX from SurfaceTexture to SurfaceProducer. (flutter/packages#6462) 2024-05-29 [email protected] Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. (flutter/packages#6461) 2024-05-29 [email protected] [dynamic_layouts] Remove the dynamic_layouts package (flutter/packages#6830) 2024-05-29 [email protected] [camerax] Add notes about Android permissions (flutter/packages#6741) 2024-05-29 [email protected] [Re-land] Bump legacy all_packages project AGP version to 7.0.0, Gradle version to 7.0.2 (flutter/packages#6742) 2024-05-29 [email protected] Roll Flutter from a1a33e6 to c85fa6a (20 revisions) (flutter/packages#6829) 2024-05-29 [email protected] [rfw] Migrate deprecated doc references (flutter/packages#6744) 2024-05-29 [email protected] [flutter_adaptive_scaffold] Migrate MaterialStateProperty to WidgetStateProperty (flutter/packages#6743) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages-flutter-autoroll Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Flutter: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

@matanlurey

…lutter#6461) _**WIP**: We do not plan to land this PR until the next stable release (>= April 3rd 2024)_. Work towards flutter/flutter#145930. ## Details Migrates uses of `createSurfaceTexture` to `createSurfaceProducer`, which is intended to have no change in behavior, but _does_ change the backend rendering path, so it will require more testing (and we're also open to minor API renames or changes before it becomes stable). ## Background Android plugins previously requested a `SurfaceTexture` from the Android embedder, and used that to produce a `Surface` to render external textures on (i.e. `video_player`). This worked because 100% of Flutter applications on Android used OpenGLES (via our Skia backend), and `SurfaceTexture` is actually an (opaque) OpenGLES-texture. Starting soon (roughly ~Q3, this is not a guarantee and just an estimate), Flutter on Android will start to use our new Impeller graphics backend, which on newer devices (`>= API_VERSION_28`), will default to the Vulkan, _not_ OpenGLES. In other words, `SurfaceTexture` will cease to work (it is possible, but non-trivial, to map an OpenGLES texture over to Vulkan). After consultation with the Android team, they helped us understand that vending `SurfaceTexture` (the _consumer-side_ API) was never the right abstraction, and we should have been vending the _producer-side_ API, or `Surface` directly. The new `SurfaceProducer` API is exactly that - it generates a `Surface`, and similar to our platform view strategy, picks the "right" _consumer-side_ implementation details _for_ the user/plugin packages. The new `SurfaceProducer` API has 2 possible rendering types (as an implementation detail): - `SurfaceTexture`, for older OpenGLES devices, which works exactly as it does today. - `ImageReader`, for newer OpenGLES _or_ Vulkan devices. These are some subtle nuances in how these two APIs work differently (one example: flutter/flutter#144407), but our theory at this point is we don't expect these changes to be observed by any users, and we have other ideas if necessary. > [!NOTE] > These invariants are [tested on CI in `flutter/engine`](https://github.com/flutter/engine/tree/main/testing/scenario_app/android#ci-configuration). Points of contact: - @matanlurey or @jonahwilliams (Flutter Engine) - @johnmccutchan or @reidbaker (Flutter on Android)

…cer`. (flutter#6461)" This reverts commit 058851b

…cer`." (#6964) Reverts #6461 Resolves flutter/flutter#150549

Migrate camera from SurfaceTexture to SurfaceProducer.

9bc2ff9

matanlurey requested a review from johnmccutchan April 4, 2024 20:28

github-actions bot added p: camera platform-android labels Apr 4, 2024

matanlurey added 4 commits April 4, 2024 14:05

Migrate test.

0e9666a

Remove private field check that no longer exists.

2eeeedf

Merge remote-tracking branch 'origin/main' into migrate-surface-produ…

d8bd607

…cer-camera-android

++

dbcafff

This was referenced Apr 8, 2024

TestUtils (flutter/packages) should be deleted/tests refactored flutter/flutter#146457

Closed

ImageReaderSurfaceProducer interacts poorly with (some) plugins due to ensureRunningOnMainThread flutter/flutter#146465

Closed

jonahwilliams mentioned this pull request Apr 16, 2024

[CP-beta][Android] post image task to main handler. flutter/engine#52166

Merged

5 tasks

matanlurey added 3 commits May 16, 2024 12:55

Merge remote-tracking branch 'upstream/main' into migrate-surface-pro…

0d68edc

…ducer-camera-android

Address merge conflicts.

5bebf83

+

8394d9b

matanlurey assigned jonahwilliams May 16, 2024

matanlurey marked this pull request as ready for review May 16, 2024 21:53

matanlurey requested a review from camsim99 as a code owner May 16, 2024 21:53

matanlurey mentioned this pull request May 16, 2024

Stage changes to flutter/packages as a PR that migrates to SurfaceProducer flutter/flutter#145930

Closed

johnmccutchan approved these changes May 20, 2024

View reviewed changes

jonahwilliams added 2 commits May 24, 2024 09:50

Merge branch 'main' into migrate-surface-producer-camera-android

f04e52a

formatting

c9a57d7

stuartmorgan reviewed May 24, 2024

View reviewed changes

redo

f7076e3

jonahwilliams reviewed May 24, 2024

View reviewed changes

autoformat

3967f1f

Merge branch 'main' into migrate-surface-producer-camera-android

913388d

Merge branch 'main' of github.com:flutter/packages into migrate-surfa…

4416cb5

…ce-producer-camera-android

camsim99 approved these changes May 29, 2024

View reviewed changes

packages/camera/camera_android/CHANGELOG.md Outdated Show resolved Hide resolved

Update packages/camera/camera_android/CHANGELOG.md

357494d

Co-authored-by: Camille Simon <[email protected]>

jonahwilliams added the autosubmit Merge PR when tree becomes green via auto submit App label May 29, 2024

auto-submit bot merged commit 058851b into main May 29, 2024
74 checks passed

auto-submit bot deleted the migrate-surface-producer-camera-android branch May 29, 2024 18:53

engine-flutter-autoroll mentioned this pull request May 30, 2024

Roll Packages from 31d332900d98 to 910fabb4886b (11 revisions) flutter/flutter#149321

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request May 30, 2024

058851b Migrate camera/android from SurfaceTexture->`SurfaceProdu…

68f8097

…cer`. (flutter/packages#6461)

AlexV525 added a commit to AlexV525/flutter-packages that referenced this pull request Jun 21, 2024

Revert "Migrate camera/android from SurfaceTexture->`SurfaceProdu…

cc885ca

…cer`. (flutter#6461)" This reverts commit 058851b

AlexV525 mentioned this pull request Jun 21, 2024

Revert "Migrate camera/android from SurfaceTexture->SurfaceProducer." #6964

Merged

auto-submit bot pushed a commit that referenced this pull request Jun 21, 2024

Revert "Migrate camera/android from SurfaceTexture->`SurfaceProdu…

8966342

…cer`." (#6964) Reverts #6461 Resolves flutter/flutter#150549

reidbaker mentioned this pull request Jul 2, 2024

i151188 update espresso dependencies v2 reidbaker/packages#6

Closed

11 tasks

AlexV525 mentioned this pull request Aug 21, 2024

[camera_android_camerax] Camera preview is rotated on the supported Impeller version flutter/flutter#149294

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. #6461

Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. #6461

matanlurey commented Apr 4, 2024

matanlurey commented May 16, 2024

stuartmorgan May 24, 2024

jonahwilliams May 24, 2024

jonahwilliams May 24, 2024

jonahwilliams commented May 24, 2024

jonahwilliams commented May 24, 2024

jonahwilliams commented May 28, 2024

camsim99 left a comment

Migrate camera/android from SurfaceTexture->SurfaceProducer. #6461

Migrate camera/android from SurfaceTexture->SurfaceProducer. #6461

Conversation

matanlurey commented Apr 4, 2024

Details

Background

matanlurey commented May 16, 2024

stuartmorgan May 24, 2024

Choose a reason for hiding this comment

jonahwilliams May 24, 2024

Choose a reason for hiding this comment

jonahwilliams May 24, 2024

Choose a reason for hiding this comment

jonahwilliams commented May 24, 2024

jonahwilliams commented May 24, 2024

jonahwilliams commented May 28, 2024

camsim99 left a comment

Choose a reason for hiding this comment

Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. #6461

Migrate `camera/android` from `SurfaceTexture`->`SurfaceProducer`. #6461