Support advanced PiP integration

[defagos]

As a developer integrating Pillarbox I want to be able to implement PiP deeper in my application, so that users can browse the application while continuing playback using PiP.

As a Pillarbox user I want to be able to use the app while watching content.

Acceptance criteria

• An API is provided so that developers can perform advanced PiP integration with dismissal and restoration.
• The Pillarbox demo can be browsed while using PiP, for all the players supporting PiP (custom player, playlist, multi-player demo).

Hints

• A unified API be compatible with Add PiP support to SystemVideoView #610 should be provided.
• PiP is not integrated in other demos (stories, simple, etc.).

Tasks

• Implement PiP lifecycle for SwiftUI and add dedicated modifiers.
• Implement for custom player:
  • Modal dismissal when entering PiP.
  • Modal restoration when exiting PiP.
• Make it work with the multi-player demo.
• Add PiP button:
  • Provide observable states for UI integration.
  • Add PiP button to relevant players.
• Ensure that PiP and usual in-app playback do not conflict:
  • Start PiP playback.
  • Open simple example without PiP support. The PiP session should end.
• Ensure that PiP is exited when navigating to the currently playing item again (instead of interrupting playback).
• Check that no resources are leaked.

[defagos]

I finally could understand why sometimes we observed Picture in Picture being paused during restoration, not from the PiP overlay itself, but by starting playback of another content. This investigation and the related fix were made as part of #702.

When do pauses occur?

Pauses occur when transitioning via PiP from some content A, played with a player supporting PiP, to some content B, played in the same player but not supporting PiP.

Compare:

1. Content A played with a player supporting PiP, transitioning via PiP to content B in the same player supporting PiP:

   PiP_A_1.mov

2. Content A played with a player supporting PiP, transitioning via PiP to content B in the same player not supporting PiP:

   PiP_A_2.mov

In the first case playback continues, while in the second one playback is paused.

Remark

You might think that this case is convoluted but actually it isn't. The fact that a view supports PiP not only supposes that the video view has been configured appropriately, but also that the view itself is installed in the hierarchy and therefore could register itself with a Picture in Picture controller.

Depending on how a player user interface is implemented the view might not be immediately available, e.g. if the implementation only displays the video view once the media type is known (see #696 for a related issue that needed to be fixed as well). This is perfectly legitimate and can happen in practice. We have no way of preventing such implementations and we cannot have behaviors break just because of subtle layout implementation refactorings that are part of any development process.

Why do pauses occur when transitioning to a layout not-supporting PiP?

Our implementation explicitly stops Picture in Picture using the dedicated API when a video view appears for the first time (via onAppear(perform:)). To better understand what happens let us have a look at what we have if we remove the stop and let PiP playback continue in the overlay. We then simulate the stop by tapping the restoration button in the PiP overlay:

1. Content A played with a player supporting PiP, transitioning via PiP to content B in the same player supporting PiP:

   PiP_B_1.mov

2. Content A played with a player supporting PiP, transitioning via PiP to content B in the same player not supporting PiP:

   PiP_B_2.mov

We can easily understand what happens now:

• In the first case the layer is the same in the PiP overlay as in the full screen player. When stopping PiP the layer is merged back without interruption.
• In the second case the layers in the PiP overlay and in the full screen player are different, yet the AVPlayer is the same (thus the content appearing twice). When stopping PiP the player is paused, which makes the other layer as well.

How can we fix this issue?

To fix the issue visible in the second scenario we need to detach the player from the PiP layer before PiP is stopped, but only when the view we are transitioning to does not support PiP anymore. This way we avoid the player being shared and pausing playback in the other layer when PiP is closed:

PIP_detach.mov

The programmatic PiP stop is not even needed in this case since this change forces PiP to close automatically. The stop is still needed for the first scenario, though, so that the PiP overlay is properly returned when transitioning to the same view supporting PiP.

[defagos]

Here are a few use cases as a reminder of expected behaviors that should possible to implement and test in the demo. This is of course no perfect replacement for proper unit testing but at least better than having nothing.

In all cases:

• Playback must never be paused in unexpected ways.
• PiP enabled with some persisted model must never be interrupted by video views associated with other models.
• Restoration with the dedicated button must never simply close the PiP overlay but always restore a user interface.
• The PiP button must have proper visibility (visible iff PiP is possible).
• Resources must never be leaked.

Note that the behaviors listed above are mostly a matter of proper route management. Our API, though, must have the flexibility so that these behaviors can be implemented.

Restoration with PiP overlay restoration button

restoration.mov

Remark

PiP must be possible simultaneously with other views backed by separate models. Restoration must also be possible.

restoration-replacement.mov

Restoration by opening the same content with the same model

restoration-to-same.mov

Restoration by opening another content with the same model

restoration-to-other.mov

Remark

If going through several contents played with the same model / player, the latest content must be restored (in the video below Couleur 3 is restored, not the first opened SWI content):

restoration-to-latest.mov

Restoration by opening another 360° content with the same model

Monoscopic display does not support PiP.

restoration-to-360.mov

Restoration by opening another failing content with the same model

restoration-to-failure.mov

Restoration to another view model with PiP-supporting VideoView

In this case PiP must be stopped since there can be at most a single VideoView enabled for PiP, no matter to which model it is connected.

restoration-to-other-model.mov

Restoration with multiple views involved

pip-with-multiple-views.mov

Restoration by opening the same content with the same model, starting PiP from a custom view and restoring it with a custom view which does not support PiP anymore

restoration-from-custom-with-to-custom-without.mov

Restoration by opening the same content with the same model, starting PiP from a system view and restoring it with a system view which does not support PiP anymore

restoration-from-system-with-to-system-without.mov

Restoration by opening the same content with the same model, starting PiP from a custom view and restoring it with the system view supporting PiP

restoration-from-custom-with-to-custom-without.mov

Restoration by opening the same content with the same model, starting PiP from the system view and restoring it with a custom view supporting PiP

restoration-from-system-with-to-custom-with.mov

Inline system player

system-pip-inline.mov