[web-animations-2][css-animations-2] Allow controlling video playback via an animation timeline

[flackr]

Similar to #9110, authors often want the ability to drive videos by animation timelines. See #11587 and #6861 (comment).

We'll need to figure out the details, but this could be something like

document.querySelector('#my-video').timeline = new ViewTimeline(...);

Possibly also with css support, e.g.:

#my-video {
  video-timeline: view();
}

[birtles]

For what it's worth, when we designed the Web Animations API, the intention was that driving video playback would happen via different types of effects and you'd synchronize it with other animations via group effects.

[ydaniv]

For what it's worth, when we designed the Web Animations API, the intention was that driving video playback would happen via different types of effects and you'd synchronize it with other animations via group effects.

Yeah, that would be a more coherent method.
If we consider a new VideoEffect that can take a <video> as target and put in side an Animation that takes a timeline and ranges. Then we can reason about synching that animation with others using group effects.

I suppose if we have video-timeline we also need to consider video-range.
But OTOH, we don't really need all the other animation-* properties.

So, how would a declarative API look like? It seems to me that all we need here is -timeline and -range. So that could be the declarative syntax, while the WAAPI side could be a new type of VideoEffect.

WDYT?

[SebastianZ]

I suppose if we have video-timeline we also need to consider video-range. But OTOH, we don't really need all the other animation-* properties.

I can see possible use cases for some other animation-* properties. E.g. @flackr mentioned video-play-state which could possibly also be covered using animation-play-state. For seconds-long videos used in chats you often see them repeated a few times before they stop, which would be covered by animation-iteration-count. animation-duration could influence the video duration. And animation-direction could influence whether the video is played forwards or backwards.

Sebastian

[bramus]

(#) Possibly also with css support, e.g.:

#my-video {
video-timeline: view();
}

Bikeshedding: maybe playback-timeline to keep it open to other media elements?

---

(#) For what it's worth, when we designed the Web Animations API, the intention was that driving video playback would happen via different types of effects and you'd synchronize it with other animations via group effects.

I feel like that’s the type of thing authors would be able to make themselves using an onProgress event, something that’s been part of the discussion for CustomEffect (#6861). I also saw a similar request in #9974 which would be solved by onProgress.

(Also see https://brm.us/scroll-driven-video#the-demos where I built this type of thing – using a rAF lacking any onProgress event)

My feeling here is that instead of trying to determine all sorts of new *Effect interfaces and requiring authors to overhaul their code to use a GroupEffect, this can all be unified in onProgress.

If the GroupEffect is considered to be more viable than onProgress, I think we should offer authors a BaseEffect class that they can extend in order to roll their own VideoEffect to drive a video progressing based on an animation. This way they would not need to wait for all vendors to implement that specific type of Effect, or any other new type of Effect.

To make these effects available in CSS, they could register it using something like CSS.registerAnimationEffect(new VideoEffect(document.querySelector('#my-video')), '--video'); … which smells a lot like Houdini at that point :)

(Personally I believe an onProgress better fits the mental model authors have about all this, in which they want to “take an animation (effect) and map its progress to something else”. Yes, GroupEffect allows more composability, but I believe that’s only for “advanced” users.)

---

Winging back to Rob’s proposal: What I like about it is that it allows an author to take a ViewTimeline/ScrollTimeline and use that as the timeline for the media file, without requiring some animation to run. This “an animation must be present” requirement has been a recurring point of friction for me, as I sometimes only need the timeline in order to drive things. In those cases I end up setting a dummy animation to goes from opacity 1 to opacity 1 on some element, which seems a bit stupid to be able to achieve my goal.

[johannesodland]

I can see possible use cases for some other animation-* properties. E.g. @flackr mentioned video-play-state which could possibly also be covered using animation-play-state. For seconds-long videos used in chats you often see them repeated a few times before they stop, which would be covered by animation-iteration-count. animation-duration could influence the video duration. And animation-direction could influence whether the video is played forwards or backwards.

I agree with @SebastianZ .

Having delay and end-delay would be useful to stagger videos, or to control the start and end of a common pre-rendered video depending on the context. We often use pre-rendered videos as illustrations or decorations in our articles where this would be useful.

Having the rest of the options/properties could be useful with a regular document timeline as well. I.e. one could use end-delay to play a video to predefined stop. Direction would allow us to do that in reverse, effectively allowing us to play a video step-by-step. (I would love a .playTo() as well :) )

[flackr]

So what I'm hearing is that we should have an easy way to make an effect that drives the video. I wonder if this would be something like

CSS:

video {
  animation-name: video();
}

Or, in JS:

new Animation(new VideoEffect(video), { /* animation options */ });

To the concern from @bramus about just wanting to drive a video, this would be as simple as:

video {
  animation-name: video();
  animation-timeline: view();
}