Support for web workers

[murillo128]

How are webcodecs objects supposed to work with web workers? Should the mediastreamtrack be transferable from/to the main thread to/from a web worker?

[sandersdan]

The first question is whether WebCodecs should work in a worker at all. The current implementation in Chromium does not, as the existing media stack in Chromium assumes there is a frame. I hope to support dedicated workers in the future.

There is a separate question about background and shared workers, which should be a separate issue.

Options for getting data in and out:

• TransferableStreams: in theory, all streams can be transferred. Support for a transferList is not in the spec yet though, and it's possible that WebCodecs could be specified in an incompatible way. Such a design precludes using an single object to represent a decode frame, which is annoying. I do not know if this feature is under active development.
• MediaStreamTrack: there has not been much work on MediaStream integration yet, but being able to transfer a MediaStream would directly solve the problem. This is probably out of scope for the first version of WebCodecs.
• Transferable encoded and decoded chunks: would enable postMessage(), but this seems like a worse API than Streams. Also requires having an explicit type, so we won't be able to use IDLDictionary.
• Manual transfer: It should be possible to deconstruct the input/output chunks, transfer them via postMessage(), and reconstitute them on the other side. This should work with Chromium's VideoDecoder implementation as soon as I implement ImageBitmap support.

[steveanton]

Note that worker integration is discussed in the explainer under "Execution environment." Might be worth expanding that section to be a bit more concrete though.

[murillo128]

My main use case is to be able to do video manipulation on gUM video track on web worker

Currently I can captureStream on an offscreencanvas, but I cannot easily grab the video contents . I would need to do a video.requestAnimationFrame get the createImageBitmap and transfer it to the web corker. If i could transfer the mediatrack (or just the reader/writter) to the web worker it would be fine for me.

[aboba]

@murillo128 In WEBRTC WG, we have discussed whether access to raw video is best provided by adding a method on the track, or on the sender/receiver (the approach currently taken by
Insertable Streams). In a number of use cases such as "funny hats" or "machine learning", the raw video processing would probably be done on the sender side, in which case adding a track method might work. The major question is how to make this efficient enough. Insertable Streams combined with web workers and transferrable streams seems performant enough to enable scenarios involving encoded frames such as e2e encryption and Virtual Reality Gaming. However, raw video frames are much larger and so a track method providing a stream of raw video frames might not provide enough performance even with web workers and transferrable streams.

[jamespearce2006]

I'd like to add my voice to the requests for web worker support.

We currently have a decode and render pipeline that happens entirely in a worker (using WebAssembly and an offscreen canvas). It'd be a real shame, and rather perverse, to have to shuffle samples to the main thread to get them decoded, then back to the worker to render them.

[tguilbert-google]

This issue hasn't been updated in a while, but Chromium does plan on supporting WebCodecs in workers. In fact, VideoDecoder should work on the latest Canary (crbug.com/1138676), and VideoEncoder will be following relatively soon (crbug.com/1094169).

Transferring MediaStreamTrack directly is still currently out of scope, as there are spec implications beyond the simple technical feasability. However, the combination of a stream based track readers and native stream transfer optimizations will allow frames to be delivered onto a worker thread directly, without hopping onto the main thread.
The TrackProcessor and TrackGenerator will ship on the same timeline as WebCodecs. Last I heard, the stream transfer optimizations were planned for Q1 2020.

[tguilbert-google]

Note that the support above is only for dedicated workers and that there are still open questions about shared workers and service workers.

[jamespearce2006]

Having access to VideoDecoder from a dedicated worker is exactly what I need, so I'm happy!

[jamespearce2006]

What about AudioDecoder? I notice that it outputs AudioBuffers, but these aren't exposed to Worker, nor are they transferable.

[tguilbert-google]

The AudioDecoder support should have been added at the same time as VideoDecoder.

It landed in 89.0.4332.2:
https://chromium.googlesource.com/chromium/src/+/acbd2eae0763947f78244814c2bdd5628778813d

It was merged back to 88.0.4324.29, so it should be available in Beta:
https://chromium.googlesource.com/chromium/src.git/+/a9b34930a44749da8e6ba5da4cf1df9daa1c3a94

I guess you are saying that underneath our worker-exposed AudioFrame lies the not-worker-exposed AudioBuffer, and therefore AudioWorker is not useable in workers? That is a bug, I've opened crbug.com/1160580.

[jianjunz]

Is it possible to make AudioFrame transferable?

AudioDecoder and AudioFrame are exposed to DedicatedWorker, but Web Audio APIs(not only AudioBuffer, but also other APIs) are exposed to Window only. At this moment, we need to getChannelData in worker and create a new AudioBuffer in the main thread. Then decoded audio frames can be processed by Web Audio.

[tguilbert-google]

This is currently planned. The spec doesn't reflect it yet, but it's on the short list of things to add in Chromium. There's currently discussion on the WebAudio spec, which needs to be modified to expose AudioBuffer (tracking bug for Chromium).

[chcunningham]

I think we can close this, but let me give a overview of where we landed.

We now support all decoder/encoder interfaces in DedicatedWorker. I think WebAudio WG is near consensus to expose AudioBuffer to workers as well.

VideoFrame and AudioFrame will be transferable. Spec work tracked in #129.

To transfer frames from a MediaStream, the idea is to setup https://w3c.github.io/mediacapture-insertable-streams/#track-processor in the main window, but transfer it's readable stream to the worker for input into the encoder. Under the hood UA's may will optimize that transfer such that frames arriving from the camera do not create main-thread contention in the original Window context.

Support for Service and Shared workers is not planned at this point.

Feel free to re-open or file new sub issues if more to discuss.

[randallb]

Are there any plans for sharedworker support or any way I can advocate on their behalf? The annoying scenario I end up with is a sharedworker spawning a dedicated worker... which I can probably deal with but it's less than ideal rn.

[sandersdan]

Can you say more about your use case?

The main reason we did not propose SharedWorker support is that the lifetime is undefined, and we were not aware of use cases that made sense in that context. In fact the use case of background encoding is still missing an appropriate worker type despite being on the original list of use cases.

A secondary reason is that that postMessage() to/from a SharedWorker needs need to serialize the VideoFrames. We (Chrome implementors) did not implement serialization, as it is expensive and the paths are synchronous.