PresentationSession should have stream interfaces!

[domenic]

Hi all,

My apologies for not looking bringing this up this sooner. But I just looked at the spec for the first time, and realized that PresentationSession is basically duplicating a lot of the work and interface that is going in to streams! I'd like to see if you are interested in, in a future revision, adding direct support for the stream data types.

Similar to promises for async operations, readable streams are meant to be a universal primitive for things that need to be read from, and writable streams for things that need to be written to. So, adding readable and writable stream interfaces to PresentationSession would let them interoperate with the rest of the streaming ecosystem, and be consumed by libraries that are meant to operate on generic streams. For example, you could do things like

fetch("http://example.com/somebytes").then(res => {
  res.body.pipeTo(myPresentationSession.writable);
});

to set up a pipe chain between the data retrieved from fetch and that sent to the presentation session. It would properly apply backpressure and so on. Or you could do streaming uploads of received data, with

fetch("http://example.com/dest", { body: myPresentationSession.readable });

---

The concrete proposal would roughly be that we add a .readable property that is a ReadableStream, and a .writable property that is a WritableStream. The chunk type would be Uint8Array, most likely, to match fetch. You could use these directly (e.g. const reader = session.readable.getReader(); reader.read().then(processNextMessage)) to get the same functionality as your current "message" event and send() APIs---although the pull-based read() API has some big advantages over the push-based "message" one. But the real benefits would come, as shown in the examples above, when used by any generic stream-consuming or producing APIs. Again, the situation is analogous to promises---slightly nicer, but the real benefit comes from building an ecosystem that uses them.

We wouldn't modify any of the existing stuff on PresentationSession---just like we have legacy APIs in the platform that have both promise and callback interfaces, this would be an API that has both ad-hoc reading/writing interfaces and standard streams-based ones. If I'd caught this sooner, maybe we'd have a chance, but as-is it seems too late, especially given that I still have some work to do before I'd say writable stream is stable enough for other specs to consume :-/.

---

WDYT?! If this sounds agreeable, I could try to put together a PR. You've already done much of the hard work by specifying "send a message through a PresentationSession S" and friends. It would largely be setting up some glue for creating the streams appropriately, as in https://domenic.github.io/streaming-mediastreams/#msr-constructor

[avayvod]

Hi Domenic!

Is there any consensus on this API from other browser vendors, Mozilla in particular?
I would support adding it to the spec when it's ready to be used by others.

[domenic]

Yes! Here is Mozilla's intent to implement. And the code is already in WebKit nightlies---they have been contributing tests back to the suite. I don't think it is ready to be turned on in shipping Safari though, last I checked.

[markafoltz]

Hi @domenic, thanks for taking the time to look at our spec and analyze it in relation to Streams. I recall we looked at it some while ago but that spec seemed to be in the formative stages, it looks like a lot of progress has been made!

The main benefit to adopting Streams is interoperability with pipes, which could greatly simplify common actions like reading the content of a URL like a photo and sending it to the presentation.

This is my first time diving into the current Streams document and I came up with several questions after reading it and your proposal. I hope you bear with me and can shed some light.

Reader Loop
The common use case for either side is to read messages in a loop and dispatch them to various application logic (i.e., advancing a slide, taking a turn in a game). With a reader each developer would have to write their own event loop:

while (true) { session.readable().getreader().read().then(handleMessage) }

How is this different/better than session.onmessage = handleMessage?

Chunk types
How are pipes and chunks typed, i.e. how do I know that a reader/writer will accept the type of data produced by the other?

Specifically, the types accepted by the PresentationSession are chosen to be serializable. How do we limit the readers and writers similarly?

Also, how does the reader recover the type of data sent by the writer?

Promise semantics
When does the Promise returned by writer.write() resolve and what guarantees are made by resolution? The actual message delivery to the display may be done by a component that is far removed from the content, and the sending UA may not be able to guarantee that the message was actually received by the other UA.

Must the writer wait until the previous promise has resolved before sending another chunk? It looks like queueing is part of the definition so writes can be pipelined. So are there N pending promises for N chunks in the queue?

Queueing
I worry a bit about adding another layer of queueing under the control of the developer. The implementation in Chromium already has layers of queuing to manage backpressure between the processes that implement the Presentation API. The more queues in use, the higher latency which will affect applications like gaming that need low latency communciation. Perhaps we can use the backpressure parameter as a signal to manage queue lengths throughout the system.

[domenic]

Reader Loop

There's a couple major ways in which you get benefits:

1. You can call read() at any time in your program, without fear of losing messages. Whereas, if you attach an onmessage handler too late, you lose data.
2. The lack of read() call, e.g. if the client is overloaded and cannot process data immediately, can be used as a backpressure signal, to stop producing so much data. I'm not sure this is applicable in the cases you mention, but it's part of the generic framework. (Maybe it would be useful to let the other side of the presentation know that its commands are not being processed in a timely fashion? It's generally applicable for any async processing of commands.)

More relevantly for your question though, you can just do

session.readable.pipeTo(new WritableStream({ write: handleMessage }));

as a starting point, with the potential for further customization (e.g. processing close signals or applying custom backpressure strategies) by adding more options to the writable stream constructor.

Chunk types

How are pipes and chunks typed, i.e. how do I know that a reader/writer will accept the type of data produced by the other?

A writable stream will usually error if fed an incompatible chunk type.

Specifically, the types accepted by the PresentationSession are chosen to be serializable. How do we limit the readers and writers similarly?

For the readable side, it's easy: just only produce serializable chunk types.

For the writable side, you would error if given an incompatible chunk type.

Also, how does the reader recover the type of data sent by the writer?

I don't think I fully understand the question...

Promise semantics

The semantics of the write() promise are entirely up to the creator of the writable stream. In general it does not signal a guarantee of delivery, but it may be useful for communicating immediately-known errors (e.g., the file handle has been closed). Or you could just have it always fulfill immediately.

Must the writer wait until the previous promise has resolved before sending another chunk?

Nope! It automatically gets queued. You can call write() many times in quick succession.

It looks like queueing is part of the definition so writes can be pipelined. So are there N pending promises for N chunks in the queue?

Yes, although if you decide to implement your writable stream so that it processes all writes immediately, the queue won't really materialize.

Queueing

So, yeah, the idea of streams is to provide an interface that exposes more directly the backpressure signals and queuing that is already presumably happening in your implementation. Either automatically, as happens with pipeTo(), or manually, if the developer does a manual read() loop or consults the writable stream's backpressure signals. As such I don't think you'd want to provide another layer---you'd just more directly expose the layer you already have. I'd be interested in digging more into your thoughts here, especially as the design of writable streams is still shaping up.

[markafoltz]

@domenic, would you be available at TPAC 2015 to discuss this in person? I think it would be a productive topic for the Second Screen F2F meeting we will be having there.

[domenic]

I'm sadly on vacation during TPAC, and can't make it :(. I'll be at BlinkOn if that helps...

[markafoltz]

Hi @domenic, this issue was tagged for consideration as the Presentation API moves towards Candidate Recommendation. Thanks for your detailed answers earlier.

I briefly reviewed the state of the Streams specification [1] and wanted to share how it might fit in with the Presentation API.

It looks like from an API point of view, the suggested integration is to expose a way to get the ReadableByteStream and WritableByteStream for a PresentationConnection, to allow two-way messaging with the browsing context that owns the other side of the PresentationConnection. The WritableByteStream object could take "any" type, but it would need to be enforced that the object is serializable (through some mechanism) to allow messaging with remote user agents.

partial interface PresentationConnection : EventTarget {
  readonly attribute Promise<ReadableByteStream> readStream;
  readonly attribute Promise<WritableByteStream> writeStream;
}

Several details need to be worked out:

1. Are these streams instantiated with the PresentationConnection, or lazily only when the connection enters a "connected" state? The approach above follows the latter, along the lines of the Fetch API in Chromium [2].
2. The canceled/errored states of the streams would need to be mapped to/and from the equivalent PresentationConnection states and event handlers. I am not sure if the Reader events would need to propagate to/from the PresentationConnection states (i.e., when should the Promise from reader.read() reject.)
3. What types are permitted to be written to writeStream? Must include at least DOMString, Blob, ArrayBuffer, and ArrayBufferView for backwards compatibility.
4. Do we leave serialization entirely in the hands of the page that obtains the WritableByteStream, specify a serialization format for any permitted type, or specify a TransformStream that the page should attach to the stream handle serialization?

In terms of implementation, it looks like the major browsers are further along on the reader side, than the writer side. Until a bidirectional API is in place, I think it will be hard to make concrete implementation progress.

One possible approach in the interim: If there were an UnderlyingSource interface that could be implemented by a polyfill around the current API, then the page could construct its own ReadableStream/WritableStream objects.

[1] https://streams.spec.whatwg.org/
[2] https://code.google.com/p/chromium/codesearch#chromium/src/third_party/WebKit/Source/modules/fetch/Response.idl&l=30

[domenic]

I briefly reviewed the state of the Streams specification [1] and wanted to share how it might fit in with the Presentation API.

Thanks for your thoughts! I think we are in general aligned on direction :). More detail below.

It looks like from an API point of view, the suggested integration is to

The names are traditionally readable and writable, and there is no need to hide them behind a promise (see below).

Are these streams instantiated with the PresentationConnection, or lazily only when the connection enters a "connected" state? The approach above follows the latter, along the lines of the Fetch API in Chromium [2].

Conceptually, they should be instantiated along with the connection. The analogy with Fetch here is that a PresentationConnection is kind of like a Request; there is no fetch-like two-stage request -> response transition in the presentation API.

The canceled/errored states of the streams would need to be mapped to/and from the equivalent PresentationConnection states and event handlers. I am not sure if the Reader events would need to propagate to/from the PresentationConnection states (i.e., when should the Promise from reader.read() reject.)

This is all taken care of by the streams infrastructure. E.g. reader.read() rejects when the stream's state is errored (which could be easily taken care of via an appropriate underlying source that ties together the stream's state and the PresentationConnection states).

What types are permitted to be written to writeStream? Must include at least DOMString, Blob, ArrayBuffer, and ArrayBufferView for backwards compatibility.

Anything you wish; how to process the incoming data is up to your spec.

Do we leave serialization entirely in the hands of the page that obtains the WritableByteStream, specify a serialization format for any permitted type, or specify a TransformStream that the page should attach to the stream handle serialization?

I think either approach is OK. It sounds like it would be more convenient for your users to allow any type and do the conversion inside the stream implementation (i.e. inside the underlying sink)

In terms of implementation, it looks like the major browsers are further along on the reader side, than the writer side. Until a bidirectional API is in place, I think it will be hard to make concrete implementation progress.

Yes; I definitely agree. As I said in the OP, the Presentation API should continue with its current API, and as streams become ready---with the writer side being the current blocker---consider adding new surface.

One possible approach in the interim: If there were an UnderlyingSource interface that could be implemented by a polyfill around the current API, then the page could construct its own ReadableStream/WritableStream objects.

Yeah, this should work really well. I expect such efforts to take place before any standardization work, and that such p(r)olyfills will be helpful when the time comes to work on something standard.

[markafoltz]

Thanks for the feedback @domenic. With your suggestions the Streams-compatible API might be:

partial interface PresentationConnection : EventTarget {
  readonly attribute ReadableByteStream readable;
  readonly attribute WritableByteStream writable;
}

In the scope of current work on the Presentation API, I plan on closing this issue, since I don't see timelines lining up for us to add a Streams API in the CR time frame.

However, I will file a new issue referencing this to experiment with the polyfill mentioned above.