Response timeouts with DirectLineSpeech

[hrazmsft]

Screenshots

Version

NPM: botframework-webchat": "^4.11.0

Describe the bug

After connecting a bot with DirectLineSpeech, every activity (except of the first one), no matter if text or speech, receive timeout while waiting for reply. In other platforms (emulator, postman, azure portal emulator) there is no timeout and all the responses are quick.

Steps to reproduce

1. Render WebChat component with DirectLineSpeech support.
2. Send one message to the bot (text or speech)
3. The response should be quick
4. Send another message to the bot (text or speech)
5. The response has timeout

Expected behavior

The response should be quick like the first response

Additional context

None

[Bug]

[corinagum]

@hraz-msft I tested this out on our sample and my behavior is slightly different from yours - when you say timeout, does the response eventually show up? Or no?

For me these timeout messages are non-blocking.

Sounds like a channel issue - Support team could someone help debug? Thanks.

[tomlm]

yes, it does show up, but 3 seconds later

[compulim]

After some investigations: since Direct Line Speech works quite differently at the protocol level, the delay that applies to Direct Line protocol could not applies to DLSpeech protocol.

Essentially, DLSpeech does not tell Web Chat if the protocol has successfully sent a message or not (a.k.a. no send receipt, no read receipt). In other words, Web Chat never know the ID of the activity just sent. That means, activities sent from user will not have an ID.

Thus, since the activity from the bot has a reply-to-ID, we can never match that ID in our transcript. Thus, we can never sequence the transcript correctly. The delay is meaningless in DLSpeech protocol.

[hrazmsft]

Look, only the WebChat component has that delay and that delay is happening from the second activity. In other platforms there is no such delay for the same bot

[compulim]

The delay is required for accessibility, and it was introduced for Direct Line protocol.

DL protocol is a distributed system: message comes in a very random order. But accessibility and screen reader doesn't work that way. It requires message to be injected on the screen in a time-sensitive manner.

As an accessible UI component which complies to WCAG 2.1, Section 508, and EN 301 549, Web Chat is required to re-sequence the message from a chaotic order. We need to inject those messages on screen in an order meaningful to people who can't see the screen.

To re-sequence the message, delay is required. It is similar to how packet defragmentation works in other systems, such as, TCP/IP.

Unfortunately, Speech SDK (used by DLSpeech protocol) doesn't give us ID, send receipt, or read receipt. It means, Web Chat has no way to verify whether a message has been sent over Web Socket, has reached the bot, or it failed due to intermittent network connectivity.

Also, DLSpeech does not fill in the details of an outgoing activity, such as conversation ID, timestamp, sender ID, activity ID, etc. And it doesn't support a lot of features that is common in transcript-based UI, such as attachments, resuming conversation, proactive messages, piggybacking data, etc.

As we mentioned in this introductory article, DLSpeech is a protocol designed for a non-transcript-based UI, such as home assistant, smart display, automotive dashboard, navigation system, etc. We have a demo here to show how to better utilize the protocol.

[hrazmsft]

Maybe @tomlm or @damarqmsft can give more data. But the weird thing is that other platforms has no delay or timeout (emulators and azure portal WebChat)

[damarqmsft]

Maybe @tomlm or @damarqmsft can give more data. But the weird thing is that other platforms has no delay or timeout (emulators and azure portal WebChat)

Emulator and portal webchat are not using DLSpeech and they might not be implementing the delay. The problem here is we were waiting for a confirmation that would never come via DLSpeech (it comes only with Direct Line).

William's PR should fix it.

[compulim]

This will be released as a daily build tomorrow ~3-4 AM.

https://github.com/microsoft/botframework-webchat/releases/daily