DesktopCaptureMonitor desktop stream does not get AudioTrack

[thedracle]

• State which version are you using
  NWJS 0.15.2-win-64
• State which OS version are you running with
  Windows 10

When obtaining a desktop stream using the DesktopCaptureMedia API described at: http://docs.nwjs.io/en/latest/References/Screen/

It doesn't seem possible to obtain an audio track for the desktop that is being recorded, even when providing aud_constraint, see the function below for an example:

    function getDesktopVideoStream(desktopNum, callback, errorCallback) {
            var vid_constraint = {
                mandatory: {
                    chromeMediaSource: 'desktop',
                    chromeMediaSourceId: gui.Screen.DesktopCaptureMonitor
                            .registerStream(`screen:${desktopNum}:0`),
                },
                optional: []
            }

            var aud_constraint = {
                mandatory: {
                    chromeMediaSource: 'system',
                    chromeMediaSourceId: gui.Screen.DesktopCaptureMonitor
                            .registerStream(`screen:${desktopNum}:0`)
                }
            }

            // Get user media by stream selected.
            navigator.webkitGetUserMedia({
                audio: aud_constraint, video: vid_constraint
            }, function (stream) {
                callback(stream);
            }, errorCallback);
    }

The above function will return a stream, however getAudioTracks() returns an empty array.

How can one go about getting a stream object with the audio track for a desktop stream?

[ghostoy]

@thedracle The feature heavily depends on Chrome's internal implementation. However Google is not planning to support audio tracks for screen cast on Linux and Mac (See crbug 223639).

I will try to enable audio tracks for Windows only.

[thedracle]

Thanks :)

I only need it on Windows for now. I was working on making a node module for grabbing Core Audio myself, but it's much better to have it as a stream that can be synced/sent with a video source.

[thedracle]

This change should appear in the following build, correct?

I'm assuming because:
0a316ed

http://dl.nwjs.io/live-build/06-30-2016/nw16-0a316ed-9a7d65c-1610340-77f39b5/v0.16.0/

I've tried the same getDesktopVideoStream example as above running it with similar results on Windows:

s = null;
getDesktopVideoStream(0, (stream) => {s = stream}, (e) => console.log(e));
s.getAudioTracks();

The result is still '[]', no audio tracks.

@ghostoy Did you see something different, or do you have a test/constraints I can use that produces an audio stream on Windows?

[thedracle]

Doing some more of my own testing on Chrome 51.0.2704.103 m on Windows 10 and Windows 7, it appears even as a Chrome extension the behavior is the same-- I.E an upstream issue maybe, or an API change of some kind?

https://bugs.chromium.org/p/chromium/issues/detail?id=223639&can=1&q=desktopCapture%20audio&colspec=ID%20Pri%20M%20Stars%20ReleaseBlock%20Component%20Status%20Owner%20Summary%20OS%20Modified

It seems to indicate this works in Windows on Chrome 39 under some circumstances:

Is this supposed to be supported on Chrome OS 37? #20 suggests it is, however
running the code below in the context of an app with desktopCapture permission yields zero audio tracks on a C720 Chromebook (39.0.2171.94). However it does yield 1 audio track on Windows.
The (whitelisted) Google Cast Extension also seems to fail to capture system audio in "entire screen" mode on ChromeOS (using getUserMedia directly).

The test example he provides:

chrome.desktopCapture.chooseDesktopMedia(['screen'], function(id){
    console.log('opening stream with id', id);
    navigator.webkitGetUserMedia({
      video: {
        mandatory: {
            chromeMediaSourceId: id, 
            chromeMediaSource:'desktop'
        }
      },
      audio: {
          mandatory: {
            chromeMediaSource: 'system',
            chromeMediaSourceId: id, 
          }
      }
    }, function(stream) {
      console.log('number of audio tracks: ', stream.getAudioTracks().length);
      stream.stop();
    }, console.error.bind(console));
});

Yields 'number of audio tracks: 0' on both Windows 10 and 7, as well as NWJS.

[thedracle]

I think there has been a regression of some kind in core chromium.

I'll file a bug on Chrome's tracker.

NWJS 13.3-win-x64 produces an AudioTrack with the same above example as does v0.14.5-win-x64

0.15.0-rc2-win-x64 demonstrates the issue of having no AudioTrack, as does 0.15.2-win-x64.

[thedracle]

See the following bug I've posted against Chromium:

https://bugs.chromium.org/p/chromium/issues/detail?id=624900

[jtg-gg]

@thedracle to capture the audio, you'll need to tick "share audio" inside the chooseDesktopMedia
and to get the tick box for the share audio, you'll need to add "audio
chooseDesktopMedia(['screen', 'audio'], function(id)

[thedracle]

@ghostoy, Any idea how this should interact with DesktopCaptureMonitor, does just adding 'audio' to the list enable audio sharing?

[ghostoy]

For chrome.desktopCapture, you need to add 'audio' to the list and check share audio in the dialog. For nw.Screen.DesktopCaptureMonitor, it should be fixed and you dont need to change your code. I will check the build later tommorrow and give you feedback.

[ghostoy]

@thedracle I've tested the original sample code with the live build for commit 0a316ed and it worked on my Windows 7 / 8.1 / 10.