CaptureElement or other UI Element to preview stream from a capture device

[BernardoPoiares]

Hello,

I want to know if the CaptureElement will be available to use or if it will be replaced for some other new element.

[StephenLPeters]

Can you please provide more info about what product you are asking about? ProjectReunion Winui3 UWP/Desktop etc.

[BernardoPoiares]

Winui3 UWP

[StephenLPeters]

@Austin-Lamb or @MikeHillberg was this removed from winui3? I think this is owned by the media team, are we asking them to do the work or enabling the feature for winui3? I think @chigy has contacts with that team.

[rmbl-xD]

I am also looking forward for the CaptureElement. I think it is necessary for the MediaCapture class to show a preview.

[Kohei-Motoya]

I had this problem too. I looked for CaptureElement, but couldn't find it.

[Environment]
OS: Windows10 (19042.985)
Windows SDK: 10.0.19041.0
Project Reunion: 0.5.7
CPU architecture: x64
Program Type: Desktop (C++)

[CarteKiwi]

Same here.

[bkudiess]

CaptureElement is not ready for consumption in WinUI 3 and thus is not available in the current releases. It is on the backlog for a future WinUI 3 release, but unfortunately, we do not currently have an estimate of when. In the meantime, you can use the following sample code to hook up to a camera:

namespace winrt::WinUIExtensions::implementation
{
    using namespace Windows::Media::Capture;
    using namespace Windows::Storage;

    CameraCaptureUI::CameraCaptureUI(const Microsoft::UI::Xaml::Window& window)
    {
        const auto nativeWindow = window.as<IWindowNative>();
        HWND hWnd = NULL;
        winrt::check_hresult(nativeWindow->get_WindowHandle(&hWnd));

        const auto init = m_launcherOptions.as<IInitializeWithWindow>();
        winrt::check_hresult(init->Initialize(hWnd));

        m_launcherOptions.TreatAsUntrusted(false);
        m_launcherOptions.DisplayApplicationPicker(false);
        m_launcherOptions.TargetApplicationPackageFamilyName(L"Microsoft.WindowsCamera_8wekyb3d8bbwe");
    }

    Windows::Foundation::IAsyncOperation<Windows::Storage::StorageFile> CameraCaptureUI::CaptureFileAsync(Windows::Media::Capture::CameraCaptureUIMode mode)
    {
        if (mode != CameraCaptureUIMode::Photo)
        {
            winrt::check_hresult(E_NOTIMPL);
        }

        const auto appDataCurrent = Windows::Storage::ApplicationData::Current();
        const auto tempLocation = appDataCurrent.TemporaryFolder();
        const auto tempFileName = L"CCapture.jpg";
        const auto tempFile = co_await tempLocation.CreateFileAsync(tempFileName, CreationCollisionOption::GenerateUniqueName);
        const auto token = Windows::ApplicationModel::DataTransfer::SharedStorageAccessManager::AddFile(tempFile);

        auto valueSet = Windows::Foundation::Collections::ValueSet();
        valueSet.Insert(L"MediaType", winrt::box_value(L"photo"));
        valueSet.Insert(L"PhotoFileToken", winrt::box_value(token));

        const auto uri = Windows::Foundation::Uri(L"microsoft.windows.camera.picker:");
        const auto result = co_await Windows::System::Launcher::LaunchUriForResultsAsync(uri, m_launcherOptions, valueSet);
        if (result.Status() == Windows::System::LaunchUriStatus::Success && result.Result())
        {
            return tempFile;
        }
        return nullptr;
    }
}

And in you xaml.cs:

private async void myButton_Click(object sender, RoutedEventArgs e)
{
    System.Console.WriteLine("MainWindow.myButton_Click()");
    myButton.Content = "Clicked";

    var captureUI = new WinUIExtensions.CameraCaptureUI(this);
    var file = await captureUI.CaptureFileAsync(Windows.Media.Capture.CameraCaptureUIMode.Photo);
    if (file != null)
    {
        var bitmapImage = new BitmapImage()
        {
            UriSource = new System.Uri(file.Path),
        };
        this.image.Source = bitmapImage;
    }
}

[zangai]

@bkudiess Any way for this to work in a C# only project? Or otherwise guide me on how to add a C++ component to a C# project?

[mallocation]

@bkudiess Wow, thank you so much for that sample! I was able to launch the camera from a WPF .NET Core app using your guidance. Is there any documentation for other parameters besides MediaType and PhotoFileToken? Specifically, I'm looking to set the equivalent of CameraCaptureUI.PhotoSettings.CroppedAspectRatio when launching.

[rgomez90]

@mallocation here is a C# version of @bkudiess answer

CameraCaptureUI

public class CameraCaptureUI
{
    private LauncherOptions _launcherOptions;

    public CameraCaptureUI(Window window)
    {
        var hndl = WindowNative.GetWindowHandle(window);

        _launcherOptions = new LauncherOptions();
        InitializeWithWindow.Initialize(_launcherOptions, hndl);

        _launcherOptions.TreatAsUntrusted = false;
        _launcherOptions.DisplayApplicationPicker = false;
        _launcherOptions.TargetApplicationPackageFamilyName = "Microsoft.WindowsCamera_8wekyb3d8bbwe";
    }

    public async Task<StorageFile> CaptureFileAsync(CameraCaptureUIMode mode)
    {
        if (mode != CameraCaptureUIMode.Photo)
        {
            throw new NotImplementedException();
        }

        var currentAppData = ApplicationData.Current;
        var tempLocation = currentAppData.TemporaryFolder;
        var tempFileName = "CCapture.jpg";
        var tempFile = await tempLocation.CreateFileAsync(tempFileName, CreationCollisionOption.GenerateUniqueName);
        var token = Windows.ApplicationModel.DataTransfer.SharedStorageAccessManager.AddFile(tempFile);

        var set = new ValueSet
        {
            { "MediaType", "photo" },
            { "PhotoFileToken", token }
        };

        var uri = new Uri("microsoft.windows.camera.picker:");
        var result = await Launcher.LaunchUriForResultsAsync(uri, _launcherOptions, set);
        if (result.Status == LaunchUriStatus.Success)
        {
            return tempFile;
        }

        return null;
    }
}

MainWindow

public sealed partial class MainWindow : Window
{
    private Image image;

    public MainWindow()
    {
        this.InitializeComponent();
    }   

    private async void Button_Click(object sender, RoutedEventArgs e)
    {           
        var captureUI = new CameraCaptureUI(this);
        var file = await captureUI.CaptureFileAsync(Windows.Media.Capture.CameraCaptureUIMode.Photo);
        if (file != null)
        {
            var bitmapImage = new BitmapImage()
            {
                UriSource = new System.Uri(file.Path),
            };
            this.image.Source = bitmapImage;
        }
    }
}

[rgomez90]

@bkudiess I tried this code (with hope) will work with WPF NET 6 as well.

I just changed the line to get the window handle to var hndl = new WindowInteropHelper(window).Handle;.

It compiles but an exception is thrown in CaptureFileAsync.

I suppose this is so, because one cannot do this in WPF but can you confirm it? Or is there a way to make it work under WPF NET 6?

[mallocation]

@rgomez90 Thanks! A while back I was able to get this "working" in WPF .NET Core 3.1. I was more so asking about additional parameters for the Camera app, specifically the cropping parameters that are available through CameraCaptureUI. I eventually discovered those parameters:

• AllowCropping: true/false
• PhotoCroppedARHeight: integer value
• PhotoCroppedARWidth: integer value

This all worked while debugging. But after packaging the WPF app using MSIX, I was not able to launch the Camera app via Launcher.LaunchUriForResultsAsync. I've since discovered that launching for results is not supported in desktop apps, though I'm not sure why.

microsoft/WindowsAppSDK#1034
microsoft/WindowsAppSDK#1938

[rgomez90]

Wooops. Sorry @mallocation I wanted to tag @zangai 😂 you didn't ask for a c# version, sorry mate! But thank you because I think you answered with your post my question🙏😇 why I am getting an exception there in my wpf net 6 test

[mallocation]

Hey no worries @rgomez90! Your C# code is sure to help the next soul that comes across this very same problem!

[ghost]

I'm interested in any update on CaptureElement on WinUI3, since the last comment from @bkudiess back in August. I'm evaluating C# application development that requires a camera preview in shared mode, and realize that we are now on a crux between sticking with UWP or moving to WinUI3. Of course, I would love to move to WinUI3 so that we are not hindered by UWP and its stagnant developments. From what I can see, UWP versions greater than 10.0.16299 don't support .NET Core beyond 2.0 in its compatibility matrix. This makes long-term development difficult because then we can't write reusable .NET Core 3.1/5/6 libraries and instead have to write all of our "camera" code in an exclusive UWP library.

Any update would be greatly appreciated.

[ghost]

@rgomez90 I am not able to get your C# sample to work. When I place it into my test WinUI3 app, this line always returns the result as LaunchUriStatus.Unknown.

var result = await Launcher.LaunchUriForResultsAsync(uri, _launcherOptions, set);

Do I need to specify any particular capabilities in my WAP project's Package.appxmanifest file? I have checked 'microphone' and 'webcam', but when I click the 'Click Me' button on the UI, nothing happens. And if I run the debugger and put in breakpoints, I see that result is equal to LaunchUriStatus.Unknown.

[tmathews]

I am also looking forward to having this implemented. 👍

[rgomez90]

I believe docs are wrong.

Since they said you should be able to use this approach with WPF and Winforms as well (need to look for docs source now) but when trying it the line getting the window handler will throw an exception if the UI framework is not WinUI....

[MajidAbuRmelah]

@rgomez90 I am not able to get your C# sample to work. When I place it into my test WinUI3 app, this line always returns the result as LaunchUriStatus.Unknown.

var result = await Launcher.LaunchUriForResultsAsync(uri, _launcherOptions, set);

Do I need to specify any particular capabilities in my WAP project's Package.appxmanifest file? I have checked 'microphone' and 'webcam', but when I click the 'Click Me' button on the UI, nothing happens. And if I run the debugger and put in breakpoints, I see that result is equal to LaunchUriStatus.Unknown.

@StephenLPeters @bkudiess
I am having the same issue. If this is not the way to do this, is there any possible way in WinUI3 we can use the webcam?
If not, what are the alternatives to solve this issue and will that option be available in the future releases?

[rgomez90]

@MajidAbuRmelah @TamBuiWork I have checked back, and created a new sample project (repo) and this solution appears only to work if you set your platform to 'x64'.

I don't really know why this is so, only found this but I am not sure if related....maybe @bkudiess can tell us something more.

[arafattehsin]

Hi @rgomez90 - It looks like either your sample project is still private or you removed it. Is it possible for you to make it available so I can have a look and use?

I already have got a UWP app that uses camera stream and draw some overlays on that control (for example, a rectangle), can I achieve this today in Win UI 3 using your example?

[castorix]

@castorix would be great if you share the code as repo!

OK... I had never used GitHub (I used AppBox for ages) I just tested with another project, then this one : WinUI3_DirectShow_Capture

I updated the test code with a Layered child window, to avoid flickering on resizing (because of the background they added in the DirectComposition window)
(tested with 1.1.0)

[mlancione]

Any update on adding CaptureElement to WinUI3? This seems like a basic control.

[TripleSM]

CaptureElement not yet available on WinAppSdk.
The simplest alternative is, Capture the media frames and render them on Image Element to make a preview.
I have made a sample with last stable release of WinAppSdk (1.1.x).
WinUI3 MediaFrame Capture

[arafattehsin]

That's a good sample @TripleSM - thanks for this. I think there's a potential need of a CaptureElement as a Video source but let's see when it becomes available.

[ArchieCoder]

@TripleSM Many thanks! https://twitter.com/archiecoder/status/1587123503739928578

[ghost]

It appears that as of this month, Microsoft has finally released a media playback control for WinUI 3. Check out the MediaPlayerElement control from their WinUI 3 Gallery from the Microsoft Store. It works similarly to the CaptureElement control from WPF, in that you specify its Source property and it will playback either a camera preview stream, or a video file. I have tested that it works on at least two different cameras.

[arafattehsin]

Thanks @TamBuiWork, were you able to capture the frames?

[ghost]

Thanks @TamBuiWork, were you able to capture the frames?

Yes, I was able to process each captured frame. Here's a snippet of my code to show the camera preview in a MediaPlayerElement, as well as processing the frames using the MediaFrameReader approach for additional frame-related scenarios:

How I defined the XAML:

<MediaPlayerElement x:Name="mediaPlayerElement" Stretch="UniformToFill" AutoPlay="True" />

The code-behind:

public MediaFrameSourceGroup MediaFrameSourceGroup;
public MediaCapture MediaCapture;

// Snippet
var deviceid = "set this to your camera's unique device id";
MediaFrameSourceGroup = await MediaFrameSourceGroup.FromIdAsync(deviceid);
MediaCapture = new MediaCapture();
var mediaCaptureInitializationSettings = new MediaCaptureInitializationSettings()

  SourceGroup = this.MediaFrameSourceGroup,
  SharingMode = MediaCaptureSharingMode.SharedReadOnly,
  StreamingCaptureMode = StreamingCaptureMode.Video,
  MemoryPreference = MediaCaptureMemoryPreference.Cpu
};
await MediaCapture.InitializeAsync(mediaCaptureInitializationSettings);

// Set the UI MediaPlayerElement control's Source property to the MediaSource you desire
mediaPlayerElement.Source = MediaSource.CreateFromMediaFrameSource(MediaCapture.FrameSources[this.MediaFrameSourceGroup.SourceInfos[0].Id]);

I was also able to capture every frame and process it using a MediaFrameReader. This might be useful if, say, you want to show some bells and whistles over the camera preview, or to simply capture and save important frames to disk.

public MediaFrameReader MediaFrameReader;

// Snippet. Remember to only do this AFTER you have called MediaCapture's InitializeAsync
MediaFrameReader = await MediaCapture.CreateFrameReaderAsync(MediaCapture.FrameSources[MediaFrameSourceGroup.SourceInfos[0].Id], MediaEncodingSubtypes.Argb32);
MediaFrameReader.FrameArrived += MediaFrameReader_FrameArrived;
await this.MediaFrameReader.StartAsync();

private void MediaFrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
{
    var mediaframeReader = sender as MediaFrameReader;
    if (mediaframeReader != null)
    {
        var mfr = mediaframeReader.TryAcquireLatestFrame();

        // From this point on, follow Microsoft's 'CameraFrames' universal sample to process the frame and do what you want with it: https://github.com/microsoft/Windows-universal-samples
    }    
}

[arafattehsin]

@TamBuiWork Thanks for this. I think the time has come when I should port my app from UWP to WinUI 3 😊

[ghost]

@TamBuiWork Thanks for this. I think the time has come when I should port my app from UWP to WinUI 3 😊

You and me both! Cheers 😊

[SprengerS]

@TamBuiWork Thanks for this. I think the time has come when I should port my app from UWP to WinUI 3 😊

This works great. But does anyone know how to decode a QR Code with the Camera Preview

[codendone]

Closing this as completed: The recommendation is to use a MediaPlayerElement to render source from a MediaCapture. There is now a sample of this in WinUI 3 Gallery.

[ashubuntu]

@codendone MediaPlayerElement can stream from MediaCapture in gallery app downloaded from Store - but frame does not come up to MediaPlayerElement from MediaCapture source when running the repo from visual studio. I am using USB for the webcam input - webcam light turns on but no frame is captured thereafter.