tvOS fixups

[tmm1]

Looks like I forgot to mark this scheme as shared.

I'm importing MoltenVK into another tvOS project now, and will leave this in WIP until I get things work in-case anything else was missed.

cc #912 #541

[tmm1]

Looks like I missed splitting out VK_USE_PLATFORM_IOS_MVK in mvk_vulkan.h. It works masquerading as iOS but that's probably not ideal.

[tmm1]

Looks like I missed splitting out VK_USE_PLATFORM_IOS_MVK in mvk_vulkan.h. It works masquerading as iOS but that's probably not ideal.

Though not ideal, I believe we are stuck with this for the moment as vulkan-headers only exposes VK_STRUCTURE_TYPE_IOS_SURFACE_CREATE_INFO_MVK

[cdavis5e]

Looks like I missed splitting out VK_USE_PLATFORM_IOS_MVK in mvk_vulkan.h. It works masquerading as iOS but that's probably not ideal.

Though not ideal, I believe we are stuck with this for the moment as vulkan-headers only exposes VK_STRUCTURE_TYPE_IOS_SURFACE_CREATE_INFO_MVK

That's from the old, deprecated surface extension. Since tvOS is a new platform, we shouldn't even need to expose that extension. The new one, VK_EXT_metal_surface, doesn't have this problem. Just use VK_USE_PLATFORM_METAL_EXT and you'll be fine.

[tmm1]

Thanks @cdavis5e! I switched over and things are making a lot more sense now.

I finally got things wired up enough that I'm hitting actual bugs. Looks like optimalBufferCopyOffsetAlignment is 0, and I may have missed adding a few more tvos paths into initProperties

[tmm1]

I'm struggling to find where the values for maxTessellationGenerationLevel and maxTessellationPatchSize come from.

[billhollings]

@tmm1

FYI...I will be pushing a simulator PR tonight or tomorrow. For no extra charge, it includes some tvOS fixes I noticed and/or needed to patch in order to get the basics of tvOS working on the tvOS Simulator.

That PR might impact what you need to include in this WIP PR.

[tmm1]

FYI...I will be pushing a simulator PR tonight or tomorrow. For no extra charge, it includes some tvOS fixes I noticed and/or needed to patch in order to get the basics of tvOS working on the tvOS Simulator.

Wonderful. Please cc me on the PR if you don't mind.

[billhollings]

Please cc me on the PR if you don't mind.

Will do. Just got some documentation updates to do...then I'll push it up.

I just had a quick look at your changes. Looks like your focus is on other areas than what I worked on, so we shouldn't see any conflicts.

[tmm1]

My application is now getting past vulkan init and failing somewhere in the render pipeline:

Execution of the command buffer was aborted due to an error during execution. Caused GPU Hang Error (IOAF code 3)
Execution of the command buffer was aborted due to an error during execution. Discarded (victim of GPU error/recovery) (IOAF code 5)
Execution of the command buffer was aborted due to an error during execution. Ignored (for causing prior/excessive GPU errors) (IOAF code 4)

[cdavis5e]

My application is now getting past vulkan init and failing somewhere in the render pipeline:

Execution of the command buffer was aborted due to an error during execution. Caused GPU Hang Error (IOAF code 3)
Execution of the command buffer was aborted due to an error during execution. Discarded (victim of GPU error/recovery) (IOAF code 5)
Execution of the command buffer was aborted due to an error during execution. Ignored (for causing prior/excessive GPU errors) (IOAF code 4)

Uh oh. Usually, that's because the command buffer got stuck waiting for something, either an MTLFence or an MTLEvent. You're not using MTLEvents for semaphores, are you? This was a common problem with using MTLEvents for semaphores--don't know if it is now. (If you don't know, you probably aren't, since the default is to use MTLFence.)

[tmm1]

I'm not sure. I do see this during boot:

[mvk-info] Using MTLFence for Vulkan semaphores.

I'm trying to use libmpv and libplacebo to render video, so there's quite a few moving pieces involved. What's the best way to get more details about where/what is getting stuck?

[tmm1]

Looking at #602 (comment), it seems the issue might be that I don't have events = true in the MVK_TVOS build.

[tmm1]

I added events = true, but the error did not go away.

[tmm1]

Okay, it looks like some videos are rendering correctly! I should be able to figure out how the shaders for the one failing earlier is different from here!

[billhollings]

@tmm1

Please cc me on the PR if you don't mind.

PR #935 is in flight now.

[tmm1]

@billhollings This is good to merge whenever you get a chance. There are no conflicts with your other PR so should be fine to merge in any order.

[tmm1]

Actually hold on merge. I just found initMemoryProperties is missing a code-path for tvOS.

[tmm1]

So it looks like MVK_VK_MEMORY_TYPE_METAL_MANAGED is only available on macOS. Does that mean none of the iOS/tvOS memory types offer VK_MEMORY_PROPERTY_HOST_CACHED_BIT?

I was able to add MoltenVK to libmpv fairly easily: mpv-player/mpv#7857

But libplacebo is trying to find a memory slab with VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT set and failing to do so:

[vo/gpu/vulkan/libplacebo] error: Found no memory type matching property flags 0xa and type bits 0x3!
[vo/gpu/vulkan/libplacebo] error: type=0 heapIdx=0 flags=0x1
[vo/gpu/vulkan/libplacebo] error: type=1 heapIdx=0 flags=0x7
[vo/gpu/vulkan/libplacebo] error: type=2 heapIdx=0 flags=0x11

https://github.com/haasn/libplacebo/blob/6a55f81a7604d3d98a9951f808b47f80d8985204/src/vulkan/malloc.c#L164-L165

[cdavis5e]

Does that mean none of the iOS/tvOS memory types offer VK_MEMORY_PROPERTY_HOST_CACHED_BIT?

No. The memory type corresponding to MTLStorageTypeShared should have VK_MEMORY_PROPERTY_HOST_CACHED_BIT. Why doesn't it? (A memory type with VK_MEMORY_PROPERTY_HOST_COHERENT_BIT but not HOST_CACHED would correspond to Metal's MTLStorageTypeShared with MTLCPUCacheModeWriteCombined. We deliberately do not offer this type, even though Metal supports it, because of potential performance concerns with apps that very naively choose memory types.)

[tmm1]

I'm not sure why it doesn't. You're saying we should make this change?

--- a/MoltenVK/MoltenVK/API/mvk_datatypes.h
+++ b/MoltenVK/MoltenVK/API/mvk_datatypes.h
@@ -443,7 +443,7 @@ static inline VkExtent3D mvkVkExtent3DFromMTLSize(MTLSize mtlSize) {
 #define MVK_VK_MEMORY_TYPE_METAL_PRIVATE       (VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT)

 /** Macro indicating the Vulkan memory type bits corresponding to Metal shared memory (host visible and coherent). */
-#define MVK_VK_MEMORY_TYPE_METAL_SHARED                (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT)
+#define MVK_VK_MEMORY_TYPE_METAL_SHARED                (VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_COHERENT_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT)

 /** Macro indicating the Vulkan memory type bits corresponding to Metal managed memory (host visible and non-coherent). */
 #define MVK_VK_MEMORY_TYPE_METAL_MANAGED       (VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT | VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT)

[cdavis5e]

You're saying we should make this change?

I don't know. git blame turns up 574e92a.

[tmm1]

I don't know. git blame turns up 574e92a.

Thanks. I'll look into #789 and #786

[billhollings]

But libplacebo is trying to find a memory slab with VK_MEMORY_PROPERTY_HOST_VISIBLE_BIT | VK_MEMORY_PROPERTY_HOST_CACHED_BIT set and failing to do so

Is libplacebo doing this with the idea that the VK_MEMORY_PROPERTY_HOST_CACHED_BIT will provide faster access? Vulkan does not require an implementation to support a memory type that includes VK_MEMORY_PROPERTY_HOST_CACHED_BIT, so that approach could fail on any implementation.

You're saying we should make this change?

VK_MEMORY_PROPERTY_HOST_CACHED_BIT is a difficult one to map to Metal, particularly given its description in the Vulkan spec:

VK_MEMORY_PROPERTY_HOST_CACHED_BIT bit specifies that memory allocated with this type is cached on the host. Host memory accesses to uncached memory are slower than to cached memory, however uncached memory is always host coherent.

In particular, the reference to uncached memory is always host coherent, but slower than cached memory doesn't seem to map well to the UMA that underlies MTLStorageTypeShared on everything Apple except the discrete GPU's on higher-end Macs.

So I think when I was reworking memory definitions with UMA in mind, I took it out, thinking that VK_MEMORY_PROPERTY_HOST_COHERENT_BIT was sufficient to define UMA memory.

However, @cdavis5e 's observation that...

A memory type with VK_MEMORY_PROPERTY_HOST_COHERENT_BIT but not HOST_CACHED would correspond to Metal's MTLStorageTypeShared with MTLCPUCacheModeWriteCombined.

...sounds reasonable (although I still don't understand what's going on under the covers if Apple needs to distinguish MTLCPUCacheModeWriteCombined on UMA).

Anyway...I'm fine with re-adding VK_MEMORY_PROPERTY_HOST_CACHED_BIT into the mix for MVK_VK_MEMORY_TYPE_METAL_SHARED, if there are real-world reasons for doing so.

[tmm1]

Thanks for the context. I'm not sure what the motivations in libplacebo were. I found this commit via git blame which last touched the flag: haasn/libplacebo@22d8002

It sounds like we all agree that the cached bit makes sense to add back. Would you prefer I do so on a new PR, or is it fine in this one? I'm struggling with a few more issues on tvOS, so this PR may sit open for a bit.

[tmm1]

I'm trying now to render 10bit HDR video. I would like to use MTLPixelFormatBGR10A2Unorm, but it appears that is not available on tvOS. Are there other 10bit formats available on iOS/tvOS which can be integrated into the EXT_swapchain_colorspace? Or am I stuck with the 16bit float format?

[tmm1]

Looks like I need the MTLPixelFormatRGB10A2Unorm equivalent of #673

[cdavis5e]

Looks like I need the MTLPixelFormatRGB10A2Unorm equivalent of #673

I'm starting to wonder why I didn't put that into #673, especially since that format is now documented as supported.

[billhollings]

Would you prefer I do so (VK_MEMORY_PROPERTY_HOST_CACHED_BIT) on a new PR, or is it fine in this one?

You might as well do it here, since you're well suited to test it, and we haven't had any other complaints about it yet.

Looks like I need the MTLPixelFormatRGB10A2Unorm equivalent of #673

If you are going to do so, please do this one as a separate PR...and include the other outstanding available surface formats in the process.

BTW...I notice that doc says MTLPixelFormatRGB10A2Unorm and MTLPixelFormatBGR10A2Unorm are only supported on macOS. Have you confirmed that they're usable on tvOS? If not, it looks like there are also the 10bit XR formats available (and we should add guard code around the use of MTLPixelFormatRGB10A2Unorm and MTLPixelFormatBGR10A2Unorm).

Or I can take care of this one (surface formats) in a separate PR if you prefer.

[tmm1]

I notice that doc says MTLPixelFormatRGB10A2Unorm and MTLPixelFormatBGR10A2Unorm are only supported on macOS.

The header looks like this:

    MTLPixelFormatRGB10A2Unorm = 90,
    MTLPixelFormatRGB10A2Uint  = 91,

    MTLPixelFormatRG11B10Float = 92,
    MTLPixelFormatRGB9E5Float = 93,

    MTLPixelFormatBGR10A2Unorm  API_AVAILABLE(macos(10.13), ios(11.0)) = 94,

I thought this meant it might be available, but when trying to use it I get an error:

*** Terminating app due to uncaught exception 'CAMetalLayerInvalid', reason: 'invalid pixel format 90'

[billhollings]

The changes to RGB10A2Unorm and RGB10A2Uint need to be modified.

[cdavis5e]

I've opened #939 to fix the supported surface formats.

[tmm1]

I made a little bit progress with my HDR video rendering goal. First I figured out how to explicitly specify the VkSurfaceFormatKHR I want to use into libplacebo. Since the 10bit formats aren't available, I'm trying to use MTLPixelFormatRGBA16Float. Here's what I'm seeing:

Apple TV 4K running 13.4.6 and iPhone X running 13.3
{VK_FORMAT_R16G16B16A16_SFLOAT, VK_COLOR_SPACE_SRGB_NONLINEAR_KHR}: video renders but colors incorrect (as expected)
{VK_FORMAT_R16G16B16A16_SFLOAT, VK_COLOR_SPACE_HDR10_ST2084_EXT}: black screen

tvOS Simulator and iOS Simulator on iMac Retina 5K 2017
{VK_FORMAT_R16G16B16A16_SFLOAT, VK_COLOR_SPACE_HDR10_ST2084_EXT}: video renders and colors are correct

iPhone Xs Max running 13.5.1
{VK_FORMAT_R16G16B16A16_SFLOAT, VK_COLOR_SPACE_HDR10_ST2084_EXT}: video renders and colors are correct

[tmm1]

Apple TV 4K running 13.4.6
{VK_FORMAT_R16G16B16A16_SFLOAT, VK_COLOR_SPACE_HDR10_ST2084_EXT}: black screen

I just upgraded this Apple TV to tvOS 14.0 beta (18J5313t) and the video is now rendering with the correct colors!

[cdavis5e]

Apple TV 4K running 13.4.6
{VK_FORMAT_R16G16B16A16_SFLOAT, VK_COLOR_SPACE_HDR10_ST2084_EXT}: black screen

I just upgraded this Apple TV to tvOS 14.0 beta (18J5313t) and the video is now rendering with the correct colors!

I guess then it was a bug, or a missing feature. Maybe we should avoid advertising the HDR color spaces unless we're on version 14EDIT: 13.5?

[tmm1]

At the moment, I am using a software decoder to decode these UHD video frames. libplacebo takes the decoded frame and uploads its pixel data into the MTLPixelFormatRGBA16Float surface it creates via vulkan/moltenvk, then uses some shaders to render it. (There appears to be some bug in the shader pipeline, as I'm seeing video corruption around the edges of the video. Happens on iPhone and Apple TV, but not on the simulators. I plan to revisit that later.)

Instead, I would ideally like to use VideoToolbox to do the decoding using hardware acceleration. That gives me a CVImageBuffer, which I can then map to a Metal texture using CVMetalTextureCacheCreateTextureFromImage. However I'm not using metal directly, and only interacting with the Vulkan API. Is there any way to access underlying Metal textures via Vulkan in a way that would allow me to use CVMetalTextureCacheCreateTextureFromImage?

[cdavis5e]

Is there any way to access underlying Metal textures via Vulkan in a way that would allow me to use CVMetalTextureCacheCreateTextureFromImage?

Currently, you can use vkSetMTLTextureMVK() from VK_MVK_moltenvk. Beware: this extension doesn't play well with the Vulkan loader (vulkan.framework on Apple platforms).

@billhollings is working on an extension to be used with VK_KHR_external_memory that will allow you to import Metal textures in a way that the Vulkan loader can support. Long term, this will be the recommended way to import Metal textures into MoltenVK.

[tmm1]

Currently, you can use vkSetMTLTextureMVK() from VK_MVK_moltenvk.

Thanks!

Beware: this extension doesn't play well with the Vulkan loader (vulkan.framework on Apple platforms).

My understanding was that the loader is currently only used on macOS, so this technique may work for me on tvOS/iOS until the new VK_KHR_external_memory extension is done.

It looks like libplacebo supports a number of other VK_KHR_external_memory extensions already, so it should be easy to add support for that when it is ready.

[tmm1]

It looks like libplacebo supports a number of other VK_KHR_external_memory extensions already, so it should be easy to add support for that when it is ready.

There appears to be quite a lot of code related to VkExternalMemoryImageCreateInfoKHR in libplacebo, and it is unclear how/where I could use vkSetMTLTextureMVK() instead, so perhaps I'll just punt on this until the new extension is ready.

(There appears to be some bug in the shader pipeline, as I'm seeing video corruption around the edges of the video. Happens on iPhone and Apple TV, but not on the simulators. I plan to revisit that later.)

This is the only outstanding issue left, and I'm not sure where exactly the bug lies. My uneducated guess is that its related to the SPIRV -> MSL conversions perhaps, especially since the same SPIRV is confirmed to be working on several other platforms. It's also interesting that it works on macOS in the simulators, but not on the actual devices.

Here's what the corruption looks like:

Only seems to happen at the beginning and end of the buffer. So perhaps some kind of alignment problem?

[tmm1]

@billhollings Please go ahead and merge this now. I'm probably going to step away from this project until later in the year, when tvOS 14 is released (and hopefully there's also some progress on the VK_KHR_external_memory extension by then).

[billhollings]

Please go ahead and merge this now. I'm probably going to step away from this project until later in the year, when tvOS 14 is released

Okay. It looks good so far. Thanks for all the work so far on bringing tvOS to MoltenVK!

and hopefully there's also some progress on the VK_KHR_external_memory extension by then

Definitely should be. But if not...there is always the existing vkSetMTLTextureMVK().