Missing getVideoFrame method

[DovgopolsSerj]

It looks like the method is lost in mdk.lib v0.30.1
void getVideoFrame(VideoFrame* frame, void* vo_opaque = nullptr)

VideoFrame *vFrame;
pl->getVideoFrame(vFrame);

Use mingw 64 on windows 11:
g++ -c mylib.cpp mylib.o
Ok
g++ -shared -o mylib.dll mylib.o mdk.lib
undefined reference to `mdk::Player::getVideoFrame(mdk::VideoFrame*, void*)'
collect2.exe: error: ld returned 1 exit status

This is the only method that causes an error. Everything is going fine without it including callback processing.
Therefore, I assume that it is lost in mdk.lib.

[wang-bin]

yes, I'm not share what to return. If returning the original decoded frame, the frame can be a hardware frame not accessible by cpu, should I convert to a cpu accessible frame?

btw, why do you need this function?

[DovgopolsSerj]

I'm making a jni plugin for JavaFX. There's a WritableImage that can use ByteBuffer as a source.
I'm using onFrame now. I take the decoded frame and convert it to a suitable one for JavaFX
frame.to(PixelFormat::BGRA, width, height)
And I send it to the java code as an array.
But there are problems here.:

1. onFrame always receives frames in their original format. If this is a 4k video, then it seems to me that this is a rather expensive conversion. Although this is not very noticeable in terms of CPU load.
2. to(PixelFormat, width, height) does not work with all pixels. The comments say "..compressed input/output formats are not supported..". And if D3D11 or MFT decoders are used, then some videos crash with an exception when trying to convert. Therefore, I always have to use DXVA or FFmpeg decoders.

I assumed that if I subscribe to the setRenderCallback in it, getVideoFrame will return an already converted size that can be set to setVideoSurfaceSize and without compression.
But if this is not the case, then the current implementation via onFrame works quite well, I would even say better than flutter with fvp)

[wang-bin]

onFrame always receives frames in their original format. If this is a 4k video, then it seems to me that this is a rather expensive conversion. Although this is not very noticeable in terms of CPU load.

getVideoFrame is the same

to(PixelFormat, width, height) does not work with all pixels. The comments say "..compressed input/output formats are not supported.."

compressed formats is only used by Hap videos decoded by mdk's hap decoder.

And if D3D11 or MFT decoders are used, then some videos crash with an exception when trying to convert

you should report a bug about the crash

But if this is not the case, then the current implementation via onFrame works quite well, I would even say better than flutter with fvp

frame.to + ByteBuffer is not efficient.

[DovgopolsSerj]

mp4 test file
onFrame callback:

onFrame<VideoFrame>([](VideoFrame &frame, int t){
  VideoFrame newFrame = frame.to(PixelFormat::BGRA, 1600, 900);
  return 0;
})

When using decoders:
setDecoders(MediaType::Video, {"MFT:d3d=11", "D3D11", "DXVA", "CUDA", "FFmpeg", "dav1d"})
The app crashes. Log:
Log_MFT_fail.txt
When using decoders:
setDecoders(MediaType::Video, {"DXVA", "FFmpeg", "CUDA", "dav1d"})
There is no crash. Log:
Log_DXVA.txt

[DovgopolsSerj]

frame.to + ByteBuffer is not efficient.

I haven't found a better way. It seems that there is no way to specify in what format and in what sizes the video should be decoded.
And converting on the Java side is much more expensive.
ByteBuffer works very well.
On the Java side, I have:

ByteBuffer byteBuffer = ByteBuffer.allocate(required Size);

private byte[] getBufferArray(){
    return byteBuffer.array();
}

It is important to use allocate rather than allocateDirect.
Then, through array(), you can get direct access to the internal state.

On the native side, I take out this array and fill it directly:

VideoFrame newFrame = frame.to(PixelFormat::BGRA, width, height);
auto frameData = newFrame.bufferData();
int size = width * height * 4;
jmethodID methodId = env->GetMethodID(cls, "getBufferArray", "()[B");
jbyteArray byteArray = (jbyteArray)env->CallObjectMethod(objRef, methodId);
env->SetByteArrayRegion(byteArray, 0, size, (const jbyte *)frameData);

It might be worth experimenting with the new FFI later. There is an option to directly allocate a Memory Segment as a ByteBuffer. But for now, this greatly limits the requirement to have Java 22+.

[DovgopolsSerj]

As a result, I understood the reason for the fall. As always, the little things of rounding. frame.to () crashes if you request odd height or width dimensions and not exactly the same aspect ratio.
Example: Original 3840:2160
frame.to: 1600 x 900 - ok.
frame.to: 1602 x 901 - fail
frame.to: 1605 x 902 - fail
....
frame.to: 1632 x 918 - ok. Exactly the same ratio and both sizes are even.

I found the best option for myself to use only FFmpeg
setDecoders(MediaType::Video, {"FFmpeg:hwaccel=cuvid"})
It offloads the processor very well by decoding on the Nvidia card.
And then I dynamically resize the frames by sending the parameters:
setProperty("video.avfilter", "scale=1632:918:flags=bilinear,format=bgra")
So in onFrame() I get exactly the frame that I need.
All I need is to use memcpy in onFrame. It turns out to have very good performance. By java standards, of course.
It's a pity that it can't be used with D3D11 and other hardware decoders.

[DovgopolsSerj]

It would be nice to add a new Render API "Software".
Which will take the PixelFormat parameter and uses it like this:
updateNativeSurface(uint8_t* data, int width, int height)
This would make it quite easy to use mdk in older frameworks and some new ones like Compose that do not provide a direct api to GPU.

[sbenmeddour]

It would be nice to add a new Render API "Software". Which will take the PixelFormat parameter and uses it like this: updateNativeSurface(uint8_t* data, int width, int height) This would make it quite easy to use mdk in older frameworks and some new ones like Compose that do not provide a direct api to GPU.

If you are interested by using mdk with compose multiplatform without effort, you can use

com.jogamp.opengl.awt.GLJPanel
com.jogamp.opengl.GLEventListener

joggl = { group = "org.jogamp.jogl", name = "jogl-all-main", version = "2.5.0" }
gluegen = { group = "org.jogamp.gluegen", name = "gluegen-rt-main", version = "2.5.0" }

System.setProperty("compose.swing.render.on.graphics", "true")
System.setProperty("compose.interop.blending", "true")
val profile = GLProfile.get(GLProfile.GL2)
val capabilities = GLCapabilities(profile)
val panel = GLJPanel(capabilities)
val animator = Animator(panel)
panel.addGLEventListener(listener)

private val glListener = object: GLEventListener {

    override fun init(drawable: GLAutoDrawable) = Unit

    override fun dispose(drawable: GLAutoDrawable) {
      LibMdkJni.setVideoSurfaceSize(handle, -1, -1, 0L)
    }

    override fun display(drawable: GLAutoDrawable) {
      LibMdkJni.renderVideo(handle, 0L)
    }

    override fun reshape(drawable: GLAutoDrawable, x: Int, y: Int, width: Int, height: Int) {
      LibMdkJni.setVideoSurfaceSize(handle, width, height, 0L)
    }

  }

This is enough to get video playback on all desktop platforms using opengl

But javafx can be indeed better if you want to use Metal or other api

[DovgopolsSerj]

This is enough to get video playback on all desktop platforms using opengl

Yes, thank you. I watched this project. It really works well. But my main goal is to keep the ability to build a completely independent native image using GraalVM.
Unfortunately, I was unable to do this with this package. The native exe is constantly crashing.

[wang-bin]

As a result, I understood the reason for the fall. As always, the little things of rounding. frame.to () crashes if you request odd height or width dimensions and not exactly the same aspect ratio. Example: Original 3840:2160 frame.to: 1600 x 900 - ok. frame.to: 1602 x 901 - fail frame.to: 1605 x 902 - fail .... frame.to: 1632 x 918 - ok. Exactly the same ratio and both sizes are even.

crash for what decoders? recently I don't have a windows device to test, but on macos no crash.

It would be nice to add a new Render API "Software".
Which will take the PixelFormat parameter and uses it like this:
updateNativeSurface(uint8_t* data, int width, int height)
This would make it quite easy to use mdk in older frameworks and some new ones like Compose that do not provide a direct api to GPU.

the renderer has to convert to rgb data in any size, so same implementation as frame.to()

[DovgopolsSerj]

crash for what decoders?

Any except FFmpeg.
FFmpeg does not crash but the picture becomes like this:

so same implementation as frame.to()

Probably yes, but this could avoid allocating memory for a new frame and an unnecessary copy operation back to the application buffer.

[wang-bin]

crash for what decoders?

Any except FFmpeg. FFmpeg does not crash but the picture becomes like this:

Can you try to run Thumbnail.exe in mdk-sdk package?

./Thumbnail.exe -scale 0 -size 1605x901 test.mp4

To test hw decoders, add -c:v MFT,FFmpeg or -c:v d3d11va,FFmpeg

then thumbnail.png is generate. the result is correct for me(tested FFmpeg and MFT software decoder). I guess the bug is from your code.

so same implementation as frame.to()

Probably yes, but this could avoid allocating memory for a new frame and an unnecessary copy operation back to the application buffer.

I can add a new api VideoFrame.to(VideoFrame& out), then out can be reused

[DovgopolsSerj]

My apologies. The error was not mdk, but in the JavaFX graphics engine. The rowPitch of the buffer was not taken into account there.
I went the other way. Now, by analogy with ftp_plugin, I made a D3D11RenderAPI and assigned it to the player. And in the RenderCallback, I copy the image from the texture through an additional STAGING texture.
This works well except that I can't resize the D3D11RenderAPI texture on the go.
This may be necessary if I have two monitors, one hd and the other 4K. While the application is on a small screen, the HD texture is sufficient, but when switching to a large screen, a large texture is already needed, otherwise distortions are visible.
If I just change the texture of my D3D11RenderAPI, the image just disappears.
If I create a new RenderAPI and and assign then a memory leak starts. It seems that the resources serving the previous renderer are not being released, but new ones are being created for the new one.
It also doesn't seem reasonable to keep the texture at the highest possible resolution all the time. It is necessary to collect the buffer line by line on the CPU.
It would be good if, when assigning a new RenderAPI, the resources of the previous one are freed not only if nullptr is passed.

[wang-bin]

I can't reproduce the leak using 1 player 2 d3d11 renderers. What is the leak speed in MB/s?

[DovgopolsSerj]

Sorry for the long time before responding. I needed time to reproduce the error accurately.
In the end, it's not really a mistake. It's just that nowhere in the documentation was it said that all methods affecting the renderer are not thread-safe (such as setScale, setVideoViewport, setVideoSurfaceSize, etc.). and they all call renderVideo.
Since I have quite a lot of processing in RenderCallback (copying from GPU to CPU, rowPitch alignment, copying to java texture) I often got into a thread race condition.
As a result, I fixed this before any image change, first reset the callback by calling setRenderCallback(nullptr) and then snap it again.