How to draw YUV buffer image on a texture

[hanna2100]

Hello. Could you please explain how to draw an image using a YUV buffer on a Texture?

I'm receiving camera images in YUV buffer from an external module.
Initially, I implemented drawing these images by using Stream and drawing a bitmap on the canvas of SurfaceTexture.
Although this method works fine, as you know, drawing a bitmap on the canvas is a heavy operation. Therefore, I want to implement it by uploading the buffer directly.

I have implemented it as below, but only a black screen is displayed.
I would appreciate your advice!

`filamentTexture = Texture.Builder()
.sampler(Texture.Sampler.SAMPLER_2D)
.format(Texture.InternalFormat.RGB8)
.levels(1)
.build(filamentEngine)

val sampler = TextureSampler(
TextureSampler.MinFilter.LINEAR_MIPMAP_LINEAR,
TextureSampler.MagFilter.LINEAR,
TextureSampler.WrapMode.REPEAT
)
filamentMaterial.setParameter("videoTexture", filamentTexture, sampler)

...

val bitmap = getRGBBitmap(yuvBuffer, context)

val byteBuffer: ByteBuffer = ByteBuffer.allocate(bitmap.byteCount)
bitmap.copyPixelsToBuffer(byteBuffer)
byteBuffer.rewind()

filamentTexture!!.setImage(
filamentEngine,
0,
PixelBufferDescriptor(
byteBuffer, Texture.Format.RGB, Texture.Type.UBYTE, 1, 0, 0, 0, null
) {
byteBuffer.clear()
}
)`

[suzp1984]

First of All, the Android's way to handle the camera images is through the Surface or SurfaceTexture, so I think you need to refactor the external module to provide the Camera's output as Surface or SurfaceTexture instead of raw image buffer, otherwise there are no high speed method to process these image.

you already knows the Stream API in filament, so I guess you already knows the sample-hello-camera sample project.

filament/android/samples/sample-hello-camera/src/main/java/com/google/android/filament/hellocam/CameraHelper.kt

Lines 45 to 49 in 2763931

	/**
	* Toy class that handles all interaction with the Android camera2 API.
	* Sets the "textureTransform" and "videoTexture" parameters on the given Filament material.
	*/
	class CameraHelper(val activity: Activity, private val filamentEngine: Engine, private val filamentMaterial: MaterialInstance) {

I will refer this project, and Steam.java and Texture.java API of the filament.
Maybe use the output of camera's SurfaceTexture to construct a Stream.

filament/android/filament-android/src/main/java/com/google/android/filament/Stream.java

Lines 129 to 146 in 2763931

	/**
	* Creates a {@link StreamType#NATIVE NATIVE} stream. Native streams can sample data
	* directly from an opaque platform object such as a {@link android.graphics.SurfaceTexture SurfaceTexture}
	* on Android.
	*
	* @param streamSource an opaque native stream handle, e.g.: on Android this must be a
	* {@link android.graphics.SurfaceTexture SurfaceTexture} object
	* @return This Builder, for chaining calls.
	* @see Texture#setExternalStream
	*/
	@NonNull
	public Builder stream(@NonNull Object streamSource) {
	if (Platform.get().validateStreamSource(streamSource)) {
	nBuilderStreamSource(mNativeBuilder, streamSource);
	return this;
	}
	throw new IllegalArgumentException("Invalid stream source: " + streamSource);
	}

Or least use the CameraHelper's method, let camera's output surface to the ImageReader, then read the image buffer as HeadwareBuffer.

[hanna2100]

While searching for a solution, I found that I could solve it by uploading y, u, and v buffers to three textures separately, without using SurfaceTexture, and then converting them to RGB in the fragment shader.
Thank you.

`private fun setTexture() {
val sampler = TextureSampler(TextureSampler.MinFilter.LINEAR, TextureSampler.MagFilter.LINEAR, TextureSampler.WrapMode.CLAMP_TO_EDGE)

    filamentTexture1 = Texture.Builder()
        .sampler(Texture.Sampler.SAMPLER_2D)
        .format(Texture.InternalFormat.RGBA8)
        .importTexture(textureIds[0].toLong())
        .build(filamentEngine)
    filamentMaterial.setParameter("y_texture", filamentTexture1, sampler)

    filamentTexture2 = Texture.Builder()
        .sampler(Texture.Sampler.SAMPLER_2D)
        .format(Texture.InternalFormat.RGBA8)
        .importTexture(textureIds[1].toLong())
        .build(filamentEngine)
    filamentMaterial.setParameter("u_texture", filamentTexture2, sampler)

...
}`

`private fun setBufferOnTexture(buffers: List, width: Int, height: Int) {
val yBuffer = buffers[0]
val uBuffer = buffers[1]
val vBuffer = buffers[2]

    val textureTarget = GLES30.GL_TEXTURE_2D

    var textureId = textureIds[0]
    GLES20.glActiveTexture(GLES20.GL_TEXTURE0)
    GLES30.glBindTexture(textureTarget, textureId)
    GLES30.glTexImage2D(
        GLES30.GL_TEXTURE_2D,
        0,
        GLES30.GL_LUMINANCE,
        width,
        height,
        0,
        GLES30.GL_LUMINANCE,
        GLES30.GL_UNSIGNED_BYTE,
        yBuffer
    )

...
}`

`
// fragment shader

fragment {
void material(inout MaterialInputs material) {
prepareMaterial(material);
vec2 uv = (materialParams.textureTransform * vec4(variable_uv.xy, 0, 1)).xy;
vec4 texColor1 = texture(materialParams_y_texture, getUV0());
vec4 texColor2 = texture(materialParams_u_texture, getUV0());
vec4 texColor3 = texture(materialParams_v_texture, getUV0());

    float y = texColor1.r;
    float v = texColor2.a;
    float u = texColor3.a;

    y = 1.1643 * (y - 0.0625);
    u = u - 0.5;
    v = v - 0.5;

    float r = y + 1.5958 * v;
    float g = y - 0.39173 * u - 0.81290 * v;
    float b = y + 2.017 * u;

    vec3 rgb = vec3(r, g, b);
    material.baseColor.rgb = inverseTonemapSRGB(rgb);
    material.baseColor.a = 1.f;
}

}`