Convert depth frame to Bitmap and save .png (Android)

[vshalkov]

Required Info
Camera Model	D435i
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	5.2
Platform	Android
SDK Version	RealSense SDK for Android OS
Language	C, opencv, kotlin, java
Segment	Smartphone

Issue Description

Hello, I need your help.
I use RealSense SDK for Android OS https://github.com/IntelRealSense/librealsense/tree/master/wrappers/android

init

Config().use { config ->
            config.enableStream(StreamType.DEPTH, 640, 480)
            config.enableStream(StreamType.COLOR, 640, 480)
            pipeline.start(config)
        }

I can convert color frame to Mat and Mat convert to Bitmap and Bitmap save to .png
Frame (this code is work):

FrameReleaser().use { fr ->
    val frames = pipeline.waitForFrames(1000).releaseWith(fr)
    
   val colour = frames.first(StreamType.COLOR).releaseWith(fr)
    val colourFrame: VideoFrame = colour.`as`(Extension.VIDEO_FRAME)
    colourFrame.releaseWith(fr)
     
    // I found the code below here:
    // https://github.com/IntelRealSense/librealsense/issues/5253
    val mRGB = Mat(colourFrame.height, colourFrame.width, CV_8UC3)
    val return_buff = ByteArray(colourFrame.dataSize)
    colourFrame.getData(return_buff)
    mRGB.put(0, 0, return_buff)

    val bmp = Bitmap.createBitmap(mRGB.cols(), mRGB.rows(), Bitmap.Config.ARGB_8888)
    Utils.matToBitmap(mRGB, bmp)
       
     val file1 = File("/sdcard/color.png")
     try {
          val out = FileOutputStream(file1)
          bmp.compress(Bitmap.CompressFormat.JPEG, 100, out)
          out.flush()
          out.close()
      } catch (e: Exception) {
          e.printStackTrace()
      }
  upload(frames)

But if i want to get this image from frame, it's not work:

I use this code for get depth Mat in FrameReleaser() :

val depth = frames.first(StreamType.DEPTH).releaseWith(fr)
val depthFrame: DepthFrame = depth.`as`(Extension.DEPTH_FRAME)
depthFrame.releaseWith(fr)

// I found the code below here:
// https://github.com/IntelRealSense/librealsense/issues/5185

val testMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1)
val size = (testMap.total() * testMap.elemSize()).toInt()
val return_buff = ByteArray(size)
depthFrame.getData(return_buff)
val shorts = ShortArray(size / 2)
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts)
testMap.put(0, 0, shorts)

And when i want to convert Mat to Bitmap, android app freezes and don't work:
Code convert:

val bmp = Bitmap.createBitmap(testMap.cols(), testMap.rows(), Bitmap.Config.ARGB_8888)
Utils.matToBitmap(testMap, bmp) // <---------- this app freezes and don't work
val file1 = File("/sdcard/depth.png")
try {
    val out = FileOutputStream(file1)
    bmp.compress(Bitmap.CompressFormat.JPEG, 100, out)
    out.flush()
    out.close()
} catch (e: Exception) {
    e.printStackTrace()
}

Please help me.
How i can to save depth frame to .png?

[kafan1986]

@vshalkov You are facing the issue because you are trying to convert a 16UC1 mat to bitmap directly. In this case you need to convert it to 8UC1 mat first then to convert to bitmap using your code. But this will give you a gray image (black and white). To achieve what you have asked, either then you can use applyColorMap yourself to get the visualization that you desire (there are multiple colourmap in opencv) https://www.learnopencv.com/applycolormap-for-pseudocoloring-in-opencv-c-python/

Or a easier approach is to use a post processing filter rs2::colorizer on your depth data which achieves the same thing with "Jet" scheme colour visualization and converts the 16UC1 depth data to 8UC3 RGB data, then you can use the same code that you use for colorFrame to save the image.

[vshalkov]

@kafan1986
Thank you!
I want to get a gray image (black and white):

Do not tell me how to correctly convert 16UC1 to 8UC1?

I do not know how to initialize a variable (mRGB) correctly that the conversion was successful:

val mRGB = Mat(depthFrame.height, depthFrame.width, CV_8UC3)

OR:

val mRGB = Mat(testMap.cols(), testMap.rows(), CV_8UC3)

or something else?

Convert code:

val testMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1)
val size = (testMap.total() * testMap.elemSize()).toInt()
val return_buff = ByteArray(size)
depthFrame.getData(return_buff)
val shorts = ShortArray(size / 2)
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts)
testMap.put(0, 0, shorts)

val mRGB = Mat(depthFrame.height, depthFrame.width, CV_8UC3)
testMap.convertTo(mRGB, CV_8UC1)

val bmp = Bitmap.createBitmap(mRGB.cols(), mRGB.rows(), Bitmap.Config.ARGB_8888)
Utils.matToBitmap(mRGB, bmp)
       
val file1 = File("/sdcard/depth.png")
try {
      val out = FileOutputStream(file1)
      bmp.compress(Bitmap.CompressFormat.JPEG, 100, out)
      out.flush()
      out.close()
  } catch (e: Exception) {
      e.printStackTrace()
  }

Now I can’t check the code that I wrote, but I would be grateful if you had thrown off the working code or link

[kafan1986]

@kafan1986
Thank you!
I want to get a gray image (black and white):

Do not tell me how to correctly convert 16UC1 to 8UC1?

I do not know how to initialize a variable (mRGB) correctly that the conversion was successful:

val mRGB = Mat(depthFrame.height, depthFrame.width, CV_8UC3)

OR:

val mRGB = Mat(testMap.cols(), testMap.rows(), CV_8UC3)

or something else?

Convert code:

val testMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1)
val size = (testMap.total() * testMap.elemSize()).toInt()
val return_buff = ByteArray(size)
depthFrame.getData(return_buff)
val shorts = ShortArray(size / 2)
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts)
testMap.put(0, 0, shorts)

val mRGB = Mat(depthFrame.height, depthFrame.width, CV_8UC3)
testMap.convertTo(mRGB, CV_8UC1)

val bmp = Bitmap.createBitmap(mRGB.cols(), mRGB.rows(), Bitmap.Config.ARGB_8888)
Utils.matToBitmap(mRGB, bmp)
       
val file1 = File("/sdcard/depth.png")
try {
      val out = FileOutputStream(file1)
      bmp.compress(Bitmap.CompressFormat.JPEG, 100, out)
      out.flush()
      out.close()
  } catch (e: Exception) {
      e.printStackTrace()
  }

Now I can’t check the code that I wrote, but I would be grateful if you had thrown off the working code or link

I have not tested the code below and there may be typos. Let me know if this works.

// Convert Depth Data to OpenCV Mat 16UC1. This approach must be much faster than the code shared by you that I posted earlier.
Mat mDepth = new Mat(depthFrame.getHeight(), depthFrame.getWidth(), CV_16UC1);
byte[] depth_buff = new byte[depthFrame.getHeight() * depthFrame.getStride()];
depthFrame.getData(depth_buff);
mDepth.put(0, 0, depth_buff);

//Now mDepth mat is having the data
//Converting the 16UC1 to 8UC1
Mat mDepth8UC1 = new Mat(mDepth.rows(),mDepth.cols(),CV_8UC1);
mDepth.convertTo(mDepth8UC1,CV_8UC1,1/255.0);

//Converting grayscale mat to bitmap
Bitmap bmpDisplay = Bitmap.createBitmap(mDepth8UC1.cols(), mDepth8UC1.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mDepth8UC1, bmpDisplay);

// Now save this "bmpDisplay" bitmap with your existing code

[vshalkov]

@kafan1986 Hello!
This code don't work (

val mDepth = Mat(depthFrame.height, depthFrame.width, CV_16UC1)
val depth_buff = ByteArray(depthFrame.height * depthFrame.stride)
depthFrame.getData(depth_buff)
mDepth.put(0, 0, depth_buff)

the program freezes when this code starts to run:

mDepth.put(0, 0, depth_buff)

[kafan1986]

@kafan1986 Hello!
This code don't work (

val mDepth = Mat(depthFrame.height, depthFrame.width, CV_16UC1)
val depth_buff = ByteArray(depthFrame.height * depthFrame.stride)
depthFrame.getData(depth_buff)
mDepth.put(0, 0, depth_buff)

the program freezes when this code starts to run:

mDepth.put(0, 0, depth_buff)

then stick with your existing code, for this part.

val testMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1)
val size = (testMap.total() * testMap.elemSize()).toInt()
val return_buff = ByteArray(size)
depthFrame.getData(return_buff)
val shorts = ShortArray(size / 2)
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts)
testMap.put(0, 0, shorts)

[vshalkov]

@kafan1986 Thank you, yes it's work!
I get this image:

I asked one more question, maybe you know the answer?
#5453