How to get Color Frame?

[lokeshgs87]

   VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);

    Log.e(TAG, "streaming:  " + videoFrame.getHeight() + " " + videoFrame.getWidth() + " " + videoFrame.getBitsPerPixel() );

    Log.e(TAG, "streaming: before  saving bitmap1 ");

    Mat mDepth = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_16UC3);

    Log.e(TAG, "streaming: before  saving ...... " + mDepth.total() + "  " +  mDepth.elemSize() );

    int size = (int) (mDepth.total() * mDepth.elemSize());
    byte[] return_buff = new byte[size];
    //  depthFrame.getData(return_buff)

    //  byte[] depth_buff = new byte[depthFrame.getHeight() * depthFrame.getStride()];
    videoFrame.getData(return_buff);
    short[] shorts = new short[size / 2];
    ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);

    Log.e(TAG, "streaming: before  saving bitmap2.1 ");
    mDepth.put(0, 0, shorts);
    Log.e(TAG, "streaming: before  saving bitmap3 ");



    Mat mDepth8UC1 = new Mat(mDepth.height(), mDepth.width(), CV_8UC3);
    mDepth.convertTo(mDepth8UC1, CV_8UC3, 1 / 255.0);


    Log.e(TAG, "streaming:  " + mDepth8UC1.cols() + " " +  mDepth8UC1.rows() + " " + Bitmap.Config.ARGB_8888 );


    Bitmap bmpDisplay = Bitmap.createBitmap(mDepth8UC1.cols(), mDepth8UC1.rows(), Bitmap.Config.ARGB_8888);
    Utils.matToBitmap(mDepth8UC1, bmpDisplay);
    String uniqueId = UUID.randomUUID().toString() + ".png";
    String path = saveToInternalStorage(bmpDisplay,"color.png");
    Log.e(TAG, "streaming: before  saving bitmap5 " + path);

The above is the code written to get the RGB image(Color Image), we are unable to get the single image, but in the single image we are getting the same face image twice. Please find attached image for reference. Could you please help us to fix this issue?

[kafan1986]

What is the format of the colour data in the config - RGB8 or BGR8 or YUYV?

[kafan1986]

The problem you are facing is not related to the librealsense but rather OpenCV. You should read about how image data is stored, specially in OpenCV like different types of Mat like 8UC3 or 16UC1 etc.

And lastly the code you have pasted, at least rename the variables correctly. The variable name suggests something and the underlying data type is something different altogether. Anyways here is the code you are looking for, assuming you have started the stream with RGB8:

VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);
Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_8UC3);
byte[] colour_buff = new byte[videoFrame.getHeight() * videoFrame.getStride()];
videoFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"color.png");

[lokeshgs87]

try (Config config = new Config()) {
config.enableDeviceFromFile(filePath);
config.enableStream(StreamType.DEPTH, 640, 480,StreamFormat.Z16);
config.enableStream(StreamType.COLOR, 640, 480, StreamFormat.BGR8);
config.enableStream(StreamType.INFRARED, 640, 480,StreamFormat.Y8);
}

What is the format of the colour data in the config - RGB8 or BGR8 or YUYV?
The follwing is the config we are using
try (Config config = new Config()) {
config.enableDeviceFromFile(filePath);
config.enableStream(StreamType.DEPTH, 640, 480,StreamFormat.Z16);
config.enableStream(StreamType.COLOR, 640, 480, StreamFormat.BGR8);
config.enableStream(StreamType.INFRARED, 640, 480,StreamFormat.Y8);
}

[kafan1986]

try (Config config = new Config()) {
config.enableDeviceFromFile(filePath);
config.enableStream(StreamType.DEPTH, 640, 480,StreamFormat.Z16);
config.enableStream(StreamType.COLOR, 640, 480, StreamFormat.BGR8);
config.enableStream(StreamType.INFRARED, 640, 480,StreamFormat.Y8);
}

What is the format of the colour data in the config - RGB8 or BGR8 or YUYV?
The follwing is the config we are using
try (Config config = new Config()) {
config.enableDeviceFromFile(filePath);
config.enableStream(StreamType.DEPTH, 640, 480,StreamFormat.Z16);
config.enableStream(StreamType.COLOR, 640, 480, StreamFormat.BGR8);
config.enableStream(StreamType.INFRARED, 640, 480,StreamFormat.Y8);
}

Stream it with RGB8 it will not require OpenCV conversion else the image will have inverted B and R channel. Look at the code pasted above.

[lokeshgs87]

The problem you are facing is not related to the librealsense but rather OpenCV. You should read about how image data is stored, specially in OpenCV like different types of Mat like 8UC3 or 16UC1 etc.

And lastly the code you have pasted, at least rename the variables correctly. The variable name suggests something and the underlying data type is something different altogether. Anyways here is the code you are looking for, assuming you have started the stream with RGB8:

VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);
Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_16UC3);
byte[] colour_buff = new byte[videoFrame.getHeight() * videoFrame.getStride()];
videoFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"color.png");

Thanks Kafan, we are getting the following error with the above code

"Mat data type is not compatible: 18"

[kafan1986]

The problem you are facing is not related to the librealsense but rather OpenCV. You should read about how image data is stored, specially in OpenCV like different types of Mat like 8UC3 or 16UC1 etc.
And lastly the code you have pasted, at least rename the variables correctly. The variable name suggests something and the underlying data type is something different altogether. Anyways here is the code you are looking for, assuming you have started the stream with RGB8:

VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);
Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_16UC3);
byte[] colour_buff = new byte[videoFrame.getHeight() * videoFrame.getStride()];
videoFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"color.png");

Thanks Kafan, we are getting the following error with the above code

"Mat data type is not compatible: 18"

Made changes to the original code. Should work now.

[lokeshgs87]

The problem you are facing is not related to the librealsense but rather OpenCV. You should read about how image data is stored, specially in OpenCV like different types of Mat like 8UC3 or 16UC1 etc.
And lastly the code you have pasted, at least rename the variables correctly. The variable name suggests something and the underlying data type is something different altogether. Anyways here is the code you are looking for, assuming you have started the stream with RGB8:

VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);
Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_16UC3);
byte[] colour_buff = new byte[videoFrame.getHeight() * videoFrame.getStride()];
videoFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"color.png");

Thanks Kafan, we are getting the following error with the above code
"Mat data type is not compatible: 18"

Made changes to the original code. Should work now.

Thanks a lot, kafan. It is working now.

Does the same code work for InfraRed Frame? or any changes to be made?.

[kafan1986]

The problem you are facing is not related to the librealsense but rather OpenCV. You should read about how image data is stored, specially in OpenCV like different types of Mat like 8UC3 or 16UC1 etc.
And lastly the code you have pasted, at least rename the variables correctly. The variable name suggests something and the underlying data type is something different altogether. Anyways here is the code you are looking for, assuming you have started the stream with RGB8:

VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);
Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_16UC3);
byte[] colour_buff = new byte[videoFrame.getHeight() * videoFrame.getStride()];
videoFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"color.png");

Thanks Kafan, we are getting the following error with the above code
"Mat data type is not compatible: 18"

Made changes to the original code. Should work now.

Thanks a lot, kafan. It is working now.

Does the same code work for InfraRed Frame? or any changes to be made?.

I have never worked with infrared stream but if it is Y8 then it means grayscale data that will be 8UC1, thus with minor changes it should work. Just change the following line to below

Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_8UC1);

[lokeshgs87]

The problem you are facing is not related to the librealsense but rather OpenCV. You should read about how image data is stored, specially in OpenCV like different types of Mat like 8UC3 or 16UC1 etc.
And lastly the code you have pasted, at least rename the variables correctly. The variable name suggests something and the underlying data type is something different altogether. Anyways here is the code you are looking for, assuming you have started the stream with RGB8:

VideoFrame videoFrame = frame.as(Extension.VIDEO_FRAME);
Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_16UC3);
byte[] colour_buff = new byte[videoFrame.getHeight() * videoFrame.getStride()];
videoFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"color.png");

Thanks Kafan, we are getting the following error with the above code
"Mat data type is not compatible: 18"

Made changes to the original code. Should work now.

Thanks a lot, kafan. It is working now.
Does the same code work for InfraRed Frame? or any changes to be made?.

I have never worked with infrared stream but if it is Y8 then it means grayscale data that will be 8UC1, thus with minor changes it should work. Just change the following line to below

Mat mColour = new Mat(videoFrame.getHeight(), videoFrame.getWidth(), CV_8UC1);

Thanks Kafan, it is working for infrared.

We have an issue with the depth image. Please find attached the depth image. Could you please help us with it.

The following is the code for generating the depth image

DepthFrame depthFrame = frame.as(Extension.DEPTH_FRAME);
Mat mColour = new Mat(depthFrame.getHeight(), depthFrame.getWidth(), CV_8UC1);
byte[] colour_buff = new byte[depthFrame.getHeight() * depthFrame.getStride()];
depthFrame.getData(colour_buff);
mColour.put(0, 0, colour_buff);
Bitmap bmpDisplay = Bitmap.createBitmap(mColour.cols(), mColour.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mColour, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"depth.png");

[kafan1986]

Somehow I never get the depth map converted to Mat directly using OpenCV. I had to use NDK and call the C++ code to convert it, which is faster, but if speed is not an issue, below code will work but takes few ms more, at least it is pure java. Also do search previous issues first. This was already answered in previous thread #5446

DepthFrame depthFrame = frame.as(Extension.DEPTH_FRAME);
val depthMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1);
val size = (depthMap.total() * depthMap.elemSize()).toInt();
val return_buff = ByteArray(size);
depthFrame.getData(return_buff);
val shorts = ShortArray(size / 2);
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);
depthMap.put(0, 0, shorts);
Mat mDepth8UC1 = new Mat(depthFrame.rows(),depthFrame.cols(),CV_8UC1);
depthMap.convertTo(mDepth8UC1,CV_8UC1,1/255.0);
Bitmap bmpDisplay = Bitmap.createBitmap(mDepth8UC1.cols(), mDepth8UC1.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mDepth8UC1, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"depth.png");

[lokeshgs87]

Somehow I never get the depth map converted to Mat directly using OpenCV. I had to use NDK and call the C++ code to convert it, which is faster, but if speed is not an issue, below code will work but takes few ms more, at least it is pure java. Also do search previous issues first. This was already answered in previous thread #5446

DepthFrame depthFrame = frame.as(Extension.DEPTH_FRAME);
val depthMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1);
val size = (depthMap.total() * depthMap.elemSize()).toInt();
val return_buff = ByteArray(size);
depthFrame.getData(return_buff);
val shorts = ShortArray(size / 2);
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);
depthMap.put(0, 0, shorts);
Mat mDepth8UC1 = new Mat(depthFrame.rows(),depthFrame.cols(),CV_8UC1);
depthMap.convertTo(mDepth8UC1,CV_8UC1,1/255.0);
Bitmap bmpDisplay = Bitmap.createBitmap(mDepth8UC1.cols(), mDepth8UC1.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mDepth8UC1, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"depth.png");

Thanks kafan. Can you please give us any sample to get depth image using c++.

[kafan1986]

Mat(Size(w, h), CV_8UC3, (void *) f.get_data(), Mat::AUTO_STEP);

[lokeshgs87]

Somehow I never get the depth map converted to Mat directly using OpenCV. I had to use NDK and call the C++ code to convert it, which is faster, but if speed is not an issue, below code will work but takes few ms more, at least it is pure java. Also do search previous issues first. This was already answered in previous thread #5446
DepthFrame depthFrame = frame.as(Extension.DEPTH_FRAME);
val depthMap = Mat(depthFrame.height, depthFrame.width, CV_16UC1);
val size = (depthMap.total() * depthMap.elemSize()).toInt();
val return_buff = ByteArray(size);
depthFrame.getData(return_buff);
val shorts = ShortArray(size / 2);
ByteBuffer.wrap(return_buff).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(shorts);
depthMap.put(0, 0, shorts);
Mat mDepth8UC1 = new Mat(depthFrame.rows(),depthFrame.cols(),CV_8UC1);
depthMap.convertTo(mDepth8UC1,CV_8UC1,1/255.0);
Bitmap bmpDisplay = Bitmap.createBitmap(mDepth8UC1.cols(), mDepth8UC1.rows(), Bitmap.Config.ARGB_8888);
Utils.matToBitmap(mDepth8UC1, bmpDisplay);
String uniqueId = UUID.randomUUID().toString() + ".png";
String path = saveToInternalStorage(bmpDisplay,"depth.png");

Thanks kafan. Can you please give us any sample to get depth image using c++.

Mat(Size(w, h), CV_8UC3, (void *) f.get_data(), Mat::AUTO_STEP);

Thanks Kafan.

We are able to get the depth, color and infrared image from frames. Please find attached images.
But we are facing issues with a color image, we are not getting the full image(getting depth image with the black background of the same part) as we got with infrared image.
The following is the alignment we are applying "mAlign = new Align(StreamType.DEPTH);

"

[kafan1986]

That is expected behaviour. In the Intel-RealSense-D400-Series-Datasheet pdf, you can go to section 4.5 about "Invalid Depth Band", to find out the reason.

In short, the depth is calculated with respect to the left stereo camera but due to lateral shift of the right camera w.r.t to left camera, there is a blind spot that happens on the extreme left edge of the left camera, where right camera can not gather the image data. Thus stereo matching can not happen.

[agrunnet]

Also I think you are using the d435 camera. The color Camera has a different field of view. 65hfov instead of 90hfov of the depth camera.

[RealSenseCustomerSupport]

---

@lokeshgs87,

Has the comment provided by @kafan1986 helped to find an answer to your question regarding depth data at the left edge of the frame? Please let us know if you need any further clarifications regarding this specific topic.

@kafan1986,

Thank you for the informative answers!

[lokeshgs87]

@RealSenseCustomerSupport All the issues were solved with the help of @kafan1986. Really with the help of @kafan1986 we were able to complete our programming so fast.

Thanks @kafan1986 for your support.

We will reach out @kafan1986 if we need any help furthur.

[neilyoung]

May I ask a non-related question? This code looks like Java for Android. In order to run IntelRealsense on Android is there still a rooted device necessary or is this obsolete meanwhile? Asking this because I didn’t had success with an ordinary device up to now. TIA

[kafan1986]

@neilyoung The android SDK will work on non-rooted device. I think they introduced non-rooted support sometime early last year, probably around 2.19 version. So current 2.31 works fine with some known issues with stability etc. Also, I think android SDK is currently having problem with android 10 and intel team is working with android team for the same. Compatibility looks OK for android 7, 8 and 9

[neilyoung]

@kafan1986 Very nice. Thanks for clarification.