Align depth to color

[nawara72]

Hi

I have some depth and color images (both in png format) saved from D435. I would like to align the depth image to the color image. I have all the intrinsics, extrinsic parameters. I studied the rs_align example, but couldn't figure out how to pass in the png images to align them.

I also studied the software_device example and was wondering is there a way to pass a depth png format to the custom_frame_source as a depth frame (I don't have the raw data), cause then that will simplify things for me and i can use rs_align way easily.

Would really appreciate your help

[dorodnic]

Hi @nawara72
I think it should be possible, and you are on the right track with software device example. We are using stbi_load_from_memory to load PNG from memory for the color frame. You can use something similar to load frames from save files. Then, instead of calculating point-cloud, just run alignment instead.
The only concern is depth file format inside the PNG file. For everything to work, you need to be able to recover original depth data.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
You can also see #2231.

[nawara72]

@dorodnic
@RealSense-Customer-Engineering
Thanks so much for your reply. I have tried reading the depth file (png/raw) using cv::imread and that all works. However if I try to pass it as a depth_frame it doesnt work. Not sure how once I have read the depth image to pass it on to the synthetic sensor as a stream. The reading of the color png file works fine and no issue with passing it on to the synthetic sensor. Any help would be greatly appreciated.
cv::Mat input = cv::imread(filename, CV_LOAD_IMAGE_ANYCOLOR | CV_LOAD_IMAGE_ANYDEPTH);
std::cout<<"Try 1 "<< input.size <<std::endl;
depth_frame.frame.assign((uint16_t*)input.datastart, (uint16_t*)input.dataend);
std::cout<<"Try 2 "<< depth_frame.x <<" "<< depth_frame.y <<" "<< depth_frame.bpp <<std::endl;
FILE* pdFile = fopen(filename.c_str(), "rb");
auto realsense_depth= stbi_load_from_file(pdFile, &depth_frame.x, &depth_frame.y, &depth_frame.bpp, false);
std::cout<<"Try 3 "<< depth_frame.x <<" "<< depth_frame.y <<" "<< depth_frame.bpp <<std::endl;
Outputs
Try 1 720 x 1280 ---- that's correct
Try 2 240 0 30 ---- that's wrong
Try 3 240 0 30 ---- that's wrong

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @nawara72,

You can try to use the code like below to read the depth raw file.

fp = fopen(fileName, "rb");
imagedata = (char )malloc(sizeof(short) * framesize);
fread(imagedata, sizeof(short), framesize, fp);
cv::Mat depthMat(Size(IMAGE_WIDTH, IMAGE_HEIGHT), CV_16U, (void*)imagedata, cv::Mat::AUTO_STEP);

[nawara72]

Hi

I have no trouble reading a png depth image into a cv::Mat. The issue is how can I pass that depth cv::Mat to the synthetic sensor's depth frame similiar to the rs_softawre-device example?

I would also appreciate any info u can give me on the raw format. I only have png depth images which I convert to raw but I am guessing the raw format is different in realsense.

Thanks again for your help, really appreciate it

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @nawara72,

The ".RAW" data layout is according to the UVC spec for the depth stream.
https://linuxtv.org/downloads/v4l-dvb-apis/uapi/v4l/pixfmt-z16.html
So far, I have no good example code to pass cv::Mat to frame. You can have a try based on "rs_softawre-device" as you mentioned.

[RealSense-Customer-Engineering]

[Realsense Customer Engineering Team Comment]
Hi @nawara72,

Still need support for this topic?

[nawara72]

@RealSense-Customer-Engineering
I hve sorted it out, thanks