Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Question of infared image #11854

Closed
huyuaaaray opened this issue May 30, 2023 · 25 comments
Closed

Question of infared image #11854

huyuaaaray opened this issue May 30, 2023 · 25 comments
Labels
d455 python Windows

Comments

@huyuaaaray
Copy link

  • Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

Required Info
Camera Model { D455 }
Firmware Version (Open RealSense Viewer --> Click info)
Operating System & Version {Win (10)
Kernel Version (Linux Only) (e.g. 4.14.13)
Platform PC
SDK Version { legacy / 2.. }
Language {python}
Segment {others }

Issue Description

Hi, i am trying to obtain the ir image using SDK with intel D455, and it return the error:
File "c:\Users\yu.hu\ultralytics\capture_frame.py", line 66, in capture_frames
infrared_image = np.asanyarray(infrared_frame.get_data())
RuntimeError: null pointer passed for argument "frame_ref"

is that because this camera having no infrared camera, or i just using a wrong function in the SDK.

Thanks,
Yu Hu
@MartyG-RealSense
Copy link
Collaborator

Hi @huyuaaaray The D455 camera has a pair of left and right infrared sensors (Infrared1 and Infrared2).

There is a Python IR script at #860 (comment) that uses np.asanyarray(infrared_frame.get_data()) that you can compare with your own script.

@huyuaaaray
Copy link
Author

huyuaaaray commented May 30, 2023 via email

@huyuaaaray
Copy link
Author

Hi,

what is the difference if i save both the RGB and IF frames as .png? should i output the IF frames as .png format? Do they use the same camera to obtain the data? Is there any essentially differences?

Sincerely,
Yu Hu

@MartyG-RealSense
Copy link
Collaborator

The D455 has a pair of infrared sensors and a separate RGB sensor. Color comes from the RGB sensor.

D455 is also able to output RGB from the left infrared sensor instead of infrared, though the image is not colored the same as the one from the RGB sensor.

The infrared image will be black & white monochrome.

It is fine to output both color and infrared as png.

@huyuaaaray
Copy link
Author

huyuaaaray commented May 30, 2023 via email

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 30, 2023 •
edited
Loading

No, the depth image is constructed from frames from the left and right IR sensor. The central large one is the IR projector that casts a dot pattern onto objects in the scene. The IR projector is a separate component from the left and right IR sensors. The dot pattern that it casts helps the camera to analyse surfaces for depth information.

@huyuaaaray
Copy link
Author

huyuaaaray commented May 31, 2023 via email

@MartyG-RealSense
Copy link
Collaborator

If you are using Python then a point cloud will typically be made from an alignment between depth (generated from the left and right IR sensors) and color (provided by the RGB camera). This merged point cloud can be referred to as a textured point cloud.

The left and right IR images that the depth frame is generated from are raw IR frames in the camera hardware and not the Infrared1 and Infrared2 streams. This is why the depth stream can be enabled even when the Infrared1 and Infrared2 streams are not enabled.

@huyuaaaray
Copy link
Author

So the infrared 1 and 2 are generated by left and right infrared sensors? And the infrared 1 generates the NIR and the infrared 2 generates the FIR, am i right? what is the difference between infrared frames and depth frames? I really appreciated your answers.
Sincerely,
Yu Hu

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 31, 2023 •
edited
Loading

Yes, Infrared1 and Infrared2 are from the left and right IR sensors respectively. Both operate on the same infrared frequency range up to near-infrared.

Intel's camera tuning guide says in its section about external projectors linked to below: "It is possible to use light in visible or near-infrared (ranging from ~400-1000 nm). Example: 850 nm is invisible to the human eye whilst still being visible to the IR sensor".

https://dev.intelrealsense.com/docs/tuning-depth-cameras-for-best-performance#use-external-projector

Infrared frames resemble a black and white RGB image, with dots overlaid on the image if the projector is enabled. It does not provide distance values.

Depth frames resemble an infrared temperature image, with the distance of each coordinate color-shaded (for example, colored blue nearest to the camera and colored red furthest from the camera). The depth frame can provide distance values from the camera to a particular coordinate on the image, or between one coordinate and another.

image

@huyuaaaray
Copy link
Author

So, when i config the script: 'config.enable_stream(rs.stream.infrared, 2, width, height, rs.format.y8, 30)', it doesn't mean I am using the FIR, It just means i am using the right ir sensor?
when I don't specify the index like: config.enable_stream(rs.stream.infrared,width, height, rs.format.y8, 30). Do the frames it returns represent the left ir sensor captured?

@huyuaaaray
Copy link
Author

huyuaaaray commented Jun 1, 2023 •
edited
Loading

I found out that it seems it will return the left frames if i don't specify the index as i mentioned above.
I am wondering is there any ways that combine the infrared 1 and infrared 2 to 1 image?
Sincerely,
Yu Hu

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Jun 1, 2023 •
edited
Loading

Infrared index number 2 uses the right-side infrared sensor (Infrared2), yes. It is only available when the camera is on a USB 3 connection and is not supported on USB 2.1.

Yes, if an index number is not specified then the left-side infrared sensor (Infrared1) is used by default.

When performing alignment with the align_to instruction, one of the two stream types should be depth, so left-right infrared alignment is not supported. The two infrared images would be almost the same anyway, with one of them slightly offset horizontally from the other due to the positions of the sensors on the front of the camera.

@huyuaaaray
Copy link
Author

Good morning,
I appreciate your answer, I hope I will not ask too much questions. But which infrared sensor will be used if I enable the depth stream.
And what are the stream indexes of each streams represent, for example, 'config.enable_stream(rs.stream.depth, 1, width, height, rs.format.y8, 30)’
I cannot find out in the official docs.
Sincerely,
Yu Hu

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Jun 1, 2023 •
edited
Loading

There is not a limit on the number of questions that can be asked, so don't worry. :)

When the depth stream is enabled, the depth image is constructed from frames from both the left and right infrared sensors. But because the frames used for constructing depth are not coming from the infrared streams but from the sensors, you can have depth, Infrared1 and Infrared2 all enabled simultaneously if you wish to. This is because the infrared frames used for depth frame creation are not the same as the ones from the Infrared1 and Infrared2 streams.

It is usually not necessary to use an index number for depth and color streams as they only have one sensor each, not a pair like infrared. Because there are two infrared sensors, the index number is required to tell the script which of the two infrared sensors to access if you are enabling both the left and right infrared.

If you are only enabling the left infrared sensor then you do not need to set an index number.

Index 1 is the left infrared sensor, index 2 is the right infrared sensor.

@huyuaaaray
Copy link
Author

So the camera will use the exactly same sensors, left and right, to construct the depth and infrared frames respectively? In the meanwhile, the depth frames must use both the left and the right sensor?
In addition, the alignment method only can align the rgb with depth, but it cannot align the infrared frames with them?

@MartyG-RealSense
Copy link
Collaborator

Both the left and right IR sensors are used to construct the depth frame.

But when using the left or right infrared stream, only the IR sensor that is being accessed will be used. So for example, if you are only using the left sensor (Infrared1) then the right IR sensor will not be used.

You can use the align_to instruction to align depth with color, or depth with infrared.

#5093 (comment) has a Python script for aligning all three stream types (depth, RGB, Infrared1). It aligns depth and color, and is also able to include Infrared1 in the alignment because the left IR sensor is already perfectly aligned with depth because depth and Infrared1 both originate from the left IR sensor.

@huyuaaaray
Copy link
Author

Thanks, Marty, i am extremely appreciate your detailed answer above : ). And when i enable the colorized depth image:
colorizer = rs.colorizer()
colorized_depth_frame = colorizer.colorize(depth_frame)
and save it as csv files, it looks different with the depth csv. Can you talk more about this format?

@MartyG-RealSense
Copy link
Collaborator

The csv format saves depth in a file representing all of the depth coordinates as an array of text information that can be imported into spreadsheet and database programs.

@huyuaaaray
Copy link
Author

Thanks, Marty, and how to edit this script: https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py
to map the depth and the RGB to a point cloud file?

@MartyG-RealSense
Copy link
Collaborator

Python has a problem with exporting color to a ply file with export_to_ply. The only script that has been known to work is at #6194 (comment)

@huyuaaaray
Copy link
Author

Thanks, is there any methods that i can project the 3d point cloud to 2d?

@MartyG-RealSense
Copy link
Collaborator

A 3D point cloud can be projected back to 2D with the SDK instruction rs2_project_point_to_pixel

https://intelrealsense.github.io/librealsense/doxygen/rsutil_8h.html#ad179d4a5e6140d9ab18ca7bd9a3e96e6

@MartyG-RealSense
Copy link
Collaborator

Hi @huyuaaaray Do you require further assistance with this case, please? Thanks!

@MartyG-RealSense
Copy link
Collaborator

Case closed due to no further comments received.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
d455 python Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@MartyG-RealSense @huyuaaaray