Real-time stream

[DuyguSerbes]

Required Info
Camera Model	D435i
Firmware Version	Open RealSense Viewer
Operating System & Version	Ubuntu 18.04
Kernel Version (Linux Only)	5.4.0-72-generic
Platform	PC
SDK Version	l2.0
Language	Python
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

Hello all,

I am looking for a real-time stream solution for realsense D435i camera. I would like to use real-time infrared and RGB stream. Can you suggest any repository written in C/C++ or Python?

Also I want to save the stream in mp4 format for other processes.

[MartyG-RealSense]

Hi @DuyguSerbes I am not aware of an example of streams being saved in mp4 format, though a RealSense user shared a Python script for saving the streams in .avi video format.

#2731 (comment)

If you require mp4 format, building librealsense with the V4L2 backend should enable raw camera data to be accessed by standard Linux tools.

#6841 (comment)

In regard to real-time streaming: the link below provides scripting for aligning depth, color and infrared. Depth is necessary to be included in that situation because the SDK needs for depth to be aligned to another type of stream, so you cannot have just RGB and infrared alignment, for example.

#5093

If you prefer for the streams to be displayed as individual panels instead of in alignment, a RealSense user in the link below shared a modified version of the opencv_viewer_example Python example program that includes infrared.

#3363

Conceivably, if you only wish RGB and infrared to be displayed then you could remove the depth stream code from the script, since alignment is not involved in this particular application.

[sam598]

@DuyguSerbes if you prefer python another alternative is ffmpeg-python https://github.com/kkroening/ffmpeg-python

There is no repository with RealSense code, but the library is well documented. With these settings I am usually able to get a 720p 30fps stream going on a Raspberry Pi 4. This also uses the Raspberry Pi's hardware encoder, which is much more efficient than OpenCV. If you have an Nvidia GPU you may be able to use Nvenc for hardware accelerated encoding.

process = (
    ffmpeg
    .input('pipe:', format='rawvideo', pix_fmt='yuv420p', s=str(width) + 'x' + str(height))
    .output('pipe:', format='h264', vcodec='h264_omx', framerate=framerate, video_bitrate='10M')
    .global_args('-an')
    .run_async(pipe_stdin=True, pipe_stdout=True)
)

You can use OpenCV to convert an RGB8 color frame to yuv420p. Unfortunately RealSense's YUYV and yuv420p are different formats.

You can feed the color frame to ffmpeg with process.stdin.write(coloryuv).

You can have ffmpeg write straight to disk, but if you are writing to an external USB drive I have had issues where ffmpeg blocks the USB bus and the RealSense camera drops frames. If you pull the encoded bytes yourself with something like data = process.stdout.read(16384), then you can write that data to disk or stream it somewhere.

This will technically be "h264" and not "mp4", but you can transcode the stream without recompressing it.

(
	ffmpeg
	.input(filePath, framerate=int(framerate))
	.output(filePath.replace(".h264", ".mp4"), c='copy')
	.global_args('-loglevel', 'error')
	.global_args('-y')
	.run()
)

[sam598]

Also if you want more detail than H264 and can deal with a higher data rate, motion JPEG might be interesting.

This library https://pypi.org/project/PyTurboJPEG/ takes about 10ms to compress a 848x480 RGB image and 20ms to compress a 1280x720 RGB image on a Raspberry Pi. A desktop computer should be able to do much better.

[MartyG-RealSense]

Hi @DuyguSerbes Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.