How to get the depth frames from recorded .bag file using pyrealsense2?

[jianjieluo]

---

Required Info
Camera Model	D415
Firmware Version	05.08.15.00
Operating System & Version	Linux (Ubuntu 16.04 LTS)
Kernel Version (Linux Only)	4.13.0-41-generic
Platform	PC
SDK Version	2

Issue Description

I record the original data in .bag file using the realsense-viewer's record button. How can I play back the .bag file and get every depth_frame in it to extract the depth information in my own way, using pyrealsense2?

I have read rs-record-playback.cpp and found that I should set configure first: config.enable_device_from_file("./20180614_105342.bag"),

and then call pipeline.poll_for_frames() rather than pipeline.wait_for_frames().

When I do these above I get such error :

TypeError: poll_for_frames(): incompatible function arguments. The following argument types are supported:
    1. (self: pyrealsense2.pipeline, frameset*: pyrealsense2.composite_frame) -> bool

Then I try

frames = rs.composite_frame()
pipeline.poll_for_frames(frames)

But it still didn't work.

---

In brief, I have succeed in using the stream:

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 1280, 720, rs.format.bgr8, 30)

try:
    while True:

        frames = pipeline.wait_for_frames()

        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()

        # Get Intrinsics & Extrinsics
        depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
        color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
        depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(
            color_frame.profile)

        ############### Extract the depth data in my preferred way below ##############
        ...

And now I want to read .bag file to get the frames rather than get them from the stream. The code I try is something like that:

pipeline = rs.pipeline()
config = rs.config()
config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 1280, 720, rs.format.bgr8, 30)

config.enable_device_from_file("./20180614_105342.bag")

try:
    while True:

        frames = pipeline.poll_for_frames()

        depth_frame = frames.get_depth_frame()
        color_frame = frames.get_color_frame()
        if not depth_frame or not color_frame:
            continue

        # Get Intrinsics & Extrinsics
        depth_intrin = depth_frame.profile.as_video_stream_profile().intrinsics
        color_intrin = color_frame.profile.as_video_stream_profile().intrinsics
        depth_to_color_extrin = depth_frame.profile.get_extrinsics_to(
            color_frame.profile)

        ############### Extract the depth data in my preferred way below ##############
        ...

Are there any example codes or instruction to show how to use pyrealsense2 to get all depth frames and work on them? Thanks a lot.

[TheMikeyR]

The code below takes two input parameter and extract the depth stream images and save them to a directory, you can just add in the color stream and do it the way you do when you have a live realsense camera.

I think what went wrong is that you never start the pipeline using pipeline.start(config)

import argparse
import pyrealsense2 as rs
import numpy as np
import cv2
import os


def main():
    if not os.path.exists(args.directory):
        os.mkdir(args.directory)
    try:
        config = rs.config()
        rs.config.enable_device_from_file(config, args.input)
        pipeline = rs.pipeline()
        config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
        pipeline.start(config)
        i = 0
        while True:
            print("Saving frame:", i)
            frames = pipeline.wait_for_frames()
            depth_frame = frames.get_depth_frame()
            depth_image = np.asanyarray(depth_frame.get_data())
            cv2.imwrite(args.directory + "/" + str(i).zfill(6) + ".png", depth_image)
            i += 1
    finally:
        pass


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-d", "--directory", type=str, help="Path to save the images")
    parser.add_argument("-i", "--input", type=str, help="Bag file to read")
    args = parser.parse_args()

    main()

[jianjieluo]

Oh my God， how careless am I! T_T

Thx @TheMikeyR

[jianjieluo]

By the way, I also want to know the difference between wait_for_frames() and poll_for_frames(). Is there anyone can help?

[dorodnic]

Hi @longjj
wait_for_frames is supposed to be blocking (for live camera that would wait until next frameset is available) while poll_for_frames always returns immediately.

Thanks @TheMikeyR
Would you like to contribute the script into python/examples folder? I'm sure it would help more people

[TheMikeyR]

wait_for_frames() will block the current thread until new frames become available.
from my understanding poll_for_frames() ask if there is a frame ready without blocking the thread, and it will retrieve the latest undelivered frameset.

Documentation for poll_for_frames()

Check if a new set of frames is available and retrieve the latest undelivered set. The frames set includes time-synchronized frames of each enabled stream in the pipeline. The method returns without blocking the calling thread, with status of new frames available or not. If available, it fetches the latest frames set. Device frames, which were produced while the function wasn't called, are dropped.
To avoid frame drops, this method should be called as fast as the device frame rate. The application can maintain the frames handles to defer processing. However, if the application maintains too long history, the device may lack memory resources to produce new frames, and the following calls to this method shall return no new frames, until resources are retained.

Documentation for wait_for_frames()

Wait until a new set of frames becomes available. The frames set includes time-synchronized frames of each enabled stream in the pipeline. In case of different frame rates of the streams, the frames set include a matching frame of the slow stream, which may have been included in previous frames set. The method blocks the calling thread, and fetches the latest unread frames set. Device frames, which were produced while the function wasn't called, are dropped. To avoid frame drops, this method should be called as fast as the device frame rate. The application can maintain the frames handles to defer processing. However, if the application maintains too long history, the device may lack memory resources to produce new frames, and the following call to this method shall fail to retrieve new frames, until resources are retained.

[TheMikeyR]

@dorodnic Sure

[jianjieluo]

Thanks @dorodnic , @TheMikeyR .

So what if I use poll_for_frames() rather than wait_for_frames() in @TheMikeyR 's script above?

I have try the script(only replace wait_for_frames) below:

import argparse
import pyrealsense2 as rs
import numpy as np
import cv2
import os


def main():
    if not os.path.exists(args.directory):
        os.mkdir(args.directory)
    try:
        config = rs.config()
        rs.config.enable_device_from_file(config, args.input)
        pipeline = rs.pipeline()
        config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
        pipeline.start(config)
        i = 0
        while True:
            print("Saving frame:", i)
            frames = rs.composite_frame(rs.frame())
            pipeline.poll_for_frames(frames)
            depth_frame = frames.get_depth_frame()
            depth_image = np.asanyarray(depth_frame.get_data())
            cv2.imwrite(args.directory + "/" + str(i).zfill(6) + ".png", depth_image)
            i += 1
    finally:
        pass


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-d", "--directory", type=str, help="Path to save the images")
    parser.add_argument("-i", "--input", type=str, help="Bag file to read")
    args = parser.parse_args()

    main()

and I get a RuntimeError:

Traceback (most recent call last):
  File "try.py", line 36, in <module>
    main()
  File "try.py", line 23, in main
    depth_image = np.asanyarray(depth_frame.get_data())
RuntimeError: null pointer passed for argument "frame_ref"

Is it possible that poll_for_frames() achieves the same result as wait_for_frames() here?

[TheMikeyR]

I guess you need some logic which waits until a new frame is ready so you can process it and save it. I think your program crash since the frame is not ready yet and you don't have a handler for that. Do it like this:

import argparse
import pyrealsense2 as rs
import numpy as np
import cv2
import os


def main():
    if not os.path.exists(args.directory):
        os.mkdir(args.directory)
    try:
        config = rs.config()
        rs.config.enable_device_from_file(config, args.input)
        pipeline = rs.pipeline()
        config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)
        pipeline.start(config)
        i = 0
        while True:
            print("Saving frame:", i)
            frames = rs.composite_frame(rs.frame())
            # Check if new frame is ready
            if pipeline.poll_for_frames(frames):
                depth_frame = frames.get_depth_frame()
                depth_image = np.asanyarray(depth_frame.get_data())
                cv2.imwrite(args.directory + "/" + str(i).zfill(6) + ".png", depth_image)
                i += 1
            # wait logic until frame is ready
            else:
                print("Waiting for frame to be ready")
    finally:
        pass


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("-d", "--directory", type=str, help="Path to save the images")
    parser.add_argument("-i", "--input", type=str, help="Bag file to read")
    args = parser.parse_args()
    main()

[jianjieluo]

Wow, it's so clear! Thank you so much! @TheMikeyR

[mohamedhassanmus]

Hi,
I want my loop to stop after the last frame in the bag. I want to loop through the whole bag once and save all frames then automatically stop.
The code above loop for ever.
Any ideas? @TheMikeyR @dorodnic

[lramati]

On the enable_device_from_file function, there is a repeat_playback argument that controls looping over the bag.

[mohamedhassanmus]

Thanks a lot @lramati

[amaanda]

When I run @TheMikeyR code using poll_for_frames(), I get the following error:

TypeError: poll_for_frames(): incompatible function arguments. The following argument types are supported:
1. (self: pyrealsense2.pyrealsense2.pipeline) -> pyrealsense2.pyrealsense2.composite_frame

[atanase2]

hello,
I had the same problem. I think to have a solution
pipeline.poll_for_frames(frames) doesn't work at all, even with composite_frame argument

compositFrame = pipeline.poll_for_frames() works like a frames= pipeline.wait_for_frame()
but it return a composite_frame type
be carefull, a poll_for_frame have not to return a frame each time

[ZackZikaiXiao]

I'm using the SDK Intel Realsense Viewer v.2.19.0 to create a 'bag' file.There are three cameras for bags. However,I can't press the Sync button.What can I do to solve this problem?

[Praneet1997]

When I run @TheMikeyR code using poll_for_frames(), I get the following error:

TypeError: poll_for_frames(): incompatible function arguments. The following argument types are supported:

1. (self: pyrealsense2.pyrealsense2.pipeline) -> pyrealsense2.pyrealsense2.composite_frame

@TheMikeyR @dorodnic Any views on this issue. I am also getting the same issue with your code

[lramati]

See the next comment by atanase2. This issue is very old and the API has changed since TheMikeyR posted their code

[atanase2]

be aware that the poll_for_frame function doesn't work with win7. I spend lots of time on this function because of that.

[filipematosinov]

If I understand correctly there is no way to get the frames apart from streaming (in which you can loose frames if your extracting speed is slower than the actual FPS or the recording?

[lbps]

@filipematosinov to read a .bag file without losing frames while script is processing images, you have to set as False a "real_time" parameter using the set_real_time function of the playback. This works for me:

pipeline = rs.pipeline()
config = rs.config()
rs.config.enable_device_from_file(config, path_video_bag,   repeat_playback=False )
profile = pipeline.start(config)
playback=profile.get_device().as_playback()
playback.set_real_time(False)

[filipematosinov]

Thanks @lbps for your suggestion it is working now :)

[milry012]

hello,
I had the same problem. I think to have a solution
pipeline.poll_for_frames(frames) doesn't work at all, even with composite_frame argument

compositFrame = pipeline.poll_for_frames() works like a frames= pipeline.wait_for_frame()
but it return a composite_frame type
be carefull, a poll_for_frame have not to return a frame each time

To check if in compositFrame = pipeline.poll_for_frames(), compositFrame is not null, check if compositFrame.is_frame() returns true.