Launching multiple 4+ D435 cameras failure - only launches some

[brycecopenhaver]

I am attempting to launch multiple (anywhere from 3-6) D435s using the same launch file structure as your [rs_multi_camera_launch.py] launch file (I just expand the same code used for 2 cams to however many I need). I am getting significant launch instability and unreliability in that when I attempt to launch multiple cameras, the wrapper only actually seems to initialize n-1 or n-2 (i.e. when I try launching 6, only 5 or 4 show up).

Each time it can be different cameras, so it's not specific to a single piece of malfunctioning hardware, and reading through the log shows that the nodes are attempting to launch each and every serial number listed in the launch file. Since I am trying to launch each node to a specific serial number, one of my assumptions is that each node attempts to search through the list of serial numbers, but since there are many SNs and the system providing them might not be ordered, the system provides the next available SN, regardless as to whether or not it has already given that SN to another node milliseconds before. This then causes overlap between each of the 4-6 nodes I am trying to launch since only one has the matching SN, and maybe the nodes timeout or otherwise fail after too many unsuccessful attempts to find attached hardware with that SN.

We are using a separate powered USB PCIe card that allows each camera to have its own USB channel, so there aren't any conflicts regarding that, which is proven again by the fact that sometimes all 4-6 cameras successfully launch (infrequently unfortunately).

Thank you!

[MartyG-RealSense]

Hi @brycecopenhaver A parameter called usb_port_id was created in the RealSense ROS wrapper in the past for addressing this kind of situation where there may be conflicts when launching multiple cameras simultaneously. If used instead of serial_no then the launch looks for a camera on a specific USB port instead of identifying it by serial number.

This method is less flexible than serial_no as a specific camera has to be looked for on a specific port. But if the cameras will never be unplugged and inserted in different ports in your particular project then it might be appropriate for you.

[brycecopenhaver]

Hi @MartyG-RealSense Great! I'll set that up and test it out early next week. I'll update this this ticket with the outcome. Appreciate the quick reply!

[brycecopenhaver]

OK @MartyG-RealSense I finally had the chance to test some things. I started by attempting to launch each camera based on its usb_port_id instead of serial_no as you suggested. While it's certainly less flexible, for the purposes of what I am trying to do, this is fine. Unfortunately, this doesn't seem to have solved the issue. I tested by launching it multiple times and I am still seeing only 3-6 of 6 cameras launch each time. I have attached the terminal log of a launch where 4 of the 6 cameras successfully launched.
4_of_6_our_launched.txt

Then, I ran a modified version of your rs_multi_camera_launch.py that I will post below. All I did here was add 4 more cameras and I will attach the terminal log for that too. In this experiment I am also only getting 3-6 cameras launching. In the instance of the log I am attaching below, 3 cameras launched, but in the test immediately before, all 6 launched.
3_of_6_launched.txt

Reading through the logs, it seems to be a power state issue. I have tried setting "initial_reset" to True and False, and it does not affect the launch. In the 4_of_6 log I am also setting "wait_for_device_timeout" to 30. and "reconnect_timeout" to 1. Neither of these parameters seem to have an effect either, whether they are set in the launch file or not. Another thing to note about the 4_of_6 log attached is that the ERRORs and WARNINGs that appear at the very end of the file continue indefinitely, so I just cut them off for the sake of log file brevity.

Here is the modified version of rs_multi_camera_launch.py file code I mentioned above:

import copy
from launch import LaunchDescription
import launch_ros.actions
from launch.actions import IncludeLaunchDescription
from launch.substitutions import LaunchConfiguration, ThisLaunchFileDir
from launch.launch_description_sources import PythonLaunchDescriptionSource
import sys
import pathlib
sys.path.append(str(pathlib.Path(__file__).parent.absolute()))
import rs_launch
import os
from ament_index_python.packages import get_package_share_directory

local_parameters = [{'name': 'camera_name1', 'default': 'camera1', 'description': 'camera unique name'},
                    {'name': 'camera_name2', 'default': 'camera2', 'description': 'camera unique name'},
                    {'name': 'camera_name3', 'default': 'camera3', 'description': 'camera unique name'},
                    {'name': 'camera_name4', 'default': 'camera4', 'description': 'camera unique name'},
                    {'name': 'camera_name5', 'default': 'camera5', 'description': 'camera unique name'},
                    {'name': 'camera_name6', 'default': 'camera6', 'description': 'camera unique name'},]

def set_configurable_parameters(local_params):
    return dict([(param['original_name'], LaunchConfiguration(param['name'])) for param in local_params])

def duplicate_params(general_params, posix):
    local_params = copy.deepcopy(general_params)
    for param in local_params:
        param['original_name'] = param['name']
        param['name'] += posix
    return local_params
    

def generate_launch_description():
    params1 = duplicate_params(rs_launch.configurable_parameters, '1')
    params2 = duplicate_params(rs_launch.configurable_parameters, '2')
    params3 = duplicate_params(rs_launch.configurable_parameters, '3')
    params4 = duplicate_params(rs_launch.configurable_parameters, '4')
    params5 = duplicate_params(rs_launch.configurable_parameters, '5')
    params6 = duplicate_params(rs_launch.configurable_parameters, '6')

    return LaunchDescription(
        rs_launch.declare_configurable_parameters(local_parameters) +
        rs_launch.declare_configurable_parameters(params1) + 
        rs_launch.declare_configurable_parameters(params2) + 
        rs_launch.declare_configurable_parameters(params3) + 
        rs_launch.declare_configurable_parameters(params4) + 
        rs_launch.declare_configurable_parameters(params5) + 
        rs_launch.declare_configurable_parameters(params6) + 

        [
        IncludeLaunchDescription(
            PythonLaunchDescriptionSource([ThisLaunchFileDir(), '/rs_launch.py']),
            launch_arguments=set_configurable_parameters(params1).items(),
        ),
        IncludeLaunchDescription(
            PythonLaunchDescriptionSource([ThisLaunchFileDir(), '/rs_launch.py']),
            launch_arguments=set_configurable_parameters(params2).items(),
        ),
        IncludeLaunchDescription(
            PythonLaunchDescriptionSource([ThisLaunchFileDir(), '/rs_launch.py']),
            launch_arguments=set_configurable_parameters(params3).items(),
        ),
        IncludeLaunchDescription(
            PythonLaunchDescriptionSource([ThisLaunchFileDir(), '/rs_launch.py']),
            launch_arguments=set_configurable_parameters(params4).items(),
        ),
        IncludeLaunchDescription(
            PythonLaunchDescriptionSource([ThisLaunchFileDir(), '/rs_launch.py']),
            launch_arguments=set_configurable_parameters(params5).items(),
        ),
        IncludeLaunchDescription(
            PythonLaunchDescriptionSource([ThisLaunchFileDir(), '/rs_launch.py']),
            launch_arguments=set_configurable_parameters(params6).items(),
        ),

        # dummy static transformation from camera1 to camera2
        launch_ros.actions.Node(
            package = "tf2_ros",
            executable = "static_transform_publisher",
            arguments = ["0", "0", "0", "0", "0", "0", "camera1_link", "camera2_link"]
        ),launch_ros.actions.Node(
            package = "tf2_ros",
            executable = "static_transform_publisher",
            arguments = ["0", "0", "0", "0", "0", "0", "camera2_link", "camera3_link"]
        ),launch_ros.actions.Node(
            package = "tf2_ros",
            executable = "static_transform_publisher",
            arguments = ["0", "0", "0", "0", "0", "0", "camera3_link", "camera4_link"]
        ),launch_ros.actions.Node(
            package = "tf2_ros",
            executable = "static_transform_publisher",
            arguments = ["0", "0", "0", "0", "0", "0", "camera4_link", "camera5_link"]
        ),launch_ros.actions.Node(
            package = "tf2_ros",
            executable = "static_transform_publisher",
            arguments = ["0", "0", "0", "0", "0", "0", "camera5_link", "camera6_link"]
        ),
        launch_ros.actions.Node(
            package="rviz2",
            executable="rviz2",
            name="rviz2",
            arguments=['-d' + os.path.join(get_package_share_directory("realsense_6_cam_test_pkg"), "rviz", "configs", "6_camera_native_multi_launch.rviz")],
        ),
    ])

[MartyG-RealSense]

In the 4_of_6_our_launched.txt log, I note that the color stream profile is being deemed by the launch for each camera to be invalid and it is instead applying 640x480 at 30 FPS as the color stream configuration. It looks as though this is because you have defined a width and height for color but not FPS, causing the custom configuration to fail.

In the ROS1 and ROS2 wrappers (but not the ROS2 beta wrapper), a custom stream configuration must provide three factors for each stream type enabled - height, width and FPS. If these 3 factors are not defined then the custom configuration fails and the default stream profile for the particular RealSense camera model being used is applied instead automatically by the wrapper.

---

As your cameras are attached to PCI-E USB ports, I wonder whether the total data usage from publishing depth and color x 6 cameras is exceeding the total bandwidth permitted by your particular model of PCI-E USB interface.

Bandwidth is more commonly a concern on USB hubs, which typically have a maximum bandwidth of 5 Gbps / 5000 Mbps to be shared between all attached devices. In Intel's original multiple camera white paper document, they provided tables with estimates of total bandwidth usage for up to 6 cameras when using various stream configurations.

https://dev.intelrealsense.com/docs/multiple-depth-cameras-configuration#section-2-multi-camera-considerations

---

Is your PCI-E USB interface an expansion card that is plugged internally into a slot on the inside of a desktop PC? If it is then I have not used one of those cards in years but I recall that there was a connector on the expansion board for optionally connecting it via cable to the power supply unit (PSU) of the PC to provide additional power for devices with a high power draw.

[brycecopenhaver]

Thank you for pointing the issue with the custom config out! Our "custom" configuration was actually just the default values anyways (640, 480, and 30) so we hadn't added the fps tag in the launch file as you suggested. However, I went ahead and made that change now so that in the future that error won't appear. I also made the change and tested it again with 6 cameras to see if that configuration failure could somehow lead to the failure of the cameras that we are seeing. I figured it wouldn't be the culprit, but wanted to double check to ensure there wasn't a strange bug associated with that regardless. Ultimately, it didn't change anything but now we won't see the custom configuration failure warning anymore.

---

So this is something we have thought about previously since we have had issues with this in the past. When we were first getting started with this tech, we tried using common USB hubs, which like you mention, only share 5Gbps between all USB ports on the hub. Due to this limitation, we ended up moving forward with the PCI-e USB cards that we are currently using, where each USB port on the card has it's own channel and bandwidth. We figure that since it sometimes succeeds in launching all 6 cameras, and when it succeeds there are no problems with bandwidth or frame dropping, that the issue we are seeing here is not directly caused by this. The cards we are using should have plenty of bandwidth for each camera as defined by their spec sheet, the white paper you linked, and our experiments.

---

As to your final set of questions:
That's correct. Our PCI-e USB cards are plugged directly into one of the PCI-e slots on the motherboard internal to our PC and you are correct that they have an additional power supply port onboard so it can provide additional power to each device. We have already plugged this in, as the type of PCI-e USB card we are using does not supply enough power without plugging into this port to run 2 at once, let alone 4.

---

Again, I appreciate your quick reply last week and apologize for only getting back to you now. I did a couple more log file captures (with the custom configuration warnings gone this time) but they essentially identical to the log files already posted (where the cameras are being detected, but 0-3 fail and the node never launches), so I will only send them if you think they are necessary to solving this issue. We're really excited to work with these cameras, and looking forward to the solution!

[MartyG-RealSense]

When there used to be problems with launching a D435 and T265 camera together at the same time with a multiple camera launch file, an approach that some RealSense users used before usb_port_id was introduced was to put a sleep period in-between launches.

http://wiki.ros.org/rospy/Overview/Time#Sleeping_and_Rates

[brycecopenhaver]

That's a good idea. Unfortunately, since that specific package is for ROS1 and we are in ROS2, we will likely need to find another way to create a sleep period in between launches. When we first had this problem, I briefly looked into it, but wasn't able to find anything at the time. I looked again just now and there doesn't seem to be a lot of official or even community solutions for adding sleep periods in launch files (actually I didn't find any for ROS2; they were exclusively ROS1)

I'll spend more time looking in the next day or so, but in the meantime I wanted to let you know in case you had other ideas or resources.

[MartyG-RealSense]

My research of your question above indicates that a popular way of implementing sleep in ROS2 is with a rate.

https://stackoverflow.com/questions/63169018/rate-and-sleep-function-in-rclpy-library-for-ros2

https://answers.ros.org/question/358343/rate-and-sleep-function-in-rclpy-library-for-ros2/?answer=358386#post-id-358386

https://get-help.robotigniteacademy.com/t/how-the-rospy-sleep-and-how-the-rospy-rate-works/10646

[brycecopenhaver]

Pardon my potential ROS ignorance, but my understanding is that the rate feature is only available inside of a node, so since we are trying to create sleep periods between launches in the launch file, that won't work in this case.

That being said, I was able to implement a sleep between launches in a different way, but tl;dr it doesn't solve the problem or help in a significant way. I just created a bash script that creates 3 new terminals (one after another with a sleep period in between), where each terminal launches a launch file that attempts to start only 2 cameras as per previous successful attempts and the example file. I'll paste the code for that at the end of this message for reference. Essentially I am creating a separate terminal for every launch file and each launch file has 15-20 seconds in between, so launch competition should be eliminated.

Instead, I am still seeing one of the cameras not showing up. For each launch file, 0-1 of the 2 cameras that I am trying to launch fails to launch. Reading through launch log, it still seems to be a "power state" failure because each time it "fails to set power state". Here is an example launch log from one of the three launch terminals (this one launches cam007, cam063, and rviz2). Please see lines 78-81 of the attached file and note that I am trying to launch based on usb_port_id instead of serial_no (but I also still see this issue with serial_no)

051920221549.txt

This is the code for the bash script in case it is useful or someone finds this thread in the future and can use it for whatever they're looking for:

#!/bin/bash
gnome-terminal -t CAMS-007-063 -- ros2 launch ./src/6_cam_pkg/launch/launch.py
sleep 20
gnome-terminal -t CAMS-696-733 -- ros2 launch ./src/6_cam_pkg/launch/launch2.py 
sleep 15
gnome-terminal -t CAMS-774-924 -- ros2 launch ./src/6_cam_pkg/launch/launch3.py

[MartyG-RealSense]

Failed to set power state can refer to any failure to access the camera. For example, because it is undetected, because insufficient power is being provided to the camera by the USB port or because the camera is busy due to already being active when a launch request is made to it.

[MartyG-RealSense]

Hi @brycecopenhaver Do you require further assistance with this case, please? Thanks!

[brycecopenhaver]

Hi @MartyG-RealSense, based on your post regarding the failed to set power state error message, it seems like the error is being thrown because the camera is busy due to already being active when a launch request is made to it (especially since we already troubleshot the first two that you list; the log shows they are being detected, and we know sufficient power is being provided to the camera by the USB ports since sometimes all 6 are successful).

That being said, I modified my workaround from 3 launch files that launch 2 cams each per terminal process, to 6 launch files that launch 1 camera each per terminal process. I launch a camera, wait 20-15 seconds, then launch another, rinse and repeat until all cameras are launched. This works for the time being but as you can imagine, it's incredibly slow from all the sleeping.

We have loved using your cameras and are excited as the possibility of continuing to use them, which is why we are disappointed that they aren't able to be launched simultaneously. For the purposes of this topic, we have achieved what we wanted to do, but I would recommend either adding this ticket to the bug pipeline or referencing it in another issue, as we should be able to launch multiple cameras simultaneously without needing to implement bulky workarounds. I appreciate your time and assistance with this issue, and look forward to seeing a more official and permanent solution in a future patch!

Thank you MartyG!

[MartyG-RealSense]

There was a multiple camera case with 6 cameras at #2022 (comment) (though with ROS1 rather than ROS2) where a RealSense team member advised to build librealsense with the V4L backend (a source code build of the SDK that is kernel patched and is very well suited to multicam) instead of the RSUSB backend installation method (which doesn't require patching but is less suited to multicam).

It was also advised that since December 2020, the Method 1 method of building librealsense and the wrapper together from packages has been based on the RSUSB build rather than V4L.

Which installation method does your project use to install librealsense and the ROS wrapper, please?

[MartyG-RealSense]

Hi @brycecopenhaver Was the advice in the comment above helpful to you, please? Thanks!

[brycecopenhaver]

Hi @MartyG-RealSense, we haven't had a chance to test this yet due to other project priorities at the moment. I will be taking a look later this week.

To answer your question though, we used the traditional method (Method 1 on the install page) to install the librealsense library and ROS wrapper, thus must be the RSUSB backend. I will try building with V4L backend in a couple days and update this ticket my results. Thank you

[MartyG-RealSense]

You are very welcome. I look forward to your test results. Good luck!

[MartyG-RealSense]

Hi @brycecopenhaver Do you have an update about this case that you can provide, please? Thanks!

[brycecopenhaver]

Hi @MartyG-RealSense Yes a brief update. I was able to successfully build librealsense with the V4L backend rather than the RSUSB backend. My original installation had in fact used RSUSB. One note is that your linked comment ultimately refers back to a thread for a Jetson, whereas we are using an x64 Intel machine, so I had to fish around for sources for that type of build instead (one reason for my delay in getting back to you). I did end up just using the links from Method 2 in the ROS2 wrapper README.

Unfortunately, it still seems like there is some instability in the launch. I am testing more today and will update this thread again once I am done identifying specific behaviors.

Additionally, we are seeing issues with missing depth frames with 5+ cameras. Originally with RSUSB, as soon as a 5th camera would connect, the depth streams across all 5 cameras would stop. Then as soon as any of the 5 cameras dropped/was disabled, all depths streams on the 4 cameras remaining would turn back on. Now, with V4L, we are seeing a similar but different issue where we can launch 6 cameras (not at the same time yet, since I have to test that today, as mentioned above), but the 6th camera consistently has no depth stream at all. This is different from the RSUSB in that the depth streams for the previous 5 launched cameras all remain running; it is just the 6th that is disabled. As this is a different issue to this thread. I need to investigate bandwidth issues with that whitepaper you sent earlier in the thread as partial reference, but once I am done I should make a separate ticket. Just wanted to let you, and anyone possibly reading this thread in the future, know how this issue changed with the V4L to RSUSB switch.

[MartyG-RealSense]

Thanks very much @brycecopenhaver for your detailed report. I look forward to your next test results and ticket. Good luck!

[MartyG-RealSense]

Hi @brycecopenhaver Do you have an update about this case that you can provide, please? Thanks!

[MartyG-RealSense]

Hi @brycecopenhaver Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.