Only a subset of cameras are detected (multi-camera configuration) #3829
Comments
|
Are you connecting the cameras to a single PC with a USB hub with many ports such as a 12 port hub, please? Doing hardware reset with scripting is the obvious way to reset without physical unplug-replug. A solution may be to iterate through all the cameras, reset them and wait for completion of reset before continuing. A RealSense community member called Westa contributed a script for doing this. It can be found at #9287 (comment) |
|
Thanks Marty.
Our typical process is to optionally [user's choice] hardware reset all the cameras at start, then enable the streams and start the pipeline. Again, usually works great. Also, to clarify, the devices become invisible to the driver, so, when this bug happens: The issue here is that the mode of failure is catastrophic and the user may not have access to the machine in our use case. Additionally, unplugging and reconnecting 12 or more sensors is not an acceptable solution for our customers, even if their operators do have access to the machine. I'd basically hate to write "go to the machine and unplug everything..." into our user documentation. |
|
Having that many cameras attached to a single PC is not usually ideal unless it is a powerful enterprise-grade PC rather than a consumer desktop one. This is because each camera added in a multi-camera setup causes resources to be consumed. For example, in a single-PC setup where 4 cameras were connected, a PC with an i7 processor would be recommendable in order to cope with the strain on the hardware. As you said though, your application is working fine when the cameras are all detectable, so it doesn't sound as though the hardware spec is a problem for you. My personal hunch for the problem would be a power instability on the USB port, causing the camera connection to drop and making it undetectable / unactivatable. RealSense cameras are sensitive to the state of the USB port. If a hub was being used, a mains electricity powered hub can increase stability of power on USB ports. If the system can see up to 7 cameras of the 12 every single time though after a reboot of the PC and it is never more or less than 7, that would suggest that something other than USB power instability is happening. |
|
Good morning, Marty! The hardware isn't the problem; everything you said about PC resource consumption is correct, including power draw. We're an OEM with a major PC manufacturer and work with their R&D unit to test bus/bridge performance and so forth. We also have a power electronics team and are capable of measuring the power draw from each camera individually and the entire array of cameras as a whole. We are certain that all cameras are adequately powered and are otherwise not lacking for bandwidth or resource. I think this is a bug in the driver. |
|
Disconnection issues have occurred with RealSense cameras even on enterprise servers with very high watt PSUs. So it is not so much a matter of the camera's power draw as a power frequency glitch on the port. Not all USB ports are made equal despite there being a USB specification, and so a camera that works fine on one make of PC may not work properly on another manufacturer's PC. The quality grade of USB cable used can also be a factor in stability. Unless a high grade cable such as an industrial grade one is used, the longer the cable is then the higher the chance of instability. On an ordinary cheap cable, I would not have expectations of the camera working on a cable longer than 2 meters. Having said all that, I have not personally dealt with setups larger than about 6 cameras, so one of the Intel staff on this forum such as @dorodnic may be able to advise you better on this subject. |
|
Right, however the bug, as described in the original message, manifests following a hardware reset of all the cameras which we invoke using the API. We are not discussing intermittent dropping of cameras, to your note about power line conditioning. I do not believe that we have power issues, as the systems are able to perform at peak for hours at a time under normal use. The bug relates to the dropping of cameras specifically following a hardware reset. It seems as though the cameras briefly disconnect, and sometimes not all of the cameras reconnect. Under which circumstance the only remedy is seemingly to unplug and reconnect them. @dorodnic, any support your able to offer would be most appreciated. |
|
Hi @evercoast-dev, |
|
OK, here we go. Updated libRealsense to 20.0.0, upgraded firmware (all cameras match) to 05.11.06.200, the system has 12 cameras plugged in, lsusb reports 12 Intel devices, then rs-enumerate devices sees none of them:
Thanks! |
|
Update on the previous post, following a reboot we see 11 Realsense cameras:
Then unplugging and reconnecting the missing 12th camera, we finally see:
|
|
Is this problem persist after subsequent reboot/hardware reset or system cold boot? What is the length(s) of the USB cable used for these cameras? Just for testing purpose, is there any improvement or difference by using the 1 M USB cable that comes with the camera? |
|
Often times we notice that when cameras "go missing" that a reboot is likely to fix the problem. However, infrequently, but often enough that we're posting a bug report, it happens that some cameras remain inactive following a reboot. It's then that the unplug/reconnection is necessary. Regarding cable length, it's not really viable to get that many cameras plugged in using only the supplied cable. But, I can say that we do normally operate without dropping frames on all cameras simultaneously. Also, our USB extensions are professional quality. |
|
I am just wondering if there's any update/potential fix that you might have discovered to workaround the issue? |
|
Closing the ticket. Please feel free to reopen if problem remains. |
|
This issue is very persistent and why is it closed? |
|
@Ashok93 An issue may be closed by RealSense team members if no further comments are received on the subject for 3 or 4 weeks and it does not have a subject that requires keeping it open for long-term tracking. Closed subjects are still findable by web search. Regarding this case's subject: improvements to multicam support were released in the latest 2.35.2 SDK version. The changes may not correct every multicam problem (such as ones related to specific USB hub models) but it may help if the problem is related to the rs2::pipeline. If you are able to use 2.35.2 then I encourage you to try it and see if it resolves your need to unplug and replug cameras. |
Camera Model: D400
Firmware Version: 5.10.03
Operating System: Ubuntu
Kernel Version: 4.15.0-43-generic (we see the problem across a variety of kernel versions)
Platform: PC
SDK Version: 2.19.0
Language: C++
Segment: VR/AR
Only a subset of cameras are detected (multi-camera configuration).
On a system with 12 D400 series cameras plugged in simultaneously:
The solution is to physically unplug and then reconnect the set of USB cables at the back of the computer. This procedure is very inconvenient. We really need a fix for this as these machines may not be accessible to an operator in the future (they may be located in a locked cabinet, for instance).
Note: We adhere to all the USB specs and power requirements and are able to operate these cameras at peak spec under regular conditions. This is an intermittent problem.
The text was updated successfully, but these errors were encountered: