Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

In Jetson Nano with L515, the point cloud from depth is slow with 'control_transfer returned error' #1758

Closed
bigbellmercy opened this issue Mar 16, 2021 · 22 comments
Closed

In Jetson Nano with L515, the point cloud from depth is slow with 'control_transfer returned error' #1758

bigbellmercy opened this issue Mar 16, 2021 · 22 comments

Comments

@bigbellmercy
Copy link

Hello, I report a problem on Jetson Nano with L515.

Error

In Jetson Nano with L515, the point cloud output from depth is slow with the following error continuously happening:

WARNING [139819288090368] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
(For reference, #1663 is a similar issue with the same error)

But, by realsense-viewer, the output is displayed fast and fluently on Jetson Nano with L515.

Symptoms

If depth resolution is set to 1024 x 768 maximally, the point cloud topic, /camera/depth/color/points, is not output as reported at #1741.
If depth resolution is set to 640 x 480, the point cloud topic is published at the rate of from about 10 to 25 Hz (usually 10 Hz) while the above error messages appearing continuously. But, after some period, the publication is stopped of itself.
If depth resolution is set to 320 x 240, it is published at the rate of from about 10 to 25 Hz (usually 10 Hz) while the error messages appearing continuously. But during about 30 minutes, the publication was not stopped. So it may be used. But the publication rate is low and fluctuated.

Condition

Only realsense-ros application is executed on the Jetson Nano.
Major parameters for L515 are as below:
enable_depth: true
enable_fisheye: false
enable_infra/1/2: false
enable_color: true (resolution 1280 x 768)
depth_fps: 30
enable_pointcloud: true
All other parameters are default values.

Environment

Jetson Nano ARM64 4GB RAM
Ubuntu 18.04
ROS melodic
L515 firmware v1.5.4.1
realsense-ros v2.2.22
librealsense2 v2.42.0

Build

I've built librealsense2 SDK and realsense-ros package all from source codes as written in #1289 and succeeded. Thanks.
@bigbellmercy bigbellmercy mentioned this issue Mar 16, 2021
Closed
@RealSenseSupport
Copy link
Collaborator

@bigbellmercy

I see that you've opened this issue up based on discussion on previous issue with D400 product.

@RealSenseSupport
Copy link
Collaborator

@bigbellmercy Do you mean you followed #1289 (comment) to build and install SDK and ROS wrapper? Could you please share your launch file when get the issue? Thanks!

@alexart13
Copy link

alexart13 commented Mar 31, 2021 •
edited
Loading

I'm facing the same issue on Jetson Xavier with L515. I followed ros-realsense installation guide and then simply executed: "roslaunch realsense2_camera rs_camera.launch"

In the logs I see a lot of warnings "Resource temporarily unavailable" and after a few minutes publication of depth stopped.
There is also a warning about the frame's time domain: "frame's time domain is HARDWARE_CLOCK. Timestamps may reset periodically." What is the reason of this warning?

When I replace L515 with D435i it works fine. So it must be something specific to L515.

L515 firmware: v1.5.4.1

[ INFO] [1617030903.157307647]: Initializing nodelet with 8 worker threads.
[ INFO] [1617030903.897582369]: RealSense ROS v2.2.22
[ INFO] [1617030903.897788328]: Built with LibRealSense v2.42.0
[ INFO] [1617030903.897928205]: Running with LibRealSense v2.42.0
[ INFO] [1617030903.998100334]:
29/03 15:15:04,244 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
[ INFO] [1617030904.903422752]: Device with serial number f0221773 was found.
[ INFO] [1617030904.903741451]: Device with physical ID 2-4-3 was found.
[ INFO] [1617030904.903879983]: Device with name Intel RealSense L515 was found.
[ INFO] [1617030904.907249474]: Device with port number 2-4 was found.
[ INFO] [1617030904.907490987]: Device USB type: 3.2
[ INFO] [1617030904.918013586]: getParameters...
[ INFO] [1617030905.163100002]: setupDevice...
[ INFO] [1617030905.163429102]: JSON file is not provided
[ INFO] [1617030905.163548210]: ROS Node Namespace: camera
[ INFO] [1617030905.163679798]: Device Name: Intel RealSense L515
[ INFO] [1617030905.163817659]: Device Serial No: f0221773
[ INFO] [1617030905.163993825]: Device physical port: 2-4-3
[ INFO] [1617030905.164147462]: Device FW version: 01.05.04.01
[ INFO] [1617030905.164285771]: Device Product ID: 0x0B64
[ INFO] [1617030905.164396559]: Enable PointCloud: Off
[ INFO] [1617030905.164532275]: Align Depth: Off
[ INFO] [1617030905.164624247]: Sync Mode: Off
[ INFO] [1617030905.164867679]: Device Sensors:
[ INFO] [1617030905.195110247]: L500 Depth Sensor was found.
[ INFO] [1617030905.226888228]: RGB Camera was found.
[ INFO] [1617030905.228237874]: Motion Module was found.
[ INFO] [1617030905.228533180]: num_filters: 0
[ INFO] [1617030905.228662944]: Setting Dynamic reconfig parameters.
[ INFO] [1617030905.616701629]: Done Setting Dynamic reconfig parameters.
[ INFO] [1617030905.619673731]: depth stream is enabled - width: 640, height: 480, fps: 30, Format: Z16
[ INFO] [1617030905.624229566]: color stream is enabled - width: 1280, height: 720, fps: 30, Format: RGB8
[ INFO] [1617030905.624715887]: setupPublishers...
[ INFO] [1617030905.638833713]: Expected frequency for depth = 30.00000
[ INFO] [1617030905.746801142]: Expected frequency for color = 30.00000
[ INFO] [1617030905.829610272]: setupStreams...
[ INFO] [1617030905.855295693]: insert Depth to L500 Depth Sensor
[ INFO] [1617030905.855610488]: insert Color to RGB Camera
29/03 15:15:06,153 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:06,294 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:06,345 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
[ INFO] [1617030906.354529555]: SELECTED BASE:Depth, 0
29/03 15:15:06,396 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:06,448 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
[ INFO] [1617030906.501151963]: RealSense Node Is Up!
[ WARN] [1617030906.515367328]:
[ WARN] [1617030906.515698923]: frame's time domain is HARDWARE_CLOCK. Timestamps may reset periodically.
29/03 15:15:07,005 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:07,348 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:07,704 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:08,600 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
29/03 15:15:09,853 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:11,105 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:12,358 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:13,611 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:14,864 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:16,116 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:17,368 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:18,621 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:19,874 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:21,126 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:22,379 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:23,631 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:24,883 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:26,136 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:27,388 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:28,641 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:29,893 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:31,145 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
29/03 15:15:32,397 WARNING [547294364032] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11

@doronhi
Copy link
Contributor

doronhi commented Apr 6, 2021

Can you compare the ROS wrapper to rs-pointcloud?

@bigbellmercy
Copy link
Author

@doronhi Thanks for reply. I'm afraid that I never used rs-pointcloud and similar ones..

@bigbellmercy
Copy link
Author

@doronhi But, I think that realsense-viewer seems to use rs-pointcloud library maybe. If then, as mentioned above in my first post, when realsense-viewer is used on Jetson Nano, its point cloud output display is very fast and fluent, FYI.

@doronhi
Copy link
Contributor

doronhi commented Apr 7, 2021

rs-pointcloud is part of the librealsense2-utils package, same as realsense-viewer.
realsense-viewer uses CUDA to calculate the pointcloud.
rs-pointcloud is a small demo for pointcloud that, like ROS, does not use CUDA.
It is interesting to see if that is the source for the difference.

@bigbellmercy
Copy link
Author

Ah ha. I didn't know that CUDA is used in Jetson Nano also. In my case, when I used the CUDA build option -DBUILD_WITH_CUDA=true when building librealsense2 sources in Jetson Nano, that was no effect for the speed up of point cloud output in Jetson Nano, because when I omitted that option the speed was the same slow.

@doronhi
Copy link
Contributor

doronhi commented Apr 7, 2021

I am sorry, I think I got confused. @alexart13 - do you have an issue with the rate of the pointcloud? I see in your log "num_filters: 0"...

@alexart13
Copy link

@doronhi I launched with the default configuration "roslaunch realsense2_camera rs_camera.launch" without applying any filters. As I understand in such case the realsense library just post depth data into the corresponding ROS topic.
I didn't measure the rate, but I expect to see the same behavior as @bigbellmercy described.

@alexart13
Copy link

@doronhi is there anything I can do to try to solve the issue? This problem blocks us to some extent. On our robot we would like to replace D435i with L515, because we need more accurate depth data. But we can't do that due to the issue.

@doronhi
Copy link
Contributor

doronhi commented Apr 8, 2021

I need some carifications:
@alexart13 - you did not add the pointcloud flag, right? Can you add and test that you indeed have the issue? If not, what is the issue that you are facing?
@bigbellmercy, It seems that maybe you have 2 copies of librealsense2 installed on your machine.
Could you post the output of apt list --installed | grep realsense?
If indeed you have another version installed from the ros repository that was compiled without CUDA, that could account for the difference in performance.

@bigbellmercy
Copy link
Author

Thanks @doronhi, here is the result:

~/catkin_ws2$ apt list --installed | grep realsense

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

librealsense2/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed,automatic]
librealsense2-dbg/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed]
librealsense2-dev/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed]
librealsense2-dkms/bionic,now 1.3.16-0ubuntu1 all [installed]
librealsense2-gl/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed,automatic]
librealsense2-net/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed,automatic]
librealsense2-udev-rules/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed,automatic]
librealsense2-utils/bionic,now 2.44.0-0~realsense0.4389 amd64 [installed]

@doronhi
Copy link
Contributor

doronhi commented Apr 8, 2021

Thanks. So that is not the issue - you have just one copy.
My other suspicion is that copying the pointcloud into the ROS structure takes too long but I'll investigate. It will take me a while to get around to it.
If you can use the pointcloud without texture it should save some running time.
If you managed to run "rs-pointcloud" and see a major difference in performance compared with the pointcloud in "realsense-viewer" it may give me a hint as to the source of the issue but it will still require debugging and modifying the wrapper.
All that has nothing to do, as far as I understand it, with the camera being D435 or L515.

@alexart13
Copy link

@doronhi with D435 it works fine. The problem is only with L515.

@bigbellmercy
Copy link
Author

@bigbellmercy Do you mean you followed #1289 (comment) to build and install SDK and ROS wrapper? Could you please share your launch file when get the issue? Thanks!

@RealSenseSupport Though it's my late reply, yes I followed #1289. And here is the launch file:

<?xml version="1.0"?>

<launch>
  <arg name="serial_no"           default=""/>
  <arg name="usb_port_id"         default=""/>
  <arg name="device_type"         default=""/>
  <arg name="json_file_path"      default=""/>
  <arg name="camera"              default="camera1"/>
  <arg name="tf_prefix"           default="$(arg camera)"/>
  <arg name="external_manager"    default="false"/>
  <arg name="manager"             default="realsense2_camera_manager"/>

  <arg name="fisheye_width"       default="640"/>
  <arg name="fisheye_height"      default="480"/>
  <arg name="enable_fisheye"      default="false"/>

  <arg name="depth_width"         default="640"/>
  <arg name="depth_height"        default="480"/>
  <arg name="enable_depth"        default="true"/>

  <arg name="infra_width"        default="640"/>
  <arg name="infra_height"       default="480"/>
  <arg name="enable_infra"        default="true"/>
  <arg name="enable_infra1"       default="false"/>
  <arg name="enable_infra2"       default="false"/>

  <arg name="color_width"         default="1280"/>
  <arg name="color_height"        default="720"/>
  <arg name="enable_color"        default="true"/>

  <arg name="fisheye_fps"         default="30"/>
  <arg name="depth_fps"           default="30"/>
  <arg name="infra_fps"           default="30"/>
  <arg name="color_fps"           default="30"/>
  <arg name="gyro_fps"            default="400"/>
  <arg name="accel_fps"           default="250"/>
  <arg name="enable_gyro"         default="false"/>
  <arg name="enable_accel"        default="false"/>

  <arg name="enable_pointcloud"         default="true"/>
  <arg name="pointcloud_texture_stream" default="RS2_STREAM_COLOR"/>
  <arg name="pointcloud_texture_index"  default="0"/>

  <arg name="enable_sync"               default="false"/>
  <arg name="align_depth"               default="false"/>

  <arg name="publish_tf"                default="true"/>
  <arg name="tf_publish_rate"           default="0"/>

  <arg name="filters"                   default=""/>
  <arg name="clip_distance"             default="-2"/>
  <arg name="linear_accel_cov"          default="0.01"/>
  <arg name="initial_reset"             default="false"/>
  <arg name="unite_imu_method"          default=""/>
  <arg name="topic_odom_in"             default="odom_in"/>
  <arg name="calib_odom_file"           default=""/>
  <arg name="publish_odom_tf"           default="true"/>
  <arg name="allow_no_texture_points"   default="false"/>

  <group ns="$(arg camera)">
    <include file="$(find realsense2_camera)/launch/includes/nodelet.launch.xml">
      <arg name="tf_prefix"                value="$(arg tf_prefix)"/>
      <arg name="external_manager"         value="$(arg external_manager)"/>
      <arg name="manager"                  value="$(arg manager)"/>
      <arg name="serial_no"                value="$(arg serial_no)"/>
      <arg name="usb_port_id"              value="$(arg usb_port_id)"/>
      <arg name="device_type"              value="$(arg device_type)"/>
      <arg name="json_file_path"           value="$(arg json_file_path)"/>

      <arg name="enable_pointcloud"        value="$(arg enable_pointcloud)"/>
      <arg name="pointcloud_texture_stream" value="$(arg pointcloud_texture_stream)"/>
      <arg name="pointcloud_texture_index"  value="$(arg pointcloud_texture_index)"/>
      <arg name="enable_sync"              value="$(arg enable_sync)"/>
      <arg name="align_depth"              value="$(arg align_depth)"/>

      <arg name="fisheye_width"            value="$(arg fisheye_width)"/>
      <arg name="fisheye_height"           value="$(arg fisheye_height)"/>
      <arg name="enable_fisheye"           value="$(arg enable_fisheye)"/>

      <arg name="depth_width"              value="$(arg depth_width)"/>
      <arg name="depth_height"             value="$(arg depth_height)"/>
      <arg name="enable_depth"             value="$(arg enable_depth)"/>

      <arg name="color_width"              value="$(arg color_width)"/>
      <arg name="color_height"             value="$(arg color_height)"/>
      <arg name="enable_color"             value="$(arg enable_color)"/>

      <arg name="infra_width"              value="$(arg infra_width)"/>
      <arg name="infra_height"             value="$(arg infra_height)"/>
      <arg name="enable_infra"            value="$(arg enable_infra)"/>
      <arg name="enable_infra1"            value="$(arg enable_infra1)"/>
      <arg name="enable_infra2"            value="$(arg enable_infra2)"/>

      <arg name="fisheye_fps"              value="$(arg fisheye_fps)"/>
      <arg name="depth_fps"                value="$(arg depth_fps)"/>
      <arg name="infra_fps"                value="$(arg infra_fps)"/>
      <arg name="color_fps"                value="$(arg color_fps)"/>
      <arg name="gyro_fps"                 value="$(arg gyro_fps)"/>
      <arg name="accel_fps"                value="$(arg accel_fps)"/>
      <arg name="enable_gyro"              value="$(arg enable_gyro)"/>
      <arg name="enable_accel"             value="$(arg enable_accel)"/>

      <arg name="publish_tf"               value="$(arg publish_tf)"/>
      <arg name="tf_publish_rate"          value="$(arg tf_publish_rate)"/>

      <arg name="filters"                  value="$(arg filters)"/>
      <arg name="clip_distance"            value="$(arg clip_distance)"/>
      <arg name="linear_accel_cov"         value="$(arg linear_accel_cov)"/>
      <arg name="initial_reset"            value="$(arg initial_reset)"/>
      <arg name="unite_imu_method"         value="$(arg unite_imu_method)"/>
      <arg name="topic_odom_in"            value="$(arg topic_odom_in)"/>
      <arg name="calib_odom_file"          value="$(arg calib_odom_file)"/>
      <arg name="publish_odom_tf"          value="$(arg publish_odom_tf)"/>
      <arg name="allow_no_texture_points"  value="$(arg allow_no_texture_points)"/>
    </include>
  </group>
</launch>

@RealSenseSupport
Copy link
Collaborator

@bigbellmercy Did you get chance to try "rs-pointcloud" with L515 to see if any issue?

@alexart13 Considering your reply "with D435 it works fine. The problem is only with L515", is this the result based on rs-pointcloud?

@RealSenseSupport
Copy link
Collaborator

@bigbellmercy @alexart13 Any update from your side? Looking forward to your update. Thanks!

@RealSenseSupport
Copy link
Collaborator

@bigbellmercy @alexart13 Could you please update? Please note that the issue will be closed if we don't hear from you for another 7 days. Thanks!

@bigbellmercy
Copy link
Author

Hi! It's quite late for me. I report that when I ran the rs-pointcloud with L515 in Jetson Nano the speed and the response of the color textured point cloud on the screen was quite fast and fluent well. Though I don't know the exact resolution of the default resolution of the rs-pointcloud, but it seems to 1024 x 176 because the screen was large enough on my 1280 x 1024 monitor.

@bigbellmercy
Copy link
Author

bigbellmercy commented May 13, 2021 •
edited
Loading

And I've got a new result: When I changed the parameter pointcloud_texture_stream from RS_STREAM_COLOR (default) to RS_STREAM_ANY in the launch file, the point cloud output rate became good as about 20 to 30 Hz in 1024 x 768 depth resolution during about 1 hour.

So, I think this issue may be closed by this solution. Thank you.

In the result above, the versions of the software are as below:

  • Realsense ROS wrapper v2.2.22 (built with LibRealSense v2.43.0)
  • LibRealSense v2.43.0 (version matched to the above)
  • L515 firmware v1.5.5.0

P.s:
But it is needed to make a new issue by me because the delay time felt is so big as as 3 seconds. When I launched rviz on the Jetson Nano its response delay was about 3 seconds. Furthermore, the time delay measured by rostopic delay command increases on and on by 2 ms per every 1 second. --> This was resolved by #1902 (comment)

@RealSenseSupport
Copy link
Collaborator

@bigbellmercy Thanks for your feedback! Glad to hear that the issue of this ticket resolved. For your new issue, please feel free to create another issue to track and we'll look into this. Thanks!

@RealSenseSupport RealSenseSupport mentioned this issue May 18, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bigbellmercy @alexart13 @doronhi @RealSenseSupport