Lagging and Shaking in Follower Arms with Camera Input at 50-60 fps Control

[RuiboLiu77]

Description:

Hi,

First, thanks for your great work on lerobot. I’ve installed it recently and followed the setup instructions, but I’m experiencing an issue when running the robot in teleoperation/record mode.

Problem Overview: When I run the robot at 30 fps without camera input, the follower arms exhibit noticeable lagging and shaking. However, if I run the control loop at 50-60 fps without the camera input, the robot moves smoothly and without issue.

Issue with Camera Input: When I add the 4 RealSense cameras and run the control loop at 50-60 fps, the follower arms start lagging again, with jerky movement and shaking. I suspect this may be due to a mismatch between the camera fps (30 fps) and the control loop fps (50-60 fps), which seems to cause delays in the movement of the arms. I also found that the dt (delta time) is only around 12-30 Hz, which could be contributing to the instability in the system.

What I’ve Tried: I’ve experimented with adjusting the number of threads and processes, but this didn’t seem to resolve the issue. The lag and shaking persist when camera input is involved, even at higher fps.

Details:
Robot: Aloha
Control fps: 50-60 fps
Camera input: 4 RealSense cameras (30 fps)
Observed issue: Lagging and shaking in follower arms when camera input is added, even at 50-60 fps control.

Could you provide suggestions on how to resolve the lag and shaking when using camera input? Specifically, is there a way to better handle the mismatch between the image input fps (30 fps) and the robot control fps (50-60 fps)?

Thanks in advance for your help!