Simulations are taking much longer to run

[malcolmst]

Over the past few days I’ve noticed my simulations are taking a lot longer to run than before. This difference seems to line up with the addition of the new visualization features in cloudsim (which look great btw!). Even in simulations where these features were not being actively used though, I am seeing a much longer run time.

This in itself is not a huge problem, except I just had one simulation from the finals practice 01 world (2417001a-75a0-4ab3-b945-31b8cf499564) which seems to have timed out after 48 hours of run time and 30 minutes of simulation time. Previously, finals practice 01 was one of the slower worlds, but it would have finished running for 60 minutes within that 48 hours.

[nkoenig]

I believe the issue is related to a trapped robot, which would create a costly collision check. In your run, one vehicle became embedded in a dynamic collapse wall.

[malcolmst]

Thanks, that is a good point. I’ll do another run with that world shortly to try to confirm if that could be the cause.

[m3d]

@nkoenig - can you please check 9fe7d49d-f9fd-4d47-a5c8-d68272e14861?

p.s. note, that it is also example where H-drone did not start

[malcolmst]

I also have a finals_qual run (da604df8-81df-43bf-9d8d-4f683175aff3) which is nearing completion (my best run yet, incidentally 😃) but has been running for 52 hours. Previously, the finals_qual world took somewhere around 24 hours to run with my configuration of robots. I have made a lot of changes recently, though I'm not sure how any of the changes on my side would have such a large impact on simulation speed.

For a comparison:

92521cc1-8d16-4e25-aa19-74ae369bb9b2 (7/15/2021):
sim_time: 3600     (60:00)
real_time: 103618   (28 hours 47 minutes)
[real time factor 3.47%]

053afa5c-e77c-4599-a529-41587244cec0 (7/16/2021):
sim_time: 2631     (43:51)
real_time: 70282   (19 hours 31 minutes)
[real time factor 3.74%]

53e2c899-99b4-4249-bb33-f6c31d291469 (7/17/2021):
sim_time: 2314     (38:34)
real_time: 109514  (30 hours 25 minutes)
[real time factor 2.11%]

I could be wrong, but based on the timing of the slowdown from ~4% to ~2% (7/17/2021) this appears like it might be somehow related to the deployment of the new visualization features.

[m3d]

p.s. now we are at 60h and based on ratio 0.7% it looks like:

>>> 100/0.7
142.85714285714286
>>> 100/0.7/24
5.9523809523809526

i.e. it will take 6 days?

[nkoenig]

@m3d , I'm going to terminate 9fe7d49d-f9fd-4d47-a5c8-d68272e14861 so that I can look at the logs.

[malcolmst]

FYI I started another run with the finals practice 01 world (59dad9e3-bdbf-4faf-ab53-0eee9012f952) and it looks about the same as the previous run despite no robots getting buried under a pile of rocks. Maybe marginally faster, but it's been running for 47 hours of real time, and 33 minutes of simulation time.

[meryl-spencer]

For reference our simulations with 7 robots took over 72 hours to simulate 45 min of competition time.

[malcolmst]

@meryl-spencer have you noticed a slowdown in recent days, or is that what you have been seeing generally?

Interestingly, most of my simulations have been slower recently, but one I started yesterday is running more quickly again (78d6166c-ab5c-47d2-89c9-11c1434e0a22).

[malcolmst]

I’ll continue to keep an eye on this, but everything I’ve been running lately appears to be back running at the normal speed, so maybe whatever was causing the slowdown is now resolved 🙂. It seems all my runs from the 17th to the 21st were affected, in case that correlates with anything relevant.

[meryl-spencer]

@malcolmst We couldn't tell if there was a slowdown since our solution was rapidly changing and the number of robots and world make a big difference. However, our runs from this weekend with 7 robots finished within 2.5 days which is significantly faster than the last few weeks.

[m3d]

my impression was that it was faster now (1.5% instead of 0.7%)

[m3d]

We were running "final" simulations also in practice world FP1 was not finished in 96 hours:

sim_time: 2344
real_time: 349748
artifact_report_count: 23 

so it was cut by the simulation limit:

>>> 349748/3600
97.15222222222222

where our teambase log says:

Total time 4 days, 1:08:42.899042

Note, that it was simulated only 39 minutes (out of 1hour).
0f3166b6-5c8d-4cff-badb-0e23bae6ed90

[m3d]

p.s. this is real-time difference as function of sim time (for the run mentioned above)

i.e. it starts with slightly 1 minute ~ 1 sim second and end with more than 3 minutes ~ 1 sim second