Tests take 15 seconds to run, if run individually

[acecilia]

I did:

git clone https://github.com/airbnb/BuckSample.git
cd BuckSample
make install_buck
tools/buck test //Libraries/ASwiftModule:ASwiftModuleTests

The result of executing those commands is as follows:

Starting new Buck daemon...
Parsing buck files: finished in 0.5 sec
Building: finished in 3.8 sec (100%) 11/11 jobs, 11 updated
  Total time: 4.6 sec
Testing: finished in 17.3 sec (1 PASS/0 FAIL)
RESULTS FOR //Libraries/ASwiftModule:ASwiftModuleTests
PASS     15.0s  1 Passed   0 Skipped   0 Failed   SomeASwiftModuleTests
TESTS PASSED

The test executes properly (it passes/fails properly), but takes 15 exact seconds to finish.
Any idea whats going on?

Thanks!

[dfed]

Did you have the simulator open before you started running the test?

I'm a little confused what you're asking for here: are you wondering why this takes 17.3 seconds to finish? Is this an aberration vs other test runners you've used before? When I ran this one target's unit test from Xcode, it took closer to a minute including booting the simulator.

A bit more information regarding your setup, expectations, and question here would be helpful.

[acecilia]

Hi @dfed, thanks for your answer.

After restarting my machine, seems fixed now. Maybe it was a problem with buckd.

Thanks for the quick answer!

[acecilia]

Okey, no. I still see this issue randomly: some tests which should be running in the order of ms take exactly 15 seconds to finish.

Maybe it is something to do with bazelbuild/bazel#2985 (comment)

[dfed]

@acecilia do you think this is particular to BuckSample? My guess is this is a Buck issue rather than something with our config, especially if it is sporadic.

The bazel issue doesn't seem related: there shouldn't be any log-running processes related to this test suite.

[acecilia]

I have a feeling it has to do with the simulator, but I do not know how to troubleshoot it further. I am seeing the same problem in Bazel: bazelbuild/rules_apple#607. I also managed to reproduce it in CI, but only with bazel, not with buck (in buck only locally in my machine).

I agree with you, it does not seem to be a problem with your setup. The reason I opened the issue here is because this seems to be the main reference around for buck setup on ios, so I was hoping that somebody experienced this before.

[acecilia]

I got a bit more information. The 15 seconds delay goes away if I have the exact simulator that is going to run the tests opened in advance:

• If the simulator app is running, but no simulators are opened, the 15 seconds delay is there.
• If the simulator app is running, but the opened simulator does not match the one used to perform the tests, the 15 seconds delay is there.
• If the simulator app is running and the opened simulator is the one used to perform the tests, the 15 seconds delay is gone.

Is this expected? Is it documented somewhere?

Thanks!

[dfed]

This sounds pretty expected, and is why my first question was:

Did you have the simulator open before you started running the test?

Simulators take awhile to boot, and they need to finish booting in order for unit test to run. I don't know of a place that this is documented, however.

[acecilia]

Okey, thanks @dfed. I just find 15 seconds a bit excessive for the simulator to boot, but so far seems like that’s the case. Now I know that booting it in advance fixes the problem 👍

[acecilia]

An update on this: apparently the simulator in Xcode 10.2 and 10.3 adds a 10-15 seconds delay randomly when running tests. Looks like it is fixed in Xcode 11. See: bazelbuild/rules_apple#607 (and more concretely bazelbuild/rules_apple#607 (comment))