Browser connected after the execution is started won't run any tests in SingleRun mode.

[bigbearzhu]

Hi guys,

I'm integrating Karma-runner into our CI system. The testing browsers could be running on remote machines/virtual machines, so we can't use the 'browsers' list in the configuration file to launch browsers. We need to keep all testing browsers running and waiting for new Karma server start. And we are using the SingleRun mode.

Interestingly, we found that sometimes one browser could get connected after the test execution is started. And that browser won't get any chance to run the code. For example, here is the time line on the server:

1. Two browsers are running (A and B).
2. Server started by CI.
3. Browser A connects. Since we are in singleRun mode and not using launcher, it will trigger "tryExecution" in server.js directly.
4. Browser B connects. At this moment the server has emitted out the 'execute' command, so the Browser A will run the test and Browser B will wait A to finish.
5. When A finishes, there is no check in "browser_complete" or "run_complete" events handling code to see whether there is any new connected browser(e.g. B) which hasn't run testing code, that, terminates the server immediately instead.

So Browser B would never have a chance to run the tests if it got in "too" late.

Possibly we could compare runningBrowsers and capturedBrowsers when "run_complete" is handled and re-trigger execution if necessary. Or can we stop existing runs if a new browser gets connected, which could save time to wait for existing runs to finish? I'm just new to Karma-runner so there could be some better ideas you have in your brilliant brains.

Cheers,
bigbearzhu

[morethanreal]

Just want to chime in that I'm having a related issue where I want to run tests from the CI environment on browserstack, where there's a limit in concurrent sessions. Since karma wants to capture all the browsers before executing the tests in single run, I needed to run karma multiple times with a subset of browsers to get all of them.

It would be great if karma can execute tests as new browsers come online, instead of waiting.

[vojtajina]

@morethanreal you are right, there's already an issue for it #57

@bigbearzhu You can use Karma to launch even remote browsers. That's exactly what we do on Angular Ci server, you can check out the scripts that start browsers on VM at https://github.com/angular/ci.angularjs.org (these are scripts for VMware, but if you check git history, you will find the same scripts for Virtual Box as well).

I guess we could support something like waitFor x number of browsers. Adding to my todo list...