Collaboration with karma

[dignifiedquire]

So after our short talk @ DTN Conf I thought I'd open up the issue here, so we could start talking about this.

• 1 - Currently karma can "in theory" launch as many browsers as you want, but they have to be added one by one to configuration, so here there would be the main need to improve. This should be actually very easy to change though.
• 2 - All communications happen centralized together with the karma server via socket.io, but I'm not a 100% sure that that communication needs to be decentralized (though it would probably be good if the amount of browsers grows very large)
• 3 - At the moment karma can only run the same thing in all browsers, but there are plans to change this Test sharding / parallelization karma-runner/karma#439 especially for test sharding, but I can imagine it being interesting for testing distributed applications as well, as you would have different types of test nodes that you want to execute
• 4 - Karma does already take care of all the business of getting your files in various ways into the browser once it is connected
• 5 - There is no automated headless way to launch browser right now, but I would love to add this, though as you have described in Launch several browsers in parallel #3 it's not clear how to get that working cross browser and cross platform.
• 6 - Karma needs to be able to work with all shitty browsers back to IE6 so if there is any usage of things like webrtc I would need to ensure that these things are either opt in, or favorably they have fallbacks to work everywhere.

Let me know what you think.
Cheers

[daviddias]

Hi @dignifiedquire, thank you for following up on this!

1- That is, pretty awesome :). Is there a "Karma daemon" that I could install in remote servers to fire up browsers from different points in the network?

2 - Having a centralised server where test results are collected and asserted is the desired architecture, it is how this module works too :)

3 - The way that I developed this module is that browsers don't even have to understand which tests they will be running. You can check here: https://github.com/diasdavid/piri-piri/blob/master/tests/serve_this.js that the file served to the browser registers function calls to be called later. Tests are defined on the .js file executed by the centralised server, which will invoke those registered function calls as the test pleases, here https://github.com/diasdavid/piri-piri/blob/master/tests/spicy-test.js#L80 . Essentially the idea was to a: instruct a module, b: load it in a browser, c: invoke those routines from a server, d: evaluate if the effect was the desired.

4 - ⭐️

5 - Since WebRTC doesn't run in headless browsers, I don't have any need for that, but I can understand that other people have for other kinds of apps.

6 - If I understand correctly, Karma wouldn't have to understand that there is WebRTC being used, as it will only load code in the browser and the browser will be the one exposing the WebRTC APIs

I believe if 3 can be achieved, I could replace all of the tests for Karma and if 1 can be achieved too, we can create something very special for P2P browser testing :)

Also, one other idea that is in the back of my mind is "Pseudo External Consistency", such like Spanner from Google, we can use time as our sorting metric if we run the tests in the same machine (checking if events happened in the desired order). For distributed browser tests (using the proposed Karma daemon in several parts of the world), we might have to bring some Vector/Lamport clocks to guarantee some way to assert the order.