Testing

[vojtajina]

The karma-intellij plugin relies on tons of Karma internals and it's very difficult to not break it. We need some e2e test, that would run a simple project in WebStorm.

@segrey do you have any framework/helpers for testing webstorm that we could take leverage ? I'd really like to get to the point where we have CI build, that starts real webstorm and runs a Karma test, because Karma is now on auto-release (patch versions) - if the Travis build passes it gets released twice a week and this would help us a lot from breaking people.

The patch release should not contain any breaking change (in the stable release), but really this plugin depends on so many internals, that it's pretty much impossible to not break it...

[segrey]

WebStorm communicates with karma using karma-intellij as a mediator. For a start, we can test how karma-intellij communicates with karma (without starting WebStorm IDE). I think that would be a good start because up until now all incompatibilities were caused by changing in karma API. I think that changing karma API is inevitable we just have to be informed about needed changes in karma-intellij.
I'll take a look at karma's e2e tests.

General suggestion: it'd be good to introduce public API for all internals karma-intellij depends on. That would help a little. But adding tests will surely help more.

[vojtajina]

I wanna follow the policy of "no breaking changes in the stable branch",
but especially with the plugins it's very hard. These plugins are super
powerful, there's almost no encapsulation, they can do almost anything, and
that makes it super easy to break it.

I need to think about this more, some kind of protocol of public API for
plugins would probably help.

I agree that having karma e2e test is probably the first step, but lot of
issues were actually in between the karma plugin and webstorm plugin
(caused by changes in Karma). The plugin would work just fine, but not
WebStorm (for instance the last removing --runner-port, the command runs
just fine, but WebStorm does not notice the browser is captured).

Maybe we could bootstrap the webstorm plugin (Java) with some kind of mock
WebStorm and run Karma through it.

On Tue, Jul 30, 2013 at 3:50 AM, Sergey Simonchik
[email protected]:

WebStorm communicates with karma using karma-intellij as a mediator. For a
start, we can test how karma-intellij communicates with karma (without
starting WebStorm IDE). I think that would be a good start because up until
now all incompatibilities were caused by changing in karma API. I think
that changing karma API is inevitable we just have to be informed about
needed changes in karma-intellij.
I'll take a look at karma's e2e tests.

General suggestion: it'd be good to introduce public API for all internals
karma-intellij depends on. That would help a little. But adding tests will
surely help more.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8#issuecomment-21783060
.

[segrey]

If I'm correct, WebStorm plugin (Java) communicates with Karma only to parse webServer port from the line:
INFO [karma]: Karma v0.9.6 server started at http://localhost:9876/

So introducing public API here (karma-runner/karma#614) would eliminate WebStorm plugin (Java) and Karma communication.

I agree that complete e2e tests should involve WebStorm Java part also.

[vojtajina]

@segrey how do you test WebStorm itself ? Or plugins ? Do you have any sort of mock version of webstorm ? Or some way to control WebStorm?

I wanna have at least one high level test, that runs before each Karma release to make sure we don't break this plugin.

[segrey]

We usually mock external dependencies (in this case it's karma itself).
For beginning I'll add e2e tests in karma-intellij.

[vojtajina]

I'd like to test at least three components: karma + karma-intellij plugin +
java part of the plugin.

On Fri, Aug 16, 2013 at 12:56 PM, Sergey Simonchik <[email protected]

wrote:

We usually mock external dependencies (in this case it's karma itself).
For beginning I'll add e2e tests in karma-intellij.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/8#issuecomment-22789906
.