CP-2272 Implemented task that allows a user to run browser based dart unit tests on saucelabs #148

jayudey-wf · 2016-04-05T14:00:27Z

Issue

There was no clear path that would allow a user to run their browser based unit tests across multiple browser configurations

Changes

Source:

Utilized sauce runner developed by @greglittlefield-wf

Tests:

n/a (there is no real clear way to test this)

Areas of Regression

n/a (separate task)

Testing

Install into a project with browser based tests and verify that browser tests can be run on saucelabs

Code Review

@trentgrover-wf
@maxwellpeterson-wf
@evanweible-wf
@dustinlessard-wf

greglittlefield-wf · 2016-04-08T01:20:11Z

README.md

@@ -26,6 +26,7 @@ development requirements:
 - Documentation generation
 - Examples for manual testing/exploration
 - Applying a LICENSE file to all source files
+- Running dart unit tests on saucelabs


#nit

s/saucelabs/Sauce Labs

greglittlefield-wf · 2016-04-08T01:35:38Z

Done with first pass. So glad to have this functionality in a common place!

dustinlessard-wf · 2016-04-08T16:22:01Z

I want this now! This is going to be awesome!

codecov-io · 2016-04-08T18:47:54Z

Current coverage is 23.56% (diff: 100%)

Merging #148 into master will not change coverage

@@             master       #148   diff @@
==========================================
  Files             7          7          
  Lines           157        157          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits             37         37          
  Misses          120        120          
  Partials          0          0

Powered by Codecov. Last update a2748c8...8ead89a

rmconsole2-wf · 2016-04-12T14:23:14Z

@jayudey-wf This pull request has merge conflicts, please resolve.

jayudey-wf · 2016-04-15T16:20:42Z

@greglittlefield-wf this is ready for re-review

greglittlefield-wf · 2016-04-29T23:44:50Z

@jayudey-wf Sorry that this fell off my radar! Looks great.

+1

dustinlessard-wf · 2016-04-30T00:26:04Z

+1 Let's do this!

trentgrover-wf · 2016-05-10T21:40:47Z

+1 - let's get this moving

1 random thought I had: would anyone be opposed to making the command just saucelabs instead of saucelabs-tests? It's possible in the future that we would have multiple saucelabs related tasks / configurations that we could root here

dustinlessard-wf · 2016-05-11T13:22:48Z

+1 to this change
+1 to naming it simply 'saucelabs'
+1 to a passing ci

evanweible-wf · 2016-05-11T18:43:08Z

lib/src/tasks/saucelabs_tests/cli.dart

+          ' please review the output above and the test report located at'
+          ' ${config.saucelabsTests.testReportPath}.');
+    } else {
+      return new CliResult.success('Success, your tests completely successfully'


evanweible-wf · 2016-05-11T18:57:24Z

I agree with the change to the shorter saucelabs

rmconsole-wf · 2016-05-12T02:01:50Z

@jayudey-wf This pull request has merge conflicts, please resolve.

evanweible-wf · 2016-05-12T21:02:45Z

QA

Manual Testing

task fails and warns about sauce credentials when they are missing
task fails and warns when missing files to test
task fails and warns when the file to test is not an HTML file
- It would be nice if it could automatically generate an HTML file like the test task does

task successfully runs tests on saucelabs

The tests to run and complete on saucelabs, but I see the following issues:

I'm seeing this in the dart2js logs:

[Warning from Dart2JS]:
test/integration/browser_integration_test.dart.sauce_browser_test.dart:44:11:
'SauceBrowserEnvironment' doesn't implement the getter 'supportsDebugging' declared in 'Environment'.
Try adding an implementation of 'supportsDebugging' or declaring 'SauceBrowserEnvironment' to be 'abstract'.
          class SauceBrowserEnvironment implements Environment {
          ^^^^^

saw this during the shutdown:

Error quitting driver: InvalidRequestException (404): ERROR The test with session id 49dacf525c3c4e4cb0412488d579ae6d has already finished, and can't receive further commands.
You can learn more at https://saucelabs.com/jobs/49dacf525c3c4e4cb0412488d579ae6d
For help, please check https://docs.saucelabs.com/reference/troubleshooting-common-error-messages

at the end, the task RTE'd with this:

Unhandled exception:
The null object does not have a method '[]'.

NoSuchMethodError: method not found: '[]'
Receiver: null
Arguments: ["failed"]
#0      Object._noSuchMethod (dart:core-patch/object_patch.dart:42)
#1      Object.noSuchMethod (dart:core-patch/object_patch.dart:45)
#2      SauceRunnerCli.run.<run_async_body> (package:dart_dev/src/tasks/saucelabs/cli.dart:81:43)
#3      _RootZone.runUnary (dart:async/zone.dart:1137)
#4      _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:553)
#5      _Future._propagateToListeners (dart:async/future_impl.dart:639)
#6      _Future._complete (dart:async/future_impl.dart:416)
#7      _SyncCompleter.complete (dart:async/future_impl.dart:52)
#8      run.<run_async_body> (package:dart_dev/src/tasks/saucelabs/sauce_runner.dart:126:5)
#9      _RootZone.runUnary (dart:async/zone.dart:1137)
#10     _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:553)
#11     _Future._propagateToListeners (dart:async/future_impl.dart:639)
#12     _Future._completeWithValue (dart:async/future_impl.dart:426)
#13     _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:481)
#14     _microtaskLoop (dart:async/schedule_microtask.dart:41)
#15     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50)
#16     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#17     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)

additionally, the tests are not showing is saucelabs up as successful/failed after completing

this can be a future improvement (see CP-2118 Report build status to Sauce Labs #154)

Manual CI

ddev format
ddev analyze
ddev test --integration

evanweible-wf · 2016-05-20T02:54:59Z

@jayudey-wf once you resolve merge conflicts, I'll put my QA stamp on it.

evanweible-wf · 2016-05-20T03:32:27Z

@jayudey-wf actually, I just found one issue. The sauceConnectTunnelIdentifier config option doesn't work - when you set it, the task fails with this:

Starting tests...
Failure starting Chrome on Windows - saucelabs_test.html (null).
UnknownException (13): Sauce could not start your job. For more information on what happened, please visit https://saucelabs.com/jobs/d285b29ffb9f473796d4dac26587bcf2

Waiting for tests to complete... (See https://saucelabs.com/tests for overall status.)
Error quitting driver: The null object does not have a method 'then'.

NoSuchMethodError: method not found: 'then'
Receiver: null
Arguments: [Closure: (dynamic) => dynamic]
- Finished: Chrome on Windows - saucelabs_test.html (null)

Test Results:
- Chrome on Windows - saucelabs_test.html: test error

Cleaning up...
- Shutting down Pub server...
Unhandled exception:
The null object does not have a method '[]'.

NoSuchMethodError: method not found: '[]'
Receiver: null
Arguments: ["failed"]
#0      Object._noSuchMethod (dart:core-patch/object_patch.dart:42)
#1      Object.noSuchMethod (dart:core-patch/object_patch.dart:45)
#2      SauceRunnerCli.run.<run_async_body> (package:dart_dev/src/tasks/saucelabs/cli.dart:81:43)
#3      _RootZone.runUnary (dart:async/zone.dart:1137)
#4      _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:553)
#5      _Future._propagateToListeners (dart:async/future_impl.dart:639)
#6      _Future._complete (dart:async/future_impl.dart:416)
#7      _SyncCompleter.complete (dart:async/future_impl.dart:52)
#8      run.<run_async_body> (package:dart_dev/src/tasks/saucelabs/sauce_runner.dart:126:5)
#9      _RootZone.runUnary (dart:async/zone.dart:1137)
#10     _Future._propagateToListeners.handleValueCallback (dart:async/future_impl.dart:553)
#11     _Future._propagateToListeners (dart:async/future_impl.dart:639)
#12     _Future._completeWithValue (dart:async/future_impl.dart:426)
#13     _Future._asyncComplete.<anonymous closure> (dart:async/future_impl.dart:481)
#14     _microtaskLoop (dart:async/schedule_microtask.dart:41)
#15     _startMicrotaskLoop (dart:async/schedule_microtask.dart:50)
#16     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:96)
#17     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:149)

evanweible-wf · 2016-05-20T03:34:12Z

Also, as a side note, the requirement of a transformer means that this task can't be used by any projects that are also a dependency of dart_dev. That's fairly low risk, I just happened to run into it trying to test this out with fluri, but since dart_dev now depends on fluri, there's a circular dependency. Normally that's fine, but with transformers it's not allowed.

evanweible-wf · 2016-06-15T18:04:54Z

@rmconsole-wf

trentgrover-wf · 2016-07-05T14:28:48Z

+1

rmconsole-wf · 2016-08-02T02:01:16Z

@jayudey-wf This pull request has merge conflicts, please resolve.

- updated documentation - renamed saucelabs-tests to saucelabs - utilized TaskProcess - reducing default browser config - added implementation of supportsDebugging - updating logic around using existing sauce tunnels and tunnel identifiers - better error handling when a test errors on sauce labs

evanweible-wf · 2016-09-21T19:10:15Z

lib/src/tasks/saucelabs/cli.dart

+  Future<CliResult> run(ArgResults parsedArgs, {bool color: true}) async {
+    if (sauceUserName == null || sauceAccessKey == null) {
+      return new CliResult.fail('Sauce Labs credentials must be available via '
+          'the `SAUCE_ACCESS_KEY` and `SAUCE_USERNAME` SauceRunnerConfig instance.');


This error message is confusing - it sounds like the key and username should be available on an instance of some Dart object, but it is actually looking for them from the shell environment via dart:io.

maxwellpeterson-wf · 2016-09-21T19:44:05Z

QA +10

task fails and warns about sauce credentials when they are missing
task fails and warns when missing files to test
task fails and warns when the file to test is not an HTML file
- it would be nice if the task could generate a default HTML file if a .dart file is given
- currently this failure results in an uncaught exception and the stack trace gets printed to the console - it would be better to just print the error and exit early
task successfully runs tests on saucelabs
- w_session tests passed with the default configuration of just Chrome
- w_session tests passed with a custom set of platforms (firefox + IE11)

evanweible-wf · 2016-09-22T13:30:10Z

+1 to the last commit

maxwellpeterson-wf · 2016-09-22T13:37:21Z

+10

see previous comment

trentgrover-wf · 2016-09-22T15:17:10Z

+1

evanweible-wf mentioned this pull request Apr 5, 2016

Implement saucelab-tests task #147

Closed

greglittlefield-wf reviewed Apr 8, 2016
View reviewed changes

rmconsole2-wf added the Merge Conflicts label Apr 12, 2016

rmconsole2-wf removed the Merge Conflicts label Apr 14, 2016

jayudey-wf force-pushed the saucelab_tests_task branch 2 times, most recently from a6bcdf4 to 18a6355 Compare April 15, 2016 15:30

evanweible-wf reviewed May 11, 2016
View reviewed changes

rmconsole-wf added the Merge Conflicts label May 12, 2016

rmconsole-wf removed the Merge Conflicts label May 12, 2016

evanweible-wf removed the testing in progress label May 12, 2016

evanweible-wf modified the milestone: 1.3.0 Jun 10, 2016

evanweible-wf added the v-minor label Jun 10, 2016

trentgrover-wf changed the title ~~Implemented task that allows a user to run browser based dart unit tests on saucelabs~~ CP-2272 Implemented task that allows a user to run browser based dart unit tests on saucelabs Aug 1, 2016

rmconsole-wf added Merge Conflicts and removed Merge Conflicts labels Aug 2, 2016

greglittlefield-wf and others added 4 commits September 21, 2016 11:42

Move Sauce Labs runner files over from web_skin_dart

570b6a8

Create saucelab-tests task

c715884

Fix Firefox test failures in Sauce Labs due to window not being active

5784dfb

evanweible-wf force-pushed the saucelab_tests_task branch from 845acdb to d3893d7 Compare September 21, 2016 19:34

evanweible-wf reviewed Sep 21, 2016

View reviewed changes

Fix readme typo. Clarify error message.

b764947

widen test dependency for test_fixture

8ead89a

trentgrover-wf merged commit 76d2cd0 into master Sep 22, 2016

maxwellpeterson-wf deleted the saucelab_tests_task branch September 22, 2016 15:17

rmconsole2-wf mentioned this pull request Sep 22, 2016

CP-2661 Release dart_dev 1.5.0 #182

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CP-2272 Implemented task that allows a user to run browser based dart unit tests on saucelabs #148

CP-2272 Implemented task that allows a user to run browser based dart unit tests on saucelabs #148

jayudey-wf commented Apr 5, 2016

greglittlefield-wf Apr 8, 2016

greglittlefield-wf commented Apr 8, 2016

dustinlessard-wf commented Apr 8, 2016

codecov-io commented Apr 8, 2016 •

edited

Loading

rmconsole2-wf commented Apr 12, 2016

jayudey-wf commented Apr 15, 2016

greglittlefield-wf commented Apr 29, 2016

dustinlessard-wf commented Apr 30, 2016

trentgrover-wf commented May 10, 2016

dustinlessard-wf commented May 11, 2016

evanweible-wf May 11, 2016

evanweible-wf commented May 11, 2016

rmconsole-wf commented May 12, 2016

evanweible-wf commented May 12, 2016 •

edited

Loading

evanweible-wf commented May 20, 2016

evanweible-wf commented May 20, 2016

evanweible-wf commented May 20, 2016

evanweible-wf commented Jun 15, 2016

trentgrover-wf commented Jul 5, 2016

rmconsole-wf commented Aug 2, 2016

evanweible-wf Sep 21, 2016

evanweible-wf Sep 21, 2016

maxwellpeterson-wf commented Sep 21, 2016

evanweible-wf commented Sep 22, 2016

maxwellpeterson-wf commented Sep 22, 2016

trentgrover-wf commented Sep 22, 2016

CP-2272 Implemented task that allows a user to run browser based dart unit tests on saucelabs #148

CP-2272 Implemented task that allows a user to run browser based dart unit tests on saucelabs #148

Conversation

jayudey-wf commented Apr 5, 2016

Issue

Changes

Areas of Regression

Testing

Code Review

greglittlefield-wf Apr 8, 2016

Choose a reason for hiding this comment

greglittlefield-wf commented Apr 8, 2016

dustinlessard-wf commented Apr 8, 2016

codecov-io commented Apr 8, 2016 • edited Loading

Current coverage is 23.56% (diff: 100%)

rmconsole2-wf commented Apr 12, 2016

jayudey-wf commented Apr 15, 2016

greglittlefield-wf commented Apr 29, 2016

dustinlessard-wf commented Apr 30, 2016

trentgrover-wf commented May 10, 2016

dustinlessard-wf commented May 11, 2016

evanweible-wf May 11, 2016

Choose a reason for hiding this comment

evanweible-wf commented May 11, 2016

rmconsole-wf commented May 12, 2016

evanweible-wf commented May 12, 2016 • edited Loading

QA

Manual Testing

Manual CI

evanweible-wf commented May 20, 2016

evanweible-wf commented May 20, 2016

evanweible-wf commented May 20, 2016

evanweible-wf commented Jun 15, 2016

trentgrover-wf commented Jul 5, 2016

rmconsole-wf commented Aug 2, 2016

evanweible-wf Sep 21, 2016

Choose a reason for hiding this comment

evanweible-wf Sep 21, 2016

Choose a reason for hiding this comment

maxwellpeterson-wf commented Sep 21, 2016

QA +10

evanweible-wf commented Sep 22, 2016

maxwellpeterson-wf commented Sep 22, 2016

trentgrover-wf commented Sep 22, 2016

codecov-io commented Apr 8, 2016 •

edited

Loading

evanweible-wf commented May 12, 2016 •

edited

Loading