Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spectron Smoketest #25291

Closed
29 tasks done
egamma opened this issue Apr 25, 2017 · 2 comments
Closed
29 tasks done

Spectron Smoketest #25291

egamma opened this issue Apr 25, 2017 · 2 comments
Assignees
@michelkaporin
Labels
engineering VS Code - Build / issue tracking / etc. vscode-build VS Code build process issues
Milestone
May 2017

Comments

@egamma
Copy link
Member

egamma commented Apr 25, 2017 •
edited
Loading

  • Fully automate parts of smoke test on Windows - npm install & run a script. Anybody in team can run it.
  • Explore visual review by talking screenshots
  • Can run smoke test against insiders/stable
  • Support to run smoketest during integration builds/CI (moved to Build definitions for smoke test #27845)
  • Support incremental wait timeouts on test failures
  • Make tests more robust by adding 'test classes' to DOM elements, add data attributes (investigated)
  • Implement the full smoke test
    • Data Migration
    • Data Loss
    • First User Experience
    • Explorer
    • Configuration and views
    • Search
    • CSS
    • Markdown (skipped, webview has nodeIntegration == false)
    • JavaScript
    • Debugging JavaScript (no debug run-time test)
    • Git
    • Integrated Terminal
    • Status bar
    • Tasks
    • Extensions
    • Accessibility (skipped due to no voice-over coverage from Spectron)
    • Localization
  • Support to run smoke test against fresh user data and populated user data
  • Contributing.md that documents how to contribute a new smoke test
  • Run on multiple platforms
  • Support Insiders (e.g. window title checks)
  • Explore smarter addressing of editor span tags

Issues

Unresolvable Issues

  • Faking native dialogs - context menu, dialog popups (e.g. in getting asked for save on dirty editor close)
@egamma egamma added the vscode-build VS Code build process issues label Apr 25, 2017
@egamma egamma added this to the May 2017 milestone Apr 25, 2017
@egamma egamma added the debt Code quality issues label Apr 25, 2017
@egamma egamma changed the title Spectron Exploration Spectron Smoketest May 3, 2017
@egamma egamma added engineering VS Code - Build / issue tracking / etc. and removed debt Code quality issues labels May 4, 2017
@kieferrm kieferrm mentioned this issue May 5, 2017
Closed
44 tasks
@michelkaporin michelkaporin mentioned this issue May 9, 2017
Closed
michelkaporin added a commit that referenced this issue May 9, 2017
@michelkaporin
Added handling for multiple '--user-data-dir' to enable Spectron smok…
f3ca4f6 
…e testing in #25291.
@michelkaporin michelkaporin mentioned this issue May 10, 2017
Closed
michelkaporin added a commit that referenced this issue May 19, 2017
@michelkaporin
Removed handling multiple '--user-data-dir' as no need for Spectron a…
e1bb941 
…ny more #25291.
@michelkaporin
Copy link
Contributor

michelkaporin commented May 19, 2017 •
edited
Loading

Looked into Make tests more robust by adding 'test classes' to DOM elements, add data attributes task in the list above and came to a conclusion that it will not make them more robust because we do not often change CSS selectors in our code for the key elements that are tested in the Smoke Test.

If we see that we do, or start doing changing them more often going forward, we can add those. At the current stage it seems an overhead. In terms of the work on the smoke test code it will be only replacing currently used selectors with another ones, however VS Code core code will grow without any big reason.

Update: #27906

This was referenced May 29, 2017
Closed
Merged
Closed
@michelkaporin
Copy link
Contributor

The implementation has been merged into master in #2784, hence closing the issue.

@michelkaporin michelkaporin mentioned this issue Jun 2, 2017
Closed
@vscodebot vscodebot bot locked and limited conversation to collaborators Nov 18, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@michelkaporin michelkaporin

Labels
engineering VS Code - Build / issue tracking / etc. vscode-build VS Code build process issues
Projects
None yet
Milestone
May 2017
Development

No branches or pull requests
2 participants
@egamma @michelkaporin