Test: Automated Smoke Test

[michelkaporin]

Ref: #25291
Complexity: 2

OS:

• Windows @alexandrudima
• OS X @weinand
• Linux @Tyriar

As part of engineering work we have automated major part of our smoke test. To ensure its stability, it is important to run it on different machines.

Running automated test

Prerequisites

1. Update stable to the latest version.
2. Close all VS Code instances that will be used in a test (e.g. latest insiders, stable) not to have any interference with the running test that will spawn them itself.

Once the test is running, please do not interfere with it by doing selection or focusing anywhere within spawned test VS Code instance.

Approximate running time

10 minutes minimum (the more failures it has, the more it runs due to adaptive retry strategy of the waiting time).

Procedure

1. Run git checkout michelkaporin/smoketest in vscode repository. Currently the code lives on my branch only.
2. cd test/smoke
3. npm install
4. npm test -- --latest "path/to/binary" --stable "path/to/binary", where latest argument is the path to VS Code executable to conduct testing, and stable is previous stable version (in order to run 'Data Migration' tests). If the latter argument is ommited, 'Data Migration' tests won't run.

Example commands:
npm test -- --latest "C:\Program Files (x86)\Microsoft VS Code Insiders\Code - Insiders.exe" --stable "C:\Program Files (x86)\Microsoft VS Code\Code.exe"
or
npm test -- --latest "/Applications/Visual Studio Code - Insiders.app/Contents/MacOS/Electron" --stable "/Applications/Visual Studio Code.app/Contents/MacOS/Electron"

Known problems

• **Extension host does not start when running Insiders with empty user-data-dir on OS X #27452 bug results in very many failing tests on OS X, once that is fixed, all should pass.
• Spectron spawns multiple cmd windows on Windows OS. This leads to VS Code window to be overlapped by another one. To look at what the test does you need to click on the VS Code title bar to focus on it in every test. (Empty terminals display on Windows electron-userland/spectron#60)

If you get failing tests, please dump the log with failed error messages here.
Any feedback is appreciated 👍

[weinand]

@michelkaporin is there a workaround for #27452 on macOS?

[michelkaporin]

@weinand yes, sorry for not providing initially. You can use the build of 921107c commit (one week old) instead of latest Insiders.

[weinand]

@michelkaporin using the old build does not work because it updates automatically and changing the "update.channel" to "none" has no effect (because the test is probably specifying its own user data folder).
I've created #27556 for this.

[michelkaporin]

@weinand Sorry to hear this. I was lucky and did not get any update prompts when running the test over old version. Would you be able to run again once #27452 is resolved please?

[weinand]

@michelkaporin there are no "update prompts". But when restarting VS Code you will get the new version automatically. And I think the test suite restarts VS Code for every test, right?

[michelkaporin]

@weinand Test suite is restarting VS Code for every test, although I am not sure how the update is applied inbetween test cases by Squirrel for Mac. If it launches already updated binary second time (i.e. on the second test case), then you should not get any test failures that are similar to what you had with the current latest Insiders.

By the way latest Insiders contains the fix for #27452 and ready to be tested.

[weinand]

With the fix for #27452 all but one test succeed on macOS:

Running tests...




  Smoke Test Suite

    Data Migration

      ✓ checks if the Untitled file is restored migrating from stable to latest

      ✓ checks if the newly created dirty file is restored migrating from stable to latest

      ✓ cheks if opened tabs are restored migrating from stable to latest

    Data Loss

      ✓ verifies that 'hot exit' works for dirty files

      1) verifies that contents of the dirty files are restored after 'hot exit'

    Explorer

      ✓ quick open search produces correct result

      ✓ quick open respects fuzzy matching

    Configuration and views

      ✓ turns off editor line numbers and verifies the live change

      ✓ changes 'workbench.action.toggleSidebarPosition' command key binding and verifies it (28496ms)

    Search

      ✓ searches for body & checks for correct result number

      ✓ searches only for *.js files & checks for correct result number

      ✓ dismisses result & checks for correct result number

      ✓ replaces first search result with a replace term

    CSS

      ✓ verifies quick outline

      ✓ verifies warnings for the empty rule

      ✓ verifies that warning becomes an error once setting changed

    JavaScript

      ✓ shows correct quick outline

      ✓ finds 'All References' to 'app'

      ✓ renames local 'app' variable

      ✓ folds/unfolds the code correctly

      ✓ verifies that 'Go To Definition' works

      ✓ verifies that 'Peek Definition' works

    Debugging JavaScript

      ✓ autodetects program attribute for launch.json

      ✓ can set a breakpoint and verify if it's set

    Git

      ✓ verifies current changes are picked up by Git viewlet

      ✓ verifies 'app.js' diff viewer changes

      ✓ stages 'app.js' changes and checks stage count

      ✓ stages, commits change to 'app.js' locally and verifies outgoing change

    Integrated Terminal

      ✓ opens terminal, runs 'echo' and verifies the output

    Status Bar

      ✓ verifies presence of all default status bar elements

      ✓ verifies that 'quick open' opens when clicking on 'Branch', 'Indentation Status, 'Encoding', 'EOL' and 'Language' status elements

      ✓ verifies that 'Problems View' appears when clicking on 'Problems' status element

      ✓ verifies that 'Tweet us feedback' pop-up appears when clicking on 'Feedback' icon

      ✓ checks if 'Go to Line' works if called from the status bar

      ✓ verifies if changing EOL is reflected in the status bar

    Tasks

      ✓ verifies that eslint task results in 1 problem

      ✓ is able to select 'Git' output

      ✓ ensures that build task produces errors in index.js

      ✓ verifies build errors are reflected in 'Problems View'

    Localization

      ✓ starts with 'DE' locale and verifies title and viewlets text is in German



  Extensions

    ✓ installs 'vscode-icons' extension and verifies reload is prompted

    ✓ installs an extension and checks if it works on restart





  41 passing (11m)

  1 failing



  1) Smoke Test Suite Data Loss verifies that contents of the dirty files are restored after 'hot exit':

      AssertionError: 'Hello, Code' == 'Hello, Code!'
      + expected - actual

      -Hello, Code
      +Hello, Code!
      
      at Context.<anonymous> (out/tests/data-loss.js:76:24)
      at Generator.next (<anonymous>)
      at fulfilled (out/tests/data-loss.js:8:58)
      at process._tickDomainCallback (internal/process/next_tick.js:129:7)




npm ERR! Test failed.  See above for more details.

[michelkaporin]

@weinand thank you. Can you tell me if you have any extensions installed in your Insiders build that you've tested? If yes, can you please list them here.

I cannot reproduce the test failure, suspecting that extensions could have caused it by changing the DOM of the editor.

[weinand]

@michelkaporin I've only the ESlint and TSLint extensions installed.