[full-ci] feat(smoke tests): automated parallel smoke tests #5675

fschade · 2021-08-15T16:50:23Z

This PR introduces a first step to have automated smoke tests.

To get a better understanding what that is, what not and how this could be useful pleas read the README.md

It introduces new concepts how and where tests are running.
It's not a final solutions, instead it's just a starting point and needs to be extended if agreed to go that road.

Things that not there yet:

Persona does not include all smoke tests
runs only local, Drone setup is yet not there

update-docs · 2021-08-15T16:50:26Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

ownclouders · 2021-08-15T17:26:40Z

Results for oC10MoveUpload https://drone.owncloud.com/owncloud/web/18295/14/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

ownclouders · 2021-08-15T19:05:14Z

Results for oC10iPhone1 https://drone.owncloud.com/owncloud/web/18296/45/1
The following scenarios passed on retry:

webUIDeleteFilesFolders/deleteFilesFolders.feature:59

ownclouders · 2021-08-15T19:09:58Z

Results for oC10XGAPortrait2 https://drone.owncloud.com/owncloud/web/18296/43/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

phil-davis · 2021-08-16T07:11:42Z

I see that this uses Playwright as the framework for accessing the browser. That moves away from Nightwatch.

We already started making some user-journeys. They are currently in the Nightwatch-based test suite.

It is easy to add whatever scenarios we want to the Nightwatch-based test suites. We can add to the user-journeys, or we can make another test suite for "basic smoke tests for local development" or we can add and choose scenarios anywhere through the test suites and put some tag on them so that they can be easily run.

There is a lot of duplication in having both Nightwatch and Playwright running scenarios that require the same knowledge of how to create a user, group, of the element ids on pages and so on.

What are the thoughts/reasons for moving from Nightwatch to Playwright?

If Playwright has significant advantage compared to Nightwatch, then should we switch all the UI tests to Playwright?

ownclouders · 2021-08-16T13:22:56Z

Results for oC10IntegrationApp2 https://drone.owncloud.com/owncloud/web/18319/66/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

fschade · 2021-08-16T15:43:38Z

I see that this uses Playwright as the framework for accessing the browser. That moves away from Nightwatch.

We already started making some user-journeys. They are currently in the Nightwatch-based test suite.

It is easy to add whatever scenarios we want to the Nightwatch-based test suites. We can add to the user-journeys, or we can make another test suite for "basic smoke tests for local development" or we can add and choose scenarios anywhere through the test suites and put some tag on them so that they can be easily run.

There is a lot of duplication in having both Nightwatch and Playwright running scenarios that require the same knowledge of how to create a user, group, of the element ids on pages and so on.

What are the thoughts/reasons for moving from Nightwatch to Playwright?

If Playwright has significant advantage compared to Nightwatch, then should we switch all the UI tests to Playwright?

Thanks for you input, it started to be a poc and at the end worked out really nice.
Playwright has a totally different concept then selenium of how it is automating the browser.

This has a lot of benefits, for example there are rare cases where a manual wait is needed.
I see this as a POC which eventually can evolve over time. It should not test everything like we do in our mature acceptance test. Instead it's a tiny piece which fills the gap of quick, local testing before the code hits the ci.

ownclouders · 2021-08-19T20:13:40Z

Results for oC10XGAPortrait1 https://drone.owncloud.com/owncloud/web/18463/42/1
The following scenarios passed on retry:

webUIDeleteFilesFolders/deleteFilesFolders.feature:59

ownclouders · 2021-08-19T20:13:49Z

Results for oCISSharingInternalUsers1 https://drone.owncloud.com/owncloud/web/18463/55/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

ownclouders · 2021-09-02T12:24:18Z

Results for oCISLocks https://drone.owncloud.com/owncloud/web/18796/48/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

ownclouders · 2021-09-02T12:24:38Z

Results for oCISFiles2 https://drone.owncloud.com/owncloud/web/18796/53/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

ownclouders · 2021-09-02T12:25:15Z

Results for oCISSharingPublic https://drone.owncloud.com/owncloud/web/18796/61/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

ocis and oc10 bdd smoke tests are now able to with playwright and in parallel, this means running tests with many users at the same time are now possible. Sessions won't conflict, this introduces a whole new level of test possibilities and speed

ownclouders · 2021-09-02T15:22:38Z

Results for oC10iPhone2 https://drone.owncloud.com/owncloud/web/18806/46/1
💥 The acceptance tests pipeline failed. The build has been cancelled.

pascalwengerter · 2021-09-02T15:23:21Z

Rebased and fixed a minor CI issue, in favor of merging this and building on top/updating the (currently WIP/draft documentation)

pascalwengerter

Lovely!

pascalwengerter · 2021-09-02T15:29:13Z

tests/smoke/support/api/http.ts

+}
+
+export const checkResponseStatus = (response: Response, message = ''): void => {
+  // response.status >= 200 && response.status < 300


Dev leftover?

think yes, should be fine to get rid of it

pascalwengerter · 2021-09-02T15:31:16Z

tests/smoke/support/api/http.ts

+import fetch, { BodyInit, Response } from 'node-fetch'
+import { User } from '../types'
+import { config } from '../config'
+import _ from 'lodash'


Why not use lodash-es here?

we could but there it doesnt matter, does not get packed

phil-davis · 2021-09-02T16:41:34Z

https://drone.owncloud.com/owncloud/web/18812/7/8

error [email protected]: The engine "node" is incompatible with this module. Expected version ">= 14.17.0". Got "8.17.0"

That is surprising - this branch is rebased to the latest master.

I suspect that to move to the later CORE_COMMITID we need to use the "new" owncoudci/core:nodejs14 docker image, like is done in core PR owncloud/core#39141

pascalwengerter · 2021-09-02T16:43:36Z

https://drone.owncloud.com/owncloud/web/18812/7/8
error [email protected]: The engine "node" is incompatible with this module. Expected version ">= 14.17.0". Got "8.17.0"
That is surprising - this branch is rebased to the latest master.

From my understanding it lacks the node14-based core img, see latest commit 😉 and error msg in CI https://drone.owncloud.com/owncloud/web/18785/14/9

phil-davis · 2021-09-02T16:46:48Z

@pascalwengerter I updated my post while you were making yours.

This needs changes like in core PR owncloud/core#39141

Should we do a separate PR here in web for that, and not mix it up with this, or?

pascalwengerter · 2021-09-02T16:50:13Z

@pascalwengerter I updated my post while you were making yours.

This needs changes like in core PR owncloud/core#39141

Should we do a separate PR here in web for that, and not mix it up with this, or?

This is where I got my inspiration from! Let's see how CI goes, I'm highly in favor of merging this if it gets green to not waste even more time waiting for it to pass

Edit: The commit + core-nodejs changes are in a separate commit and will show up in the git history accordingly

…mmand

ownclouders · 2021-09-02T17:21:10Z

Results for oC10SharingPublicManagement https://drone.owncloud.com/owncloud/web/18815/34/1
The following scenarios passed on retry:

webUISharingPublicManagement/publicLinkIndicator.feature:27

sonarqubecloud · 2021-09-03T09:29:47Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

fschade added Type:CI Related to our Continouus Integration Solution QA:testing labels Aug 15, 2021

fschade requested review from phil-davis, kulmann, pascalwengerter, LukasHirt and ScharfViktor August 15, 2021 16:50

fschade changed the title ~~feat(smoke tests): Introduce dedicated automated smoke tests~~ feat(smoke tests): automated parallel smoke tests Aug 19, 2021

fschade force-pushed the add-smoke-tests branch from ad2d279 to 962ccfc Compare August 19, 2021 19:26

pascalwengerter mentioned this pull request Aug 30, 2021

Update accounts, idp, settings JS dependencies owncloud/ocis#2420

Merged

pascalwengerter force-pushed the add-smoke-tests branch from 962ccfc to e39516e Compare September 2, 2021 12:10

pascalwengerter force-pushed the add-smoke-tests branch from c06f637 to 94a25b4 Compare September 2, 2021 14:48

pascalwengerter approved these changes Sep 2, 2021

View reviewed changes

pascalwengerter changed the title ~~feat(smoke tests): automated parallel smoke tests~~ [full-ci] feat(smoke tests): automated parallel smoke tests Sep 2, 2021

pascalwengerter force-pushed the add-smoke-tests branch from 94a25b4 to 6784a61 Compare September 2, 2021 15:56

pascalwengerter mentioned this pull request Sep 2, 2021

[full-ci] [Tests-Only] Update commit id #5766

Closed

10 tasks

pascalwengerter force-pushed the add-smoke-tests branch 2 times, most recently from 15a4a71 to a9bae68 Compare September 2, 2021 16:17

pascalwengerter force-pushed the add-smoke-tests branch from a9bae68 to 1373d02 Compare September 2, 2021 16:42

Bump commit IDs & core img with node14 & update cypress in CI yarn co…

8748cef

…mmand

pascalwengerter force-pushed the add-smoke-tests branch from 1373d02 to 8748cef Compare September 2, 2021 16:57

wrap file unlinking in try catch

8dc1a13

pascalwengerter mentioned this pull request Sep 3, 2021

Bump core & ocis commit IDs & core node version in CI #5773

Closed

pascalwengerter merged commit 144706c into master Sep 3, 2021

delete-merged-branch bot deleted the add-smoke-tests branch September 3, 2021 10:18

LukasHirt mentioned this pull request Sep 6, 2021

Not possible to run single scenario after updating cucumber #5781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[full-ci] feat(smoke tests): automated parallel smoke tests #5675

[full-ci] feat(smoke tests): automated parallel smoke tests #5675

fschade commented Aug 15, 2021 •

edited

Loading

update-docs bot commented Aug 15, 2021

ownclouders commented Aug 15, 2021

ownclouders commented Aug 15, 2021

ownclouders commented Aug 15, 2021

phil-davis commented Aug 16, 2021

ownclouders commented Aug 16, 2021

fschade commented Aug 16, 2021

ownclouders commented Aug 19, 2021

ownclouders commented Aug 19, 2021

ownclouders commented Sep 2, 2021

ownclouders commented Sep 2, 2021

ownclouders commented Sep 2, 2021

ownclouders commented Sep 2, 2021

pascalwengerter commented Sep 2, 2021

pascalwengerter left a comment

pascalwengerter Sep 2, 2021

fschade Sep 2, 2021

pascalwengerter Sep 2, 2021

fschade Sep 2, 2021

phil-davis commented Sep 2, 2021 •

edited

Loading

pascalwengerter commented Sep 2, 2021

phil-davis commented Sep 2, 2021

pascalwengerter commented Sep 2, 2021 •

edited

Loading

ownclouders commented Sep 2, 2021

sonarqubecloud bot commented Sep 3, 2021

[full-ci] feat(smoke tests): automated parallel smoke tests #5675

[full-ci] feat(smoke tests): automated parallel smoke tests #5675

Conversation

fschade commented Aug 15, 2021 • edited Loading

update-docs bot commented Aug 15, 2021

ownclouders commented Aug 15, 2021

ownclouders commented Aug 15, 2021

ownclouders commented Aug 15, 2021

phil-davis commented Aug 16, 2021

ownclouders commented Aug 16, 2021

fschade commented Aug 16, 2021

ownclouders commented Aug 19, 2021

ownclouders commented Aug 19, 2021

ownclouders commented Sep 2, 2021

ownclouders commented Sep 2, 2021

ownclouders commented Sep 2, 2021

ownclouders commented Sep 2, 2021

pascalwengerter commented Sep 2, 2021

pascalwengerter left a comment

Choose a reason for hiding this comment

pascalwengerter Sep 2, 2021

Choose a reason for hiding this comment

fschade Sep 2, 2021

Choose a reason for hiding this comment

pascalwengerter Sep 2, 2021

Choose a reason for hiding this comment

fschade Sep 2, 2021

Choose a reason for hiding this comment

phil-davis commented Sep 2, 2021 • edited Loading

pascalwengerter commented Sep 2, 2021

phil-davis commented Sep 2, 2021

pascalwengerter commented Sep 2, 2021 • edited Loading

ownclouders commented Sep 2, 2021

sonarqubecloud bot commented Sep 3, 2021

fschade commented Aug 15, 2021 •

edited

Loading

phil-davis commented Sep 2, 2021 •

edited

Loading

pascalwengerter commented Sep 2, 2021 •

edited

Loading