Add playwright tests job

[taghreed86]

What changed?
Just adding a new job in deploy.yaml file and this job triggers the Playwright-tests workflow and run it

You can check the run status here and this is the Playwright tests run.

This PR contribute towards #3520 especially for the Regression Test section as after merge this PR you can run the already added playwright acceptance tests against each commit into your branch or into main by just select the Deploy workflow to run and choose from which branch you want to run it as in the below screen shot. This way will give you an indication if your new feature will broke something or not by checking the playwright acceptance tests result after running the workflow.

[enekofb]

please @taghreed86 could you add in the PR description how this Pr contributes towards this ticket #3520

For example, if aim to solve the regression journey I think that this job does not uses the code from weave gitops enterprise repo.

[taghreed86]

please @taghreed86 could you add in the PR description how this Pr contributes towards this ticket #3520

For example, if aim to solve the regression journey I think that this job does not uses the code from weave gitops enterprise repo.

Done

[enekofb]

I think that the unique question that I have is about the expectations and error handling scenario for a failing scenario like the following

As a developer, i have merged the PR intro enterprise
Then smoke tests run passing but playwright tests fails

Doubts from my side are:

• do we expect this workflow to fail ?
• would the playwright-smoke-tests job be marked as green?
• in case that is marked as green, given that we want to understand the stability of old vs new test suite, how we are going to understand in quantitative terms?

I think the ideal behavior is:

• if playwright tests fail we are able to see a red failure (because we want to use it for measuring stability vs current smoke tests).
• deploy workflow does not fail cause developers are not expected to recover from this situation (as we are doing PoC).

[taghreed86]

I think that the unique question that I have is about the expectations and error handling scenario for a failing scenario like the following

As a developer, i have merged the PR intro enterprise Then smoke tests run passing but playwright tests fails

Doubts from my side are:

• do we expect this workflow to fail ?
• would the playwright-smoke-tests job be marked as green?
• in case that is marked as green, given that we want to understand the stability of old vs new test suite, how we are going to understand in quantitative terms?

I think the ideal behavior is:

• if playwright tests fail we are able to see a red failure (because we want to use it for measuring stability vs current smoke tests).
• deploy workflow does not fail cause developers are not expected to recover from this situation (as we are doing PoC).

1. We expect the playwright- tests workflow to Pass when a developer merged something into main branch of weve-gitops-enterprise repo unless the change that the developer merged break something in the workflows that we covered in the playwright tests.
2. If the Playwright tests fail in this case the playwright job will mark as Red.
3. If the playwright tests fail in this case the deploy workflow will fail and from my point of view that's what we need to give the developer indication that there is something wrong in the workflows that the playwright tests are covered were working well and the changes he made leads to the tests become failed.

[enekofb]

I think that the unique question that I have is about the expectations and error handling scenario for a failing scenario like the following
As a developer, i have merged the PR intro enterprise Then smoke tests run passing but playwright tests fails
Doubts from my side are:

• do we expect this workflow to fail ?
• would the playwright-smoke-tests job be marked as green?
• in case that is marked as green, given that we want to understand the stability of old vs new test suite, how we are going to understand in quantitative terms?

I think the ideal behavior is:

• if playwright tests fail we are able to see a red failure (because we want to use it for measuring stability vs current smoke tests).
• deploy workflow does not fail cause developers are not expected to recover from this situation (as we are doing PoC).

1. We expect the playwright- tests workflow to Pass when a developer merged something into main branch of weve-gitops-enterprise repo unless the change that the developer merged break something in the workflows that we covered in the playwright tests.
2. If the Playwright tests fail in this case the playwright job will mark as Red.
3. If the playwright tests fail in this case the deploy workflow will fail and from my point of view that's what we need to give the developer indication that there is something wrong in the workflows that the playwright tests are covered were working well and the changes he made leads to the tests become failed.

Yes, that makes sense to me beyond the PoC stage. However, for this initial stage, given that we dont really have real datapoints that the workflow wont produce false positives (this is one of the elements to validate), it feels that we should favour not getting in the way of the developer workflow.

I would consider in this poc stage to run it with something like continue-on-error https://github.com/weaveworks/weave-gitops-enterprise/pull/3595/files#diff-07e563e9e96c4e400b69036f9a4ac603dea520acbef2f83237c744ea2ef383e0R173

Then once we have passed the poc stage, to remove it.

Let me know your thoughts.

[taghreed86]

I think that the unique question that I have is about the expectations and error handling scenario for a failing scenario like the following
As a developer, i have merged the PR intro enterprise Then smoke tests run passing but playwright tests fails
Doubts from my side are:

• do we expect this workflow to fail ?
• would the playwright-smoke-tests job be marked as green?
• in case that is marked as green, given that we want to understand the stability of old vs new test suite, how we are going to understand in quantitative terms?

I think the ideal behavior is:

• if playwright tests fail we are able to see a red failure (because we want to use it for measuring stability vs current smoke tests).
• deploy workflow does not fail cause developers are not expected to recover from this situation (as we are doing PoC).

1. We expect the playwright- tests workflow to Pass when a developer merged something into main branch of weve-gitops-enterprise repo unless the change that the developer merged break something in the workflows that we covered in the playwright tests.
2. If the Playwright tests fail in this case the playwright job will mark as Red.
3. If the playwright tests fail in this case the deploy workflow will fail and from my point of view that's what we need to give the developer indication that there is something wrong in the workflows that the playwright tests are covered were working well and the changes he made leads to the tests become failed.

Yes, that makes sense to me beyond the PoC stage. However, for this initial stage, given that we dont really have real datapoints that the workflow wont produce false positives (this is one of the elements to validate), it feels that we should favour not getting in the way of the developer workflow.

I would consider in this poc stage to run it with something like continue-on-error https://github.com/weaveworks/weave-gitops-enterprise/pull/3595/files#diff-07e563e9e96c4e400b69036f9a4ac603dea520acbef2f83237c744ea2ef383e0R173

Then once we have passed the poc stage, to remove it.

Let me know your thoughts.

@enekofb Done, please check the last run

[enekofb]

looks good, i think we just need to remove the need from smoke-test-results: to approve it

[enekofb]

lets go!