[BUG]Visual Regression Test screenshots differ from each pipeline run

[badigerpramod]

System info

• Playwright Version: [v1.32.3]
• Operating System: [ubuntu-20.04]
• Browser: [Chromium]
• Other info:

Source code

• I provided exact source code that allows reproducing the issue locally.

implementing based on https://www.npmjs.com/package/@web/test-runner-visual-regression

[badigerpramod]

Previously had version 1.22.1 version of playwright, everything worked very well, tried updating 1.31.1 and 1.32.3 versions then started getting different results for different runs

[aslushnikov]

This is known issue; browsers do not have deterministic rendering. You can try experimental comparator that was designed
to fight some of these:

#13873 (comment)

If you do try it, then please let us know how it goes. Thanks!

[badigerpramod]

The problem with our approach is that, we are comparing the images very strictly without any threshold and other observations are,

our test works without any issues from 1.22.0 1.23.0 1.24.0 then issue started coming from 1.25.0 to 1.33.0, there always disturbance in the corners of the component like above screenshots, that too only in Chromium, do we need to set any property or adjust the chromium here?

[aslushnikov]

@badigerpramod someone reported some success with a bunch of extra flags, however in my experiments they didn't help much.

In general, Chromium does not guarantee consistent rendering for the same inputs, so you should be prepared that some pixels might be different.

[badigerpramod]

@aslushnikov i found solution for the problem, somehow chrome was too fast to run the tests this is where screenshots were wrongly taken/compared, by adding 100ms or required delay before each test solved the issue for me

beforeEach(async () => {
      await new Promise((resolve) => setTimeout(resolve, 100));
});

Now I am getting same results every run!

Thank you

[sijakret]

@badigerpramod someone reported some success with a bunch of extra flags, however in my experiments they didn't help much.

In general, Chromium does not guarantee consistent rendering for the same inputs, so you should be prepared that some pixels might be different.

@aslushnikov do you have any references for this?
I understand that there might be differences when using different environments (like different os, different rendering hardware, different drivers, etc..).
However, somehow I have doubts that rendering on an identical environment (same os, hardware, driver, etc..) is not deterministic. That would also make it kind of hard for chrome itself to be tested.