feat(autoedit): Add more E2E test scenarios #6573
Conversation
|
Going to iterate on these for additional telemetry testing but will do so in a follow up branch |
| width: 1024, | ||
| height: 741, | ||
| width: 1920, | ||
| height: 1080, |
There was a problem hiding this comment.
nit: Curious if we chaging the view port results in some improvements. I set these values arbitrary but can we add a small comment if we think there is any preferred view port size.
There was a problem hiding this comment.
It makes it possible to see the diff on some lines (e.g. a line where there are two separate insertion decorations)
I'll leave a comment here :)
| }) | ||
|
|
||
| it.only('first hunk', () => { | ||
| expect(citationSubstring(`hello("world") // comment`, `hello("world", 32, "goodbye")`)).toStrictEqual(`hello("world")`) |
There was a problem hiding this comment.
@umpox I this this example creates a very small ghost text and when I was testing it last time, I observed that such small diffs are not detected because we set certain maxDiffPixelRatio: 0.02 in the snapshot testing.
So, just wanted to cross check if we are actually able to detect such minor changes in the testing. (for eg: if I just replace the image with no decoration as ground truth, will the test still pass ?)
There was a problem hiding this comment.
Thanks for raising @hitesh-1997. This test fails if I remove the decoration, but I did notice two of the tests still pass.
I've adjusted the maxDiffPixels from 1000 to 500, now all tests fail if I remove the decoration. I think we'll have to keep an eye out for this in future so will leave some comments.
Tests seem stable on my machine, but running 100 times in CI to check for any flakiness
There was a problem hiding this comment.
That's awesome !!
Thanks for looking into this Tom, approving to unblock.
There was a problem hiding this comment.
This is so awesome !! Thanks for adding these test cases @umpox !!
Mostly changes looks good to me, just wanted to check if are are actually detecting regression in certain tests (particularly the ones which have very small diff) or will they just pass by default even if they are broken ?
umpox
force-pushed
the
tr/e2e-autoedits
branch
from
709e3ea to
f5eb8af
Compare
umpox
force-pushed
the
tr/e2e-autoedits
branch
from
f5eb8af to
8ba5be3
Compare
umpox
force-pushed
the
tr/e2e-autoedits
branch
from
8ba5be3 to
d898e40
Compare
| * Linear issue: https://linear.app/sourcegraph/issue/CODY-4650/fix-rendering-issues-with-suffix-suggestions-at-the-end-of-a-file | ||
| * Expected behaviour: We do not show any suggestion, as there is not enough room in the file to show the full decorations.. | ||
| * Actual behaviour: We *do* hide the suffix decorations, but we still show the deletion decorations. |
Description
closes https://linear.app/sourcegraph/issue/CODY-4617/improve-e2e-tests-to-cover-more-cases
Covers different rendering logic cases for:
Test plan
This PR adds tests, and is tested by running the tests and verifying the results