Considerations for a playwright adapter

[thenbe]

Hi. I've been hacking away at a neotest adapter for playwright and there are few issues I'm wondering about. These are not necessarily feature requests or well thought out suggestions, but more like me thinking out loud as I mentally plan out the implementation.

Representing multiple different results for the same test

For example, take this playwright config shown in the docs. It has 3 projects. In this case, each project runs on a different browser. So when you do:

npx playwright test ./mytest.test.ts

your test is run three times- once for each project. These may have different results (your chrome and firefox tests may pass, but your safari test could fail). How would you recommend representing these three results in neotest (a test ran with 3 different projects, 2 passed but 1 failed)? To get info about what failed, the user could hover/open the output. But in the tree do we have to mark the test as failed, or is there a way pass a custom (yellow?) status/indicator?

Including or excluding projects in the next test run

Update: Completed

The main reason I started working on this adapter is because I wanted a seamless way to switch between projects as I ran my tests. Setting them on the command line quickly became very tiring. Therefore, I'm really motivated to find a way to configure projects on the fly 🤓.

Currently the official playwright vscode extension does this using a panel, as can be seen here around the 5:43 mark. You can tick/untick projects and they will be included/excluded the next time you run a test. I feel like that interface would be the way to go- a list of checkboxes of the project names resolved from the playwright.config.ts file. What I'm not very sure about is how/where to persist the data of what's selected and what's not? Which of vim/nvim/neotest's api would come in handy here? Any advice on how/where to place the panel?

Flaky tests

In it's test output, playwright will show you which tests were flaky during the run. It'd be nice to indicate this somehow. It also reports slow tests, but that info isn't as important as flaky tests.

Global errors

These don't belong to any test, most of the time these happen in global setup or global teardown. Maybe we should just print an error message?

---

Finally, thank you for creating this. 🙏🏼

p.s. If you've already answered some of these, or have come across neotest adapter implementations that are dealing with some of the same considerations, please feel free to point me towards them.

[thenbe]

After some reflection on the first point, I think the solution that makes the most sense would be to display the tests separately. Yes, it's slightly more verbose. But the flexibility gained from treating each node in its own right feels like a net gain.

Anyways, I've hit a road bump while trying to return multiple positions using a custom build_position. I've extracted the test info from the playwright cli- I have test names, locations, unique id's, etc, from playwright, but I'm not sure how/where to pair this up with the data resolved by neotest/treesitter. I played around with build_position and position_id, but could never quite get the desired result. While I was able to get neotest to display the correct amount of test children as dynamically resolved by the user's playwright config/code, for some reason they would show up as children of each other, instead of siblings.

Finally, it'd be really helpful if there was some more info about the relationship between build_position and position_id, if any.