Proposal: BrowserStack testing links on MDN BCD tables

[chrisdavidmills]

Summary: BrowserStack approached the Mozilla MDN team with the intention of cultivating a partnership that will give them exposure on MDN, and in return provide MDN with both financial support and semi-exclusive access to BrowserStack tooling for MDN’s readership. This document explores one proposal on the table: putting BrowserStack testing links on Browser Compatibility tables.

What would this look like?

A similar partnership has already been forged between BrowserStack and the web compatibility information site caniuse.com, so you can get an idea of what this would look like already.

To check it out:

1. Go to the caniuse Flexbox compat page.
2. Click on the “132-134” support row in the “Chrome” column.
3. In the popup that appears, click on “Test on Chrome 132 - 134”.
4. You will be asked to enter an email and create a free account.
5. You will be redirected to a BrowserStack live testing session, in which Chrome 133 dev is being run. This session will last a few minutes and allow you to test the feature you are interested in, provided you have an example you can load in the address bar. In addition, the live testing session provides access to other features such as:
   a. Screenreader testing
   b. IP-location setting
   c. Responsive testing (resolution adjustment)

The proposal is to insert similar links on MDN browser compat tables.

How would the feature differ on MDN?

On MDN, the feature has the promise of being more useful. This is because caniuse.com does not host examples, whereas MDN does. In addition, when opening a BrowserStack live testing session, you can pass in a URL parameter containing a URL to load in the virtual machine-run browser when the session starts.

This means that you could, for example:

1. Click on a BrowserStack testing link in the Firefox support column of the flex CSS property page: https://developer.mozilla.org/en-US/docs/Web/CSS/flex#browser_compatibility
2. Send the URL of the interactive example (or one of the live examples) over to BrowserStack as a URL parameter.
3. Have an example prepopulated in the BrowserStack live session, ready to test.

What benefits would the two parties get?

• For BrowserStack, the benefit would be promotion/reach/sign-ups — this would give them daily exposure to MDN’s multi-million-strong monthly audience, and by extension, the organizations/communities/networks that the audience is a part of. Down the line, this has the potential to bring in a significant number of new paying customers.
• For MDN, having BrowserStack live testing available on our site would be a significant value add for our users. In addition, BrowserStack would provide financial support to MDN, which can be invested into making BCD even better, fortifying Mozilla’s commitment to having engineering resources on BCD.

What is the proposed timeline?

This is as yet unclear, but if folks are OK to go ahead with this, we could aim for early 2025.

[wbamberg]

You will be redirected to a BrowserStack live testing session, in which Chrome 133 dev is being run. This session will last a few minutes and allow you to test the feature you are interested in, provided you have an example you can load in the address bar. In addition, the live testing session provides access to other features such as:

What happens with me is: I get asked to give my email to BrowserStack, to create an account with them. Is this the intended flow, that I can only use the feature if I have a BrowserStack account? That seems a bit stronger than just "promotional".

On MDN, the feature has the promise of being more useful. This is because caniuse.com does not host examples, whereas MDN does. In addition, when opening a BrowserStack live testing session, you can pass in a URL parameter containing a URL to load in the virtual machine-run browser when the session starts.

This means that you could, for example:

1. Click on a BrowserStack testing link in the Firefox support column of the flex CSS property page: https://developer.mozilla.org/en-US/docs/Web/CSS/flex#browser_compatibility

2. Send the URL of the interactive example (or one of the live examples) over to BrowserStack as a URL parameter.

3. Have an example prepopulated in the BrowserStack live session, ready to test.

The UX needs some thought. in this case, say the page contains several executable examples. Does the page choose which one to send to BrowserStack? Or is the user presented with a list? And if the latter, how will they know which item in the list corresponds to which example in the page?

I think it might be more useful to attach the UI to the example, so a user can click on an icon in the example's iframe that allows them to select a browser+version to run that example in.

[chrisdavidmills]

What happens with me is: I get asked to give my email to BrowserStack, to create an account with them. Is this the intended flow, that I can only use the feature if I have a BrowserStack account? That seems a bit stronger than just "promotional".

Apologies. When writing this, I thought I was testing the flow in a fresh browser, but it must've been a browser where I already had an account active. Yes, you need to create a free account to get the benefit. I've updated the instructions to take this into account and also added "sign-ups" to the BrowserStack benefits section.

The UX needs some thought. in this case, say the page contains several executable examples. Does the page choose which one to send to BrowserStack? Or is the user presented with a list? And if the latter, how will they know which item in the list corresponds to which example in the page?

This is to be decided, but I was thinking that we would choose a single example to preload, rather than offering them a choice or something like that. Probably the interactive example, or the first live example available if an interactive example does not exist on the page.

I think it might be more useful to attach the UI to the example, so a user can click on an icon in the example's iframe that allows them to select a browser+version to run that example in.

Again, this is all up for discussion. My immediate thought was that a user is more likely to want to do some testing after looking at the state of a feature's browser support, but equally, they could also want to do a test after reading the baseline banner. We could even do both.

[wbamberg]

Can you elaborate a bit on the use case for this? The user sees a compat table, it tells them that a feature is supported in some browser versions and not in others. So they can then go to BrowserStack to...? I can see how BrowserStack is useful if you're not sure whether a feature is supported, but isn't BCD exactly telling you that? I mean in your example, the table tells you flexbox is supported in Chrome 132-134. So why would someone want to go and test the feature in those versions?

[chrisdavidmills]

Hi Will, sorry for the delay in responding. The Mozilla MDN team got busy with other things for a while, but we just picked this back up and had a meeting with the BrowserStack folks today to get things moving again.

Akash from BrowserStack folks provided this rather useful bit of description, which I'm sure he doesn't mind me sharing here:

The intention behind testing on Live is not for users to check whether a browser supports a specific API—that's what MDN docs are for. Instead, the goal is to verify whether their actual test website is functioning as expected in an end-to-end manner. This aligns with the typical approach developers take when manually testing their websites. Simply confirming API support isn’t enough because:

• The code might still fail due to bugs or incorrect usage of an API.
• The integration of different components may introduce emergent behavior that requires testing from an end-user’s perspective.
• Bugs can arise from the use of unsupported APIs that users may not even be aware they are utilizing.

So from this standpoint, you can imagine a developer with a new site running locally. They read on MDN that the features they are using work on all modern browsers, but maybe they don't have a particular browser available to test with on their machine, so BrowserStack is useful there. Maybe their site isn't working on that browser even though MDN says all the features should work, so again, BrowserStack is useful for investigating further, and figuring out what bug or combination of features is causing the problem?

Anyway, we have agreed that UX needs to be considered carefully if such a feature is to be added to MDN. To help explore what it might look like, the MDN dev team is going to create a few prototypes to visualize the ideas we've received so far. Once these are available, we can pass them around for testing and comment.

[estelle]

I have a browserstack account, so i made sure i was logged out before trying the following....

2. Click on the “132-134” support row in the “Chrome” column.

I got a requirement to enter an email address. I didn't want to use mine as I already have an account, and wanted a new user experience. was unable to do anything without providing a confirmable email address.

I entered a fake but legit looking email address, and the form was not submittable. After one try, I received an error message that there were too many requests from my server (not sure what server? caniuse's server?), and was not able to try a second email, which means if there was a typo in a legit email, there would be a semi-permanent failure. If two people try at the same time from developer.mozilla.org, will they get the same questionable user experience?

I switched browsers (so, i left Firefox, which i don't think is what Moz wants) and tried again, this time in Edge. I was able to get through using someone else's email address. This means, it accepts real email addresses, even if that email is not the user's email. I am not sure if they're going to spam the person whose email I used. This would be important to know before any decisions are made.

The BrowserStack experience:
I got a google landing page in a Firefox browser. I took notes on the experience above. By this point, my time had expiered, so I retried, and received the message "You have already used the available Free Trial minutes of this browser. Upgrade now to enjoy unrestricted testing." So I couldn't really test anything.

Had I not left to take notes, and stayed on, i would have had about 5 minutes of a free trial I guess? Not sure.
The google page was not an example of any feature i want to test. From our user's perspective, I am not clear what the point of the link would bw. There is no way to figure out the browser stack UI or create a page using the tech that I don't know well (the main use case for looking up a feature on MDN.) The experience felt like I gave browser stack my personal email address for no return value; in other words, gave away my personal information, I didn't sell it, as there was no useful-for-me in this exchange.

A better proposal would be for BrowserStack to include a link clearly marked as an advertisement, with the link opening an example of a live example in a better CodePen-live-mode-like version of Play. As it stands, the user experience is not useful for MDN readers. I fear moving forward with the proposal as is will erode trust in MDN.

Addendum: I just noticed "Each device is available for up to 1 minute during Free Trial. For full access". I noted above, you can't retry. So there is no way to make this useful for our users.

[chrisdavidmills]

I share some of your concerns here, Estelle. If this has any chance of being included on MDN, we need to make sure the experience over on the BrowserStack side is transparent, useful, and not frustrating. We've shared all these concerns with the BrowserStack folks, and they are looking into how the experience can be improved. Once testing prototypes are available (see above comment), we can start to collect more feedback more a wider audience.