Extension Usage Telemetry

[hoovercj]

This is a feature request for basic extension usage telemetry.

Many platforms show app/extension developers basic statistics such as installs per day, uninstalls per day, daily active users, etc.

If it is done in the core product and made available to developers through the marketplace management dashboard, the data will benefit ALL extension developers while ensuring proper information and protections for users. If extension developers have to do this themselves, they may not all reach an appropriate level of security, anonymization, consent, or user notification.

[waderyan]

Thanks for the feature request. Although you'd have no way of knowing, this - starting with download and update numbers - is on the backlog for the Marketplace team. We don’t have a firm timeline but initial investigations are underway.

[hoovercj]

Is there any update on what is or isn't being considered for extension telemetry?

[waderyan]

Hey @hoovercj no firm update yet, but we have made some progress. I think you can count on download and update numbers. What other telemetry would you like to see?

[hoovercj]

My first thought is that it would be nice to know daily/weekly/monthly users that have activated the extension/theme.

That would be a great start, but it isn't the full picture, of course, because unlike an app on a phone, users that have an extension active still might not be using it - They might have your extension installed but never actually use the commands it provides, for example.

So a logical next step would be to track how often users use commands contributed by an extension. Since all commands go through a central dispatch in VS Code, I know this is at least theoretically possible.

Another case is that an extension might be enabled, but the user has settings which have disabled it. So Again, getting telemetry on how contributed settings are being used. That one is a bit trickier because it could contain PII (executable paths, for example), but I know that the telemetry service already does its best to scrub PII.

I think stats like this are something that could benefit all extension developers but I know that I personally would not feel confident building and add this to my extensions myself because I'd be nervous of violating privacy or security standards. I'm more confident that with microsoft's legal support and security practices, it could be done safely and open up great opportunities for extension developers.

[hoovercj]

Also, uninstall numbers to accompany the install + update numbers

[Chillee]

@waderyan One thing that I would really like is settings. It would be very useful to see how many of our users use a setting when it comes to decisions.

For example, for VSCodeVim we have a setting called vim.foldfix that provides a certain hacky fix for an issue users have. I don't know whether it's worth spending the time to work on that fix.

There is a privacy issue here, but I think that limiting it to certain kinds of settings (not say, strings) would be fine.

And of course, configurable telemetry through some kind of API would be great, although there would be privacy concerns there...

[Almenon]

Any update on this? Currently I keep track of downloads manually which is a pain. When I updated my extension my download count spiked, but then I realized updates must count as downloads too... uggg.

Because the downloads count shows both downloads and updates I don't have an accurate count of either!

[tboby]

A separate install/update count would be amazing. I have ~550 downloads, but I've released at least 28 versions.

I could have anywhere between 15 and 500 users!

Another very useful (and relatively anonymous) statistic would be the frequency specific settings are changed from their default value.

[Almenon]

In the meantime you can use azure application insights for telemetry. It's actually pretty easy to set up (and free).

I wrote a small guide on how to do it: https://medium.com/@almenon214/adding-telemetry-to-your-vscode-extension-f3d52d2e573c

It won't help you with your download count but it can show you the number of actual users, a much more important statistic.

[aaron-bond]

@waderyan In terms of the basic download / update numbers, is that something that will be available before the other statistics or is it an all-or-nothing type of deal?

[NipunaMarcus]

Hi, Is there any progress made on providing these telemetries as I couldn't find any place where we can count download/update numbers (for month, for week, for days, for hours) ??

[rangav]

This feature will be very useful for extension developers, any update on this?

[vinistock]

👋 @seaniyer. I was talking to Isidor about VS Code extension metrics and he shared this issue and your contact. Having this in the marketplace would be useful for understanding adoption. I wanted to suggest a few other possible metrics for the team's consideration, which would also help track/understand adoption.

• DAU/MAU ratio (daily active users over monthly active users)
• Churn Rate (rate of people who used the extension for some time and uninstalled)
• Bounce Rate (this could be measured as the number of people that see the extension page on the marketplace or inside VS Code but leave without installing)
• Retention Rate (kind of the opposite of Churn Rate)
• Install Growth Rate (how rapidly installs are increasing)