Begin work on a data request API

[mikeshardmind]

Type

• Bugfix
• Enhancement
• New feature

Description of the changes

This adds an API for showing users what type of data red stores (including statements from cog creators) as well as providing the framework for and commands for users to request data deletion.

This framework provides enough information for cog creators to determine whether or not they need to comply with various requests. (ie a user should not be able to request deleting a record of themself using the bot to spam)

This is a breaking change (technically speaking, blocked by breaking change that this PR depends on in #4085) as it changes the used method names of the cog base class and could conflict with an existing cog.

this resolves #3239

Things worked on in this PR (check indicating ready for review)

• Working deletion of core data
• Working calling of methods provided for 3-rd party deletion of data
• Working presentation of 3rd-party End User Data Statements
• Cog Creator documentation
• A User facing docs page which explains what data red stores in core.
• User feedback for crashed calls to extension methods
• Tuned default cooldowns
• Owner setting for user self deletion availability.

Things which will need doing in other PRs

• Downloader handling of a new info.json key
• Handling requests for getting what data is stored, not just deleting what data is stored.

[Dav-Git]

Would handling the delete/request command be the responsibility of the CC or do you plan to have something like [p]data request <User> and [p]data delete <User> somewhere in core?
I think other than those two types of operations not much more is needed in terms of the API

[Drapersniper]

I think is infesable to leave this to core and core alone.

Core can't know all the data stored (Custom scopes) so it should be left to CC unless there is a way to make core away of EUD stored in custom scopes.

In adition to the above, I feel this should be made a requirement for Approved CCs

[Dav-Git]

Couldn't the invocation of the command do an event thingie which CCs would then be able to handle?

[Dav-Git]

I just think it would be pretty complicated if the deletion request would have to be done individually per COG

[Drapersniper]

Oh I see what you meant, when a request is done it is dispatched as an event that cogs have to handle.

Yeah that doesnt sound bad and leave it to CC To implement their data deletion on a per cog basis while the user would only have to send the request once. via a global command that dispatch the event to core and cogs to do the right thing.

[Dav-Git]

Exactly what I meant with better words

[mikeshardmind]

The intent is that CCs only have to handle those 2 methods, core handles every other facet of this process. I'll have more to add to this later and just wanted to ensure since this is the part cog creators will need to interact with to support, that there was visibility on this part first.

[mikeshardmind]

I need to take the time to write up a few additional thoughts that came up for this (which probably need discussion before continuing) and have not yet. This PR is not being abandoned but may end up slightly delayed.

[mikeshardmind]

So one of the goals this is intended to work towards is allowing Red to be used in use cases that need to consider various laws that allow users to request their data. I'm struggling to come up with a universal API here, as there are various things cogs can store that won't fit into a simple solution of "just shove it all in a JSON file" and am currently thinking that each cog on the bot will have to handle sending the user a message with data the cog has.

Clearly this is not ideal, but I'm not sure how else this can remain compliant with requirements that include the data be human-understandable, and clear as to what the data represents.

The request API might need to shift in this direction, but I'm open to suggestions which allow something more streamlined.

[mikeshardmind]

Proposed APIs have been updated, these may need more review before commands, events, and UX are designed around this.

[mikeshardmind]

I've linked a PR from my cog repo to this to serve as a demonstration for the intended way of using the deletion portion of this, as well as how some of the handlings of this may be subject to some discretion.

None of my cogs make great candidates for needing to share data to end-users, except the one which was already documented as being a potential liability for bot owners and included a giant warning on install that it did not and would not respect this sort of API and to only use if you could manually accommodate requests as legally required.

I'm hoping for feedback on the design from both a compliance standpoint and from cog creators to ensure the ease of use before attaching too many more mechanisms to this.

[mikeshardmind]

Cog creators need enough time to get their cogs in line with what we need prior to discord's deadline in mid August, and I need enough agreement on how we're handling this to have the remainder of this finished in time for then as well

For technical handling, the above APIs cover the cogs actually interacting with the bot on removal requests.

The following are also things I'd like approval on being the way to handle

• A new info.json key named end_user_data_statement which should hold the same info as the similarly named field in the cog class, but be accessible prior to loading the cog. This one is for use by downloader to ensure that the bot owner is aware of if a cog is going to be a liability for them prior to loading it.

• Unloading all cogs forcibly on the next bot update so that owners get an opportunity to see these statements.

• Downloader warning users if these statements change on a cog update

• Adding a core command which ejects a cog from the bot and which wipes its data folder (for use with non-compliant cogs that the owner does not want anymore)

As a matter of policy, I'm suggesting other things in an additional comment, but as long as I can get approval on the technical portions above, we can present this to cog creators to give them enough notice, and I can get to finishing work on this.

[Dav-Git]

One thing about this I would like to amend is the forced unloading of cogs. When we do this (which I agree is a good way to raise awareness) I think we should show which cogs were loaded prior to forcefully unloading all.

[mikeshardmind]

Policy:

The following are merely suggestions for how this should interact with the cog ecosystem

Core cogs

Core cogs should support as much of it as possible, including user sourced deletion requests which don't have conflicts of interest (like removing self from a blocked user list) as well as explicitly saying what data each cog + core itself stores and for what purposes. This should additionally be documented in our user-facing docs somewhere

Approved CCs

Cogs in an approved repo should at least mark what data they store and why + how much they support the Data API

This avoids current approved Cog Creators needing to be on this immediately if they are not capable, while still giving end-users enough information to make an informed decision about the cogs (However, note on this below)

Approved CCs which do not comply with this should be removed.

Sr CCs

(Note: this references an original term which was intended for use and designed as part of Cog Creator applications being opened, but the Sr Apps have never been available. Should this remain so, all of these requirements need to shift to Approved CCs for that distinction to do its job of protecting users and not just being a label handed out)

Sr. Cog Creators should mark what data they store and why + how much they support the Data API
They should also remove data if it's a request from discord or the bot owner.
They may keep some EUD as needed for the bot.
If they receive a user request, they can keep data needed for the cog
If they receive a user_strict request, they can keep data (such as snowflakes and timestamps of interactions) needed for anti-abuse measures, but not EUD

[Drapersniper]

Cog creators need enough time to get their cogs in line with what we need prior to discord's deadline in mid August, and I need enough agreement on how we're handling this to have the remainder of this finished in time for then as well

For technical handling, the above APIs cover the cogs actually interacting with the bot on removal requests.

The following are also things I'd like approval on being the way to handle

• A new info.json key named end_user_data_statement which should hold the same info as the similarly named field in the cog class, but be accessible prior to loading the cog. This one is for use by downloader to ensure that the bot owner is aware of if a cog is going to be a liability for them prior to loading it.
• Unloading all cogs forcibly on the next bot update so that owners get an opportunity to see these statements.
• Downloader warning users if these statements change on a cog update
• Adding a core command which ejects a cog from the bot and which wipes its data folder (for use with non-compliant cogs that the owner does not want anymore)

As a matter of policy, I'm suggesting other things in an additional comment, but as long as I can get approval on the technical portions above, we can present this to cog creators to give them enough notice, and I can get to finishing work on this.

I have nothing wrong with this section.

Sure it can be percieved as a one time annoyance ... but its for the best, ensure bot owners see the EUD statement, so +1 from me.

[Flame442]

Two things that cannot be easily handled when a data deletion request occurs are logs and data from unloaded cogs. There should be some additional documentations / warnings to inform users that these may still be storing data after a deletion.

[Kowlin]

@mikeshardmind You have my go ahead.

[mikeshardmind]

the "discord" arg will instead be "discord_deleted_user" to ensure future proofness on this, as this is currently the only request like this discord has informed devs of, but may not be the only one they ever have.

[mikeshardmind]

    #: This is a string which may be presented to users about what data you store and why
    `__red_end_user_data_statement__ = None`

This is being removed from the cog and moved to the extension level (parallel with the setup function in an extension) to better support varied cog types.

[mikeshardmind]

Due to a valid concern that was brought up, the API for getting user data has been changed to not make each cog responsible for sending to the user. Sorry for any disruption this causes to people already planning.

[LGACode]

This PR looks closely related to an issue that I posted a while back for a Privacy API. I can't speak much to how it should be implemented technically, but that issue discusses what I would expect as a business owner to be able to provide to my end users. My concerns are mostly rooted in the new Colorado privacy law which was recently passed and came into effect and can be viewed below, however it's not necessary to consider it specifically per say because it follows the standard of many other laws already in place such as California's and GPDR which I suspect to be a major factor in this change.

#3239

https://leg.colorado.gov/bills/hb18-1128

[mikeshardmind]

oh right, this does close that issue too (updates the PR description)

Resolved

[Jackenmen]

Just few smaller enhancements left, everything works functionally just fine.

[Jackenmen]

Looks good to me, thanks for all the help with making this happen!

This PR is temporarily blocked, look in Discord for more info.