Define charter review process to require addressing comments as in CR transitions

[fantasai]

As there have been multiple instances of AC comments on a proposed charter being effectively ignored, I would like the charter approval process to be made more explicit in a similar manner to a CR transition approval. Specifically, I am requesting that the charter review process require a proper Disposition of Comments in the same manner as a CR transition request, and that its enactment requires the approval of both the AB and the TAG to ensure that

• Review has been solicited from relevant interested parties, including prospective members of the WG as well as current/former members (for WGs like SVG which are mostly continuous).
• All comments have been meaningfully and reasonably addressed.
• The charter (and its creation) is consistent with the W3C Process (including the above two points).
• The intended work is compatible with Web Platform architecture.

[ Copied from https://lists.w3.org/Archives/Member/w3c-ac-forum/2018AprJun/0009.html ]

[wseltzer]

Team practice is to send any changes proposed to a charter after review to all of the reviewers and to the member-charters-review list, asking whether any of these changes would affect the members' review. I hear you requesting that we include the WG/public in that loop, where it's a continuing WG, which makes sense to me.

[dwsinger]

who's on member-charter-review? would that loop in any AC member who failed to vote on the first but would object to the change? (but maybe this is a good reason to vote: you'll be notified of the resolution process and will have an opportunity to object, even if you vote 'abstain').

[dwsinger]

mindless thought of the morning: maybe the tooling can be improved that the response to a comment could be seen in the same table as the responses to the ballot, i.e. an extra column "response/action"?

[wseltzer]

re who's on member-charter-review? Any member who subscribes to it.

[dwsinger]

I kinda like the idea that the set of people who get to hear of resolution is the set of people who bothered to vote (at all, including abstain) in the first place. I doubt many members want to hear of all charter reviews.

[vfournier17]

@wseltzer How do you subscribe to it? Do people know how to subscribe to it?

[TzviyaSiegman]

I agree with @fantasai. Having been on the side of writing charters and responding to objections, the process is can be clunky. Responding to objections seems to often be about placating individuals than about finding consensus and the best way forward for the Web. When I read final versions of charters, I can pick out the spots that are responses to formal objections.

[fantasai]

@wseltzer Sending changes for re-review makes sense, but it only deals with making sure that addressed comments are addressed in a way that's acceptable to anyone notified of the changes. It doesn't solve the problem of comments which are rejected or otherwise not addressed.

[dwsinger]

Somehow we have to find a way to achieve a charter which gets consensus, addresses the review comments for which there is consensus that they should be addressed, and does this in a timely fashion. The current informal process achieves the timeliness but not the transparency and evident consensus.

Saying that charter-comment-resolution will involve the community of those who voted on the charter (including a vote to abstain), plus of course the team and chairs, addresses the transparency and consensus parts. Would it help or hinder timeliness?

[chaals]

I can only think of one example, and it was a few years ago, where objections to charters were worked on collectively by a a community of equivalent size.

Opinions probably vary, but my memory of the exercise was that the transparency and consensus part didn't particularly impact the timeline. (I am not convinced that the current approach is particularly effective in achieving timeliness, although nor am I convinced that it is the root cause of common delays).

[wseltzer]

Talked with @fantasai about

• sharing draft dispositions of comments as team is proposing changes to address them
• with a list (cc, rather than bcc, +member-charters-review) of all who have voted on the charter

We could do this under current Process.

[vfournier17]

@wseltzer Are you saying that changes would be made to the charter after it's been voted on? I'm not sure how someone could approve a charter that's not in final form? Also, would the reviewers need to review multiple drafts? It seems like it would make more sense to review one final revised draft with all comments, so you could see how all the language holds together and make sure there aren't any conflicts. I would not be in favor of reviewing multiple drafts with piecemeal comments.
@chaals It's not only objections that are a concern. Sometimes review comments are made on a charter that don't get incorporated, and the commenter is left wondering why.

[dwsinger]

@vfournier17 that's the problem we are struggling with; charters are adjusted after the ballot, to respond to comments. But we're not very clear as to who to recirculate to, to make sure that those fixes don't introduce new problems. We'd like to tighten this up without having multiple rounds of voting...

and yes, we don't circulate a response document either (I would personally like a third column in the ballot response form, filled in with a short response; all the AC would be able to see that).

[frivoal]

I agree something like this is needed, primarily for 2 reasons:

• I have seen multiple comments, including comments made as part of formal objections, that were dismissed with some generic phrasing that boils down to "objections have been considered, but it was judged better to proceed." This would not be acceptable within a WG: comments have to be addressed individually (even if the way to address them is to disagree). I don't understand why it is in a charter review.
• Changes made to the charter during / after the vote, without giving a say to people how had approved the charter before the change.

I think taking inspiration from how specs are developed, and doing something similar with charters would be a good thing. Here is an example of how the process could look, step by step (this is a first draft, to be discussed, not a set-in-stone proposal I want to push as-is):

1. A first draft of the charter is prepared. If it is a renewed charter for an existing group, it must include a change section.
2. It is announced, with a minimum review period, to:
   • the TAG (to check that intended work is compatible with Web Platform architecture)
   • the AB (to check that the charter is consistent with the W3C Process)
   • [email protected]
   • [email protected]
   • the public mailing list of all groups listed as liaisons in the charter
   • the public mailing list of the existing WG in case of renewal, or the public mailing list of the CG in case of incubation
3. all issues raised get discussed, and the charter is updated accordingly
4. Each change to the charter since the announced draft is documented in the change section.
5. A Disposition of Comments is produced. It lists every comment/change request, and for each, gives:
   • Who raised the issue (can be an individual, or a WG, CG, the AB, the TAG…)
   • Its status: Accepted / Accepted (Editorial), Retracted, Rejected, Invalid, Duplicate.
   • A link (or links) to the discussion and to the resolution that concludes it.
   • In the case of Rejected or Invalid, a rationale for rejecting (or a link to it)
6. Once all issues have been closed (but no sooner than then end of the minimum review period),
   1. The AB validates that all the above has been done properly (if not, rerun until it has)
   2. Call for an AC vote, and announce it to:
      • [email protected]
      • [email protected]
      • [email protected]
      • the public mailing list of all groups listed as liaisons in the charter
      • the public mailing list of the existing WG in case of renewal, or the public mailing list of the CG in case of incubation
      • each person who filed a comment
7. If all votes are in favor, the charter is accepted and published as is. Otherwise, rerun from step 3, with the following differences:
   • each person who previously filled a comment or voted must be informed of proposed changes and be included in the corresponding discussions
   • if none of the issues raised during the vote lead to a substantive change (all changes are editorial, or no changes)), there is no need for a new vote (6.ii). The AB still needs to validate that comments have been addressed properly (6.i)

• At any point, the whole process may be abandoned if there are irreconcilable differences.
• TBD: during the review + editing period (steps 3 to 5), who has the authority to determine consensus & declare resolutions, and particular, to close close issues as rejected? (Today, it is The Director after the vote, and undefined / the team before).

[fantasai]

Thanks @florian for the overview / proposal. :) To help people who aren't familiar with Dispositions of Comments, which is what we use to track how comments are addressed for CR and PR transition requests, here's a CSS Grid Layout Level 1 Disposition of Comments as an example.

The key benefits of creating a DoC are:

• It helps ensure that all comments are tracked, by requiring them to be collected, collated, and listed individually, reducing the chances of something falling through the cracks.
• Ensures that each comment is addressed because it is individually tracked by issue (whether multiple people raised the same issue, or one comment raised multiple issues).
• Ensures that each comment receives a response explaining either how it was addressed or why it was rejected, which a) is courteous to the commenter, b) ensures that the commenter—and others—can respond to errors in the fix or misconceptions about the issue, and c) helps make sure the issue is thoroughly addressed and not waived off because it was skimmed, forgotten, or seemed unimportant to attend to.
• Highlights cases where there was disagreement for further review.

W3C doesn't mandate any particular tool for compiling DoCs; I've been generating mine (as you see above) with https://drafts.csswg.org/bin/issuegen.pl We have some additional fields which I've added as I've found them useful, but the critical ones for a DoC to fulfill its goals are:

• Summary of the issue
• Link to the discussion raising the issue
• Link to the response which closes the issue (explaining the changes made or rationale for rejecting)
• Status of the issue (open, accepted, rejected, invalid, etc.)

[wseltzer]

As the Strategy team is now responsible for bringing charters the the AC review and Director approval, I'd recommend that we iterate on this model before freezing it in the Process document. We have been working to adopt these points in recent charter reviews (see), and I'd be interested to hear feedback: is it effective? are we missing important elements or sending too much email?

[dwsinger]

Thanks for Florian's detailed exposition. Do we need a formal update to the Process document?

[frivoal]

I've been thinking about this, and I think:

• We do need to put this kind of requirements formally in the process document. It is possible to run a sane chartering workflow without the process mandating it specifically, but being able to do chartering with sufficient transparency/oversight is not sufficient. We have to be actually required to do so.

• On the other hand, I don't think we should include it in the formal process this year. While I think the steps I proposed are good, this is a somewhat complex multi-step process, and it is quite likely that it has some rough edges.

My suggestion instead, is that I will set up a wiki page with a cleaned up / refined version of the proposal above, with sufficient detail and rigor that it is actually a process we can follow to the letter (and an explicit list of open issues / points to clarify). Then, we can discuss and tweak for a bit. Then ask the team to try and rigorously follow it (for now, merely out of good will, not out of an actual process requirement) for a while, and to report back any difficulty with that process. If they report back any rough edge, we tweak accordingly.

By the end of the summer, we'll have either agreed that this was a terrible idea (I suspect not), or we'll have reasonably solid and well exercised set of rules that we can then just put into the official Process, for next years update.

[frivoal]

First draft of the wiki page proposed above: https://github.com/w3c/w3process/wiki/Charter-development-and-review-process

[dwsinger]

see #28 which points out that in 7.1.2 the requirements on "substantive changes" in a charter links to the definition of "substantive changes" in a Rec., which is not the same thing at all. This needs addressing as well.

[frivoal]

Changing labels, as we've decided to run this as an experiment for this year.

[wseltzer]

I discussed this with the W3C Strategy Team, which is responsible for chartering new work and, with the Project Team, for re-charters of existing groups. We hear the concerns with predictability, understandability, and transparency of process, and still think it better to address through updates to the Guide.

Some concerns with the process proposed above and on the wiki:

• adding Process overhead and timetables to chartering and rechartering hinders agility, and serves as inducement to leave work in CGs or take it elsewhere, rather than bring it to the Rec track.
• combined with the push from members for shorter (2-year or less) charters, this overhead adds substantial burden in frequent rechartering.
• some of the specifics in the wiki, such as TAG and AB review, and AB review of changes, are new and don't necessarily match the scope or expectations of those groups

Instead of changing Process, I invite comments on the Guide and suggestions for changes there. Earlier this year, I documented recent experience and didn't hear complaints. Thanks.

[samuelweiler]

It seems to me that:

1. We're always going to make mistakes in charters, and being in a dynamic world, charters will often need changes along the way. It's important for us to be able to fix those mistakes - and adapt to changes in the world. So long as we can do that effectively, making the chartering process harder is not necessary - if we mess up and overlook an objection, we can fix it.

2. I'm seeing more and more work happening in groups that don't need charters - namely CGs, particularly WICG. I wonder how much of that is that the chartering process itself is too burdensome already. As a team member responsible for chartering, it certainly feels too burdensome.

3. The process as a whole feels too heavy-weight for the quantity of work we're pushing through it. Adding this seems like it's going in the wrong direction.

In practice, I think we're doing many of these things already. e.g. I prepared a disposition of comments on the Privacy IG charter: https://www.w3.org/2019/09/ping-disp-of-comments.html (member only link)

I would rather see us - both the team and the community - focus on being agile and responsive. If we encounter issues, let's address the substance. Inventing process just makes our work more painful.

Also, are both the TAG and AB on board for being potential roadblocks to chartering? From the complaints I'm hearing about the TAG's spec review load - primarily from the Blink process - I wonder if they're ready to take on something more.

[frivoal]

As per https://www.w3.org/2020/01/29-w3process-minutes.html and https://www.w3.org/2020/02/12-w3process-minutes.html, this issue is deferred to a subsequent cycle of the Process.

[fantasai]

This has been addressed by the adoption of the Charter Refinement phase and stricter rules about charter review and approval so closing.