Proposal: Base Distribution Developers = Derivative Developers

[adrelanos]

The problem you're addressing

Low review bandwidth for contributions to Qubes OS.

The solution you'd like

Proposal: Base Distribution Developers = Derivative Developers

The value to a user, and who that user might be

Higher review bandwidth for contributions to Qubes OS, faster review, merge process.

Proposal: Base Distribution Developers = Derivative Developers

This proposal outlines a strategy for increasing review bandwidth and development efficiency within Linux derivative projects by granting base distribution developers commit access for derivative distributions.

This proposal suggests a collaborative framework whereby developers from base Linux distributions, such as Debian or Fedora, are granted commit access within derivative Linux projects like Qubes. The aim is to leverage the established trust and vetting processes of larger distributions to expedite development, enhance security, and improve the implementation of bug fixes and feature requests in derivative distributions.

Version: 3.0

Definitions

• reviewer: A "reviewer" in this context is someone with the right to review and merge (commit) source code contributions (such as pull requests on GitHub) without necessarily requiring approval from the derivative distribution core team.
  • Similar names in different contexts: In the case of Debian, these are called Debian Developer (DD) or maintainer, a different but similar concept. A DD can upload packages to Debian which they maintain, and any other package (NMU - non-maintainer upload), subject to Debian policy.
• base distribution: Examples include Debian, Fedora.
• derivative distribution: Examples include Kicksecure, Whonix, Qubes.

Issue

• Low Review Bandwidth: The review bandwidth of many derivative Linux distributions (such as Kicksecure, Whonix, Qubes) is low. This means there are sometimes complex or large source code or documentation contributions than the existing derivative core team can thoroughly merge, maintain and review without jeopardizing the security and/or quality of the operating system. This slows down development, growth, implementation of bug fixes and feature requests.
• Trust Decisions Difficulty: Establishing trust with people far from the internet is challenging. Physical meetups such as conferences are not a magical solution either. Deciding who gets commit access is difficult. This is probably why the named Linux derivatives have only a small team of people with commit access, and new committers are rarely added.

Current State versus Potential Future State

Base Distributions:

• How to acquire commit access for the base distribution?
  • Answer: By becoming a base distribution developer.
    • Example: Is it possible to get commit access in Debian? Yes.
      • How to get Debian commit access? By becoming a Debian Developer (DD). How to become a DD? By going through the Debian New Member Process.

Derivative Distributions - Current State:

• Possibility: Is it possible to acquire commit access for the derivative distribution? No. Realistically, unlikely.
  • How: There is no defined process/path that one could point to that more or less "guarantees" acquiring heritable commit access, whether technologically or via a shared set of widely agreed upon protocols and principles.

Derivative Distributions - Potential Future State:

• Possibility: Is it possible to acquire commit access for the derivative distribution? Yes.
  • How: By becoming a base distribution developer one has a viable path to acquire commit access for the derivative distribution.
    • Example for Kicksecure: Debian developers would have an easy path to request Kicksecure commit access.
    • Examples for Qubes:
      • Debian developers would have an easy path to request commit access for Qubes Debian Templates.
      • Fedora developers would have an easy path to request commit access for Qubes Fedora Templates.
      • dom0? Intentionally left out in this proposal to see first if this proposal receives positive feedback. Could be considered later or stay as is.

Solution

• Base distribution developer proposal: Any properly authenticated base distribution developer should be able to nominate themself for review and/or commit access rights for the derivative distribution. In other words, there should be a process to allow base distribution merge access to be inherited by the derivative distribution.
• Prior contributions requirement: This may be limited to individuals who have made prior contributions to the derivative.
• Shortcut solution rationale: This is an imperfect shortcut solution for the low bandwidth review queue by derivatives such as Qubes. What would be the "perfect" solution? See footnote [1].

Advantages

• More speedy review of contributions: The review of pending contributions (pull requests) could be sped up because potentially there might be more reviewers.
• Simplified vetting process: The reviewer vetting process would be outsourced to larger Linux distribution(s) that have existing infrastructure to handle that. Want commit access? Simply make a number of useful source code contributions (to be defined), have credentials from eligible Linux base distributions and complete a brief application requesting access.
• Closer Cooperation with Upstream: By encouraging more people to become base distribution developers, cooperation with upstream would improve. One can only become a base distribution developer by making useful and accepted contributions to the base distribution.

Eligible Linux Base Distributions

• Debian: Debian is a natural choice because Debian Developers must be trusted as they have the capability to backdoor Debian and therefore any distribution based on it (Kicksecure, Whonix, Qubes Debian Template) anyway.
• Extension to other distributions: Could potentially be extended to include Fedora and other Linux base distributions pending the definition of eligibility criteria.
• Discussion on expansion: This should be discussed separately in case this proposal receives positive feedback. No need to engage in distribution polemics if this concept is completely rejected.

Why Commit Access

• Commit access versus review rights: Why commit access, review, and rights and not only review rights? Because then the responsibility would still be on the shoulders of the person with commit access. The merge at the end is what gets recorded in the source control system (git). With only review rights, not much would change. Even now, anyone can make a comment that can serve as a community review.
• Importance of commit access: Without commit access, the usefulness of this proposal would be severely limited.

What this is not

• Hard condition: Is this an exclusive, hard condition? No. This does not create a hard condition. It's an additional way. It does not curtail the rights of the existing decision-makers to grant review privileges to other people of their choice.
• Exclusion of existing members: Should existing people with commit access be removed if they are not base distribution developers? No. This proposal does not suggest curtailing the rights of any existing members.

Questions and Answers

• rights grant for non-base distribution developers: Will it still be possible for the current decision makers to "sidestep" this proposal and grant commit access to any person, including those who are not base distribution developers? Yes.
• refusal of rights to base distribution developers: Can the project current decision makers still refuse to grant commit access to any arbitrary number of base distribution developers? Yes.
• revocation of commit access: Can the project current decision makers still revoke commit access at any time? Yes.
• public pull requests: Can any member of the public still send pull requests as is possible prior to this proposal? Yes.
• absolute rule: Is this an absolute rule? No. This is a living document that aims to grow the pool of trusted developers and removing development friction in order to maximize the effect of their contributions to the FLOSS ecosystem. It can be further extended or amended as needed based on feedback.
• vetting for new reviewers: Could there be a (short, cursory, or long) vetting before adding new reviewers? Optional.
• base distribution developers' contributions: Did any base distribution developers ever contribute to derivative distributions? Yes.
• examples of contributions: Examples of base distribution developers contributing to derivative distributions? Intentionally left out to avoid putting unnecessary spotlight on these contributors. Since everything is public and Open Source, it's on the public record.
• realism of contributions: How realistic is it that base distribution developers would contribute to derivative distributions? Unknown, but see above question. Worth trying. Effort is comparatively low. Maybe some will comment here.
• point of proposal if there is disinterest from current base distribution developers: What's the point if no base distribution developers are interested to apply for derivative distribution merge access anyhow? Then at least derivative contributors can be pointed to a viable path on how they can get commit access to the derivative distribution - by first becoming a base distribution developer.
• distribution developer rationale: What is so special about becoming a base distribution developer? It is hard, time-consuming, requires a learning process, going through the proverbial school of the base distribution, social skills and includes a vetting process.
• upload rights: Build/upload rights for releases and/or binary packages are a separate right and are not covered by merely having source commit access.

Qubes Specific

• prerequisite knowledge: Qubes Core Reviewer (QCR)
• commit access to Qubes core packages: Does this mean, a base distribution developer would necessarily get commit access to Qubes core packages such as qubes-core-agent-linux? No, not necessarily. The concept of maintainers could be kept. A package maintainer of a package or QCR could remain the only reviewer deciding what is permissible to be merged.
  • NMU policy: There is no policy suggestion for NMU (non-maintainer upload) in Qubes yet. Such as when no uploader (QCR?) is (planned or unexpectedly) unavailable. For example, a selection and/or quorum of a minimum number of base distribution developers might be allowed to perform NMU. Such things could be considered at a later point in case this proposal receives positive feedback.
• QubesOS-contrib eligibility: A base distribution developer could be eligible to add packages to QubesOS-contrib, perhaps without even needing a second reviewer, but this would be optional. Another option would be that two base distribution developers would be eligible to add packages without requiring a review by QCR.

Footnotes

• [1] The "perfect" solution would be for the derivative to develop and maintain a member mechanism similar to Debian New Member Process. This would also require forking related policies such as Debian Constitution, Debian Free Software Guidelines (DFSG), Debian Policy, among an unknown number of other important policies and processes. However, this would be very hard, time-consuming, require reducing development activities, and a strong focus on the development of an organisational structure. This is presumably why that has not been done yet and is unlikely to happen soon.

Improvements to this Proposal

• Suggestions welcome.
• This is just a proposal. Instead of dismissing it, how can it be improved to be made suitable?
• The core team is obviously free to make any changes of their liking.

Disclaimer

• no conflict of interest: I am not a base distribution developer myself, neither did I start the process to become one at this point. I would therefore not personally benefit from this policy suggestion.
• proposal origin: Did any base distribution developer suggest this? No, Patrick invented this idea without prior counseling with any base distribution developers.
• Speaking for myself only.

See Also

• https://producingoss.com/en/committers.html
• https://developer.blender.org/docs/handbook/contributing/commit_rights/
• https://medium.com/@elombrozo/the-bitcoin-core-merge-process-74687a09d81d

[marmarek]

Generally, I don't think this proposal named as it is makes sense. But I do agree some process changes are needed.

First of all, commit rights should be given based on (at least) two conditions:

• trust
• competence

Qubes as a whole is not just tweaked Fedora or tweaked Debian, it's a whole another abstraction level. You can be very proficient Debian Developer and not really good candidate for Qubes developer (or the other way around). But also, for the trust factor, just being Debian Developer means Debian template users needs to trust you anyway, but not necessarily Fedora template users, so the fact that Debian project trust a given developer shouldn't automatically give them right to commit to all qubes templates (*).

Historically, there was another issue: qubes-builder did not support different maintainers for different packages. This is fixed in qubes-builder v2 (including support for requiring multiple signatures if desired). This is not perfect, as being able to upload any package to the repository one can still compromise all users even if that package is not installed by default (for example by setting Enhances: dependency or similar), but still better than direct control of all packages, and de facto shell access to build servers as it was in legacy qubes-builder...
BTW, this limitation wasn't the case for template build scripts (builder plugin), which is why we could grant access there (or in fact - use repository from outside QubesOS github org) for al long time already.
So, starting with qubes-builder v2, we have technical means to extend maintainers for individual packages. In that case, I'd prefer this to be "at least 2 signatures", but possibly from a larger set of people. But they should be chosen based on their interaction with qubes community, past contributions etc, not necessarily (and definitely not only) with other projects (even if related).

BTW, I'd like to make a comment to the "Commit access versus review rights" point: yes, direct commit access is much more than review rights and possibly make things quicker. But also, it has huge impact on the trustworthiness of the whole project. You need just one backdoor (or more likely "bugdoor") to attack somebody (and heavily impact project reputation while at it). The more people have such power, the more likely it can happen (and that isn't only about intentional malice, but also about compromised development environment, signing keys etc). That's also why the "at least 2 signatures" rule above.
On the other hand, posting reviews do help. And in fact it is a way to gain trust over time (and prove understanding of the system) - exactly the trait future maintainer should have. Of course the level of trust in external review very much depends on who does it, but for example your @adrelanos or @rustybird reviews are very valuable and help a lot. "Just" checking if a change is not actively malicious vs checking if it's functionally correct have very different complexities, with the latter requiring usually more time, and often some testing, but also not as much trust as the former.

(*) we don't separate packaging for different distributions from the code itself; while such split would indeed allow separate people maintaining those packages (Debian developers maintaining Debian packages, Fedora devs maintaining Fedora packages etc), it also means that many changes would require more work (now when you introduce new file, or change some location, you'd need it change it several repositories - code + all the packages, possibly involving several people). And since packaging-only changes are very rare (looking at commits history), such split in our case would not improve anything.

[adrelanos]

First of all, commit rights should be given based on (at least) two conditions:

• trust
• competence

Right.

But also, for the trust factor, just being Debian Developer means Debian template users needs to trust you anyway, but not necessarily Fedora template users, so the fact that Debian project trust a given developer shouldn't automatically give them right to commit to all qubes templates (*).

Yes, this is covered in the proposal here:

Examples for Qubes:

• Debian developers would have an easy path to request commit access for Qubes Debian Templates.
• Fedora developers would have an easy path to request commit access for Qubes Fedora Templates.
• dom0? Intentionally left out in this proposal to see first if this proposal receives positive feedback. Could be considered later or stay as is.

[marmarek]

But also, for the trust factor, just being Debian Developer means Debian template users needs to trust you anyway, but not necessarily Fedora template users, so the fact that Debian project trust a given developer shouldn't automatically give them right to commit to all qubes templates (*).

Yes, this is covered in the proposal here:

Yes, and see the footnote to that sentence.

[unman]

This proposal reads like a boilerplate for derivatives, and not particularly relevant to Qubes. In what way does the proposal ease the path for Debian developers to request commit access? Why cant they request commit access now? Are there any examples of DDs requesting commit access? And those requests not being dealt with? More to the point, if a DD want to fix a bug in a Debian template, they should do it upstream. If it's a Qubes specific bug, then it's either appropriate to fix upstream, (as pertaining to Debian on virtualization), or it's not, in which case their DD status is irrelevant.

[adrelanos]

unman:

This proposal reads like a boilerplate for derivatives, and not
particularly relevant to Qubes.

Written in a generic way indeed. But not copied/pasted from elsewhere. Invented by me.

In what way does the proposal ease the path for Debian developers to
request commit access?

They're blessed eligible or at least being told they an easiser path may be ahead of them.

Why cant they request commit access now?

They could.

• Current state: They have no reason to believe that the process would be any simpler than for example the Debian new maintainer process. Their existing credentials aren't expressively honored.
• This proposal - new state: They knew that the process might be simpler.

Also quoting from the proposal.

• realism of contributions: How realistic is it that base distribution developers would contribute to derivative distributions? Unknown, but see above question. Worth trying. Effort is comparatively low. Maybe some will comment here.

Then also this doesn't matter if no existing base distribution developers request Qubes commit access. Then at least other interested contributors knew a way forward how that one day could be accomplished.

• point of proposal if there is disinterest from current base distribution developers: What's the point if no base distribution developers are interested to apply for derivative distribution merge access anyhow? Then at least derivative contributors can be pointed to a viable path on how they can get commit access to the derivative distribution - by first becoming a base distribution developer.

Are there any examples of DDs requesting commit access?

None that I know.

More to the point, if a DD want to fix a bug in a Debian template,
they should do it upstream.

Right.

If it's a Qubes specific bug, then it's either appropriate to fix
upstream, (as pertaining to Debian on virtualization), or it's not, in
which case their DD status is irrelevant.

If granted commit access to Qubes then...

Quote #1939 (comment)

In theory, it would be nice to have a trusted repository, in reality, it is very limiting of the packages that are included. In theory, the review process would be nice, but in reality, there was no deep review of the external formulas most used by qubes users, Shaker and it was also never added to Qubes repos.

• Current state: Stalled / slow.
• This proposal - new state: At least a chance. If new trusted people could be found, then they could review, merge for example Shaker.

[IOZZYS]

I'd be lying if I say I haven't been daydreaming for weeks about a more extensive platform feature for people to confirm they have inspected a piece of code and found it to be safe and functional, instead of feeling it out based on stars, people "watching" and forks (A lot of this might be inactive people or people who use the code without actually inspecting it) whenever it's too difficult for me to understand. Combine this with the ability to leave notes wherever you find bugs/backdoors, something to make duplicate accounts difficult, a sizable user base and something like blockchain instead of a centralized platform and you've got quite some community based "trustworthy" code reviewing.

(This could have become a way longer off topic explanation of the concept above, but for illustrative purposes of my feelings toward the issue at hand, this will suffice;)

However, even with something like that working perfectly, when it comes to Qubes OS I would prefer to put the threshold of required confirmations (quantity and quality of accounts making it) unreasonable high if possible.

TLDR; No, I don't want to give template developers any additional power that they don't already have, I wish I had the luxury of trusting the current team less (no offence intended <3 ) by having them + even more people review the code before commits can be made and it taking even longer (with exceptions to critical security updates).

[adrelanos]

That seems to address other issues (using github, using centralized web servers instead of serverless as in no server, no cloud) and more and would imo be more suitable as a separate discussion.

[IOZZYS]

As I mentioned, the concept described above and my level of scepticism of trust even within a way more elaborate system were merely meant to indicate how much the proposed change to the commit system was off from what I would like to see for Qubes OS.

Since, if I understand it correctly, the proposed change would lead to less trusted individuals being able to approve code to highly trusted components of Qubes OS, for speed of reviews sake, without needing additional approval from currently ultimately trusted individuals. Whilst I dream of requiring as many additional approvals as is realistically possible.
I'm sorry if I was unclear, I was in a certain mood.

If you are interested in me expanding on the idea above somewhere separate, because you think it would gain a lot of support and help, I can do that in 2 months but I will be quite unable to help set it up for at least another half a year. (Busy + still expanding my knowledge of code)

[bi0shacker001]

This feels like the antithesis of qubes current security practices. The limited number of people with commit access is a deliberate choice from the team, and this SHOULD NOT be considered a valid proposal, from my perspective. Qubes is not a standard distribution, it's a security -focused one. Keeping that in mind, nobody should have an easy path to commit access. It needs to be earned, someone needs to be explicitly trusted. The current system is the best one for that.

[marmarek]

In the light of recent XZ attack, I believe it is clear now that my reservations explained above are not purely theoretical.

[github-actions]

This issue has been closed as "declined." This means that the issue describes a legitimate bug (in the case of bug reports) or proposal (in the case of enhancements and tasks), and it is actionable, at least in principle. Nonetheless, it has been decided that no action will be taken on this issue. Here are some examples of reasons why an issue may be declined:

• No solution can be found.
• The proposed action is not possible.
• The proposed action would weaken security to an unacceptable degree.
• The proposed action would be too costly (in time, money, or other resources) relative to the benefits it would provide.
• The proposed action would make some things better while making other things worse, and the trade-off is not worthwhile.

These are just general examples. If the specific reason for this particular issue being declined has not already been provided, please feel free to leave a comment below asking for an explanation.

We respect the time and effort you have taken to file this issue, and we understand that this outcome may be unsatisfying. Please accept our sincere apologies and know that we greatly value your participation and membership in the Qubes community.

If anyone reading this believes that this issue was closed in error or that the resolution of "declined" is not accurate, please leave a comment below saying so, and the Qubes team will review this issue again. For more information, see How issues get closed.

[adrelanos]

I think this is an invalid comparison.

• xz backdoor / @JiaT75:
  • Was anonymous. Nobody ever met @JiaT75 in real life.
  • Did not pass Debian New Maintainers process.
• Debian Developers:
  • Are non-anonymous. It's difficult to "get accepted into club" without having social connections.
  • Did pass Debian New Maintainers process.

Despite of not having this policy as suggested in this ticket, the xz backdoor made it into Qubes' community templates (#9071) and presumably would have made it into ITL templates and dom0 too given enough time would it not have been accidentally detected thanks to Andres Freund investigating the unexpected 0.5 seconds SSH delay.