Replace rule/group prodtype alignment with explicit inclusion in a product profile/benchmark

[evgenyz]

Share the Context

The prodtype property of a rule (or a group of rules) allows the build system to selectively discriminate inclusion of rules in benchmarks for products that share benchmark_root pool of rules.

Description of the Problem

Current implementation of this mechanism has multiple flaws, most important being:

• The 'all' value causes a lot of rules to be implicitly included in products where they might not be welcome
• Any set of values causes the need to process a huge amount of rule files when a product is being added, removed or updated (e.g. new products)
• It duplicates the platform and product properties in the most blunt and inflexible way

Proposed Change

The proposal is to reverse the connection between product and rule by referring to a rule from the product profile instead of referring to a product from the rule: roughly rule[prodtype] -> product will be replaced with product[profile][rule_id] -> rule.

This change will fix the most painful problems of prodtype and also will open the way for improvements with detailed control over datastream contents:

• While rules will retain the ability to be included in any product (like with all), they won't be part of any data stream until SME consciously add them
• All rules additions or removals (goes for whole products as well) will be confined to product's directory, preventing spaghetti-like PRs
• It will eliminate the possibility of one SME accidental interference with other SME's product content
• It will allows SME to clean up their benchmarks (remove stray rules etc) without massive disruptive changes between 'all' and a set of prodtypes
• It will finally allow to have a good answer to the question 'Why the hell this rule belongs to this product` without too much effort and mental gymnastic

Obviously, one of the requirements for this change would be to retain backward-compatibility in the behavior of the build system and minimization of the turbulence. Ideally SMEs should not notice the change at all up until the moment they would like to fiddle with benchmark content.

Implementation Details

There are 3 paths the conversion could be applied:

1) For a rule that has product's prodtype (or all, or implicit all) but is not listed in any profile (or a control file):

We add it to a new 'default' profile to keep the rule in the benchmark. In order to not disrupt the product final benchmark we make this profile hidden. It is a novelty to the build system: a hidden profile is not rendered as XCCDF Profile, but takes rules with it into the Benchmark.

This usually is a case of a stray rule (of which we have about 800), the rule that is a part of a product because god knows why. Later on SMEs would be able to remove there rules from the profile, move them to real profiles or even create a set of hidden profiles to structure them.

2.1) For a rule that has product's prodtype (or all, or implicit all) and is listed in a profile (or a control file):

and

2.2) For a rule that doesn't have product's prodtype (or all, or implicit all) and is not listed in a profile (or a control file):

The most straightforward thing: do nothing, all good.

3.1) For a rule that does not have product's prodtype and listed in a profile (or a control file):

We add the rule exclude id to the profile (or a profile that expands a control set): !rule_id, which yield the same result, but
in a more visible way from SMEs point of view as the exclusion accompanies the control expansion property.

The same goes for a regular profile (instead of plain removal of the entry from profile file) for two reasons:

• It is visible for SME without the need to check the actual PR changelog
• This also works for regular profiles that inherit each other

3.2) There are 2 special cases with control files:

• When the control file is being used by multiple products with the same pool of rules (benchmark_root)

We only exclude rules that match the product-profile pair, which creates a visible and clear difference between same (similar) profile files.

• When the control file is being used by multiple products with different pools of rules (I'm looking at you OpenShift, you naughty boy!)

We only exclude rules that match benchmark_root-product-profile triplet. This might be perceived as a disruption, while it, in fact, just highlights the awkwardness of the approach.

Roadmap

The contingency plan is quite simple.

• We ignore the prodtype, while excluding rules that are not part of a profile and make the conversion:
  Refactoring: Remove 'prodtype' Mk.2 #11378
• We remove all the remnants of prodtype: build system, rules that escaped mass conversion, documentation, tests, anything that is related to 'prodtype': Refactoring: Remove all references to prodtype (code/tests/docs) #11505

References:

1. Example of PRs with new products: https://github.com/ComplianceAsCode/content/pulls?q=is%3Apr+label%3A%22New+Product%22

[jan-cerny]

I like this proposal. The problem description is really spot on! I think the proposed changes will help us simplify and streamline the project.

[marcusburghardt]

It is good to remove what is not necessary and simplify things. Currently, the prodtype usage is not consistent and this makes it less beneficial.

Ideally the applicability of rules should be done by CPE-AL (platform: ...) and this is much more tangible after the #10609, but not all rules have well defined platform yet and when they also don't have a correct definition of prodtype we eventually see issues such as rules not relevant to a product being included in the product Datastream. This ultimately causes problems if the rule is tried to be used.

This proposal can mitigate this issue caused by lack or incorrect definition of prodtype and can also help us to more efficiently review the rules platform definition.

My only remark is to be careful about how to keep the backward-compatibility simple.
But I consider this as implementation details to be discussed in PRs since the idea is already expressed in the proposal.

[marcusburghardt]

FYI @ComplianceAsCode/oracle-maintainers @ComplianceAsCode/red-hatters @ComplianceAsCode/suse-maintainers @ComplianceAsCode/ubuntu-maintainers

[yuumasato]

We add the rule exclude id to the profile (or a profile that expands a control set): !rule_id, which yield the same result, but
in a more visible way from SMEs point of view as the exclusion accompanies the control expansion property.

Is my understanding correct that the !rule_id added to the profiles will be removed once the implicit prodtype all is not in effect anymore?

[evgenyz]

We add the rule exclude id to the profile (or a profile that expands a control set): !rule_id, which yield the same result, but
in a more visible way from SMEs point of view as the exclusion accompanies the control expansion property.

Is my understanding correct that the !rule_id added to the profiles will be removed once the implicit prodtype all is not in effect anymore?

No. It might have to stay for a moment. There are 2 options:

• if this !rule_id was set in a profile where there is rule_id present directly, one could just remove the pair and have the rule not included in a profile (if the rule is not referenced in a different profile of that product or a control file, it will also be excluded from the data stream).
• if this !rule_id was set to counter rule_id inclusion in a control file, it'd have to stay in the profile that expand the control file to replace prodtype filter effect. This is actually the case only for some OpenShift content with very peculiar control files. Once these files are cleaned — these profiles could then be cleaned from !rule_ids as well.