compare_ds tool is reporting that rules are not part of ssg-product-oval anymore

[ggbecker]

As observed in #10406 (comment)

excerpt:

Rule 'xccdf_org.ssgproject.content_rule_prefer_64bit_os' points to 'ssg-rhel8-oval.xml' which isn't a part of the old datastream
Rule 'xccdf_org.ssgproject.content_rule_prefer_64bit_os' points to 'ssg-rhel8-oval.xml' which isn't a part of the new datastream
Rule 'xccdf_org.ssgproject.content_rule_rpm_verify_hashes' points to 'ssg-rhel8-oval.xml' which isn't a part of the old datastream
Rule 'xccdf_org.ssgproject.content_rule_rpm_verify_hashes' points to 'ssg-rhel8-oval.xml' which isn't a part of the new datastream
Rule 'xccdf_org.ssgproject.content_rule_rpm_verify_ownership' points to 'ssg-rhel8-oval.xml' which isn't a part of the old datastream
Rule 'xccdf_org.ssgproject.content_rule_rpm_verify_ownership' points to 'ssg-rhel8-oval.xml' which isn't a part of the new datastream
Rule 'xccdf_org.ssgproject.content_rule_rpm_verify_permissions' points to 'ssg-rhel8-oval.xml' which isn't a part of the old datastream
Rule 'xccdf_org.ssgproject.content_rule_rpm_verify_permissions' points to 'ssg-rhel8-oval.xml' which isn't a part of the new datastream
Rule 'xccdf_org.ssgproject.content_rule_package_aide_installed' points to 'ssg-rhel8-oval.xml' which isn't a part of the old datastream
Rule 'xccdf_org.ssgproject.content_rule_package_aide_installed' points to 'ssg-rhel8-oval.xml' which isn't a part of the new datastream
Rule 'xccdf_org.ssgproject.content_rule_aide_build_database' points to 'ssg-rhel8-oval.xml' which isn't a part of the old datastream

From this block of code:

content/ssg/content_diff.py

Lines 171 to 186 in 366ba8c

	def get_check_docs(self, system, identifier, old_check_file_name, new_check_file_name):
	try:
	old_check_doc = self.old_content.components.get(system)[old_check_file_name]
	except (KeyError, TypeError):
	print(
	"Rule '%s' points to '%s' which isn't a part of the "
	"old datastream" % (identifier, old_check_file_name))
	old_check_doc = None
	try:
	new_check_doc = self.new_content.components.get(system)[new_check_file_name]
	except (KeyError, TypeError):
	print(
	"Rule '%s' points to '%s' which isn't a part of the "
	"new datastream" % (identifier, new_check_file_name))
	new_check_doc = None
	return old_check_doc, new_check_doc

There is likely some change needed in the content_diff.py script to handle this exception properly

[ggbecker]

I guess this makes the compare_ds service kind of useless if there is always this kind of output

[ggbecker]

Same observed in: #10403 (comment)

[ggbecker]

Apparently it's loading the cpe-oval content instead of oval content:
Rule 'xccdf_org.ssgproject.content_rule_prefer_64bit_os' points to 'ssg-rhel9-oval.xml' which isn't a part of the old datastream 'OVAL' '{'OVAL': {'ssg-rhel9-cpe-oval.xml': <ssg.xml.XMLComponent object at 0x7fb08a148910>}, 'OCIL': {'ssg-rhel9-ocil.xml': <ssg.xml.XMLComponent object at 0x7fb08a14b550>}}'

[ggbecker]

The new DS has a new entry of cpe in:

    <ds:checklists>
      <ds:component-ref id="scap_org.open-scap_cref_ssg-rhel9-xccdf.xml" xlink:href="#scap_org.open-scap_comp_ssg-rhel9-xccdf.xml">
        <cat:catalog>
          <cat:uri name="ssg-rhel9-cpe-oval.xml" uri="#scap_org.open-scap_cref_ssg-rhel9-cpe-oval.xml"/>
          <cat:uri name="ssg-rhel9-oval.xml" uri="#scap_org.open-scap_cref_ssg-rhel9-oval.xml"/>
          <cat:uri name="ssg-rhel9-ocil.xml" uri="#scap_org.open-scap_cref_ssg-rhel9-ocil.xml"/>
          <cat:uri name="security-data-oval-com.redhat.rhsa-RHEL9.xml.bz2" uri="#scap_org.open-scap_cref_security-data-oval-com.redhat.rhsa-RHEL9.xml.bz2"/>
        </cat:catalog>
      </ds:component-ref>
    </ds:checklists>

whereas the old datastream doesn't have:

    <ds:checklists>
      <ds:component-ref id="scap_org.open-scap_cref_ssg-rhel9-xccdf.xml" xlink:href="#scap_org.open-scap_comp_ssg-rhel9-xccdf.xml">
        <cat:catalog>
          <cat:uri name="ssg-rhel9-oval.xml" uri="#scap_org.open-scap_cref_ssg-rhel9-oval.xml"/>
          <cat:uri name="ssg-rhel9-ocil.xml" uri="#scap_org.open-scap_cref_ssg-rhel9-ocil.xml"/>
          <cat:uri name="security-data-oval-com.redhat.rhsa-RHEL9.xml.bz2" uri="#scap_org.open-scap_cref_security-data-oval-com.redhat.rhsa-RHEL9.xml.bz2"/>
        </cat:catalog>
      </ds:component-ref>
    </ds:checklists>

The function:

content/ssg/xml.py

Lines 224 to 242 in 366ba8c

	def _find_all_component_contents(self):
	component_doc_dict = dict()
	for component in self.root.findall("ds:component", self.ns):
	for check_spec in self.check_engines:
	def_doc = component.find(check_spec[1], self.ns)
	if def_doc is not None:
	def_doc_dict = dict()
	comp_id = component.get("id")
	comp_href = "#" + comp_id
	try:
	filename = self.uris["#" + self.component_refs[comp_href]]
	except KeyError:
	continue
	def_doc_dict[filename] = XMLComponent(def_doc)
	component_doc_dict[check_spec[0]] = def_doc_dict
	# This component matched one of the checking engines,
	# thre is no need to continue further
	break
	return component_doc_dict

loads the date and since CPE also has the oval_definitions it is loading the CPE instead of the ssg-product-oval.xml file.

[ggbecker]

index 5eaf1c3de2..ae8696650d 100644
--- a/ssg/xml.py
+++ b/ssg/xml.py
@@ -234,6 +234,8 @@ class XMLContent(XMLElement):
                         filename = self.uris["#" + self.component_refs[comp_href]]
                     except KeyError:
                         continue
+                    if "cpe" in filename:
+                        continue
                     def_doc_dict[filename] = XMLComponent(def_doc)
                     component_doc_dict[check_spec[0]] = def_doc_dict
                     # This component matched one of the checking engines,

If we skip loading the cpe it seems to work, but it might not be the correct way of fixing it.

@jan-cerny can you take a look at this one?

[jan-cerny]

@ggbecker The fix seems to work. Thanks for the investigation! I think that when we are here we can make the method less clumsy and more generic. I will emit a PR.