Profile resolution flattens nested control structure

[GaryGapinski]

Describe the bug

Profile resolution of a nested control includes ancestor control but the control nesting is not maintained.

Who is the bug affecting

Users of oscal-profile-RESOLVE.xsl.

What is affected by this bug

Tooling & API

How do we replicate this issue

I was wondering if inclusion of a control enhancement would include the parent control.

It does. This is in accord with SP 800-53 rev5 (§2.2 ¶2 final sentence) and the profile resolution specification. However, the nested control structure is flattened.

Using this profile

<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="https://github.com/usnistgov/OSCAL/raw/v1.0.4/xml/schema/oscal_complete_schema.xsd" schematypens="http://www.w3.org/2001/XMLSchema" title="OSCAL complete schema" ?>
<profile xmlns="http://csrc.nist.gov/ns/oscal/1.0" uuid="3353e8e1-3eef-42ef-8163-1f71616bafca">
    <metadata>
        <title />
        <last-modified>2023-02-18T13:06:18Z</last-modified>
        <version>2023-02-18T13:06:18Z</version>
        <oscal-version>1.0.4</oscal-version>
    </metadata>
    <import href="c-ce-c.xml">
        <include-controls>
            <with-id>c1.1</with-id>
        </include-controls>
    </import>
</profile>

and catalog

<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="https://github.com/usnistgov/OSCAL/raw/v1.0.4/xml/schema/oscal_complete_schema.xsd" schematypens="http://www.w3.org/2001/XMLSchema" title="OSCAL complete schema" ?>
<catalog xmlns="http://csrc.nist.gov/ns/oscal/1.0" uuid="37b7acd4-ab3f-4217-99d1-2ab9e490db49">
    <metadata>
        <title />
        <last-modified>2023-02-18T13:06:18Z</last-modified>
        <version>2023-02-18T13:06:18Z</version>
        <oscal-version>1.0.4</oscal-version>
    </metadata>
    <control id="c1">
        <title />
        <control id="c1.1">
            <title />
        </control>
    </control>
</catalog>

perform a profile resolution.

gapinski@flexion-mac-C02FCBVSMD6N ~ % cd ~/Projects/github/usnistgov/OSCAL/src/utils/util/resolver-pipeline  
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % git status                                                             
On branch main
Your branch is up to date with 'origin/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	c-ce-c.xml
	c-ce-p.xml
	c-ce.zip

nothing added to commit but untracked files present (use "git add" to track)
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % cat c-ce-p.xml                                                         
<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="https://github.com/usnistgov/OSCAL/raw/v1.0.4/xml/schema/oscal_complete_schema.xsd" schematypens="http://www.w3.org/2001/XMLSchema" title="OSCAL complete schema" ?>
<profile xmlns="http://csrc.nist.gov/ns/oscal/1.0" uuid="3353e8e1-3eef-42ef-8163-1f71616bafca">
    <metadata>
        <title />
        <last-modified>2023-02-18T13:06:18Z</last-modified>
        <version>2023-02-18T13:06:18Z</version>
        <oscal-version>1.0.4</oscal-version>
    </metadata>
    <import href="c-ce-c.xml">
        <include-controls>
            <with-id>c1.1</with-id>
        </include-controls>
    </import>
</profile>
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % cat c-ce-c.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-model href="https://github.com/usnistgov/OSCAL/raw/v1.0.4/xml/schema/oscal_complete_schema.xsd" schematypens="http://www.w3.org/2001/XMLSchema" title="OSCAL complete schema" ?>
<catalog xmlns="http://csrc.nist.gov/ns/oscal/1.0" uuid="37b7acd4-ab3f-4217-99d1-2ab9e490db49">
    <metadata>
        <title />
        <last-modified>2023-02-18T13:06:18Z</last-modified>
        <version>2023-02-18T13:06:18Z</version>
        <oscal-version>1.0.4</oscal-version>
    </metadata>
    <control id="c1">
        <title />
        <control id="c1.1">
            <title />
        </control>
    </control>
</catalog>
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % alias xslt='java -cp ~/saxon/saxon-he-12.0.jar net.sf.saxon.Transform' 
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % xslt
No source file name
SaxonJ-HE 12.0 from Saxonica
Usage: see http://www.saxonica.com/documentation/index.html#!using-xsl/commandline
Format: net.sf.saxon.Transform options params
Options available: -? -a -catalog -config -cr -diag -dtd -ea -expand -explain -export -ext -im -init -it -jit -json -l -lib -license -nogo -now -ns -o -opt -or -outval -p -quit -r -relocate -repeat -s -sa -scmin -strip -t -T -target -threads -TJ -Tlevel -Tout -TP -TPxsl -traceout -tree -u -val -versionmsg -warnings -x -xi -xmlversion -xsd -xsdversion -xsiloc -xsl -y --?
Use -XYZ:? for details of option XYZ
Params: 
  param=value           Set stylesheet string parameter
  +param=filename       Set stylesheet document parameter
  ?param=expression     Set stylesheet parameter using XPath
  !param=value          Set serialization parameter
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % xslt -xsl:oscal-profile-RESOLVE.xsl -s:c-ce-p.xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://csrc.nist.gov/ns/oscal/1.0"
          uuid="2f121ef6-3503-4731-afdc-05e7fb3ba092">
   <metadata>
      <title/>
      <last-modified>2023-02-19T08:08:20.01928-05:00</last-modified>
      <version>2023-02-18T13:06:18Z</version>
      <oscal-version>1.0.4</oscal-version>
      <link rel="resolution-source" href=""/>
   </metadata>
</catalog>
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % 

Note that the resolution is faulty when operating on main branch of the usnistgov/OSCAL repo.

In order to demonstrate the problem, use oscal-profile-RESOLVE.xsl from #1639.

gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % xslt -xsl:https://raw.githubusercontent.com/galtm/OSCAL/saxon11/src/utils/util/resolver-pipeline/oscal-profile-RESOLVE.xsl -s:c-ce-p.xml
<?xml version="1.0" encoding="UTF-8"?>
<catalog xmlns="http://csrc.nist.gov/ns/oscal/1.0"
          uuid="00000000-0000-4000-B000-000000000000">
   <metadata>
      <title/>
      <last-modified>2023-02-19T08:22:04.145451-05:00</last-modified>
      <version>2023-02-18T13:06:18Z</version>
      <oscal-version>1.0.4</oscal-version>
      <prop name="resolution-tool"
             value="OSCAL Profile Resolver XSLT Pipeline OPRXP"/>
      <link href="file:/Users/gapinski/Projects/github/usnistgov/OSCAL/src/utils/util/resolver-pipeline/c-ce-p.xml"
             rel="source-profile"/>
   </metadata>
   <control id="c1">
      <title/>
   </control>
   <control id="c1.1">
      <title/>
   </control>
</catalog>
gapinski@flexion-mac-C02FCBVSMD6N resolver-pipeline % 

The controls are not nested. This flattening seems at odds with the (explicit) with-parent-controls attribute description.

Worse, the the OSCAL XML Schema lacks the with-parent-controls attribute for the include-controls element (#1662).

Expected behavior (i.e. solution)

The nesting structure of the controls in the catalog should be preserved.

Other comments

The specimen instance documents are in the attached archive.
c-ce.zip

[aj-stein-nist]

First up in Sprint 64, we need to confirm if this is an issue (or actually a bug) in the profile resolver itself, the specification, or both.

• Verify if the bug is in resolver first.
• Check draft unit tests (if not merged in develop) have coverage to test for this area.
• Check the specification if the functionality in the resolver matches the current draft specification.

We would like to loop in @galtm from the community for feedback as she has recently worked on components of the profile resolver that pertain to this issue.

[wendellpiez]

We do indeed have some unit tests in a WIP branch here: https://github.com/wendellpiez/OSCAL/tree/feature-profile-resolution-unittestingA (needs rebasing).

Suggest we spike merging all that outstanding specification-and-testing work, then come back to this. This branch needs rebasing plus resolving of conflicts, which we can do expeditiously (I think I know what needs to be done). Then we have a basis for more/better tests, as well as a better sense of where we are with test coverage.

[aj-stein-nist]

The first step in the checklist was confirming the bug exists in the XSLT profile resolver. Does it exist in the current version in the develop branch as reported, @wendellpiez?

[wendellpiez]

Sorry, I was trying to be 'efficient'.

I trust that what Gary reports over his data is true, although I have not made a perfect repro nor diagnosed the cause.

I can see from static examination that the develop resolver does have support for @with-parent-controls, including testing for its behavior. This does not mean it does not have a bug or that the bug being reported is being tested against.

However, there are three other unstable pieces: (1) draft Profile Resolution Specification (which defines usage), (2) schemas missing this flag (#1662), and (3) oscal-cli Java implementation (not just XSLT implementation).

I mention (3) because in order to be fully stable, we need not only a clear specification of the correct behavior, schema models and tests, but also the tools. And one explanation for the missing flag in the schema is that it was removed on purpose. If oscal-cli has no support for this, we should know about it.

Should I continue to diagnose/assess the XSLT resolver in light of tests (which I could find or make) constructed against the Specification as currently drafted? Enabling us to float issues and fix 'bugs' without engaging the question of other profile resolver(s)? In doing so I might bring more issues to light (if the spec is faulty) but we presumably need to address (and test) them across implementations, as well. Picking up Gary's demo sample as a model for a test or two certainly makes sense.

Here is testing for with-parent-controls in develop:

OSCAL/src/utils/util/resolver-pipeline/testing/1_selected/select.xspec

Line 296 in e3c4432

<x:scenario label="Default value of 'with-parent-controls' (yes)">

and also elsewhere in this file.

At least one interpretation here is that the bug is in the schemas (metaschema) -- assuming the behavior defined by the current Spec is also what is tested and implemented. If it is not, then the question is, is the Spec wrong to have this at all and is the correction to remove all support?

[aj-stein-nist]

Sorry, I was trying to be 'efficient'.

No worries, I appreciate that!

I trust that what Gary reports over his data is true, although I have not made a perfect repro nor diagnosed the cause.

I can see from static examination that the develop resolver does have support for @with-parent-controls, including testing for its behavior. This does not mean it does not have a bug or that the bug being reported is being tested against.

OK this is a good summary, it seems we are heading in a good direction and I think we should focus on that.

However, there are three other unstable pieces: (1) draft Profile Resolution Specification (which defines usage), (2) schemas missing this flag (#1662), and (3) oscal-cli Java implementation (not just XSLT implementation).

OK, this is super important for bringing up, thanks for calling it out. Let's look into 2 first. Once we hammer that out, we can check for alignment on this bug report in conjunction with 3 (that is issue 106 in oscal-cli, also reported by Gary and soon to be tracked), and then we can tighten up 1 later. But for now, let's head over to 1662.

I mention (3) because in order to be fully stable, we need not only a clear specification of the correct behavior, schema models and tests, but also the tools. And one explanation for the missing flag in the schema is that it was removed on purpose. If oscal-cli has no support for this, we should know about it.

I do not disagree, we just have to focus on one manageable piece at a time, so let's go to 1662, then the other things in that order (over the next several sprints).

Should I continue to diagnose/assess the XSLT resolver in light of tests (which I could find or make) constructed against the Specification as currently drafted? Enabling us to float issues and fix 'bugs' without engaging the question of other profile resolver(s)? In doing so I might bring more issues to light (if the spec is faulty) but we presumably need to address (and test) them across implementations, as well. Picking up Gary's demo sample as a model for a test or two certainly makes sense.

Let's hold off on this stuff for 1662 for now. If we do not make headway there before end of sprint, we come back to this.

Here is testing for with-parent-controls in develop:

OSCAL/src/utils/util/resolver-pipeline/testing/1_selected/select.xspec

Line 296 in e3c4432

<x:scenario label="Default value of 'with-parent-controls' (yes)">

and also elsewhere in this file.

Thanks for the reference.

At least one interpretation here is that the bug is in the schemas (metaschema) -- assuming the behavior defined by the current Spec is also what is tested and implemented. If it is not, then the question is, is the Spec wrong to have this at all and is the correction to remove all support?

I guess once we have a handle on 1662, we can come back to answering this question. It seems that based on some others issues and comments by Dave, we need to track down "is the issue in the metaschemas or in the resulting schema generation?" first before the spec issue (as backwards as it sounds), but I do agree: it is an important material question to this issue.