PCS hardening updates

[jgleissner]

This moves the hardening from config.sh to images.sh so it is executed last. Also disables rules that need running systemd or require /proc, and adds remediation for the disabled rules.

[brett060102]

This change will beak the link between what is in the upstream hardening profiles and what we are doing at image build time. If we remove these rules from the upstream profile, then what do at image build time will be missing from that profile. It also means that any change to the upstream profile might break our image builds. We do have a
requirement a user to have a supported process to get an existing unhardened image to the same state as hardened image. The goal was to use the SAP hardening profile to do this.

[jgleissner]

There is already a disconnect between what we do at build time and what those profiles do due to the fact that they do different things in an online system and a chroot env. Plus we hand-pick rules that are not in the SAP hardened profile. So AFAICS this is something that needs to be sorted upstream.

[brett060102]

Agreed that this needs to be sorted out upstream. And while there is a disconnect between what happens on live systems and during image hardening, currently what come out of the hardening process matches very well with what users see if they apply the profile to a running unhardened system. I think with Robert's PR any inconsistencies were addressed.

I think we need to move in a direction That will match where we would like upstream to end at, and I think that would be to either skip retest when doing hardening or just the portion of the test that requires system services is skipped.

In either case the desired upstream solution is more inline with not failing if the retest fails, than not using oscap at all.

And we do have a customer requirement to provide a path for running instances and that requirement does require oscap. It needs to be driven from SUSE manager.

So, we are between a rock and hard place and we need to pick the least offensive solution.

[brett060102]

I think our goals the following
1: determine what fixes need to be made
2: make those fixes
3: have process driven by upstream profiles

The problem we have right now, is that there is a recheck after making a fix and the recheck fails because the checks are not smart enough to know we are working not working on a live system.

We could just execute the sle15-script-pcs-hardening-sap.sh or sle15-script-pcs-hardening.sh from the security guide, but that would have us potentially trying to fix things that are not broken.

I have tested the following and it looks like it does the right thing:

keg: included from hardened-config

get failing SAP rules

echo "get failing SAP rules,run oscap xccdf eval --profile pcs-hardening-sap"
oscap xccdf eval --results /tmp/SAP_fixes.xml --profile pcs-hardening-sap /usr/share/xml/scap/ssg/content/ssg-sle15-ds.xml || {
echo "!!!expected FAILED: we expect initial check to fail for some rules, --profile pcs-hardening-sap"
}

generate remediation script

echo "run oscap -profile pcs-hardening-sap"
oscap xccdf generate fix --fix-type bash --output /tmp/SAP_fixes.sh --result-id xccdf_org.open-scap_testresult_xccdf_org.ssgproject.content_profile_pcs-hardening-sap /tmp/SAP_fixes.xml || {
echo "!!!FAILED: creating /tmp/SAP_fixes.sh"
/bin/fail
}

run sap fix script

/bin/bash /tmp/SAP_fixes.sh

The first part generates a xml file with all of the failing rules from the SAP hardening profile, it is expected that oscap returns an error on that run, because it returns an error code if any test fails. And at this point we expect failing tests.

The second uses those results to create a remediation shell script.
Third, is to just execute the shell script.
This avoids the retest after remediation, but is still only driven by the upstream hardening profiles.

It is a bit convoluted, but given how upstream currently works, I think it is a good as we can get.

I tested this in:
https://build.suse.de/package/show/home:esampson:keg-images:SLE-15-SP4/SLES15-SP4-SAP-Hardened-BYOS

In my test, I also added a retest after running the script and with the exception of ensure_logrotate_activated which requires a running system for test, all tests passed. I added the retest just to check how well the script did, It would not be in production.
Since it is the retest we are trying to avoid.

[jgleissner]

We could run the script, but we might as well just ignore oscap returning errors (maybe limited to what we know fails in chroot), so I'm not sure what benefit that it would give us. Btw, running that script without /proc throws a lot of /dev/fd/62: No such file or directory errors due to usage of < <(...) but it seems remediation works for the most part, except for pam_disable_automatic_configuration AFAICS.

Also, what about the extra rules that are currently applied? Are they not actually required?

[rjschwei]

@brett060102 I do not follow your reasoning. The extra rules that are in config.sh and that we loop over are not in the upstream hardening profile for SAP, they are in the cis-l1 profile, I think. As such there is a mismatch. This is why I had asked for an upstream PR that adds the rules to the pcs-hardening-sap profile. As such I think we have a good plan to get to the consistency we are looking for.

Adding the config file to sysctl.d allows us to turn on error checking for rules application which will help us when the next security-guide gets picked up, otherwise we are in a situation where we are basically blind to additional errors. As such I think we should move forward as we have a plan to get to the consistency we want to achieve and we have a pth to enabling error checking as quickly as possible.

[brett060102]

@rjschwei I have PR ComplianceAsCode/content#10965 to add the rules in the CIS loop to the SAP hardening profile. The loop is only there now because of the lead time between landing a change upstream and when it shows up in SLE. The goal is to have this all driven but what is in the security guide package.

[jgleissner]

Given that it'll take a while until the upstream PR will trickle down into a package, should we just merge this work-around so we're not blocked anymore? It gives us reasonable confidence that remediation is applied because it will abort the build if any of the rules fail, with the ones that don't work in the build env being handled as extra code and overlay files. Once upstream changes land we can start removing the work-arounds.

One small thing though, I've just pushed another commit that changes execution of the extra rules from one at a time to a single oscap call. This avoids parsing the 16 MB rules xml file over and over again and should speed things up a bit. Would be good if someone could give this another glance over.

[jgleissner]

There seems to be the concern that is once a security-guide update is available the image builds will fail, which is true due to the sysctl rules not playing nicely when there's no /proc. I wouldn't consider this a big problem as we will have to touch the descriptions anyway.

As long as there is no true chroot mode we will probably have to actually disable them, or mount /proc so they don't fail, or use the bash remediation script route and lose error checking. All not great, but I suggest we cross the bridge when we get there.

To move forward with this, we can either merge this PR as-is, change it back to use oscap for sysctl rules and ignore errors, or go the bash route which ignores all errors.

[rjschwei]

Well, having the sysctl rules included upstream at this time was not what I had hoped for it would sure have been a lot easier to not have those in the upstream profile at this time until the sysctl (ComplianceAsCode/content#10967) issue is resolved. Anyway this means pcs-hardening-workarounds.sh needs to be expanded to disable the sysctl rules. @jgleissner can you please add the syctl rules that are in https://github.com/ComplianceAsCode/content/pull/10965/files to this PR and then we should be good to go.

[jgleissner]

Can do. We could simply disable all rules in the catalog that have sysctl in the name to be on the safe side. We may miss some remediation but not sure what else to do. This may stay messy for a while.

[brett060102]

Based in 5.2 and 5.3 in https://static.open-scap.org/openscap-1.3/oscap_user_manual.html off line remediation is supported when using the test and generate the shell script model. I have created an image using that model and it has the same pass rate as the ignore failures code.

I can put a PR for that next week if that is the desired solution.

If the preferred solution is to not attempt remediation any remediation on systemd based rules at all, then just list the offending rules here and I will pull those from the SAP harding profile.

I think that we should only do remediation via the oscap process though. I think duplicating remediation through other means might create maintenance problems in the future.

[jgleissner]

What they call offline is just a two-step execution and still requires an online system for systemd and sysctl rules AFAICS. I don't see how that changes things.

Using the rules instead of the captured remediation files is definitely preferable, not doubt about that. But for that they need to work in the first place. We could use the sysctl rules by running them through oscap and ignoring the errors to avoid duplication at this point. I don't think there's a way to make it work with systemd rules, outside using the shell script.

[brett060102]

Section 5 of that doc says offline is supported.

[rjschwei]

After reading the docs I agree with @jgleissner. What is stated in the doc as "offline" is not equal to "not a running system". The remediation and the checks for the way systemd and sysctl rules are applied and verified still require a running system. We need to continue to push upstream for a mode that does not require a running system. Until then we have to remove the rules that have such a requirement and for sysctl rules we can put he configuration files in place as is implemented in this PR.

[jgleissner]

It sure reads the word offline, but it appears they refer to "remediation from a results file" as offline and don't mean the target system, which does need to be online for this to work.

There are two related tools, oscap-chroot and oscap-remediate-offline, which at a first glance look like they could be useful, but oscap-chroot can really only do evaluation which doesn't help us. But, when running it I got messages like the following:

W: oscap:         Requested offline mode is not supported by uname probe.

That hints to the fact that oscap probes are intended to have an offline mode, at least optionally, so perhaps the basic framework for offline mode is actually in oscap, which would be good news.

The oscp-remediate-offline tool uses the systemd offline update feature to perform remediation at boot time, which isn't really what we want either.

[brett060102]

oscap supports env vars for scanning and remediating mount FS. OpenSCAP/openscap#778 Has some details. Can talk more when I get back, but given this PR has merged is right place to continue the discussion?

[rjschwei]

That (#778) is only about scanning in a chroot. As long as remediation includes something like sudo systemctl start auditd.service it will never work on a non running system. This code would have to be surrounded by an if condition that checks is there is a pid 1 and if that pid 1 is systemd as described in ComplianceAsCode/content#10945. Similar applies for sysctl rules where changes are made in /proc