cdk synth returns exit code 0 despite policy validation failures

[rittneje]

Describe the bug

If a custom policy validation reports a failure, cdk synth still exits successfully (exit code 0).

Expected Behavior

A policy validation failure should result in a non-zero exit code.

Current Behavior

It reports the failure on stdout/stderr but still exits successfully.

Reproduction Steps

@jsii.implements(aws_cdk.IPolicyValidationPluginBeta1)
class MyPlugin:
    def validate(self, context):
        return aws_cdk.PolicyValidationPluginReportBeta1(success=False)

def main():
    app = aws_cdk.App(policy_validation_beta1=[MyPlugin()])
    env = aws_cdk.Environment(region="us-east-1", account="...")
    stack = aws_cdk.Stack(app, "test-stack", env=env)
    app.synth()

if __name__ == "__main__":
    main()

cdk synth
echo $?

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.85.0 (build 4e0d726)

Framework Version

No response

Node.js Version

v18.16.1

OS

Alpine 3.18

Language

Python

Language Version

Python 3.11.4

Other information

No response

[peterwoodworth]

We are setting the exitCode to 1 here

aws-cdk/packages/aws-cdk-lib/core/lib/private/synthesis.ts

Line 162 in 9ccab73

process.exitCode = 1;

We know that the code is getting to here because the message right above it is printing in the console before succeeding

So I tried this on TypeScript, and the process successfully failed. So this would be an issue or limitation of JSII

It's also interesting that the Performing Policy Validations loggings statement seems to appear late in Python compared to TypeScript

[peterwoodworth]

Hey, we're going to keep the design this way. Thanks for bringing this up still 🙂

The reason for that is that policy validation plugin is not an enforcement tool. As a developer, you need to opt in for the plugin, you also have an option to skip some rules. There is nothing that stops you from deploying the generated template, despite of policies violations. So, we decided that there is no point to return different values, since this can be confusing (return non-zero exit code can be trggered by other issues)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[rittneje]

@peterwoodworth Then the plugin is completely worthless. Please re-open this issue until it is properly resolved.