Tests for secret handling in provider configuration #2440
Comments
|
TBD move to the bridge |
|
Jotting this down as part of discussion re: #2439 |
|
@iwahbe brought up a good point that due to pulumi/pulumi#15032 the language dimension is pertinent here. We do not know from a YAML-based test that C# or TypeScript are going to work because languages differ due to SDK handling in how they pass config information over the wire. We discussed a reluctance to add language-specific testes to the bridge codebase as in an idealized world language variance would be guaranteed by pulumi/pulumi machinery and out of scope for the bridged codebase. In the immediate term Ian is to introduce replay tests that mimic TypeScript behavior. It is however not entirely satisfactory as we will not be able to push #2439 forward without having tests that assure us that bridged providers behave correctly for configuration in each language after the change, and there is the Gordian knot that the bridge does not bottom out at a supported Go SDK. This could be done either by language-variant tests in the bridge or else by introducing tests in pulumi/pulumi that depend on bridged providers. Expecting some pushback both ways but we'll need to pick our poison to move the 2439 forward. |
iwahbe
added
kind/engineering
|
Put it on the list for M111 ( pulumi/pulumi#15032 ).. even if we don't put language tests in the bridge we should be able to solidly test 15032 through Tier1 providers, even if we need temporary tests to do this. |
|
Thanks for picking this up! I've been collecting some more feedback and developing the ideas in https://docs.google.com/document/d/1cnBJzAGWMnjjj2Q5fDdVcq0DcrWd-OqvPPlSxQV50R4/edit?usp=sharing TLDR trying to take care of the language dimension in pulumi/pulumi conformance tests. But this is very much needed too. |
The interface for configure cross-tests is:
```go
func Configure(
t *testing.T,
schema schema.Schema, // The provider's schema
tfConfig map[string]cty.Value, // Config values for HCL/TF
puConfig resource.PropertyMap, // Config values for Pulumi YAML/Pulumi
)
```
Because the mapping between `tfConfig` and `puConfig` is non-trivial,
and to allow for secret injection, I have chosen to allow setting
`tfConfig` and `puConfig` separately.
This is a pre-requisite for #2440.
---
When the underlying provider configuration doesn't match between Pulumi
and TF, the error looks like this:
```go
func TestFailure(t *testing.T) {
crosstests.Configure(t,
schema.Schema{Attributes: map[string]schema.Attribute{
"k": schema.StringAttribute{Optional: true},
}},
map[string]cty.Value{"k": cty.StringVal("foo")},
resource.PropertyMap{"k": resource.NewProperty("oo")},
)
}
```
```
go test -test.run \^TestFailure\$
warning: resource:test_res.Docs received a non-zero custom value &{ [32] false false} that is being ignored. Plugin Framework based providers do not yet support this feature.
--- FAIL: TestFailure (2.25s)
configure.go:129: Pulumi.yaml:
backend:
url: file://./data
name: project
resources:
p:
properties:
k: oo
type: pulumi:providers:test
runtime: yaml
--- FAIL: TestFailure/compare (0.00s)
configure.go:174:
Error Trace: /Users/ianwahbe/go/src/github.com/pulumi/pulumi-terraform-bridge/pf/tests/internal/cross-tests/configure.go:174
Error: Not equal:
expected: tfsdk.Config{Raw:tftypes.Value{typ:tftypes.Object{AttributeTypes:map[string]tftypes.Type{"k":tftypes.primitive{name:"String", _:[]struct {}(nil)}}, OptionalAttributes:map[string]struct {}(nil), _:[]struct {}(nil)}, value:map[string]tftypes.Value{"k":tftypes.Value{typ:tftypes.primitive{name:"String", _:[]struct {}(nil)}, value:"foo"}}}, Schema:schema.Schema{Attributes:map[string]schema.Attribute{"k":schema.StringAttribute{CustomType:basetypes.StringTypable(nil), Required:false, Optional:true, Sensitive:false, Description:"", MarkdownDescription:"", DeprecationMessage:"", Validators:[]validator.String(nil)}}, Blocks:map[string]schema.Block(nil), Description:"", MarkdownDescription:"", DeprecationMessage:""}}
actual : tfsdk.Config{Raw:tftypes.Value{typ:tftypes.Object{AttributeTypes:map[string]tftypes.Type{"k":tftypes.primitive{name:"String", _:[]struct {}(nil)}}, OptionalAttributes:map[string]struct {}(nil), _:[]struct {}(nil)}, value:map[string]tftypes.Value{"k":tftypes.Value{typ:tftypes.primitive{name:"String", _:[]struct {}(nil)}, value:"oo"}}}, Schema:schema.Schema{Attributes:map[string]schema.Attribute{"k":schema.StringAttribute{CustomType:basetypes.StringTypable(nil), Required:false, Optional:true, Sensitive:false, Description:"", MarkdownDescription:"", DeprecationMessage:"", Validators:[]validator.String(nil)}}, Blocks:map[string]schema.Block(nil), Description:"", MarkdownDescription:"", DeprecationMessage:""}}
Diff:
--- Expected
+++ Actual
@@ -18,3 +18,3 @@
},
- value: (string) (len=3) "foo"
+ value: (string) (len=2) "oo"
}
Test: TestFailure/compare
FAIL
exit status 1
FAIL github.com/pulumi/pulumi-terraform-bridge/pf/tests 3.059s
```
|
This PR has been shipped in release v3.97.0. |
|
This issue has been addressed in PR #2716 and shipped in release v3.98.0. |
For the following combinations:
Check that first-class secrets work as expected:
Check that schema-based secrets work as expected:
Some suggestions on "bulletproof" test suite at integration level to work around concerns of mixing things up around provider configuration due to Config Encoding.
CC @VenelinMartinov @iwahbe @guineveresaenger
One other thing to call out is checking upgrade scenarios where DiffConfig() might be in play and receive a previous version of the schema.
The text was updated successfully, but these errors were encountered: