Integration test TestAccSimpleTs is flaky

[thomas11]

gh flaky -workflows release,master,acceptance-test,autorest-scheduled -lookback 21
Running as thomas11

Checking last 100 runs of the last 21 days of workflows:
    {2576859 release .github/workflows/release.yml}
    {2342475 master .github/workflows/master.yml}
    {46530939 acceptance-test .github/workflows/acceptance-test.yml}
    {127814901 autorest-scheduled .github/workflows/autorest-scheduled.yml}
Starting search from 2024-12-10

Found 1 runs with retries (out of 4) for workflow 'release'
Checking failed attempts: .

Found 4 runs with retries (out of 19) for workflow 'master'
Checking failed attempts: ....

Found 1 runs with retries (out of 81) for workflow 'acceptance-test'
Checking failed attempts: .

Found 5 runs with retries (out of 19) for workflow 'autorest-scheduled'
Checking failed attempts: .....

Examined 123 workflow runs, out of which 11 succeeded after retries.
Found 11 failed jobs, out of which 11 had logs available.
Failed tests by test output: map[Gotestfmt:0 Plain Go:11]

TestAccSimpleTs   9 failures
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12463606736/job/34786978863
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12462133808/job/34783202203
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12420208944/job/34678091864
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12319625744/job/34388101906
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12460662563/job/34779655418
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12405659040/job/34633435748
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12365771126/job/34511973802
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12289257385/job/34294999012
    https://github.com/pulumi/pulumi-azure-native/actions/runs/12554999262/job/35004805873

[thomas11]

ProgramTest runs up, then refresh, then preview, and asserts that the refresh didn't cause any changes, i.e., the preview is empty. Which is not the case here.

error: error: no changes were expected but changes were proposed
    command.go:98: Invoke '/opt/homebrew/bin/pulumi --logflow --logtostderr -v=9 preview --non-interactive --expect-no-changes' failed: exit status 255

command.go:46: **** Invoke '/opt/homebrew/bin/pulumi refresh --non-interactive --yes --skip-preview' in '/var/folders/xm/974bx_2j3cxb3v_7czt_3l7c0000gn/T/p-it-tmplocal-simple-7171b914-741257735'
Refreshing (p-it-tmplocal-simple-7171b914):
@ refreshing....
 ~  azure-native:network:VirtualNetwork vnet updated (1s) [diff: ~subnets]
 ~  azure-native:web:WebApp app updated (3s) [diff: +virtualNetworkSubnetId~siteConfig]
 ~  azure-native:web:WebAppSlot slot updated (3s) [diff: +siteConfig,virtualNetworkSubnetId]
Resources:
    ~ 3 updated
    16 unchanged

WebApp and WebAppSlot compete for siteConfig and virtualNetworkSubnetId, which non-deterministically flop between them.

command.go:46: **** Invoke '/opt/homebrew/bin/pulumi preview --non-interactive --expect-no-changes --diff' in '/var/folders/xm/974bx_2j3cxb3v_7czt_3l7c0000gn/T/p-it-tmplocal-simple-7171b914-741257735'
Previewing update (p-it-tmplocal-simple-7171b914):
  pulumi:pulumi:Stack: (same)
    [urn=urn:pulumi:p-it-tmplocal-simple-7171b914::simple::pulumi:pulumi:Stack::simple-p-it-tmplocal-simple-7171b914]
    ~ azure-native:web:WebAppSlot: (update)
        [id=/subscriptions/0282681f-7a9e-424b-80b2-96babd57a8a1/resourceGroups/rg69016507/providers/Microsoft.Web/sites/app1db95ed6/slots/slot]
        [urn=urn:pulumi:p-it-tmplocal-simple-7171b914::simple::azure-native:web:WebAppSlot::slot]
        [provider=urn:pulumi:p-it-tmplocal-simple-7171b914::simple::pulumi:providers:azure-native::default_2_0_0_alpha_0_dev::7e2cd651-d993-44d4-9043-f26c8ee32e01]
      - siteConfig: { everything removed }

Resources:
    ~ 1 to update
    18 unchanged
error: error: no changes were expected but changes were proposed

The reason we're not seeing more changes in the preview is that there already are some ignoreChanges in the test.

const appService = new web.WebApp("app", {
    resourceGroupName: resourceGroup.name,
    serverFarmId: appServicePlan.id,
    kind: "app",
    siteConfig: {
        appSettings: [{
            name: "test",
            value: "this is a slot setting",
        }],
    },
// Subnet is associated by WebAppSwiftVirtualNetworkConnection below, so it should be ignored here to avoid
// overrides on refresh-update cycles.
// SiteConfig is modified outside of this resource via the WebApp* resources.
}, { ignoreChanges: ["virtualNetworkSubnetId", "siteConfig", "siteConfig.*"] });

const slot = new web.WebAppSlot("slot", {
    resourceGroupName: resourceGroup.name,
    name: appService.name,
// Subnet is associated by WebAppSwiftVirtualNetworkConnection, so it should be ignored here to avoid
// overrides on refresh-update cycles.
}, { ignoreChanges: ["virtualNetworkSubnetId"] });

[pulumi-bot]

This issue has been addressed in PR #3830 and shipped in release v2.81.0.