Cognitive Services RaiPolicy sending invalid content according to the ARM API response

[pierskarsenbarg]

What happened?

Trying to create a RaiPolicy resource . Using the same properties as in the example on the resource documentation (which is also in the Azure docs.

Getting error message:

azure-native:cognitiveservices:RaiPolicy (raiPolicy-pk):
    error: autorest/azure: Service returned an error. Status=400 Code="InvalidRequestContent" Message="The request content was invalid and could not be deserialized."

Example

import * as pulumi from "@pulumi/pulumi";
import * as resources from "@pulumi/azure-native/resources";
import {
    AllowedContentLevel,
    RaiPolicy,
    RaiPolicyContentSource,
    RaiPolicyMode,
    Account,
    ResourceIdentityType,
    
  } from '@pulumi/azure-native/cognitiveservices';

// Create an Azure Resource Group
const resourceGroup = new resources.ResourceGroup("pk-cognitiveservices-rg");

const account = new Account("cs-account", {
    accountName: "pk-test",
    identity: {
        type: ResourceIdentityType.SystemAssigned
    },
    kind: "CognitiveServices",
    resourceGroupName: resourceGroup.name,
    sku: {
        name: "S0"
    }
})

new RaiPolicy("raiPolicy-pk", {
    accountName: account.name,
    properties: {
        basePolicyName: "112",
        contentFilters: [
            {
                policyName: "hate",
                allowedContentLevel: "Low",
                blocking: true
            },
            {
                policyName: "sexual",
                allowedContentLevel: "Low"
            },
            {
                policyName: "violence",
                enabled: false
            },
            {
                policyName: "DefaultHateSpeechBlockList",
                enabled: false
            }
          ],
    },
    raiPolicyName: "pkraipolicyname",
    resourceGroupName: resourceGroup.name,
});

Output of pulumi about

CLI          
Version      3.100.0
Go Version   go1.21.5
Go Compiler  gc

Plugins
NAME          VERSION
azure-native  2.25.0
nodejs        unknown

Host     
OS       darwin
Version  14.2.1
Arch     x86_64

This project is written in nodejs: executable='/Users/piers/.nvm/versions/node/v20.10.0/bin/node' version='v20.10.0'

Current Stack: pierskarsenbarg/azure-openai-raipolicy/dev

TYPE                                    URN
pulumi:pulumi:Stack                     urn:pulumi:dev::azure-openai-raipolicy::pulumi:pulumi:Stack::azure-openai-raipolicy-dev
pulumi:providers:azure-native           urn:pulumi:dev::azure-openai-raipolicy::pulumi:providers:azure-native::default_2_25_0
azure-native:resources:ResourceGroup    urn:pulumi:dev::azure-openai-raipolicy::azure-native:resources:ResourceGroup::pk-cognitiveservices-rg
azure-native:cognitiveservices:Account  urn:pulumi:dev::azure-openai-raipolicy::azure-native:cognitiveservices:Account::cs-account


Found no pending operations associated with dev

Backend        
Name           pulumi.com
URL            https://app.pulumi.com/pierskarsenbarg
User           pierskarsenbarg
Organizations  pierskarsenbarg, karsenbarg, team-ce, demo
Token type     personal

Dependencies:
NAME                  VERSION
@pulumi/azure-native  2.25.0
@pulumi/pulumi        3.101.1
@types/node           18.19.7

Additional context

I have a log file. Here's a snippet including the PUT request:

I0116 11:35:15.436880   77692 log.go:81] eventSink::Infoerr(<{%reset%}>I0116 11:35:15.435898   77726 log.go:81] HTTP Request Begin PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/pk-cognitiveservices-rge3872768/providers/Microsoft.CognitiveServices/accounts/pk-test/raiPolicies/pkraipolicyname?api-version=2023-10-01-preview ===================================================
<{%reset%}>)
I0116 11:35:15.436912   77692 log.go:81] eventSink::Infoerr(<{%reset%}>PUT /subscriptions/00000000-0000-0000-0000-000000000000/providers/Microsoft.CognitiveServices/accounts/pk-test/raiPolicies/pkraipolicyname?api-version=2023-10-01-preview HTTP/1.1
<{%reset%}>)
I0116 11:35:15.436938   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Host: management.azure.com
<{%reset%}>)
I0116 11:35:15.436964   77692 log.go:81] eventSink::Infoerr(<{%reset%}>User-Agent: Go/go1.21.5 (amd64-darwin) go-autorest/v14.2.1 pulumi-azure-native/2.25.0 pid-a90539d8-a7a6-5826-95c4-1fbef22d4b22
<{%reset%}>)
I0116 11:35:15.436990   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Content-Length: 278
<{%reset%}>)
I0116 11:35:15.437012   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Content-Type: application/json; charset=utf-8
<{%reset%}>)
I0116 11:35:15.437042   77692 log.go:81] eventSink::Infoerr(<{%reset%}>{"properties":{"basePolicyName":"112","contentFilters":[{"allowedContentLevel":"Low","blocking":true,"policyName":"hate"},{"allowedContentLevel":"Low","policyName":"sexual"},{"enabled":false,"policyName":"violence"},{"enabled":false,"policyName":"DefaultHateSpeechBlockList"}]}}
<{%reset%}>)
I0116 11:35:15.437143   77692 log.go:81] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Request End PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/pk-cognitiveservices-rge3872768/providers/Microsoft.CognitiveServices/accounts/pk-test/raiPolicies/pkraipolicyname?api-version=2023-10-01-preview

and the response:

I0116 11:35:15.609780   77692 log.go:81] eventSink::Infoerr(<{%reset%}>I0116 11:35:15.609644   77726 log.go:81] HTTP Response Begin PUT [https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/pk-cognitiveservices-rge3872768/providers/Microsoft.CognitiveServices/accounts/pk-test/raiPolicies/pkraipolicyname?api-version=2023-10-01-preview ===================================================
<{%reset%}>)
I0116 11:35:15.609856   77692 log.go:81] eventSink::Infoerr(<{%reset%}>HTTP/2.0 400 Bad Request
<{%reset%}>)
I0116 11:35:15.609877   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Content-Length: 117
<{%reset%}>)
I0116 11:35:15.609891   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Cache-Control: no-cache
<{%reset%}>)
I0116 11:35:15.609905   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Content-Type: application/json; charset=utf-8
<{%reset%}>)
I0116 11:35:15.609932   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Date: Tue, 16 Jan 2024 11:35:15 GMT
<{%reset%}>)
I0116 11:35:15.609979   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Expires: -1
<{%reset%}>)
I0116 11:35:15.610001   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Pragma: no-cache
<{%reset%}>)
I0116 11:35:15.610022   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Server: istio-envoy
<{%reset%}>)
I0116 11:35:15.610041   77692 log.go:81] eventSink::Infoerr(<{%reset%}>Strict-Transport-Security: max-age=31536000; includeSubDomains
<{%reset%}>)
I0116 11:35:15.610064   77692 log.go:81] eventSink::Infoerr(<{%reset%}>X-Content-Type-Options: nosniff
<{%reset%}>)
I0116 11:35:15.610093   77692 log.go:81] eventSink::Infoerr(<{%reset%}>X-Envoy-Upstream-Service-Time: 9
<{%reset%}>)
I0116 11:35:15.610143   77692 log.go:81] eventSink::Infoerr(<{%reset%}>X-Ms-Correlation-Request-Id: 109b0102-f10b-419b-958b-0edf7e03e139
<{%reset%}>)
I0116 11:35:15.610176   77692 log.go:81] eventSink::Infoerr(<{%reset%}>X-Ms-Ratelimit-Remaining-Subscription-Writes: 1199
<{%reset%}>)
I0116 11:35:15.610191   77692 log.go:81] eventSink::Infoerr(<{%reset%}>X-Ms-Request-Id: ec70b44e-01de-49a7-9851-5b46b0dc434e
<{%reset%}>)
I0116 11:35:15.610212   77692 log.go:81] eventSink::Infoerr(<{%reset%}>X-Ms-Routing-Request-Id: UKSOUTH:20240116T113515Z:109b0102-f10b-419b-958b-0edf7e03e139
<{%reset%}>)
I0116 11:35:15.610233   77692 log.go:81] eventSink::Infoerr(<{%reset%}>{"error":{"code":"InvalidRequestContent","message":"The request content was invalid and could not be deserialized."}}
<{%reset%}>)
I0116 11:35:15.610296   77692 log.go:81] eventSink::Infoerr(<{%reset%}>===================================================== HTTP Response End PUT https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/pk-cognitiveservices-rge3872768/providers/Microsoft.CognitiveServices/accounts/pk-test/raiPolicies/pkraipolicyname?api-version=2023-10-01-preview

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[thomas11]

The request the provider generates looks identical to the example (as it should). Could the problem be that the DefaultHateSpeechBlockList policy is disabled but Only customers who have been approved for modified content filtering have full content filtering control and can turn content filters partially or fully off?

[pierskarsenbarg]

@thomas11 asked the customer, and you have to apply to disable the content filter but not to modify it. Both me and them can create the resource manually.

[thomas11]

Both me and them can create the resource manually.

@pierskarsenbarg manually in what way - Azure portal, az cli, directly using the REST API?

[pierskarsenbarg]

@thomas11 Azure portal

It looks like we might be missing an input in our resource. The ARM docs reference a policyType value to send over: https://learn.microsoft.com/en-us/rest/api/cognitiveservices/accountmanagement/rai-policies/create-or-update?view=rest-cognitiveservices-accountmanagement-2023-10-01-preview&tabs=HTTP#raipolicytype

I can see it in the version of the spec that we're referencing: https://github.com/Azure/azure-rest-api-specs/blob/8b5aae78857c5f3cc4ecd41a4c084c884816e964/specification/cognitiveservices/resource-manager/Microsoft.CognitiveServices/preview/2023-10-01-preview/cognitiveservices.json#L5485

But it's not in our SDKs (node for example: https://github.com/pulumi/pulumi-azure-native/blob/master/sdk/nodejs/cognitiveservices/v20231001preview/raiPolicy.ts)

We do reference it as an output (docs and here in the .NET SDK since the nodejs outputs file is too big to show in github)

So I think we're missing a property

[thomas11]

Thanks for digging into this, @pierskarsenbarg. The policyType you linked to in the spec has "readOnly": true, making it an output-only property. I believe the issue is somewhere else.

[smcmanis]

@thomas11 Terraform uses the property in their implementation: hashicorp/terraform-provider-azurerm#22822.

[thomas11]

@smcmanis that's a good find, although the issue you linked is still open so it doesn't seem Terraform supports this yet. One of the commenters managed to do it via azapi which simply invokes the REST API via TF.

I don't understand how they're using API version 2023-06-01-preview - it doesn't exist in the API spec.

Independent of that, I used the payload @pierskarsenbarg captured to make requests directly using the REST API az rest. I can reproduce the InvalidRequestContent error, no matter whether I add "type": "UserManaged" or not. Leaving out all contentFilters passes that validation stage, but even a single filter with a variety of property permutations that I tried causes the problem.

I also tried the above with API version 2023-06-01-preview, which does indeed exist although undocumented. Made no difference for me, though.

[thomas11]

Hey @pierskarsenbarg and @smcmanis! After much trial and error with the woefully under-documented Cognitive Services API, I think I found the problem. It's a bug (or two) in the Azure spec for this endpoint, but fortunately, you can work around it on your end.

First issue: the blocking and source properties of content filters are annotated as optional in the spec, but they aren't. Adding them to each filter gets past the InvalidRequestContent error.

        contentFilters: [
            {
                name: "hate",
                allowedContentLevel: AllowedContentLevel.Low,
                blocking: true,
                source: RaiPolicyContentSource.Completion,
            },
            {
                name: "sexual",
                allowedContentLevel: AllowedContentLevel.Low,
                blocking: false,
                source: RaiPolicyContentSource.Completion,
            },
            {
                name: "violence",
                enabled: false,
                blocking: false,
                source: RaiPolicyContentSource.Completion,
            },
            {
                name: "DefaultHateSpeechBlockList",
                enabled: false,
                blocking: false,
                source: RaiPolicyContentSource.Prompt,
            }          
        ],

The second issue is with the base policy. I don't think the "112" policy from the sample actually exists, although I could be wrong - I couldn't find any docs on base policies. I found other examples that used Microsoft.Default as base policy but that didn't work for me either. After some digging around the portal, I realized it's due to the account type. When you provision the cognitiveservices.Account with kind: "CognitiveServices", it doesn't have the Microsoft.Default policy. (Or you need to do additional steps to get it.) When you provision the account as kind: "OpenAI", it does, and the sample now works.

A third issue I found is that the Account.kind is incorrectly annotated as well. It cannot be changed on an existing kind but the spec doesn't say that, so Pulumi will try to change it and fail with an Azure error.

I filed bugs against the Azure spec:

• [BUG] Cognitive Services Account "kind" not annotated as immutable #27777
• [BUG] Cognitive Services RaiPolicyContentFilter required properties not marked as such #27778

[smcmanis]

Hi @thomas11,

First off, thank you for the in-depth analysis and for pinpointing the potential causes behind the issues we've been facing with the Cognitive Services API. Your effort in experimenting and documenting your findings is appreciated.

Upon reviewing your workaround suggestions, it seems that the approaches you've outlined (specifying blocking and source properties in content filters, addressing the base policy confusion, and the Account.kind annotation) align closely with the configuration we've already implemented on our end. To ensure clarity and to aid in further troubleshooting, here are the relevant parts of our Pulumi code reflecting this:

import * as resources from "@pulumi/azure-native/resources";
import {
  Account,
  AllowedContentLevel,
  RaiPolicy,
  RaiPolicyContentSource,
  RaiPolicyMode,
} from '@pulumi/azure-native/cognitiveservices';

const resourceGroup = new resources.ResourceGroup("pk-cognitiveservices-rg");

const account = new Account('cs-account', {
  accountName: 'pk-test',
  // identity: {},
  kind: 'OpenAI',
  location: 'westus',
  properties: {
    customSubDomainName: 'test-subdomain',
    networkAcls: {
      defaultAction: 'Allow',
      ipRules: [],
      virtualNetworkRules: [],
    },
    publicNetworkAccess: 'Enabled',
  },
  resourceGroupName: resourceGroup.name,
  sku: {
    name: 'S0',
  },
});

new RaiPolicy(`raipolicy-pk`, {
  accountName: account.name,
  properties: {
    basePolicyName: 'Microsoft.Default',
    mode: RaiPolicyMode.Default,
    contentFilters: [
      {
        policyName: 'hate',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Prompt,
      },
      {
        policyName: 'sexual',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Prompt,
      },
      {
        policyName: 'violence',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Prompt,
      },
      {
        policyName: 'selfharm',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Prompt,
      },
      {
        policyName: 'hate',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Completion,
      },
      {
        policyName: 'sexual',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Completion,
      },
      {
        policyName: 'violence',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Completion,
      },
      {
        policyName: 'selfharm',
        allowedContentLevel: AllowedContentLevel.High,
        blocking: true,
        enabled: true,
        source: RaiPolicyContentSource.Completion,
      },
    ],
  },
  raiPolicyName: 'pkraipolicyname',
  resourceGroupName: resourceGroup.name,
});

Running pulumi up fails with these diagnostics:

Diagnostics:
  azure-native:cognitiveservices:RaiPolicy (raipolicy-pk):
    error: autorest/azure: Service returned an error. Status=400 Code="InvalidRequestContent" Message="The request content was invalid and could not be deserialized."

I'm curious if there were any specific nuances or additional configurations in your successful deployment that might not have been captured in the initial summary. Could you confirm you were able to successfully deploy RaiPolicy using Pulumi without encountering the issues mentioned?

Thank you again for your assistance and for filing the bugs against the Azure spec.

[thomas11]

Hi @smcmanis, yes, I was able to successfully a RaiPolicy. In fact, just now I was even able to deploy your above program, verbatim! I see the policy in the portal, too.

I'm currently puzzled as to what the difference might be. The Pulumi program is the same, your provider version is very recent as well. What's the output of pulumi config in your Pulumi.yaml directory?

[lukehoban]

In fact, just now I was even able to deploy your above program, verbatim! I see the policy in the portal, too.

FWIW - I just tried the code in #3022 (comment) as well, and was able to deploy it succesfully. I did have to change policyName to name in the policies. It appears maybe the upstream API changed the name of this property? Is it possible that that difference is leading to the error you are seeing @smcmanis?

[smcmanis]

Thanks @lukehoban and @thomas11. I managed to get it working by upgrading @pulumi/azure-native from 2.18 to 2.28. I also had to change policyName to name.

[thomas11]

I'm glad this worked for you. Yes, the change of policyName to name was due to a change Azure did within the same API version. Not a best practice, but since it's a preview API it's fair game.