GenAI Client safety settings does not support multiple SafetySetting

[victai]

Description of the bug:

VertexAI Client works fine, fine, but the new GenAI Client (google.genai.Client) does not support multiple SafetySetting.

SafetySettings applied

safety_settings = [
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_HATE_SPEECH,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_HARASSMENT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_CIVIC_INTEGRITY,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
]

Error message:

400 INVALID_ARGUMENT. {'error': {'code': 400, 'message': 'Multiple safety settings with the same category.', 'status': 'INVALID_ARGUMENT'}}

The error message is incorrect and misleading, which makes me think this is a bug.

If I only apply one safety setting it'll work, but whenever there are multiple, it claims that it is setting the same category.

Actual vs expected behavior:

Actual behavior: If I only apply one safety setting it'll work, but whenever there are multiple, it claims that it is setting the same category.
Expected behavior: should work the same as the vertexai client.

Any other information you'd like to share?

Both synchronous and asynchronous clients/functions have the issue mentioned above. Here's an asynchronous script that demonstrates the issue

import asyncio
from google import genai
from google.genai.types import GenerateContentConfig
from vertexai.generative_models import HarmBlockThreshold, HarmCategory, SafetySetting, GenerativeModel

safety_settings = [
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_HATE_SPEECH,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_HARASSMENT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
    SafetySetting(
        category=HarmCategory.HARM_CATEGORY_CIVIC_INTEGRITY,
        threshold=HarmBlockThreshold.BLOCK_NONE,
    ),
]

async def main():
    query = "How does AI work?"
    model = "gemini-1.5-flash-002"
    vertex_client = GenerativeModel(model)
    try:
        response = await vertex_client.generate_content_async(
            contents=query,
            safety_settings=safety_settings[:1],
        )
        print("vertex client single safety settings response: ", response.text[:50])
    except Exception as e:
        print("vertex client single safety settings error: ", e)

    try:
        response = await vertex_client.generate_content_async(
            contents=query,
            safety_settings=safety_settings,
        )
        print("vertex client multiple safety settings response: ", response.text[:50])
    except Exception as e:
        print("vertex client multiple safety settings error: ", e)

    genai_client = genai.Client(vertexai=True, location="us-east4")
    try:
        response = await genai_client.aio.models.generate_content(
            model=model, contents=query, config=GenerateContentConfig(safety_settings=safety_settings[:1])
        )
        print("genai client single safety settings response: ", response.text[:50])
    except Exception as e:
        print("genai client single safety settings error: ", e)

    try:
        response = await genai_client.aio.models.generate_content(
            model=model, contents=query, config=GenerateContentConfig(safety_settings=safety_settings)
        )
        print("genai client multiple safety settings response: ", response.text[:50])
    except Exception as e:
        print("genai client multiple safety settings error: ", e)

asyncio.run(main())

[Gunand3043]

Hi @victai

The new Google GenAI SDK expects the safety category values to be strings. You need to pass the safety settings in the format below.

safety_settings = [
     types.SafetySetting(
         category="HARM_CATEGORY_DANGEROUS_CONTENT",
         threshold="BLOCK_NONE",
     ),
     types.SafetySetting(
         category="HARM_CATEGORY_HATE_SPEECH",
         threshold="BLOCK_NONE",
     ),
     types.SafetySetting(
         category="HARM_CATEGORY_HARASSMENT",
         threshold="BLOCK_NONE",
     ),
      types.SafetySetting(
         category="HARM_CATEGORY_SEXUALLY_EXPLICIT",
         threshold="BLOCK_NONE",
     ),
     types.SafetySetting(
         category="HARM_CATEGORY_CIVIC_INTEGRITY",
         threshold="BLOCK_NONE",
     ),
 ]

Thanks

[victai]

@Gunand3043 that doesn't work for me. The result is the same. I'm using google-generativeai==0.8.3, which is the latest version.

[Gunand3043]

Oh, sorry to hear that you're still facing the issue. Since you're using google-generativeai, we have a quickstart tutorial on customizing multiple safety settings. Here’s the notebook link: notebook.

If you're using the new Google Gen AI SDK, I’ve also prepared a colab gist that shows how to pass multiple safety settings.

Hope this helps!

[victai]

Thanks for the response. The issue was that I was using SafetySettings from vertexai.generative_models not google.genai.types