Answer relevancy metric is worse in languages other than English

[icejean]

I set up Neo4j KGBuilder on my Ubuntu22, and find that the answer relevancy metric is worse with Chinese, and I find out at last that the cause is that the questions generated by LLM is in English, not in Chinese, so the embedding of them are quite different with the original question in Chinese.
I'v posted an issue and the solution to the ragas project's home page at here: Answer relevancy metric is worse in languages other than English, please read it for details.

I think a solution for all languages and all metrics is needed.

Best regards
Jean from China

[icejean]

Well, the issue can be addressed in this way: Adapting metrics to target language
For example:

from datasets import Dataset
import os, json
from langchain_openai import ChatOpenAI

llm2 = ChatOpenAI(model="gpt-4o") 

# messages is the message list of LangGraph state
question = messages[-4].content
print(question)
answer = json.dumps(messages[-1].content, ensure_ascii=False)
print(answer)
context = json.dumps(messages[-2].content, ensure_ascii=False)
print(context)
dataset = Dataset.from_dict(
      {"question": [question], "reference": [answer], "answer": [answer], "contexts": [[context]]}
        )

# from ragas.metrics import answer_relevancy, faithfulness,context_entity_recall
from ragas.metrics import faithfulness,context_entity_recall
from ragas import evaluate
from ragas.cost import get_token_usage_for_openai

from ragas.llms import LangchainLLMWrapper

# This LLM instance is used to translate the prompt into target language.
llm3 = LangchainLLMWrapper(llm2)


from ragas.metrics import AnswerRelevancy

answer_relevancy = AnswerRelevancy(
    name="answer_relevancy", strictness=3, embeddings=embeddings
)

prompts = answer_relevancy.get_prompts()

import asyncio

async def adapt_prompt():
    adapted_prompts = await answer_relevancy.adapt_prompts(language="chinese", llm=llm3)
    print(adapted_prompts)
    return adapted_prompts

# run adapt_prompt() with  asyncio.run()
adapted_prompts = asyncio.run(adapt_prompt())

answer_relevancy.set_prompts(**adapted_prompts)


score = evaluate(
    dataset=dataset,
    metrics=[answer_relevancy],
    llm=llm2,
    embeddings=embeddings,
)

df = score.to_pandas()[['answer_relevancy']]
df

[kartikpersistent]

Thank you @icejean we will surely incoperate into our application