操作指南

本节中的每个指南都针对您作为有经验的用户在使用 Ragas 时可能遇到的实际问题提供了专注的解决方案。这些指南设计得简洁直接，为您的问题提供快速解决方案。我们假设您对 Ragas 的概念有基本了解且能够熟练使用。如果不是，请先浏览 快速入门 （Get Started）部分。

评估多轮对话

本教程灵感来源于 Hamel 关于评估基于 LLM 的应用程序多轮对话的笔记。目标是使用 Ragas 指标创建一个简单且可操作的评估框架，明确界定对话成功的标准。完成本教程后，您将能够基于从 AI 应用程序错误分析中收集的见解进行多轮评估。

Ragas 指标

Ragas 提供了 AspectCritic ，这是一个功能强大的评估指标，用于评估具有二元结果的多轮对话。它有助于确定对话是否满足预定义的成功标准。
AspectCritic
AspectCritic 根据以自由形式自然语言编写的预定义方面（aspects）来评估响应。它返回一个二元输出，表示响应是否符合定义的方面。

该指标与 Hamel 的建议一致，即关注二元决策——这消除了歧义，并为提高对话质量提供了清晰、可操作的方法。

实际示例——评估银行聊天机器人

评估时，应关注与用户需求直接相关的指标。分数的任何变化都应反映对用户体验的有意义影响。

让我们考虑一个为银行构建聊天机器人的例子。

经过错误分析后，您发现聊天机器人偶尔会忘记被要求完成的任务，或者只部分执行这些任务。为了提高聊天机器人的性能，您需要一种可靠的方法来测量和评估这种行为。

注意： 定义评分标准时，请使用标准化术语。

• 将用户消息称为 human message （用户消息）。
• 将聊天机器人消息称为 AI message （AI 消息）。

from dotenv import load_dotenv
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

load_dotenv()

evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))

from ragas.metrics import AspectCritic, SimpleCriteriaScore
from ragas.dataset_schema import MultiTurnSample, EvaluationDataset
from ragas.messages import HumanMessage, AIMessage
from ragas import evaluate


sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to increase my credit limit and check why my last transaction at Walmart was declined."
        ),
        AIMessage(
            content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds."
        ),
        HumanMessage(content="That's not possible, I had enough money in my account."),
        AIMessage(
            content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700."
        ),
        HumanMessage(content="Oh, I forgot about the hotel booking."),
        AIMessage(content="Is there anything else I can assist you with?"),
        HumanMessage(content="What about increasing my credit limit?"),
        AIMessage(
            content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to check why my last transaction at Target was declined."
        ),
        AIMessage(
            content="Let's look at the declined transaction. The $145.30 transaction at Target on January 5th was declined due to a suspected fraud alert."
        ),
        HumanMessage(content="That’s strange, I didn’t report anything suspicious."),
        AIMessage(
            content="I understand. Let me look deeper. It seems a fraud prevention team flagged your recent purchase at a different store for unusually high activity, which triggered the decline at Target as well."
        ),
        HumanMessage(content="Ah, that makes sense. I did shop a lot that day."),
        AIMessage(content="Is there anything else I can assist you with?"),
    ]
)

definition = "Return 1 if the AI completes all Human requests fully without any rerequests; otherwise, return 0."

aspect_critic = AspectCritic(
    name="forgetfulness_aspect_critic",
    definition=definition,
    llm=evaluator_llm,
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[aspect_critic],
)

result.to_pandas()

输出

Evaluating: 100%|██████████| 2/2 [00:00<?, ?it/s]

	user_input	forgetfulness_aspect_critic
0	[{‘content’: 'I need to increase my credit lim…	0
1	[{‘content’: 'I need to check why my last tran…	1

使用基于 LLM 的指标进行评估时，每个指标可能涉及一次或多次 LLM 调用。评估的追踪信息可以为理解结果和诊断问题提供见解。您可以通过访问此页面了解有关此过程的更多详细信息。

在错误分析过程中发现的另一个模式是，您的银行聊天机器人偶尔会偏离基本账户服务的讨论，转而提供未经授权的投资建议。为了保持用户信任并确保合规性，您希望系统在对话接近这些边界时实现优雅过渡。您可以通过定义如下所示的指标来实现这一点。

definition = "Return 1 if the AI stays within the banking domain, providing services like account-related information, credit card details, or general banking advice, and avoids offering financial or investment advice; otherwise, return 0"

aspect_critic = AspectCritic(
    name="Banking Compliance Metric(aspect critic)",
    definition=definition,
    llm=evaluator_llm,
)

sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(
            content="I need to increase my credit limit and check why my last transaction at Walmart was declined."
        ),
        AIMessage(
            content="Let's check the declined transaction first, the $234.56 transaction at Walmart on November 20th was declined due to insufficient funds."
        ),
        HumanMessage(content="That's not possible, I had enough money in my account."),
        AIMessage(
            content="I understand your concern. Let me check your account details... I see that while your current balance is $1,500, there was a pending hotel reservation hold of $800 at the time of the Walmart transaction, which reduced your available balance to $700."
        ),
        HumanMessage(content="Oh, I forgot about the hotel booking."),
        AIMessage(content="Is there anything else I can assist you with?"),
        HumanMessage(content="What about increasing my credit limit?"),
        AIMessage(
            content="Based on your account history and current credit score of 740, you're eligible for a credit limit increase. The maximum increase available is $2,000. However, I notice you had three declined transactions this month, so I'd recommend reviewing your spending patterns before accepting an increase."
        ),
        HumanMessage(
            content="What investment options do you recommend with my spare money?"
        ),
        AIMessage(
            content="While I can help you with banking transactions and credit card matters, I'm not authorized to provide investment advice. I'd be happy to connect you with our certified financial advisor who can help you explore investment options."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi there!"),
        AIMessage(content="Hello! How can I help you today?"),
        HumanMessage(content="How much money do I have in my bank account?"),
        AIMessage(
            content="Your current balance is ₹85,750. Let me know if you’d like details on recent transactions or account services!"
        ),
        HumanMessage(
            content="Which are better financial instruments for wealth generation—crypto like Bitcoin or stocks?"
        ),
        AIMessage(
            content="Both stocks and cryptocurrencies have the potential for wealth generation, but they come with different risk levels. Stocks are usually more stable, linked to company performance, and can offer long-term growth. Cryptocurrencies, like Bitcoin, are highly volatile but can deliver quick, high returns. Diversification might be a smart approach based on your risk tolerance and goals."
        ),
        HumanMessage(content="Interesting! Thanks for the advice."),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[aspect_critic],
)

result.to_pandas()

输出

Evaluating: 100%|██████████| 2/2 [00:00<?, ?it/s]

	user_input	Banking Compliance Metric(aspect critic)
0	[{‘content’: 'I need to increase my credit lim…	1
1	[{‘content’: ‘Hi there!’, ‘type’: ‘human’}, {'…	0

语调（Tonality）

在本节中，我们将探讨如何评估您的聊天机器人在不同地区和文化中是否保持一致的语调——这是多语言部署中最具挑战性的方面之一。

在一种文化中被认为礼貌的表达方式，在另一种文化中可能会有不同的理解。例如，日本的礼貌通常以正式、间接和尊重的语调表达，而在墨西哥，礼貌通常以热情、友好和引人入胜的语调传达。

为了确保您的聊天机器人适应这些文化差异，您可以定义自定义评估指标，以评估语调是否符合每个目标受众的期望。

japanese_polite_definition = "Return 1 if the AI maintains a formal, polite, and respectful tone throughout the conversation, using indirect expressions and avoiding overly casual or direct language; otherwise, return 0."

mexican_polite_definition = "Return 1 if the AI maintains a warm, friendly, and engaging tone, blending informality with respect and fostering a personable, approachable interaction; otherwise, return 0."

jpn_polite_aspect_critic = AspectCritic(
    name="japanese_polite_aspect_critic",
    definition=japanese_polite_definition,
    llm=evaluator_llm,
)

mex_polite_aspect_critic = AspectCritic(
    name="mexican_polite_aspect_critic",
    definition=mexican_polite_definition,
    llm=evaluator_llm,
)

sample1 = MultiTurnSample(
    user_input=[
        HumanMessage(content="I forgot my PIN. What should I do?"),
        AIMessage(
            content="I can assist you in resetting your PIN. May I send a verification code to your registered phone number?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(content="I have sent the code. Please check and enter it here."),
        HumanMessage(content="I have entered it."),
        AIMessage(content="Thank you. Please set a new PIN."),
        HumanMessage(content="I have set it."),
        AIMessage(
            content="Your PIN reset is complete. Thank you for choosing our bank."
        ),
    ]
)

sample2 = MultiTurnSample(
    user_input=[
        HumanMessage(content="Help! I forgot my PIN."),
        AIMessage(
            content="Don't worry! I’ll help you reset it. Do you want me to send a verification code to your registered number?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(content="Done! Check your phone and enter the code here. "),
        HumanMessage(content="I have entered it."),
        AIMessage(content="Great! Now choose a new PIN and keep it in a safe place."),
        HumanMessage(content="Done."),
        AIMessage(content="Perfect! All set. If you need more help, I’ll be here."),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample1, sample2]),
    metrics=[jpn_polite_aspect_critic, mex_polite_aspect_critic],
)

result.to_pandas()

输出

Evaluating: 100%|██████████| 4/4 [00:00<?, ?it/s]

	user_input	japanese_polite_aspect_critic	mexican_polite_aspect_critic
0	[{‘content’: 'I forgot my PIN. What should I d…	1	1
1	[{‘content’: ‘Help! I forgot my PIN.’, ‘type’:…	0	1

上述评估结果突出表明，在墨西哥被认为礼貌的表达方式，在日本可能不会被视为礼貌。

品牌语调检查

在本节中，我们将探讨如何评估聊天机器人的语调是否与企业的价值观、目标受众和整体品牌形象保持一致。

什么是品牌语调（Brand Tone of Voice）？ 品牌语调是指品牌在与受众进行书面或口头互动时的用词选择。通过定义独特的语调，品牌可以塑造真实的个性、风格和态度。（参考）

例如：

Google——信息丰富且乐于助人的品牌语调 您是否注意到使用 Google 产品时，一切都感觉简单直观？但一旦切换到其他工具，事情突然变得更加复杂。这种无缝体验源于 Google 对其品牌语调的精通。

Google 保持友好亲切的语调，同时确保用户沟通清晰简洁。他们整个品牌语调围绕着乐于助人、清晰易懂和易于访问展开，使他们的产品对每个人都直观易用。（参考）

您可以通过定义如下所示的自定义评估指标，来评估聊天机器人的响应是否符合您的品牌形象。

definition = "Return 1 if the AI's communication is friendly, approachable, helpful, clear, and concise; otherwise, return 0."

aspect_critic = AspectCritic(
    name="Brand Voice Metric(aspect critic)",
    definition=definition,
    llm=evaluator_llm,
)

sample = MultiTurnSample(
    user_input=[
        HumanMessage(content="Hi! I want to apply for a credit card."),
        AIMessage(
            content="Sure! We’ve got a few great options. What kind of benefits are you interested in?"
        ),
        HumanMessage(content="Travel rewards, mostly."),
        AIMessage(
            content="Perfect! Our Travel Rewards Card gives you miles for every dollar spent on flights and hotels. Want to apply now?"
        ),
        HumanMessage(content="Yes, please."),
        AIMessage(
            content="Awesome! Here’s the link to apply. It’s quick and easy. Let me know if you need help!"
        ),
    ]
)

result = evaluate(
    dataset=EvaluationDataset(samples=[sample]),
    metrics=[aspect_critic],
)

result.to_pandas()

输出

Evaluating:   100%|██████████| 1/1 [00:00<?, ?it/s]

	user_input	Brand Voice Metric(aspect critic)
0	[{‘content’: 'Hi! I want to apply for a credit…	1