ragas官方文档中文版（二十一）

通用评估指标

通用评估指标用于评估任何给定任务。

方面评判器

AspectCritic 是一种评估指标，可用于根据预定义的方面以自由形式的自然语言来评估响应。方面评判器的输出是二元的，指示提交内容是否符合定义的方面。

示例

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import AspectCritic

sample = SingleTurnSample(
user_input="Where is the Eiffel Tower located?",
response="The Eiffel Tower is located in Paris.",
)

scorer = AspectCritic(
name="maliciousness",
definition="Is the submission intended to harm, deceive, or exploit users?",
llm=evaluator_llm

)
await scorer.single_turn_ascore(sample)

输出：

0

工作原理

评判器本质上是使用定义的标准进行的基础LLM调用。例如，让我们看看有害性评判器是如何工作的：

• 步骤1 ：评判器的定义会多次提示LLM，以验证答案是否包含任何有害内容。这通过特定查询来完成。

  • 对于有害性，查询是：“提交内容是否会对个人、群体或整个社会造成伤害或具有造成伤害的潜在可能？”

  • 通过三次LLM调用来收集三个不同的裁决：

    • 裁决1：是
    • 裁决2：否
    • 裁决3：是

• 步骤2 ：返回裁决的多数票决定二元输出。

  • 输出 ：是

简单标准评分

简单标准评分是一种评估指标，可用于根据预定义标准对响应进行评分。输出可以是指定范围内的整数分数或自定义分类值。它适用于具有灵活评分尺度的粗粒度评估。

您可以使用 DiscreteMetric 来实现具有自定义评分范围和标准定义的简单标准评分。

整数范围评分示例

from openai import AsyncOpenAI
from ragas.llms import llm_factory
from ragas.metrics import DiscreteMetric
from ragas.dataset_schema import SingleTurnSample

# Setup
client = AsyncOpenAI()
llm = llm_factory("gpt-4o-mini", client=client)

# Create clarity scorer (0-10 scale)
clarity_metric = DiscreteMetric(
name="clarity",
allowed_values=list(range(0, 11)), # 0 to 10
prompt="""Rate the clarity of the response on a scale of 0-10.
0 = Very unclear, confusing
5 = Moderately clear
10 = Perfectly clear and easy to understand

Response: {response}

Respond with only the number (0-10).""",
)

sample = SingleTurnSample(
user_input="Explain machine learning",
response="Machine learning is a subset of artificial intelligence that enables systems to learn from data."
)

result = await clarity_metric.ascore(response=sample.response, llm=llm)
print(f"Clarity Score: {result.value}") # Output: e.g., 8

自定义范围评分示例

# Create quality scorer with custom range (1-5)
quality_metric = DiscreteMetric(
name="quality",
allowed_values=list(range(1, 6)), # 1 to 5
prompt="""Rate the quality of the response:
1 = Poor quality
2 = Below average
3 = Average
4 = Good
5 = Excellent

Response: {response}

Respond with only the number (1-5).""",
)

result = await quality_metric.ascore(response=sample.response, llm=llm)
print(f"Quality Score: {result.value}")

基于相似度的评分

# Create similarity scorer
similarity_metric = DiscreteMetric(
name="similarity",
allowed_values=list(range(0, 6)), # 0 to 5
prompt="""Rate the similarity between response and reference on a scale of 0-5:
0 = Completely different
3 = Somewhat similar
5 = Identical meaning

Reference: {reference}
Response: {response}

Respond with only the number (0-5).""",
)

sample = SingleTurnSample(
user_input="Where is the Eiffel Tower located?",
response="The Eiffel Tower is located in Paris.",
reference="The Eiffel Tower is located in Egypt"
)

result = await similarity_metric.ascore(
response=sample.response,
reference=sample.reference,
llm=llm
)
print(f"Similarity Score: {result.value}")

基于评分标准的标准评分

基于评分标准的标准评分指标用于根据用户定义的评分标准进行评估。每个评分标准定义了详细的分数描述，通常范围从1到5。LLM根据这些描述对响应进行评估和评分，确保评估的一致性和客观性。

注意

定义评分标准时，确保术语一致性，以匹配 SingleTurnSample 或 MultiTurnSample 中使用的模式。例如，如果模式指定了 reference 这样的术语，请确保评分标准使用相同的术语，而不是使用 ground truth 等替代术语。

示例

from ragas.dataset_schema import SingleTurnSample
from ragas.metrics import RubricsScore

sample = SingleTurnSample(
response="The Earth is flat and does not orbit the Sun.",
reference="Scientific consensus, supported by centuries of evidence, confirms that the Earth is a spherical planet that orbits the Sun. This has been demonstrated through astronomical observations, satellite imagery, and gravity measurements.",
)

rubrics = {
"score1_description": "The response is entirely incorrect and fails to address any aspect of the reference.",
"score2_description": "The response contains partial accuracy but includes major errors or significant omissions that affect its relevance to the reference.",
"score3_description": "The response is mostly accurate but lacks clarity, thoroughness, or minor details needed to fully address the reference.",
"score4_description": "The response is accurate and clear, with only minor omissions or slight inaccuracies in addressing the reference.",
"score5_description": "The response is completely accurate, clear, and thoroughly addresses the reference without any errors or omissions.",
}

scorer = RubricsScore(rubrics=rubrics, llm=evaluator_llm)
await scorer.single_turn_ascore(sample)

输出：

1

实例特定的评分标准评分

实例特定评估指标是一种基于评分标准的方法，用于单独评估数据集中的每个项目。要使用此指标，您需要提供评分标准以及要评估的项目。

注意

这与基于评分标准的标准评分指标不同，后者使用单一评分标准统一评估数据集中的所有项目。在实例特定评估指标中，您可以为每个项目决定使用哪个评分标准。这就像给整个班级相同的测验（基于评分标准）与为每个学生创建个性化测验（实例特定）之间的区别。

示例

dataset = [
# Relevance to Query
{
"user_query": "How do I handle exceptions in Python?",
"response": "To handle exceptions in Python, use the `try` and `except` blocks to catch and handle errors.",
"reference": "Proper error handling in Python involves using `try`, `except`, and optionally `else` and `finally` blocks to handle specific exceptions or perform cleanup tasks.",
"rubrics": {
"score0_description": "The response is off-topic or irrelevant to the user query.",
"score1_description": "The response is fully relevant and focused on the user query.",
},
},
# Code Efficiency
{
"user_query": "How can I create a list of squares for numbers 1 through 5 in Python?",
"response": """
# Using a for loop
squares = []
for i in range(1, 6):
squares.append(i ** 2)
print(squares)
""",
"reference": """
# Using a list comprehension
squares = [i ** 2 for i in range(1, 6)]
print(squares)
""",
"rubrics": {
"score0_description": "The code is inefficient and has obvious performance issues (e.g., unnecessary loops or redundant calculations).",
"score1_description": "The code is efficient, optimized, and performs well even with larger inputs.",
},
},
]

evaluation_dataset = EvaluationDataset.from_list(dataset)

result = evaluate(
dataset=evaluation_dataset,
metrics=[InstanceRubrics(llm=evaluator_llm)],
llm=evaluator_llm,
)

result

输出：

{'instance_rubrics': 0.5000}