opencompass评测框架部署

某大模型评测岗面试临时抱佛脚搞的，最后也没用上就是了，分享一下部署教程，用qwen 2.5 7B跑了MMLU的一个子集

1 环境配置

1.1 基本环境

• 显卡：RTX 5070
• 内存：32G
• 操作系统：Windows11 + WSL2（Ubuntu 22.04.5 LTS），Windows可能存在不兼容的问题（issue #799）
• opencompass：0.5.2

1.2 python环境

1.2.1 创建虚拟环境

使用：

conda create -n opencompass python=3.10 -y
conda activate opencompass

创建虚拟环境，注意先行配置镜像源或加速源。

1.2.2 安装pytorch

根据显卡架构及cuda版本安装合适版本的torch，opencompass给出的版本要求中，torch版本需≥1.13.1，此处5070是sm120，安装cu128版本的torch：

pip install torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cu128

1.2.3 安装opencompass

由于没有编辑代码的需求，直接从pypi源安装：

pip install -U opencompass

若需要安装为可编辑形式，则：

git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .

1.2.4 修复transformers

opencompass要求transformers>=4.29.1，但不支持transformers==5，将transformers降级至4.46.3：

pip install transformers==4.46.3

1.2.5 安装bitsandbytes

实验所用显卡5070为12G显存，全量推理会OOM，需要通过bitsandbytes进行量化，量化的的具体配置在后文呈现，现在先安装依赖：

pip install bitsandbytes

1.2.6 安装modelscope

本文采用modelscope下载模型。

pip install modelscope

1.3 模型下载

提醒：模型会占用10G以上的存储空间，而WSL默认在系统分区上创建环境，如果存在磁盘分区，务必注意存储空间分配，建议优先导出并重新导入WSL至存储空间充裕的位置。

本文的实验主要对Qwen 2.5 7B进行评测，通过modelscope下载，创建一个下载脚本：

# download_model.py

from modelscope import snapshot_download

model_dir = snapshot_download(
model_id='Qwen/Qwen2.5-7B-Instruct',
cache_dir='/home/llm_location'
)

print(model_dir)

通过命令行运行脚本：

python /mnt/c/Users/username/projects/OpenCompassExperiment/tools/download_model.py

或直接通过命令行下载：

modelscope download \\
–model Qwen/Qwen2.5-7B-Instruct \\
–cache_dir /home/llm_location

此时，我们将模型下载到了/home/llm_location目录下。如果你的项目和我一样在WSL外部的IDE中开发，不建议将模型下载到/mnt目录，其IO性能可能受限。

cache_dir参数和local_dir参数均为指定模型存放路径，前者提供了一个跨项目共享模型的缓存库，后者聚焦于独立部署等场景，个人开发或实验、无交付需求的情况下建议优先选择前者。

对于Qwen 2.5 7B而言，完整的下载结果中应该包含如下文件：

2 评测

2.1 检查评测集prompt

编写python脚本：

# dataset_prompt_viewer.py

from mmengine.config import read_base
from copy import deepcopy
import pprint

with read_base():
from opencompass.configs.datasets.mmlu.mmlu_gen import mmlu_datasets

def inspect_dataset(ds):
print("\\n========== DATASET KEYS ==========")
print(ds.keys())

print("\\n========== INFER CFG ==========")
pprint.pprint(ds.get('infer_cfg', None), width=120)

print("\\n========== PROMPT TEMPLATE ==========")
pt = ds['infer_cfg'].get('prompt_template', None)
pprint.pprint(pt, width=120)

if pt is not None:
print("\\n========== TEMPLATE TYPE ==========")
print(type(pt))

# 重点：OpenCompass通常在这里出错
if isinstance(pt, dict):
print("\\nKeys in prompt_template:")
print(pt.keys())

if 'template' in pt:
print("\\n— template —")
pprint.pprint(pt['template'], width=120)

print("\\n— template type —")
print(type(pt['template']))

if isinstance(pt['template'], dict):
print("\\n— template keys —")
print(pt['template'].keys())

if isinstance(pt['template'], list):
print("\\n template is LIST (likely error source)")
for i, item in enumerate(pt['template']):
print(f"\\n— item {i} —")
pprint.pprint(item, width=120)

if __name__ == "__main__":
ds = deepcopy(mmlu_datasets[0])
inspect_dataset(ds)

opencompass的数据集存在多层嵌套结构，建议相关拷贝操作统一采用深拷贝避免对象获取错误。

在命令行中运行这段代码，执行结果为：

========== DATASET KEYS ==========
dict_keys(['abbr', 'type', 'path', 'name', 'reader_cfg', 'infer_cfg', 'eval_cfg'])

========== INFER CFG ==========
{'inferencer': {'type': <class 'opencompass.openicl.icl_inferencer.icl_gen_inferencer.GenInferencer'>},
'prompt_template': {'template': {'round': [{'prompt': 'Answer the following multiple choice question. The last line '
"of your response should be of the following format: 'ANSWER: "
"$LETTER' (without quotes) where LETTER is one of ABCD. Think "
'step by step before answering.\\n'
'\\n'
'{input}\\n'
'\\n'
'A) {A}\\n'
'B) {B}\\n'
'C) {C}\\n'
'D) {D}',
'role': 'HUMAN'}]},
'type': <class 'opencompass.openicl.icl_prompt_template.PromptTemplate'>},
'retriever': {'type': <class 'opencompass.openicl.icl_retriever.icl_zero_retriever.ZeroRetriever'>}}

========== PROMPT TEMPLATE ==========
{'template': {'round': [{'prompt': 'Answer the following multiple choice question. The last line of your response '
"should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER "
'is one of ABCD. Think step by step before answering.\\n'
'\\n'
'{input}\\n'
'\\n'
'A) {A}\\n'
'B) {B}\\n'
'C) {C}\\n'
'D) {D}',
'role': 'HUMAN'}]},
'type': <class 'opencompass.openicl.icl_prompt_template.PromptTemplate'>}

========== TEMPLATE TYPE ==========
<class 'dict'>

Keys in prompt_template:
dict_keys(['type', 'template'])

— template —
{'round': [{'prompt': 'Answer the following multiple choice question. The last line of your response should be of the '
"following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by "
'step before answering.\\n'
'\\n'
'{input}\\n'
'\\n'
'A) {A}\\n'
'B) {B}\\n'
'C) {C}\\n'
'D) {D}',
'role': 'HUMAN'}]}

— template type —
<class 'dict'>

— template keys —
dict_keys(['round'])

如果存在修改prompt的需求，需要在正确的嵌套位置进行修改，这步操作的作用就是找到修改prompt的位置，下文会展示这一修改过程。

2.2 评测配置文件

编写python配置：

# qwen25_7b_mllu.py

from mmengine.config import read_base
from transformers import BitsAndBytesConfig
import torch
from copy import deepcopy

with read_base():
from opencompass.configs.datasets.mmlu.mmlu_gen import mmlu_datasets

from opencompass.models import HuggingFacewithChatTemplate

model_path = '/home/llm_location/Qwen/Qwen2___5-7B-Instruct'

datasets = deepcopy([
d for d in mmlu_datasets
if d['abbr'] == 'lukaemon_mmlu_college_biology'
])

datasets[0]['infer_cfg']['prompt_template']['template']['round'][0]['prompt'] = (
"Answer the following multiple choice question.\\n"
"Choose one option from A, B, C or D.\\n"
"Do not explain.\\n"
"Output only:\\n"
"ANSWER: <LETTER>\\n\\n"
"{input}\\n\\n"
"A) {A}\\n"
"B) {B}\\n"
"C) {C}\\n"
"D) {D}"
)

models = [
dict(
type=HuggingFacewithChatTemplate,

abbr='qwen25-7b-mmlu',

path=model_path,
tokenizer_path=model_path,

max_seq_len=4096,
max_out_len=128,

batch_size=1,

run_cfg=dict(
num_gpus=1,
),

model_kwargs=dict(
device_map='auto',
quantization_config=dict(
load_in_4bit=True,
),
),
)
]

下面分别介绍各语句块的作用和值得关注的细节，介绍顺序为语句块引入逻辑而非出现顺序。

2.2.1 评测集

datasets指定评测集，这里我们只评测lukaemon_mmlu_college_biology子集验证框架是否可用。如需评测其它子集，可通过以下方法获取子集名称：

from opencompass.configs.datasets.mmlu.mmlu_gen import mmlu_datasets
for ds in mmlu_datasets: print(ds["abbr"])

'''
lukaemon_mmlu_college_biology
lukaemon_mmlu_college_chemistry
lukaemon_mmlu_college_computer_science
lukaemon_mmlu_college_mathematics
lukaemon_mmlu_college_physics
lukaemon_mmlu_electrical_engineering
lukaemon_mmlu_astronomy
lukaemon_mmlu_anatomy
lukaemon_mmlu_abstract_algebra
lukaemon_mmlu_machine_learning
lukaemon_mmlu_clinical_knowledge
lukaemon_mmlu_global_facts
lukaemon_mmlu_management
lukaemon_mmlu_nutrition
lukaemon_mmlu_marketing
lukaemon_mmlu_professional_accounting
lukaemon_mmlu_high_school_geography
lukaemon_mmlu_international_law
lukaemon_mmlu_moral_scenarios
lukaemon_mmlu_computer_security
lukaemon_mmlu_high_school_microeconomics
lukaemon_mmlu_professional_law
lukaemon_mmlu_medical_genetics
lukaemon_mmlu_professional_psychology
lukaemon_mmlu_jurisprudence
lukaemon_mmlu_world_religions
lukaemon_mmlu_philosophy
lukaemon_mmlu_virology
lukaemon_mmlu_high_school_chemistry
lukaemon_mmlu_public_relations
lukaemon_mmlu_high_school_macroeconomics
lukaemon_mmlu_human_sexuality
lukaemon_mmlu_elementary_mathematics
lukaemon_mmlu_high_school_physics
lukaemon_mmlu_high_school_computer_science
lukaemon_mmlu_high_school_european_history
lukaemon_mmlu_business_ethics
lukaemon_mmlu_moral_disputes
lukaemon_mmlu_high_school_statistics
lukaemon_mmlu_miscellaneous
lukaemon_mmlu_formal_logic
lukaemon_mmlu_high_school_government_and_politics
lukaemon_mmlu_prehistory
lukaemon_mmlu_security_studies
lukaemon_mmlu_high_school_biology
lukaemon_mmlu_logical_fallacies
lukaemon_mmlu_high_school_world_history
lukaemon_mmlu_professional_medicine
lukaemon_mmlu_high_school_mathematics
lukaemon_mmlu_college_medicine
lukaemon_mmlu_high_school_us_history
lukaemon_mmlu_sociology
lukaemon_mmlu_econometrics
lukaemon_mmlu_high_school_psychology
lukaemon_mmlu_human_aging
lukaemon_mmlu_us_foreign_policy
lukaemon_mmlu_conceptual_physics
'''

2.2.2 模型

models指定模型，可以指定本地或API模型，本文使用本地模型，通过变量model_path传入模型配置。

max_seq_len表示模型最大输入上下文长度，一般为了保证问题完整，设置为4096。

max_out_len表示模型最大输出长度，需要注意的是，模型无法先验地根据规定长度调整输出内容，这是一个硬截断机制，设置过短将导致模型无法输出答案。

quantization_config表示量化配置，本文通过load_in_4bit=True进行了4bit量化。事实上，opencompass当前版本也支持直接使用load_in_4bit=True配置，如：

model_kwargs=dict(
device_map='auto',
load_in_4bit=True,
),

但会在推理时提示：

The `load_in_4bit` and `load_in_8bit` arguments are deprecated and will be removed in the future versions. Please, pass a `BitsAndBytesConfig` object in `quantization_config` argument instead.

2.2.3 修改评测集prompt

2.1中检查评测集prompt时，MMLU的评测集：

Answer the following multiple choice question. The last line of your response should be of the following format: 'ANSWER: $LETTER' (without quotes) where LETTER is one of ABCD. Think step by step before answering.

这会导致不带有隐式推理机制的模型输出显式思考过程，虽然可能提高准确率，但易被max_out_len截断。为此，需要修改prompt，datasets[0]['infer_cfg']['prompt_template']['template']['round'][0]['prompt']对应了2.1中输出的嵌套结构，将prompt修改为：

"Answer the following multiple choice question.\\n"     "Choose one option from A, B, C or D.\\n"     "Do not explain.\\n"     "Output only:\\n"     "ANSWER: <LETTER>\\n\\n"     "{input}\\n\\n"     "A) {A}\\n"     "B) {B}\\n"     "C) {C}\\n"     "D) {D}"

此处并非经过精心设计的prompt engineering，仅防止硬截断的措施。注意对齐原始的输出格式，否则框架无法解析推理结果，会造成Acc=0。

2.3 进行评测

在命令行中，输入：

opencompass …/qwen25_7b_mllu.py -w ./outputs/qwen25_7b

即可开始评测。其中，…/qwen25_7b_mllu.py是2.2中python配置文件所在位置。-w参数指定输出路径，评测时会在该路径下创建一个时间戳目录，如图：

logs保存推理及评测时的日志信息，包括具体错误堆栈，如遇报错可查询这两个日志。

predictions是模型对于全部问题的预测结果，可从中观察bad case。

results和predictions的内容类似，但保存了解析后的结果（将标准输出格式答案“Answer: C”解析为“C”）及Acc。

summary为评测结果，三个文件内容一致，格式不同，包含信息如下：

关于mode列，mmlu除了gen模式外，还支持ppl模式，在文末会贴出配置文件，不再演示过程。

参考资料

https://github.com/open-compass/opencompass

https://github.com/modelscope/modelscope

MMLU_ppl配置

from mmengine.config import read_base
from transformers import BitsAndBytesConfig
import torch
from copy import deepcopy

with read_base():
from opencompass.configs.datasets.mmlu.mmlu_ppl import mmlu_datasets

from opencompass.models import HuggingFaceCausalLM

model_path = '/home/llm_location/Qwen/Qwen2___5-7B-Instruct'

datasets = deepcopy([
d for d in mmlu_datasets
if d['abbr'] == 'lukaemon_mmlu_college_biology'
])

# ppl模式无需修改prompt，它要求模型输出的是模型对每个选项的概率分配，不会面临gen模式那么严重的截断问题

models = [
dict(
type=HuggingFaceCausalLM,

abbr='qwen25-7b-mmlu_ppl',

path=model_path,
tokenizer_path=model_path,

max_seq_len=4096,
max_out_len=128,

batch_size=1,

run_cfg=dict(
num_gpus=1,
),

model_kwargs=dict(
device_map='auto',
load_in_4bit=True,
),
)
]