Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.
{question}
A. {choice_a}
B. {choice_b}
C. {choice_c}
D. {choice_d}
{
"input": [
{
"id": "544443c0",
"content": [
{
"audio": "[BASE64_AUDIO: mp3, ~25.9KB]",
"format": "mp3"
},
{
"text": "Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.\nWhich age range do you believe best matches the speaker's voice?\nA. teens to twenties\nB. thirties to fourties\nC. fifties to sixties\nD. seventies to eighties"
}
]
}
],
"target": "B",
"id": 0,
"group_id": 0,
"subset_key": "Speaker_Age_Prediction_common_voice_13.0_en",
"metadata": {
"uniq_id": 5973,
"task_name": "Speaker_Age_Prediction",
"dataset_name": "common_voice_13.0_en",
"category": "speech",
"answer_gt_text": "thirties to fourties",
"choices": {
"A": "teens to twenties",
"B": "thirties to fourties",
"C": "fifties to sixties",
"D": "seventies to eighties"
}
}
}
提示模板
提示模板:
Choose the most suitable answer from options A, B, C, and D to respond the question in next line, you may only choose A or B or C or D.
额外参数
参数
类型
默认值
描述
subsets
list
None
可选的 Foundation 源数据集目录列表,用于评估。默认为所有 25 个目录。当仅下载了部分子集时非常有用。
使用方法
使用 CLI
evalscope eval\
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets air_bench_foundation \
--limit 10 # Remove this line for formal evaluation
使用 Python
fromevalscopeimportrun_taskfromevalscope.configimportTaskConfigtask_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['air_bench_foundation'],
dataset_args={
'air_bench_foundation': {
# subset_list: ['Speaker_Age_Prediction_common_voice_13.0_en', 'Speaker_Emotion_Recontion_iemocap', 'Speaker_Emotion_Recontion_meld'] # optional, evaluate specific subsets
# extra_params: {} # uses default extra parameters
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)