概述
SimpleQA 是由 OpenAI 设计的一个基准测试,用于评估语言模型准确回答简短、事实性问题的能力。该基准专注于衡量事实准确性,并为正确、错误和未作答的答案提供了明确的评分标准。
任务描述
主要特点
评估说明
is_correct
is_incorrect
is_not_attempted
属性
数据统计
样例示例
子集: default
default
{ "input": [ { "id": "9dd57f4c", "content": "Answer the question:\n\nWho received the IEEE Frank Rosenblatt Award in 2010?" } ], "target": "Michio Sugeno", "id": 0, "group_id": 0, "metadata": { "topic": "Science and technology", "answer_type": "Person", "urls": [ "https://en.wikipedia.org/wiki/IEEE_Frank_Rosenblatt_Award", "https://ieeexplore.ieee.org/author/37271220500", "https://en.wikipedia.org/wiki/IEEE_Frank_Rosenblatt_Award", "https://www.nxtbook.com/nxtbooks/ieee/awards_2010/index.php?startid=21#/p/20" ] } }
提示模板
提示模板:
Answer the question: {question}
使用方法
使用 CLI
evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets simple_qa \ --judge '{"strategy":"llm","models":[{"model_id":"YOUR_JUDGE_MODEL"}]}' \ --limit 10 # 正式评估时请删除此行
使用 Python
from evalscope import run_task from evalscope.config import TaskConfig task_cfg = TaskConfig( model='YOUR_MODEL', api_url='OPENAI_API_COMPAT_URL', api_key='EMPTY_TOKEN', datasets=['simple_qa'], judge={'strategy': 'llm', 'models': [{'model_id': 'YOUR_JUDGE_MODEL'}]}, limit=10, # 正式评估时请删除此行 ) run_task(task_cfg=task_cfg)