概述
SPaRC(空间路径推理挑战)基于游戏《见证者》测试二维网格谜题中的逐步空间推理。模型必须从起始节点到结束节点绘制一条线,这条线必须遵守网格上的所有规则。
任务描述
####
(x, y)
主要特点
评估说明
subset_list: ['Level 5']
limit
limit=4
accuracy
invalid_path_rate
sparc-puzzle
evalscope/SPaRC
all
max_tokens
属性
数据统计
各子集统计:
样例示例
子集: Level 1
Level 1
{ "input": [ { "id": "a7ac7618", "content": "You are an expert at solving puzzles games." }, { "id": "468c49a5", "content": "\n ## Objective\n You are a specialized AI proficient in spatial reasoning and solving puzzles from the game 'The Witness'. Your goal is to find a valid path (a continuous line) from the specified Start Node to the End Node on the provide ... [TRUNCATED 7215 chars] ... marker string `####` followed immediately by the solution path as a list of node coordinates `(x, y)`. Include all intermediate nodes from start to end.\n\n **Example Solution Path Format:**\n ####\n (0,0), (1,0), (2,0), (2,1), ...\n " } ], "target": "(5,0), (6,0), (6,1), (6,2), (5,2), (4,2), (4,3), (4,4), (5,4), (6,4), (6,5), (6,6), (5,6), (4,6), (3,6), (2,6), (1,6), (0,6), (0,5), (0,4), (1,4), (2,4), (2,3), (2,2), (1,2), (0,2), (0,1), (0,0), (1,0), (2,0), (3,0), (4,0)", "id": 0, "group_id": 0, "subset_key": "Level 1", "metadata": { "difficulty_level": 1, "difficulty_score": 0.842447049854198, "solution_count": 3, "puzzle": { "id": "3fe8a85b768ae5b7", "grid_size": { "height": 3, "width": 4 }, "puzzle_array": [ [ "+", ".", "+", "+", "E", "S", "+", "+", "+" ], [ "+", "N", "+", "N", "+", "N", "+", "P-B-1824", "+" ], [ "+", "+", "+", "+", "+", "+", "+", "+", "+" ], [ "+", "N", "+", "N", "+", "N", "+", "N", "G" ], [ ".", "+", "+", "+", "+", "+", "+", "+", "+" ], [ "+", "N", "+", "N", "+", "N", "+", "N", "+" ], [ "+", "+", "+", "+", "+", ".", "+", "+", "+" ] ], "polyshapes": "{\"1824\":[[0,0,1,0],[0,1,1,0],[0,0,1,0],[0,0,0,0]]}", "solutions": [ { "index": 0, "path": [ { "x": 5, "y": 0 }, { "x": 6, "y": 0 }, { "x": 6, "y": 1 }, { "x": 6, "y": 2 }, { "x": 5, "y": 2 }, { "x": 4, "y": 2 }, { "x": 4, "y": 3 }, { "x": 4, "y": 4 }, { "x": 5, "y": 4 }, { "x": 6, "y": 4 }, "... [TRUNCATED 22 more items] ..." ], "pathLength": 32 }, { "index": 1, "path": [ { "x": 5, "y": 0 }, { "x": 6, "y": 0 }, { "x": 6, "y": 1 }, { "x": 6, "y": 2 }, { "x": 5, "y": 2 }, { "x": 4, "y": 2 }, { "x": 4, "y": 3 }, { "x": 4, "y": 4 }, { "x": 5, "y": 4 }, { "x": 6, "y": 4 }, "... [TRUNCATED 18 more items] ..." ], "pathLength": 28 }, { "index": 2, "path": [ { "x": 5, "y": 0 }, { "x": 6, "y": 0 }, { "x": 6, "y": 1 }, { "x": 6, "y": 2 }, { "x": 5, "y": 2 }, { "x": 4, "y": 2 }, { "x": 4, "y": 3 }, { "x": 4, "y": 4 }, { "x": 5, "y": 4 }, { "x": 6, "y": 4 }, "... [TRUNCATED 18 more items] ..." ], "pathLength": 28 } ] } } }
注:部分内容因显示原因被截断。
提示模板
系统提示:
You are an expert at solving puzzles games.
提示模板:
{question}
使用方法
使用 CLI
evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets sparc \ --limit 10 # Remove this line for formal evaluation
使用 Python
from evalscope import run_task from evalscope.config import TaskConfig task_cfg = TaskConfig( model='YOUR_MODEL', api_url='OPENAI_API_COMPAT_URL', api_key='EMPTY_TOKEN', datasets=['sparc'], dataset_args={ 'sparc': { # subset_list: ['Level 1', 'Level 2', 'Level 3'] # optional, evaluate specific subsets } }, limit=10, # Remove this line for formal evaluation ) run_task(task_cfg=task_cfg)