概述
MBPP Plus 是 MBPP 基准测试的增强版本,旨在提高基础 Python 编程合成的评估可靠性。它解决了原始数据集中的质量问题,并显著增加了每个问题的测试覆盖率。
任务描述
主要特点
评估说明
pass@k
属性
数据统计
样例示例
子集: default
default
{ "input": [ { "id": "be634cb0", "content": "You are an expert Python programmer, and here is your task: Write a function to find the shared elements from the given two lists. Your code should pass these tests:\n\nassert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))\nassert set(similar_elements((1, 2, 3, 4),(5, 4, 3, 7))) == set((3, 4))\nassert set(similar_elements((11, 12, 14, 13),(17, 15, 14, 13))) == set((13, 14))" } ], "target": "\ndef similar_elements(test_tup1, test_tup2):\n return tuple(set(test_tup1) & set(test_tup2))\n", "id": 0, "group_id": 0, "metadata": { "task_id": 2, "source_file": "Benchmark Questions Verification V2.ipynb", "test_imports": [], "test_list": [ "assert set(similar_elements((3, 4, 5, 6),(5, 7, 4, 10))) == set((4, 5))", "assert set(similar_elements((1, 2, 3, 4),(5, 4, 3, 7))) == set((3, 4))", "assert set(similar_elements((11, 12, 14, 13),(17, 15, 14, 13))) == set((13, 14))" ], "test": "import numpy as np\nfrom math import inf\n\ndef is_floats(x) -> bool:\n # check if it is float; List[float]; Tuple[float]\n if isinstance(x, float):\n return True\n if isinstance(x, (list, tuple)):\n return all(isinstance(i, fl ... [TRUNCATED 8976 chars] ... wvS', 'tUqF', 'ITntCqEvPi', 'kx'), (1, 2, 3, 4, 5, 6, 7, 8, 9), (11, 12, 13, 14, 15, 16, 17, 19, 20), (19, 5), (1, 2, 3, 6, 7, 8, 9, 10, 12)]\nfor i, (inp, exp) in enumerate(zip(inputs, results)):\n assertion(similar_elements(*inp), exp, 0)\n" } }
注: 部分内容为了显示而被截断。
提示模板
提示模板:
You are an expert Python programmer, and here is your task: {question} Your code should pass these tests: {tests}
沙箱配置
此基准测试需要沙箱环境来执行代码。
{ "image": "python3.11-numpy", "tools_config": { "shell_executor": {}, "python_executor": {} } }
使用方法
使用 CLI
evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets mbpp_plus \ --sandbox '{"enabled": true}' \ --limit 10 # Remove this line for formal evaluation
使用 Python
from evalscope import run_task from evalscope.config import TaskConfig task_cfg = TaskConfig( model='YOUR_MODEL', api_url='OPENAI_API_COMPAT_URL', api_key='EMPTY_TOKEN', datasets=['mbpp_plus'], sandbox={'enabled': True}, limit=10, # Remove this line for formal evaluation ) run_task(task_cfg=task_cfg)