概述
MultiPL-E MBPP 是一个源自 MBPP(Mostly Basic Python Programming)的多语言代码生成基准测试。它将原始 MBPP 扩展至 18 种编程语言,支持对代码生成能力进行跨语言评估。
任务描述
主要特性
评估说明
属性
数据统计
各子集统计数据:
样例示例
子集:mbpp-cpp
mbpp-cpp
{ "input": [ { "id": "209e8d38", "content": "```cpp\n#include<assert.h>\n#include<bits/stdc++.h>\n// Write a cppthon function to identify non-prime numbers.\nbool is_not_prime(long n) {\n\n```\n\nPlease complete the above code according to the requirements in the docstring. Write the complete code and wrap it in markdown fenced code. The code should not contain `Main` function." } ], "target": "", "id": 0, "group_id": 0, "metadata": { "tests": "}\nint main() {\n auto candidate = is_not_prime;\n assert(candidate((2)) == (false));\n assert(candidate((10)) == (true));\n assert(candidate((35)) == (true));\n assert(candidate((37)) == (false));\n}\n", "stop_tokens": [ "\n}" ], "task_id": "mbpp_3_is_not_prime", "language": "cpp", "doctests": "transform" } }
提示模板
提示模板:
{prompt}
沙箱配置
此基准测试需要沙箱环境来执行代码。
{ "image": "volcengine/sandbox-fusion:server-20250609", "tools_config": { "shell_executor": {}, "python_executor": {}, "multi_code_executor": {} }, "memory_limit": "2g", "cpu_limit": "2.0" }
使用方法
使用 CLI
evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets multiple_mbpp \ --sandbox '{"enabled": true}' \ --limit 10 # 正式评估时请删除此行
使用 Python
from evalscope import run_task from evalscope.config import TaskConfig task_cfg = TaskConfig( model='YOUR_MODEL', api_url='OPENAI_API_COMPAT_URL', api_key='EMPTY_TOKEN', datasets=['multiple_mbpp'], sandbox={'enabled': True}, dataset_args={ 'multiple_mbpp': { # subset_list: ['mbpp-cpp', 'mbpp-ts', 'mbpp-sh'] # 可选,用于评估特定子集 } }, limit=10, # 正式评估时请删除此行 ) run_task(task_cfg=task_cfg)