概述
MultiPL-E HumanEval 是一个源自 OpenAI HumanEval 的多语言代码生成基准测试。它将原始 HumanEval 扩展至 18 种编程语言,支持对代码生成能力进行跨语言评估。
任务描述
主要特性
评估说明
属性
数据统计
各子集统计信息:
样例示例
子集: humaneval-cpp
humaneval-cpp
{ "input": [ { "id": "8f096947", "content": "```cpp\n#include<assert.h>\n#include<bits/stdc++.h>\n// Check if in given vector of numbers, are any two numbers closer to each other than\n// given threshold.\n// >>> has_close_elements((std::vector<float>({(float)1.0f, (float)2.0f, (float)3.0f}) ... [TRUNCATED] ... ents(std::vector<float> numbers, float threshold) {\n\n```\n\nPlease complete the above code according to the requirements in the docstring. Write the complete code and wrap it in markdown fenced code. The code should not contain `Main` function." } ], "target": "", "id": 0, "group_id": 0, "metadata": { "tests": "}\nint main() {\n auto candidate = has_close_elements;\n assert(candidate((std::vector<float>({(float)1.0f, (float)2.0f, (float)3.9f, (float)4.0f, (float)5.0f, (float)2.2f})), (0.3f)) == (true));\n assert(candidate((std::vector<float>({( ... [TRUNCATED] ... (std::vector<float>({(float)1.1f, (float)2.2f, (float)3.1f, (float)4.1f, (float)5.1f})), (1.0f)) == (true));\n assert(candidate((std::vector<float>({(float)1.1f, (float)2.2f, (float)3.1f, (float)4.1f, (float)5.1f})), (0.5f)) == (false));\n}\n", "stop_tokens": [ "\n}" ], "task_id": "HumanEval_0_has_close_elements", "language": "cpp", "doctests": "transform" } }
注:部分内容因展示需要已被截断。
提示模板
提示模板:
{prompt}
沙箱配置
此基准测试需要沙箱环境来执行代码。
{ "image": "volcengine/sandbox-fusion:server-20250609", "tools_config": { "shell_executor": {}, "python_executor": {}, "multi_code_executor": {} }, "memory_limit": "2g", "cpu_limit": "2.0" }
使用方法
使用 CLI
evalscope eval \ --model YOUR_MODEL \ --api-url OPENAI_API_COMPAT_URL \ --api-key EMPTY_TOKEN \ --datasets multiple_humaneval \ --sandbox '{"enabled": true}' \ --limit 10 # 正式评估时请删除此行
使用 Python
from evalscope import run_task from evalscope.config import TaskConfig task_cfg = TaskConfig( model='YOUR_MODEL', api_url='OPENAI_API_COMPAT_URL', api_key='EMPTY_TOKEN', datasets=['multiple_humaneval'], sandbox={'enabled': True}, dataset_args={ 'multiple_humaneval': { # subset_list: ['humaneval-cpp', 'humaneval-ts', 'humaneval-sh'] # 可选,用于评估特定子集 } }, limit=10, # 正式评估时请删除此行 ) run_task(task_cfg=task_cfg)