{
"input": [
{
"id": "803b7f31",
"content": "{\"state\": \"Here are three expense lines from my trip. Turn them into a JSON list with fields merchant, date, amount_usd.\\n- Blue Cafe, 2024-03-02, $14.50\\n- MetroCab, 2024-03-02, $23.00\\n- Hotel Lira, 2024-03-04, $189.90\", \"question\": {\"type\" ... [TRUNCATED 79 chars] ... .\", \"criteria\": {\"chitchat\": \"Casual small talk, no task.\", \"creative_writing\": \"Compose original text.\", \"extraction_json\": \"Pull given details into a structured JSON shape.\", \"data_analysis\": \"Compute statistics or trends over the data.\"}}}",
"internal": {
"systemone": {
"kind": "choice",
"state": "Here are three expense lines from my trip. Turn them into a JSON list with fields merchant, date, amount_usd.\n- Blue Cafe, 2024-03-02, $14.50\n- MetroCab, 2024-03-02, $23.00\n- Hotel Lira, 2024-03-04, $189.90",
"instructions": "Classify the task type of this request for routing.",
"criteria": {
"chitchat": "Casual small talk, no task.",
"creative_writing": "Compose original text.",
"extraction_json": "Pull given details into a structured JSON shape.",
"data_analysis": "Compute statistics or trends over the data."
},
"answer_prefix": "ANSWER: "
}
}
}
],
"target": "extraction_json",
"id": 0,
"group_id": 0,
"metadata": {
"source_id": "jb16-llm_router-9a24706d03",
"decision_gold": "extraction_json",
"decision_kind": "choice",
"source_group": null,
"family": "llm_router",
"tier": "easy",
"language": "en",
"continuous": false,
"decision_labels": [
"chitchat",
"creative_writing",
"extraction_json",
"data_analysis"
],
"fixed_classes": false
}
}
注:部分内容因显示需要被截断。
提示模板
未定义提示模板。
使用方法
使用 CLI
evalscope eval\
--model YOUR_MODEL \
--eval-type systemone_api \
--api-url http://127.0.0.1:18131/v1 \
--api-key EMPTY_TOKEN \
--datasets open_decisions \
--limit 10 # Remove this line for formal evaluation
使用 Python
fromevalscopeimportrun_taskfromevalscope.configimportTaskConfigtask_cfg = TaskConfig(
model='YOUR_MODEL',
eval_type='systemone_api',
api_url='http://127.0.0.1:18131/v1',
api_key='EMPTY_TOKEN',
datasets=['open_decisions'],
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)