概述
句子相似度在0到5六个有序级别上进行评分;保留连续的黄金标准。
任务描述
主要特点
sentence-transformers/stsb
评估说明
属性
数据统计
样例示例
子集: default
default
{ "input": [ { "id": "8eaa765d", "content": "{\"state\": {\"sentence1\": \"A girl is styling her hair.\", \"sentence2\": \"A girl is brushing her hair.\"}, \"question\": {\"type\": \"score\", \"instructions\": \"Rate the semantic similarity of sentence1 and sentence2.\", \"criteria\": [\"Unrelated\", \"Very different\", \"Some shared meaning\", \"Mostly similar\", \"Nearly equivalent\", \"Completely equivalent\"]}}", "internal": { "systemone": { "kind": "score", "state": { "sentence1": "A girl is styling her hair.", "sentence2": "A girl is brushing her hair." }, "instructions": "Rate the semantic similarity of sentence1 and sentence2.", "criteria": [ "Unrelated", "Very different", "Some shared meaning", "Mostly similar", "Nearly equivalent", "Completely equivalent" ], "answer_prefix": "ANSWER: " } } } ], "target": "2.5", "id": 0, "group_id": 0, "metadata": { "source_id": null, "decision_gold": 2.5, "decision_kind": "score", "source_group": null, "family": "", "tier": "", "language": "", "continuous": true, "decision_labels": [ "0", "1", "2", "3", "4", "5" ], "fixed_classes": false } }
提示模板
未定义提示模板。
使用方法
使用CLI
evalscope eval \ --model YOUR_MODEL \ --eval-type systemone_api \ --api-url http://127.0.0.1:18131/v1 \ --api-key EMPTY_TOKEN \ --datasets stsb \ --limit 10 # Remove this line for formal evaluation
使用Python
from evalscope import run_task from evalscope.config import TaskConfig task_cfg = TaskConfig( model='YOUR_MODEL', eval_type='systemone_api', api_url='http://127.0.0.1:18131/v1', api_key='EMPTY_TOKEN', datasets=['stsb'], limit=10, # Remove this line for formal evaluation ) run_task(task_cfg=task_cfg)