仅准确率以全部样本数为分母;若答案不是严格意义上的 YES/NO,则不会计入精确率、召回率和 F1 的计算,因此当答案格式错误较多时,这三个指标可能虚高
支持带推理示例的 few-shot 评估
属性
属性
值
基准测试名称
coin_flip
数据集 ID
extraordinarylab/coin-flip
论文
N/A
标签
Reasoning, Yes/No
指标
accuracy, precision, recall, f1, yes_ratio
默认示例数量
0-shot
评估分割
test
训练分割
validation
聚合方式
f1
数据统计
指标
值
总样本数
3,333
提示词长度(平均)
500.15 字符
提示词长度(最小/最大)
453 / 551 字符
样例示例
子集: default
{"input":[{"id":"05503706","content":[{"text":"\nSolve the following coin flip problem step by step. The last line of your response should be of the form \"ANSWER: [ANSWER]\" (without quotes) where [ANSWER] is the answer to the problem.\n\nQ: A coin is heads up. rushawn flips the coin. yerania ... [TRUNCATED] ... the coin. jostin does not flip the coin. Is the coin still heads up?\n\nRemember to put your answer on its own line at the end in the form \"ANSWER: [ANSWER]\" (without quotes) where [ANSWER] is the answer YES or NO to the problem.\n\nReasoning:\n"}]}],"target":"NO","id":0,"group_id":0,"metadata":{"answer":"NO"}}
注:部分内容为显示目的已截断。
提示模板
提示模板:
Solve the following coin flip problem step by step. The last line of your response should be of the form "ANSWER: [ANSWER]" (without quotes) where [ANSWER] is the answer to the problem.
{question}
Remember to put your answer on its own line at the end in the form "ANSWER: [ANSWER]" (without quotes) where [ANSWER] is the answer YES or NO to the problem.
Reasoning:
Few-shot 模板
Here are some examples of how to solve similar problems:
{fewshot}
Solve the following coin flip problem step by step. The last line of your response should be of the form "ANSWER: [ANSWER]" (without quotes) where [ANSWER] is the answer to the problem.
{question}
Remember to put your answer on its own line at the end in the form "ANSWER: [ANSWER]" (without quotes) where [ANSWER] is the answer YES or NO to the problem.
Reasoning: