{
"input": [
{
"id": "3f044b9c",
"content": "Below is a table and some paragraphs from a financial report, followed by a question.\n\nTable:\n | June 30, | \n | 2019 | 2018\nDeferred tax assets | | \nNon-capital loss carryforwards | $161,119 | $129,436\nCapital loss carryforwards | 155 | 417\n ... [TRUNCATED 1480 chars] ... a number, give the number only and put its scale (thousand, million, billion or percent) on the Scale line.\nThink step by step, then end your response with these two lines:\nAnswer: <answer>\nScale: <none, thousand, million, billion or percent>"
}
],
"target": "948,578",
"id": 0,
"group_id": 0,
"subset_key": "table",
"metadata": {
"uid": "107efaa11617ac41f5f9b3b5adf1e98c",
"answer": [
"948,578"
],
"answer_type": "span",
"answer_from": "table",
"scale": ""
}
}
注:部分内容因显示原因被截断。
提示模板
提示模板:
Below is a table and some paragraphs from a financial report, followed by a question.
Table:
{table}
Paragraphs:
{paragraphs}
Question: {question}
Answer the question using the table and the paragraphs:
- If the answer is text from the table or the paragraphs, copy it exactly. If there are several, separate them with " | ".
- If the answer is a number, give the number only and put its scale (thousand, million, billion or percent) on the Scale line.
Think step by step, then end your response with these two lines:
Answer: <answer>
Scale: <none, thousand, million, billion or percent>
使用方法
使用CLI
evalscope eval\
--model YOUR_MODEL \
--api-url OPENAI_API_COMPAT_URL \
--api-key EMPTY_TOKEN \
--datasets tat_qa \
--limit 10 # Remove this line for formal evaluation
使用Python
fromevalscopeimportrun_taskfromevalscope.configimportTaskConfigtask_cfg = TaskConfig(
model='YOUR_MODEL',
api_url='OPENAI_API_COMPAT_URL',
api_key='EMPTY_TOKEN',
datasets=['tat_qa'],
dataset_args={
'tat_qa': {
# subset_list: ['table', 'text', 'table-text'] # optional, evaluate specific subsets
}
},
limit=10, # Remove this line for formal evaluation
)
run_task(task_cfg=task_cfg)