通用能力评估
在完成模型训练后,您可以通过 llamafactory-cli eval examples/train_lora/llama3_lora_eval.yaml 来评估模型效果。
llamafactory-cli eval examples/train_lora/llama3_lora_eval.yaml
配置示例文件 examples/train_lora/llama3_lora_eval.yaml 具体如下:
examples/train_lora/llama3_lora_eval.yaml
### examples/train_lora/llama3_lora_eval.yaml ### model model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct adapter_name_or_path: saves/llama3-8b/lora/sft # 可选项 ### method finetuning_type: lora ### dataset task: mmlu_test # mmlu_test, ceval_validation, cmmlu_test template: fewshot lang: en n_shot: 5 ### output save_dir: saves/llama3-8b/lora/eval ### eval batch_size: 4
NLG 评估
此外,您还可以通过 llamafactory-cli train examples/extras/nlg_eval/llama3_lora_predict.yaml 来获得模型的 BLEU 和 ROUGE 分数以评价模型生成质量。
llamafactory-cli train examples/extras/nlg_eval/llama3_lora_predict.yaml
配置示例文件 examples/extras/nlg_eval/llama3_lora_predict.yaml 具体如下:
examples/extras/nlg_eval/llama3_lora_predict.yaml
### examples/extras/nlg_eval/llama3_lora_predict.yaml ### model model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct adapter_name_or_path: saves/llama3-8b/lora/sft ### method stage: sft do_predict: true finetuning_type: lora ### dataset eval_dataset: identity,alpaca_en_demo template: llama3 cutoff_len: 2048 max_samples: 50 overwrite_cache: true preprocessing_num_workers: 16 ### output output_dir: saves/llama3-8b/lora/predict overwrite_output_dir: true ### eval per_device_eval_batch_size: 1 predict_with_generate: true ddp_timeout: 180000000
同样,您也通过在指令 python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo 中指定模型、数据集以使用 vllm 推理框架以取得更快的推理速度。
python scripts/vllm_infer.py --model_name_or_path path_to_merged_model --dataset alpaca_en_demo
评估相关参数
表:EvalArguments
evaluation
4
42
en
zh
5
None
DownloadMode.REUSE_DATASET_IF_EXISTS