基于局部偏好优化的大模型机器翻译方法研究

doi:10.19678/j.issn.1000-3428.0252213

摘要/Abstract

摘要： 基于直接偏好优化（DPO）的强化学习方法在大语言模型（LLM）的诸多下游任务中展现了良好的效果，然而该方法直接应用在机器翻译中时，常会因为全局奖励最大化策略而产生过度优化问题，具体表现为模型过度关注与参考译文的分布一致性，而丧失了局部翻译多样性和全局优化的潜力。为了解决上述问题，探究DPO方法在LLM机器翻译中表现劣化的根本原因，在此基础上提出一种基于局部偏好优化（LPO）的LLM机器翻译方法。该方法对LLM进行动态温度采样，并通过无参考评估找出翻译中的易错低频短语，提出一种结合全局差异和局部关键差异的偏好数据构造方法，在综合考虑模型全局翻译效果和局部多样性的前提下，设计token级的全局损失和局部损失函数，最后利用两阶段课程学习策略逐步调整模型对低频短语的输出偏好。在FLORES-200数据集上选取14种形态复杂的多语言翻译任务进行测试，实验结果表明，所提方法在XCOMET、COMET-22和BLEU上的得分结果分别为80.7、89.9和30.2。此外，与多个多语言机器翻译强基线进行对比，所提方法在所有翻译语向上均优于基线模型，验证了该方法的有效性。

关键词: 机器翻译, 大语言模型, 多语言, 强化学习, 直接偏好优化

Abstract: Reinforcement learning based on Direct Preference Optimization (DPO) has demonstrated excellent performance in numerous downstream tasks of Large Language Model (LLM). However, when directly applied to machine translation, this method often leads to over-optimization owing to the global reward maximization strategy. Specifically, the model focuses excessively on distribution consistency with reference translations, thus losing local translation diversity and the potential for global optimization. To address these problems and explore the root cause of performance degradation of the DPO method in LLM-based machine translation, this paper proposes an LLM machine translation method based on Local Preference Optimization (LPO). This method performs dynamic temperature sampling on the LLM, identifies error-prone low-frequency phrases in translation through a reference-free evaluation, and proposes a preference data construction method that combines global and local key differences. By comprehensively considering the model's global translation performance and local diversity, it designs token-level global and local loss functions and finally adopts a two-stage curriculum learning strategy to gradually adjust the model's output preference for low-frequency phrases. Fourteen multilingual translation tasks with complex morphologies are selected from the FLORES-200 dataset for testing. Experimental results show that the proposed method achieves scores of 80.7, 89.9 and 30.2 on XCOMET, COMET-22, and BLEU, respectively. In addition, compared with several strong baselines for multilingual machine translation, the proposed method outperforms the baseline models in all translation directions, thus validating its effectiveness.

Key words: machine translation, Large Language Model (LLM), multilingual, reinforcement learning, Direct Preference Optimization (DPO)

中图分类号:

TP391

鲁凯文, 杨雅婷, 董瑞, 马博, 王磊, 周喜, 马荣. 基于局部偏好优化的大模型机器翻译方法研究[J]. 计算机工程, 2026, 52(11): 412-422.

LU Kaiwen, YANG Yating, DONG Rui, MA Bo, WANG Lei, ZHOU Xi, MA Rong. Research on Large Language Model Machine Translation Method Based on Local Preference Optimization[J]. Computer Engineering, 2026, 52(11): 412-422.

https://www.ecice06.com/CN/Y2026/V52/I11/412

参考文献

[1] TOUVRON H, LAVRIL T, IZACARD G, et al. LLaMA: open and efficient foundation language models[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2302.13971.
[2] ANIL R, DAI A M, FIRAT O, et al. PaLM 2 technical report[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2305.10403.
[3] CHOWDHERY A, NARANG S, DEVLIN J, et al. PaLM: scaling language modeling with pathways[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2204.02311.
[4] FUJII K, NAKAMURA T, LOEM M, et al. Continual pre-training for cross-lingual LLM adaptation: enhancing Japanese language capabilities[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2404.17790.
[5] ZHANG S L, FANG Q K, ZHANG Z C, et al. BayLing: bridging cross-lingual alignment and instruction following through interactive translation for large language models[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2306.10968.
[6] LU K W, YANG Y T, YANG F Y, et al. Low-resource language expansion and translation capacity enhancement for LLM: a study on the uyghur[C]//Proceedings of the 31st International Conference on Computational Linguistics. [ S.l.]:ACL,2025: 8360-8373.
[7] DEEPSEEK-AI, GUO D Y, YANG D J, et al. DeepSeek-R1: incentivizing reasoning capability in LLMs via reinforcement learning[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2501.12948.
[8] 刘鹏程, 汪永伟, 余欣鋆, 等. 一种面向博弈场景的PPO-Dueling DQN策略优化方法[J]. 小型微型计算机系统, 2025, 46(11): 2594-2599.LIU P C, WANG Y W, YU X J, et al. PPO-Dueling DQN strategy optimization method for game scenarios[J]. Journal of Chinese Computer Systems, 2025, 46(11): 2594-2599. (in Chinese)
[9] RAFAILOV R, SHARMA A, MITCHELL E, et al. Direct preference optimization: your language model is secretly a reward model[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2305.18290.
[10] WANG P F, XU A, ZHOU Y L, et al. Direct judgement preference optimization[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2409.14664.
[11] CHEN G X, LIAO M P, LI C X, et al. Step-level value preference optimization for mathematical reasoning[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2406.10858.
[12] XU H R, SHARAF A, CHEN Y M, et al. Contrastive preference optimization: pushing the boundaries of LLM performance in machine translation[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2401.08417.
[13] HE Z W, WANG X, JIAO W X, et al. Improving machine translation with human feedback: an exploration of quality estimation as a reward model[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [S.l.]:ACL,2024: 8164-8180.
[14] TEAM N, COSTA-JUSSÀ M R, CROSS J, et al. No language left behind: scaling human-centered machine translation[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2207.04672.
[15] MAO Z Y, YU Y. Tuning LLMs with contrastive alignment instructions for machine translation in unseen, low-resource languages[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2401.05811.
[16] ZHANG X, RAJABI N, DUH K, et al. Machine translation with large language models: prompting, few-shot learning, and fine-tuning with QLoRA[C]//Proceedings of the 8th Conference on Machine Translation. [S.l.]:ACL,2023: 468-481.
[17] 侯钰涛, 阿布都克力木·阿布力孜, 史亚庆, 等. 面向“一带一路”的低资源语言机器翻译研究[J]. 计算机工程, 2024, 50(4): 332-341.HOU Y T, ABUDUKELIMU A, SHI Y Q, et al. Research on low-resource language machine translation for the “Belt an Road”[J]. Computer Engineering, 2024, 50(4): 332-341. (in Chinese)
[18] YIN Y J, ZENG J L, LI Y F, et al. LexMatcher: dictionary-centric data collection for LLM-based machine translation[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2406.01441.
[19] FENG Z P, ZHANG Y, LI H, et al. TEaR: improving LLM-based machine translation with systematic self-refinement[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2402.16379.
[20] RAUNAK V, SHARAF A, WANG Y R, et al. Leveraging GPT-4 for automatic translation post-editing[C]//Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2023. [S.l.]:ACL,2023: 12009-12024.
[21] 李博, 季佰军, 段湘煜. 基于译文易错词纠正机制的大语言模型机器翻译[J]. 计算机工程, 2026, 52(2): 372-382.LI B, JI B J, DUAN X Y. Machine translation with large language models based on correction mechanism of error-prone words in translations[J]. Computer Engineering, 2026, 52(2): 372-382. (in Chinese)
[22] MA Y J, LIANG W, WANG G Z, et al. EUREKA: human-level reward design via coding large language models[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2310.12931.
[23] WU Y, FAN Y W, LIANG P P, et al. Read and reap the rewards: learning to play atari with the help of instruction manuals[EB/OL]. [2025-02-05]. https://proceedings.neurips.cc/paper_files/paper/2023/file/034d7bfeace2a9a258648b16fc626298-Paper-Conference.pdf.
[24] ZHOU Z H, HU B, ZHANG P, et al. Large language model as a policy teacher for training reinforcement learning agents[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2311.13373.
[25] 罗焕坤, 葛一烽, 刘帅. 大语言模型在数学推理中的研究进展[J]. 计算机工程, 2024, 50(9): 1-17.LUO H K, GE Y F, LIU S. Research progress of large language models in mathematical reasoning[J]. Computer Engineering, 2024, 50(9): 1-17. (in Chinese)
[26] YANG G Y, CHEN J H, LIN W Z, et al. Direct preference optimization for neural machine translation with minimum Bayes risk decoding[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [S.l.]:ACL,2024: 391-398.
[27] OUYANG L, WU J, JIANG X, et al. Training language models to follow instructions with human feedback[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2203.02155.
[28] OPENAI, ACHIAM J, ADLER S, et al. GPT-4 technical report[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2303.08774.
[29] XU H R, MURRAY K, KOEHN P, et al. X-ALMA: plug & play modules and adaptive rejection for quality translation at scale[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2410.03115.
[30] POST M. A call for clarity in reporting BLEU scores[EB/OL]. [2025-02-05]. https://arxiv.org/abs/1804.08771.
[31] REI R, STEWART C, FARINHA A C, et al. COMET: a neural framework for MT evaluation[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2009.09025.
[32] GUERREIRO N M, REI R, VAN S D, et al. XCOMET: transparent machine translation evaluation through fine-grained error detection[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2310.10482.
[33] BARRAULT L, BOJAR O, COSTA-JUSS M R, et al. Findings of the 2019 conference on machine translation (WMT19) [EB/OL]. [2025-02-05]. https://aclanthology.org/W19-5301.pdf.
[34] FEDERMANN C, KOCMI T, XIN Y. NTREX-128-news test references for MT evaluation of 128 languages[C]//Proceedings of the 1st Workshop on Scaling up Multilingual Evaluation. [S.l.]:ACL,2022: 21-24.
[35] ÜSTÜN A, ARYABUMI V, YONG Z, et al. Aya model: an instruction finetuned open-access multilingual language model[C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. [S.l.]:ACL,2024: 15894-15939.
[36] XUE L T, CONSTANT N, ROBERTS A, et al. MT5: a massively multilingual pre-trained text-to-text transformer[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [S.l.]:ACL,2021: 483-498.
[37] LU Y Q, ZHU W H, LI L, et al. LLaMAX: scaling linguistic horizons of LLM by enhancing translation capabilities beyond 100 languages[C]//Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2024. [S.l.]:ACL,2024: 10748-10772.
[38] TOUVRON H, MARTIN L, STONE K, et al. Llama 2: open foundation and fine-tuned chat models[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2307.09288.
[39] DUBEY A, JAUHRI A, PANDEY A, et al. The Llama 3 herd of models[EB/OL]. [2025-02-05]. https://arxiv.org/abs/2407.21783.
[40] REI R, DE SOUZA J G C, ALVES D, et al. COMET-22: unbabel-IST 2022 submission for the metrics shared task[C]//Proceedings of the 7th Conference on Machine Translation (WMT). [S.l.]:ACL,2022: 578-585.

选择文件类型/文献管理软件名称

选择包含的内容