Abstract: Reinforcement Learning from Human Feedback (RLHF) is a key technology for achieving professional value alignment in large legal language models. However, existing reviews lack systematic analysis of the specific adaptation mechanisms and challenges of legal scenarios. This paper aims to fill this gap, systematically construct a panoramic view of the research and application of RLHF in legal large language models, and provide a reference for method selection in engineering practice. This article takes the six-dimensional framework of "technical foundation - algorithm evolution - task migration - evaluation system - application challenges - future path" as the main analytical thread, systematically reviewing 80 core papers from top conferences and journals such as NeurIPS, ICML, ICLR, and ACL from 2017 to 2025. Analyze the complete evolution logic of RLHF in the legal field from algorithm adaptation to evaluation system. Analysis shows that the core predicament of legal RLHF lies in the inaccurate reward modeling caused by the conflict of multi-dimensional rewards and the sparsity of expert preferences. The PPO algorithm has insufficient stability in the generation of long text laws, while new algorithms such as DPO and GRPO simplify the process but face the bottleneck of preference data quality. Further research reveals that different legal tasks vary significantly in terms of risk levels, reasoning complexity, and feedback acquisition costs. Therefore, there is no unified optimal solution for the RLHF algorithm. For structured tasks such as the generation of legal documents and summaries of regulations, DPO has good applicability due to its advantages such as a simple training process and relatively high stability. For tasks such as legal reasoning and case analysis that require multi-path argumentation, GRPO uses intra-group relative ordering to construct learning signals and shows great potential in handling complex legal arguments. In high-risk scenarios such as judicial auxiliary decision-making and sentencing recommendations, it is necessary to combine expert feedback, rule constraints, and knowledge retrieval mechanisms to achieve reliable alignment. The evaluation system is evolving from automated metrics to process interpretability and bias auditing. RLHF provides a key evolutionary path for legal large language models from usability to credibility. However, its implementation relies on endogenous rewards and hybrid feedback mechanisms to reduce annotation costs, and the development of reward traceability and process evaluation to meet judicial transparency requirements. Meanwhile, with the development of new methods such as direct preference optimization and group relative strategy optimization, legal RLHF is gradually evolving from the traditional reward model-driven mode to lightweight preference optimization and relative ranking optimization, providing a new technical path for reducing training complexity and enhancing the adaptability to legal scenarios. Existing research indicates that relying solely on a single reward model is insufficient to fully depict the complex trade-off relationships within the legal value system. In the future, it is necessary to further integrate rule knowledge, expert experience, and process feedback information to construct a more stable and interpretable legal alignment mechanism. This paper not only constructs a six-dimensional analysis framework for legal RLHF research, but also further proposes a legal task-oriented RLHF adaptability analysis system, systematically comparing the applicable boundaries and failure mechanisms of algorithms such as PPO, DPO, and GRPO in different tasks such as legal document generation, legal reasoning, and compliance review. In addition, this paper conducts a horizontal comparison of different RLHF methods by combining the risk levels of legal tasks and the characteristics of algorithms, providing a reference basis for algorithm selection and engineering deployment in legal intelligent systems. Research shows that the core contradiction of legal RLHF is not merely the insufficiency of algorithm performance, but rather a structural contradiction among multi-dimensional reward conflicts in legal scenarios, inconsistent expert preferences, and the difficulty in quantifying judicial value. This paper, for the first time, comprehensively reveals the entire process logic of legal RLHF from algorithm adaptation to engineering implementation from four aspects: technological evolution, reward modeling, evaluation system, and judicial governance, providing a systematic research framework and practical guidance for the trustworthy construction of legal large language models.

摘要： 人类反馈强化学习（Reinforcement Learning from Human Feedback, RLHF）是法律大语言模型实现专业价值对齐的关键技术，但现有综述缺乏对法律场景特有适配机制与挑战的系统分析。本文旨在填补这一空白，系统构建RLHF在法律大语言模型中的研究与应用全景，并为工程实践提供方法选型参考。本文以“技术基础—算法演进—任务迁移—评估体系—应用挑战—未来路径”六维框架为分析主线，系统回顾2017年至2025年间NeurIPS、ICML、ICLR、ACL等顶会顶刊中80篇核心文献，剖析RLHF在法律领域从算法适配到评估体系的完整演进逻辑。分析表明，法律RLHF的核心困境在于多维奖励冲突与专家偏好稀疏导致的奖励建模失准，PPO算法在长文本法律生成中稳定性不足，而DPO、GRPO等新算法虽简化流程但面临偏好数据质量瓶颈。进一步研究发现，不同法律任务在风险等级、推理复杂度以及反馈获取成本方面存在显著差异，因此RLHF算法并不存在统一最优方案。对于法律文书生成、法规摘要等结构化任务，DPO凭借训练流程简洁、稳定性较高等优势具有较好的适用性；对于法律推理、案例分析等需要多路径论证的任务，GRPO利用组内相对排序构建学习信号，在处理复杂法律论证方面展现出较大潜力；而在司法辅助决策、量刑建议等高风险场景中，则需要结合专家反馈、规则约束以及知识检索机制共同实现可靠对齐。评估体系正从自动化指标向过程可解释性与偏见审计演进。 RLHF为法律大语言模型提供了从可用到可信的关键进化路径，但其落地需依赖内生奖励与混合反馈机制以降低标注成本，并发展奖励溯源与过程评估以满足司法透明性要求。同时，随着直接偏好优化和群体相对策略优化等新方法的发展，法律RLHF正逐步由传统奖励模型驱动模式向轻量化偏好优化与相对排序优化方向演进，为降低训练复杂度和提升法律场景适配能力提供了新的技术路径。现有研究表明，仅依赖单一奖励模型难以充分刻画法律价值体系中的复杂权衡关系，未来需要进一步融合规则知识、专家经验以及过程反馈信息，构建更加稳定和可解释的法律对齐机制。本文不仅构建了面向法律RLHF研究的六维分析框架，还进一步提出法律任务导向的RLHF适配性分析体系，系统比较PPO、DPO、GRPO等算法在法律文书生成、法律推理、合规审查等不同任务中的适用边界与失效机制。此外，本文结合法律任务风险等级与算法特征，对不同RLHF方法进行了横向比较，为法律智能系统中的算法选型与工程部署提供参考依据。研究表明，法律RLHF的核心矛盾并非单纯算法性能不足，而是法律场景中多维奖励冲突、专家偏好不一致与司法价值难以量化之间的结构性矛盾。本文首次从技术演进、奖励建模、评估体系与司法治理四个层面完整揭示法律RLHF从算法适配到工程落地的全流程逻辑，为法律大语言模型可信化构建提供系统性研究框架与实践指引。