Abstract:

Bayesian Optimization (BO) is a powerful tool for sequential optimal design in formulation development. However, conventional BO is sensitive to initial samples, easily falls into a local optimum, and cannot dynamically adapt to changes in the process easily. By combining the global search capability of Reinforcement Learning (RL) with the efficient exploration strategy of BO, this study proposes a sequential formulation optimization design method based on a Deep Deterministic Policy Gradient (DDPG) to achieve adaptive formulation optimization. By leveraging real-time feedback from the data, this approach dynamically adjusts the optimization strategy, enabling efficient global optimization starting from random initial points. It effectively addresses challenges such as limited data, inaccuracies in surrogate models, and measurement noise. During the optimization process, the method employs a Gaussian Process (GP) to construct a surrogate model of the objective function and adopts an inner-outer dual-loop framework to enhance optimization efficiency. In the inner loop, the actor network generates candidate actions, and the Upper Confidence Bound (UCB) strategy is used to select optimal experimental points. In the outer loop, the experimental strategy is dynamically adjusted based on historical experience, and the surrogate model is updated using new observations. The results from simulation cases and wine blending experiments demonstrate that the proposed method outperforms traditional optimization methods in terms of optimization efficiency, convergence speed, and quality of the final solution. Moreover, it exhibits a strong generalization ability in new environments, providing a novel solution for complex formulation optimization problems.

Key words: sequential design, formulation optimization, Bayesian Optimization (BO), Reinforcement Learning (RL), Gaussian Process (GP)

摘要：

贝叶斯优化(BO)是实现序贯配方优化设计的有力工具。然而, BO对初始点敏感, 易陷入局部最优, 且难以动态适应原料、目标函数及工艺条件的变化。结合强化学习(RL)的全局搜索能力与BO的高效探索策略, 提出一种基于深度确定性策略梯度(DDPG)的序贯配方优化设计方法, 实现自适应配方优化。该方案通过数据的实时反馈, 动态调整优化策略, 能够利用随机初始点实现高效的全局寻优, 并有效应对数据量少、代理模型不准确及测量噪声等问题。在优化过程中, 该方法利用高斯过程(GP)构建目标函数的代理模型, 并采用内外双循环框架提升优化效率。其中, 内循环通过Actor网络生成候选动作, 并结合置信上界(UCB)策略选择最优实验点, 外循环则基于历史经验动态调整实验策略并利用新观测值更新代理模型。仿真案例和白酒勾调实验的结果表明, 该方法在优化效率、收敛速度及质量方面均优于传统优化方法, 并在新环境下展现出良好的泛化能力, 为复杂配方优化问题提供了一种新的解决方案。

关键词: 序贯设计, 配方优化, 贝叶斯优化, 强化学习, 高斯过程