基于融合状态预测的深度强化学习A2C的交通信号控制

doi:10.19678/j.issn.1000-3428.0069478

计算机工程 ›› 2025, Vol. 51 ›› Issue (5): 33-42. doi: 10.19678/j.issn.1000-3428.0069478

• 空天地一体化算力网络 • 上一篇下一篇

基于融合状态预测的深度强化学习A2C的交通信号控制

叶宝林^1,2, 孙瑞涛^1,2, 李灵犀³, 吴维敏⁴

1. 嘉兴大学信息科学与工程学院, 浙江嘉兴 314001;
2. 浙江理工大学信息科学与工程学院, 浙江杭州 310018;
3. 普渡大学埃尔莫尔家族电气与计算机工程学院, 美国西拉法叶 47907;
4. 浙江大学智能系统与控制研究所工业控制技术国家重点实验室, 浙江杭州 310027

收稿日期:2024-03-04 修回日期:2024-07-25 出版日期:2025-05-15 发布日期:2025-05-10
通讯作者: 叶宝林,E-mail:yebaolin@zjxu.edu.cn E-mail:yebaolin@zjxu.edu.cn
基金资助:
国家自然科学基金(61603154);浙江省自然科学基金(LTGS23F030002);浙江省尖兵领雁研发攻关计划项目(2023C01174);嘉兴市应用性基础研究项目(2023AY11034);工业控制技术国家重点实验室开放课题(ICT2022B52)。

Traffic Signal Control Based on Deep Reinforcement Learning A2C Integrated with State Prediction

YE Baolin^1,2, SUN Ruitao^1,2, LI Lingxi³, WU Weimin⁴

1. School of Information Science and Engineering, Jiaxing University, Jiaxing 314001, Zhejiang, China;
2. School of Information Science and Engineering, Zhejiang Sci-Tech University, Hangzhou 310018, Zhejiang, China;
3. Elmore Family School of Electrical and Computer Engineering, Purdue University, West Lafayette 47907, USA;
4. State Key Laboratory of Industrial Control Technology, Institute of Cyber-Systems and Control, Zhejiang University, Hangzhou 310027, Zhejiang, China

Received:2024-03-04 Revised:2024-07-25 Online:2025-05-15 Published:2025-05-10

摘要/Abstract

摘要： 现有基于强化学习的交通信号控制方法主要使用历史交通状态和当前时间步的实时交通状态来确定下一个时间步的控制策略,造成控制策略始终滞后于交通状态一个时间步。为了解决该问题,提出一种基于融合交通状态预测的深度强化学习优势演员评论家(A2C)的交通信号控制方法。首先,为了获取未来时间步的交通状态,以确保制定的控制策略能够更精准地响应实时交通状态下的决策需求,设计一个长短时记忆(LSTM)网络预测路网未来时间步的交通状态。然后,为了提高输入深度强化学习模型中数据的准确性和鲁棒性,设计一个卡尔曼滤波器对采集的历史交通状态数据和LSTM网络预测的未来交通状态数据进行融合。其次,为了使深度强化学习模型能够更全面地理解交通流量中包含的时间依赖关系,并实现更高效和稳定的交通信号控制决策,提出一种融合双向LSTM网络的A2C算法。最后,基于微观交通仿真(SUMO)平台的仿真测试结果表明,与传统交通信号控制方法和基于深度强化学习A2C的交通信号控制方法相比,该方法在低峰、平峰和高峰两种不同交通流量状态下均能够取得更好的交通信号控制效益。

关键词: 交通信号控制, 优势演员评论家, 交通状态预测, 双向长短时记忆网络

Abstract: Existing reinforcement learning-based traffic signal control methods primarily utilize historical and real-time traffic states at the current time step to determine the control strategy for the next time step. However, this approach results in the control strategy to lag behind the actual traffic state by one time step. To address this issue, this study proposes a traffic signal control method based on Advantage Actor Critic (A2C) using deep reinforcement learning. First, a Long Short-Term Memory (LSTM) network is designed to predict the future traffic states of a road network, to obtain the traffic state of future time steps and ensure that the formulated control strategy can respond more accurately to decision-making requirements under real-time traffic conditions. Second, a Kalman filter is designed to fuse collected historical traffic state data with the future traffic state data predicted by the LSTM, to improve the accuracy and robustness of the data being input into the deep reinforcement learning model. Additionally, a bidirectional LSTM-integrated A2C algorithm is proposed that allows the deep reinforcement learning model to fully capture the time-dependent relationships within traffic flow and achieve more efficient and stable traffic signal control decisions. Finally, simulations conducted on the Simulation of Urban Mobility (SUMO) platform demonstrate that the proposed method achieves superior traffic signal control efficiency under both low-peak, off-peak and peak traffic conditions compared to traditional traffic signal control methods and deep reinforcement learning A2C-based traffic signal control method.

Key words: traffic signal control, Advantage Actor Critic (A2C), traffic state prediction, bidirectional Long Short-Term Memory (LSTM) network

中图分类号:

TN929

叶宝林, 孙瑞涛, 李灵犀, 吴维敏. 基于融合状态预测的深度强化学习A2C的交通信号控制[J]. 计算机工程, 2025, 51(5): 33-42.

YE Baolin, SUN Ruitao, LI Lingxi, WU Weimin. Traffic Signal Control Based on Deep Reinforcement Learning A2C Integrated with State Prediction[J]. Computer Engineering, 2025, 51(5): 33-42.

https://www.ecice06.com/CN/Y2025/V51/I5/33

参考文献

[1] 杨笑笑,柯琳,陈智斌.深度强化学习求解车辆路径问题的研究综述[J].计算机工程与应用, 2023, 59(5):1-13. YANG X X, KE L, CHEN Z B. Review of deep reinforcement learning model research on vehicle routing problems[J]. Computer Engineering and Applications, 2023, 59(5):1-13.(in Chinese)
[2] ZUO Z Y, HAN Q L, NING B D, et al. An overview of recent advances in fixed-time cooperative control of multiagent systems[J]. IEEE Transactions on Industrial Informatics, 2018, 14(6):2322-2334.
[3] LOPEZ A, JIN W L, AL FARUQUE M A. Security analysis for fixed-time traffic control systems[J]. Transportation Research, Part B:Methodological, 2020, 139:473-495.
[4] ZHAO D B, DAI Y J, ZHANG Z. Computational intelligence in urban traffic signal control:a survey[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C:Applications and Reviews, 2012, 42(4):485-494.
[5] 屈新明,姚红云,王玉刚,等.基于有效绿灯时间利用率的自适应控制策略研究[J].交通运输研究, 2015, 1(1):54-58. QU X M, YAO H Y, WANG Y G, et al. Adaptive control strategy based on effective utilization ratio of green light time[J]. Transport Research, 2015, 1(1):54-58.(in Chinese)
[6] LI S R, WEI C, YAN X D, et al. A deep adaptive traffic signal controller with long-term planning horizon and spatial-temporal state definition under dynamic traffic fluctuations[J]. IEEE Access, 2020, 8:37087-37104.
[7] 叶宝林.城市路网交通信号协调控制理论与方法研究[D].杭州:浙江大学, 2015. YE B L. Study on the theory and method of traffic signal coordinated control in urban road network[D]. Hangzhou:Zhejiang University, 2015.(in Chinese)
[8] BELLOMO N, DOGBE C. On the modeling of traffic and crowds:a survey of models, speculations, and perspectives[J]. SIAM Review, 2011, 53(3):409-463.
[9] ZHENG Y, JIN L S, JIANG Y Y, et al. Research on cooperative vehicle intersection control scheme without using traffic lights under the connected vehicles environment[J]. Advances in Mechanical Engineering, 2017, 9(8):115789974.
[10] HAYES C F, R ǍG DULESCU R, BARGIACCHI E, et al. A practical guide to multi-objective reinforcement learning and planning[J]. Autonomous Agents and Multi-Agent Systems, 2022, 36(1):26.
[11] YE B L, ZHU S W, LI L X, et al. Short-term traffic flow prediction at isolated intersections based on parallel multi-task learning[J]. Systems Science& Control Engineering, 2024, 12(1):2316160.
[12] WAN C H, HWANG M C. Value-based deep reinforcement learning for adaptive isolated intersection signal control[J]. IET Intelligent Transport Systems, 2018, 12(9):1005-1010.
[13] 马东方,陈曦,吴晓东,等.基于强化学习的干线信号混合协同优化方法[J].交通运输系统工程与信息, 2022, 22(2):145-153. MA D F, CHEN X, WU X D, et al. Mixed-coordinated decision-making method for arterial signals based on reinforcement learning[J]. Journal of Transportation Systems Engineering and Information Technology, 2022, 22(2):145-153.(in Chinese)
[14] YE B L, WU W M, RUAN K Y, et al. A survey of model predictive control methods for traffic signal control[J]. IEEE/CAA Journal of Automatica Sinica, 2019, 6(3):623-640.
[15] YE B L, WU W M, LI L X, et al. A hierarchical model predictive control approach for signal splits optimization in large-scale urban road networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(8):2182-2192.
[16] MA W J, WAN L J, YU C H, et al. Multi-objective optimization of traffic signals based on vehicle trajectory data at isolated intersections[J]. Transportation Research, Part C:Emerging Technologies, 2020, 120:102821.
[17] 孔凌辉,饶哲恒,徐彦彦,等.基于深度强化学习的无线网络智能路由算法[J].计算机工程, 2023, 49(9):199-207, 216. KONG L H, RAO Z H, XU Y Y, et al. Intelligent routing algorithm for wireless networks based on deep reinforcement learning[J]. Computer Engineering, 2023, 49(9):199-207, 216.(in Chinese)
[18] 刘朝阳,穆朝絮,孙长银.深度强化学习算法与应用研究现状综述[J].智能科学与技术学报, 2020, 2(4):314-326. LIU Z Y, MU C X, SUN C Y. An overview on algorithms and applications of deep reinforcement learning[J]. Chinese Journal of Intelligent Science and Technology, 2020, 2(4):314-326.(in Chinese)
[19] 刘智敏,叶宝林,朱耀东,等.基于深度强化学习的交通信号控制方法[J].浙江大学学报(工学版), 2022, 56(6):1249-1256. LIU Z M, YE B L, ZHU Y D, et al. Traffic signal control method based on deep reinforcement learning[J]. Journal of Zhejiang University (Engineering Science), 2022, 56(6):1249-1256.(in Chinese)
[20] MA D F, ZHOU B, SONG X, et al. A deep reinforcement learning approach to traffic signal control with temporal traffic pattern mining[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(8):11789-11800.
[21] 叶宝林,孙瑞涛,吴维敏,等.基于异步优势演员-评论家的交通信号控制方法[J].浙江大学学报(工学版), 2024, 58(8):1671-1680, 1703. YE B L, SUN R T, WU W M, et al. Traffic signal control method based on asynchronous advantage actor-critic[J]. Journal of Zhejiang University (Engineering Science), 2024, 58(8):1671-1680, 1703.(in Chinese)
[22] YE B L, WU P, LI L X, et al. Uniformity of Markov elements in deep reinforcement learning for traffic signal control[J]. Electronic Research Archive, 2024, 32(6):3843-3866.
[23] 张尊栋,王岩楠,刘雨珂,等.基于Nash-Stackelberg分层博弈模型的路网交通控制强化学习算法[J].东南大学学报(自然科学版), 2023, 53(2):334-341. ZHANG Z D, WANG Y N, LIU Y K, et al. Road network traffic control reinforcement learning algorithms based on Nash-Stackelberg hierarchical game model[J]. Journal of Southeast University (Natural Science Edition), 2023, 53(2):334-341.(in Chinese)
[24] 陈喜群,朱奕璋,吕朝锋.基于混合近端策略优化的交叉口信号相位与配时优化方法[J].交通运输系统工程与信息, 2023, 23(1):106-113. CHEN X Q, ZHU Y Z, LV C F. Signal phase and timing optimization method for intersection based on hybrid proximal policy optimization[J]. Journal of Transportation Systems Engineering and Information Technology, 2023, 23(1):106-113.(in Chinese)
[25] YE B L, WU W M, MAO W J. A two-way arterial signal coordination method with queueing process considered[J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(6):3440-3452.

选择文件类型/文献管理软件名称

选择包含的内容

基于融合状态预测的深度强化学习A2C的交通信号控制

Traffic Signal Control Based on Deep Reinforcement Learning A2C Integrated with State Prediction

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 7

编辑推荐

Metrics

本文评价

[1]	褚跃跃, 闫飞, 李浦. 含扰动迭代学习补偿的城市交通信号预测控制方法[J]. 计算机工程, 2023, 49(7): 305-312.
[2]	石磊, 张吉涛, 高宇飞, 卫琳, 陶永才. 基于Transformer与BiLSTM的网络流量入侵检测[J]. 计算机工程, 2023, 49(3): 29-36,57.
[3]	廖涛, 黄荣梅, 张顺香, 段松松. 基于交互式特征融合的嵌套命名实体识别[J]. 计算机工程, 2022, 48(12): 119-126,133.
[4]	江旭, 钱雪忠, 宋威. 结合残差BiLSTM与句袋注意力的远程监督关系抽取[J]. 计算机工程, 2022, 48(10): 110-115,122.
[5]	张冬梅, 陈永乐, 杨玉丽. 基于分层特征的代码克隆检测方法[J]. 计算机工程, 2021, 47(10): 125-131.
[6]	许莹莹, 黄浩. 基于标签分解的口语理解模型[J]. 计算机工程, 2019, 45(7): 237-241.
[7]	张应成,杨洋,蒋瑞,全兵,张利君,任晓雷. 基于BiLSTM-CRF的商情实体识别模型[J]. 计算机工程, 2019, 45(5): 308-314.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于融合状态预测的深度强化学习A2C的交通信号控制

Traffic Signal Control Based on Deep Reinforcement Learning A2C Integrated with State Prediction

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 7

编辑推荐

Metrics

本文评价