基于混合动作强化学习的电动汽车聚合商决策优化算法

doi:10.19678/j.issn.1000-3428.0068701

摘要/Abstract

摘要：

电动汽车可以在聚合商的集中式管理下形成规模化灵活可调资源, 从而在能源市场上套利并为电网提供辅助服务。为此, 提出一种基于混合动作强化学习的电动汽车聚合商决策优化算法。该算法利用连续动作优化市场投标决策, 根据离散动作控制不同功率分解策略的动态切换, 从而实现市场投标与功率分解决策的联合优化。此外, 还提出了一种考虑单位灵活性价值的电动汽车聚合灵活性建模方法, 在最大化日总灵活性价值的同时确保每台汽车的充电需求得到满足。仿真实验结果表明, 动态策略切换能够充分利用优先级分解策略和比例分解策略在延缓电池衰减、维持电池双向调节灵活性方面的各自优势, 与仅考虑投标决策优化的算法相比, 所提算法可以进一步提升电动汽车充电站的运行经济性。

关键词: 强化学习, 混合动作输出, 电动汽车聚合商, 功率分解, 市场投标

Abstract:

Electric vehicles (EV), when managed centrally by aggregators, can be utilized as flexible and adjustable resources to participate in energy market arbitrage and provide ancillary services to the grid. To optimize this potential, this study introduces an advanced decision-making algorithm for EV aggregators based on hybrid action reinforcement learning. The algorithm uses continuous actions to optimize market bidding decisions and discrete actions to manage the dynamic switching between different power disaggregation strategies, realizing a joint optimization of market bidding and power disaggregation. In addition, the study presents an EV aggregator flexibility modelling method that considers the value of unit flexibility, aiming to maximize the total daily flexibility value while ensuring that the charging demand of each vehicle is met. Simulation results show that dynamic policy switching effectively leverages the strengths of both priority decomposition and proportional decomposition strategies, helping to reduce battery degradation and maintain the flexibility of two-way battery regulation. The proposed algorithm enhances the operational economy of EV charging stations, outperforming algorithms that focus solely on optimizing the bidding decision.

Key words: reinforcement learning, hybrid action output, Electric Vehicle(EV) aggregator, power allocation, market bidding

孔月萍, 杨世海, 段梅梅, 丁泽诚, 方凯杰. 基于混合动作强化学习的电动汽车聚合商决策优化算法[J]. 计算机工程, 2024, 50(10): 418-428.

KONG Yueping, YANG Shihai, DUAN Meimei, DING Zecheng, FANG Kaijie. Optimal Decision-making Algorithm for Electric Vehicle Aggregator Based on Hybrid Action Reinforcement Learning[J]. Computer Engineering, 2024, 50(10): 418-428.

https://www.ecice06.com/CN/Y2024/V50/I10/418

图/表 13

图1 支持混合动作生成的网络结构

Fig.1 Network architecture supporting hybrid action generation

图2 电动汽车在站情况统计结果

Fig.2 Statistical results on the presence of electric vehicles at stations

图3 电动汽车聚合灵活域

Fig.3 Aggregate flexibility of electric vehicles

图4 聚合灵活性大小

Fig.4 Magnitude of aggregate flexibility

图5 日内能量及平衡市场价格

Fig.5 Intraday energy and balanced market prices

图6 投标结果

Fig.6 Bidding results

图7 功率分解结果

Fig.7 Power allocation results

图8 电池电量变化曲线

Fig.8 Battery energy change curve

图9 训练曲线

Fig.9 Training curves

图10 电池衰减成本

Fig.10 Battery degradation cost

参考文献 31

1	姚颖蓓, 陆建忠, 傅业盛, 等. 华东地区电动汽车发展趋势及用电需求预测. 电力系统保护与控制, 2021, 49(4): 141- 145.
	YAO T B, LU J Z, FU Y S, et al. Development trends and electricity demand forecast of electric vehicles in east China. Power System Protection and Control, 2021, 49(4): 141- 145.
2	刘坚. 电动汽车退役电池储能应用潜力及成本分析. 储能科学与技术, 2017, 6(2): 243- 249.
	LIU J. Second use potential of retired EV batteries in power system and associated cost analysis. Energy Storage Science and Technology, 2017, 6(2): 243- 249.
3	刘哲. 车网互动V2G中电动汽车充放电效益优化研究[D]. 杭州: 杭州电子科技大学, 2023.
	LIU Z. Study on optimization of charging and discharging benefits of electric vehicles in vehicle-network interaction V2G[D]. Hangzhou: Hangzhou Dianzi University, 2023. (in Chinese)
4	沙广林, 刘璐, 马春艳, 等. 考虑车网互动的电动汽车有序充电策略. 供用电, 2023, 40(10): 46- 54.
	SHA G L, LIU L, MA C Y, et al. Orderly charging strategy for electric vehicles considering the vehicle-network interaction. Distribution & Utilization, 2023, 40(10): 46- 54.
5	刘敦楠, 王玲湘, 汪伟业, 等. 基于深度强化学习的大规模电动汽车充换电负荷优化调度. 电力系统自动化, 2022, 46(4): 36- 46.
	LIU D N, WANG L X, WANG W Y, et al. Optimal scheduling of electric vehicle load for large-scale battery charging and swapping based on deep reinforcement learning. Automation of Electric Power Systems, 2022, 46(4): 36- 46.
6	刘卫亮, 闫倩文, 张启亮, 等. 基于虚拟电厂区间主从博弈的车网互动优化调度. 系统仿真学报, 2024, 36(7): 1559- 1572.
	LIU W L, YAN Q W, ZHANG Q L, et al. Vehicle-to-grid interaction optimization scheduling based on virtual power plant interval master-slave game. Journal of System Simulation, 2024, 36(7): 1559- 1572.
7	PENG C, ZOU J X, LIAN L, et al. An optimal dispatching strategy for V2G aggregator participating in supplementary frequency regulation considering EV driving demand and aggregator's benefits. Applied Energy, 2017, 190, 591- 599. doi: 10.1016/j.apenergy.2016.12.065
8	LI B S, WANG X, SHAHIDEHPOUR M, et al. Robust bidding strategy and profit allocation for cooperative DSR aggregators with correlated wind power generation. IEEE Transactions on Sustainable Energy, 2019, 10(4): 1904- 1915. doi: 10.1109/TSTE.2018.2875483
9	CHEN X Y, LEUNG K C. Non-cooperative and cooperative optimization of scheduling with vehicle-to-grid regulation services. IEEE Transactions on Vehicular Technology, 2020, 69(1): 114- 130. doi: 10.1109/TVT.2019.2952712
10	SORTOMME E, EL-SHARKAWI M A. Optimal charging strategies for unidirectional vehicle-to-grid. IEEE Transactions on Smart Grid, 2011, 2(1): 131- 138. doi: 10.1109/TSG.2010.2090910
11	JIN C R, TANG J, GHOSH P. Optimizing electric vehicle charging with energy storage in the electricity market. IEEE Transactions on Smart Grid, 2013, 4(1): 311- 320. doi: 10.1109/TSG.2012.2218834
12	LI B, XIE K, ZHONG W F, et al. Operation management of electric vehicle battery swapping and charging systems: a bilevel optimization approach. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(1): 528- 540. doi: 10.1109/TITS.2022.3211883
13	徐湘楚, 米增强, 詹泽伟, 等. 考虑多重不确定性的电动汽车聚合商参与能量-调频市场的鲁棒优化模型. 电工技术学报, 2023, 38(3): 793- 805.
	XU X C, MI Z Q, ZHAN Z W, et al. A robust optimization model for electric vehicle aggregator participation in energy and frequency regulation markets considering multiple uncertainties. Transactions of China Electrotechnical Society, 2023, 38(3): 793- 805.
14	ZHANG H C, HU Z C, XU Z W, et al. Evaluation of achievable vehicle-to-grid capacity using aggregate PEV model. IEEE Transactions on Power Systems, 2017, 32(1): 784- 794. doi: 10.1109/TPWRS.2016.2561296
15	WANG L S, KWON J, SCHULZ N, et al. Evaluation of aggregated EV flexibility with TSO-DSO coordination. IEEE Transactions on Sustainable Energy, 2022, 13(4): 2304- 2315. doi: 10.1109/TSTE.2022.3190199
16	孙波, 李思敏, 谢敬东, 等. 基于IGDT理论的电动汽车负荷聚合商需求侧放电投标决策模型. 现代电力, 2020, 37(5): 484- 490.
	SUN B, LI S M, XIE J D, et al. IGDT-based demand side discharge bidding decision strategy for electric vehicle load aggregator. Modern Electric Power, 2020, 37(5): 484- 490.
17	宋艺航, 王秀丽, 匡熠, 等. 含风电和电动汽车的VPP现货市场投标鲁棒优化模型. 电力工程技术, 2020, 39(3): 120- 127.
	SONG Y H, WANG X L, KUANG Y, et al. Robust optimization model for spot market bidding of virtual power plant with wind power and electric vehicles. Electric Power Engineering Technology, 2020, 39(3): 120- 127.
18	李永刚, 孙浩潮, 周一辰, 等. 考虑新增电动汽车充放电中断风险的聚合商调频辅助服务投标策略. 电力自动化设备, 2022, 42(10): 3- 12.
	LI Y G, SUN H C, ZHOU Y C, et al. Frequency regulation auxiliary service bidding strategy for aggregators considering charging and discharging interruption risk of newly added electric vehicles. Electric Power Automation Equipment, 2022, 42(10): 3- 12.
19	ZHAO L, ZHANG W, HAO H, et al. A geometric approach to aggregate flexibility modeling of thermostatically controlled loads. IEEE Transactions on Power Systems, 2017, 32(6): 4721- 4731. doi: 10.1109/TPWRS.2017.2674699
20	CHEN X, DALL'ANESE E, ZHAO C H, et al. Aggregate power flexibility in unbalanced distribution systems. IEEE Transactions on Smart Grid, 2020, 11(1): 258- 269. doi: 10.1109/TSG.2019.2920991
21	WU H Y, SHAHIDEHPOUR M, ALABDULWAHAB A, et al. A game theoretic approach to risk-based optimal bidding strategies for electric vehicle aggregators in electricity markets with variable wind energy resources[C]//Proceedings of IEEE Power and Energy Society General Meeting. Washington D. C., USA: IEEE Press, 2016: 374-385.
22	RIVERA J, GOEBEL C, JACOBSEN H A. Distributed convex optimization for electric vehicle aggregators. IEEE Transactions on Smart Grid, 2017, 8(4): 1852- 1863. doi: 10.1109/TSG.2015.2509030
23	TAN X Q, QU G N, SUN B, et al. Optimal scheduling of battery charging station serving electric vehicles based on battery swapping. IEEE Transactions on Smart Grid, 2019, 10(2): 1372- 1384. doi: 10.1109/TSG.2017.2764484
24	ZHENG J H, WANG X Y, MEN K, et al. Aggregation model-based optimization for electric vehicle charging strategy. IEEE Transactions on Smart Grid, 2013, 4(2): 1058- 1066. doi: 10.1109/TSG.2013.2242207
25	KANG Q, WANG J B, ZHOU M C, et al. Centralized charging strategy and scheduling algorithm for electric vehicles under a battery swapping scenario. IEEE Transactions on Intelligent Transportation Systems, 2016, 17(3): 659- 669. doi: 10.1109/TITS.2015.2487323
26	HABIBIFAR R, ARIS LEKVAN A, EHSAN M. A risk-constrained decision support tool for EV aggregators participating in energy and frequency regulation markets. Electric Power Systems Research, 2020, 185, 106367. doi: 10.1016/j.epsr.2020.106367
27	LI H P, WAN Z Q, HE H B. Constrained EV charging scheduling based on safe deep reinforcement learning. IEEE Transactions on Smart Grid, 2020, 11(3): 2427- 2439. doi: 10.1109/TSG.2019.2955437
28	YU P P, ZHANG H C, SONG Y H. District cooling system control for providing regulation services based on safe reinforcement learning with barrier functions. Applied Energy, 2023, 347, 121396. doi: 10.1016/j.apenergy.2023.121396
29	MA R, YI Z H, XIANG Y M, et al. A blockchain-enabled demand management and control framework driven by deep reinforcement learning. IEEE Transactions on Industrial Electronics, 2023, 70(1): 430- 440. doi: 10.1109/TIE.2022.3146631
30	LI H P, HE H B. Learning to operate distribution networks with safe deep reinforcement learning. IEEE Transactions on Smart Grid, 2022, 13(3): 1860- 1872. doi: 10.1109/TSG.2022.3142961
31	张文昕, 栗然, 臧向迪, 等. 基于强化学习的电动汽车换电站实时调度策略优化. 电力自动化设备, 2022, 42(10): 134- 141.
	ZHANG W X, LI R, ZANG X D, et al. Real-time scheduling strategy optimization for electric vehicle battery swapping station based on reinforcement learning. Electric Power Automation Equipment, 2022, 42(10): 134- 141.

[1]	石琼, 段辉, 师智斌. 基于深度强化学习的可信任务卸载方案[J]. 计算机工程, 2024, 50(8): 142-152.
[2]	钱清, 龙永, 蒋忠远, 段春红, 王宏. 基于深度强化学习的自适应图像隐写算法[J]. 计算机工程, 2024, 50(8): 319-327.
[3]	高家豪, 胡创业, 丁男, 刘战东. 智能网联汽车中联合驾驶风格的交通流数据有效性分析[J]. 计算机工程, 2024, 50(6): 367-376.
[4]	孙文洁, 李宗民, 孙浩淼. 基于图神经网络的多智能体强化学习值函数分解方法[J]. 计算机工程, 2024, 50(5): 62-70.
[5]	傅明建, 郭福强. 基于深度强化学习的无信号灯路口决策研究[J]. 计算机工程, 2024, 50(5): 91-99.
[6]	张斯力, 李梓健, 蔡瑞初, 郝志峰, 闫玉光. 基于因果机制约束的强化推荐系统[J]. 计算机工程, 2024, 50(5): 279-290.
[7]	冯雄波, 黄于欣, 赖华, 高玉梦. 基于多策略强化学习的低资源跨语言摘要方法研究[J]. 计算机工程, 2024, 50(2): 68-77.
[8]	杜海军, 余粟. 基于时空图注意力网络的服务机器人动态避障[J]. 计算机工程, 2024, 50(2): 105-112.
[9]	张俊娜, 韩超臣, 陈家伟, 赵晓焱, 袁培燕. 一种联合边缘服务器部署与服务放置的方法[J]. 计算机工程, 2024, 50(10): 266-280.
[10]	蔡梓越, 谭北海, 余荣, 黄旭民, 王思明. 面向6G物联网设备协同的区块链动态分片[J]. 计算机工程, 2024, 50(1): 50-59.
[11]	张晓天, 王雅文, 谢志庆, 金大海, 宫云战. 面向类集成测试序列确定的强化学习方法[J]. 计算机工程, 2024, 50(1): 68-78.
[12]	王少桐, 况立群, 韩慧妍, 熊风光, 薛红新. 基于优势后见经验回放的强化学习导航方法[J]. 计算机工程, 2024, 50(1): 313-319.
[13]	孔凌辉, 饶哲恒, 徐彦彦, 潘少明. 基于深度强化学习的无线网络智能路由算法[J]. 计算机工程, 2023, 49(9): 199-207, 216.
[14]	胡水. 基于深度强化学习的智能兵棋推演决策方法[J]. 计算机工程, 2023, 49(9): 303-312.
[15]	张冠莹, 伊鹏, 李丹, 朱棣, 毛明. 面向大规模网络的服务功能链部署方法[J]. 计算机工程, 2023, 49(8): 122-129.

选择文件类型/文献管理软件名称

选择包含的内容