SSD-SMR Write Cache Strategy for Optimizing Long-Tail Latency Based on Q-Learning

doi:10.19678/j.issn.1000-3428.0070280

Computer Engineering ›› 2026, Vol. 52 ›› Issue (3): 287-298. doi: 10.19678/j.issn.1000-3428.0070280

• Computer Architecture and Advanced Computing • Previous Articles Next Articles

SSD-SMR Write Cache Strategy for Optimizing Long-Tail Latency Based on Q-Learning

LIU Jian¹, ZHANG Buhao², FANG Kuangchi¹, LIU Xuanfeng¹, SUN Guodao¹, LIANG Ronghua¹, LIANG Haoran¹^,*()

1. College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou 310023, Zhejiang, China
2. Enjoyor Technology Co., Ltd., Hangzhou 311400, Zhejiang, China

Received:2024-08-22 Revised:2024-10-29 Online:2026-03-15 Published:2024-12-23
Contact: LIANG Haoran

基于Q-Learning长尾延迟优化的SSD-SMR写缓存策略研究

刘健¹, 章步镐², 方匡弛¹, 刘宣锋¹, 孙国道¹, 梁荣华¹, 梁浩然¹^,*()

1. 浙江工业大学计算机科学与技术学院, 浙江杭州 310023
2. 银江技术股份有限公司, 浙江杭州 311400

通讯作者: 梁浩然
作者简介:
刘健(CCF会员)，男，讲师、博士，主研方向为时序数据库、存储系统
章步镐，副高级工程师
方匡弛，硕士
刘宣锋，硕士
孙国道，副教授、博士
梁荣华，教授、博士
梁浩然(通信作者)，副研究员、博士
基金资助:
国家自然科学基金区域创新发展联合基金重点项目(U24A20247); 国家自然科学基金(62202430); 国家自然科学基金(62176235); 国家自然科学基金(62432014); 浙江省自然科学基金(LY24F020018); 浙江省自然科学基金(LR23F020003); 浙江省智能交通工程技术研究中心开放课题(2023ERCITZJ-KF02)

Abstract

Abstract:

With the continuous increase in the scale of global data, the effective and inexpensive improvement of data access performance is an important challenge faced by storage systems. An effective solution is to build cache systems using low-latency, high-bandwidth Solid-State Drives (SSD) and low-cost, high-storage-density Shingled Magnetic Recording (SMR). However, the inherent mechanical motion and multitrack stacking characteristics of SMR result in poor write performance, and the frequent write-back of dirty data in SSD to SMR may cause severe long-tail latency owing to the large number of Read-Merge-Write (RMW) operations. To this end, a cache replacement optimization strategy combining a reinforcement learning Q-Learning algorithm is proposed based on the SSD-SMR hybrid storage architecture. By learning the empirical relationship between the I/O load status and the latency of the SMR devices, write operations to the SMR can be controlled. When the SMR load is high, controlling the eviction of dirty data in the cache can reduce the number of RMW operations caused by SMR write-backs, thereby optimizing the tail latency overhead of the system under different loads. The Q-Learning algorithm is combined with the data-popularity-based caching algorithm LRU and the SMR aware caching algorithm SAC and tested using real enterprise Trace and simulated Trace generated by YCSB. The experimental results show that the proposed method can effectively improve the performance of existing caching algorithms, reducing the average latency by 57.06% and tail latency by 87.49%.

Key words: Q-Learning algorithm, I/O load, long-tail latency, cache replacement algorithm, hybrid storage

摘要：

随着全球数据规模的不断增大, 如何以低成本的方式有效提升数据的访问性能是存储系统面临的一项重要挑战, 使用低延迟、高带宽的固态硬盘(SSD)和低成本、高存储密度的叠瓦式磁盘(SMR)来构建缓存系统, 成为一种有效的解决方案。但是, SMR固有的机械运动和多磁道堆叠的特性导致其写性能较差, SSD中的脏数据频繁写回SMR所导致的大量读-合并-写(RMW)操作可能会引起严重的长尾延迟现象。为此, 基于SSD-SMR混合存储架构提出一种结合强化学习Q-Learning算法的缓存替换优化策略。通过学习SMR设备的I/O负载状况与延迟之间的经验知识来控制对SMR的写入, 当SMR负载较大时, 通过控制缓存中脏数据的逐出来减少SMR因写回而产生的大量RMW操作, 从而优化系统在不同负载下的尾部延迟开销。将Q-Learning算法与基于数据流行度的缓存算法LRU以及SMR感知的缓存算法SAC进行结合, 使用真实企业Trace和YCSB生成的模拟Trace进行测试, 实验结果表明, 所提方法能够有效提升现有缓存算法的性能, 可以降低57.06%的平均延迟和87.49%的尾部延迟。

关键词: Q-Learning算法, I/O负载, 长尾延迟, 缓存替换算法, 混合存储

LIU Jian, ZHANG Buhao, FANG Kuangchi, LIU Xuanfeng, SUN Guodao, LIANG Ronghua, LIANG Haoran. SSD-SMR Write Cache Strategy for Optimizing Long-Tail Latency Based on Q-Learning[J]. Computer Engineering, 2026, 52(3): 287-298.

刘健, 章步镐, 方匡弛, 刘宣锋, 孙国道, 梁荣华, 梁浩然. 基于Q-Learning长尾延迟优化的SSD-SMR写缓存策略研究[J]. 计算机工程, 2026, 52(3): 287-298.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070280

https://www.ecice06.com/EN/Y2026/V52/I3/287

Figures/Tables 12

Fig.1 Q-Learning framework

Fig.2 SSD — SMR hybrid storage system

Fig.3 SAC architecture optimized with Q-Learning

Fig.4 Performance comparison under different read-write ratios

Fig.5 The impact of cache size on performance

Fig.6 Performance comparison under different distributions

Fig.7 Effectiveness verification results of I/O sensing strategy

References 26

1	RYDNING D R J G J , REINSEL J , GANTZ J . The digitization of the world from edge to core. Framingham: International Data Corporation, 2018, 16, 1- 28. URL
2	祝青, 李小勇. 混合存储综述. 微型电脑应用, 2013, 29 (2): 33- 38.
	ZHU Q , LI X Y . A review on hybrid storage. Microcomputer Applications, 2013, 29 (2): 33- 38.
3	余帅文. 基于近存计算的固态硬盘缓存分割研究[D]. 重庆: 西南大学, 2025.
	YU S W. Research on cache partitioning of solid-state drives based on near-memory computing[D]. Chongqing: Southwest University, 2025. (in Chinese)
4	梅文娟, 蔡鹏. 面向存算分离架构的混合粒度缓存策略. 华东师范大学学报(自然科学版), 2023 (5): 26- 39. doi: 10.3969/j.issn.1000-5641.2023.05.003
	MEI W J , CAI P . Mixed granularity caching strategy for storage-computing separation architecture. Journal of East China Normal University (Natural Science), 2023 (5): 26- 39. doi: 10.3969/j.issn.1000-5641.2023.05.003
5	ZHANG C , NIE S Q , WANG J Y , et al. MCB: a multidevice cooperative buffer management strategy for boosting the write performance of the SSD-SMR hybrid storage. The Journal of Supercomputing, 2023, 79 (12): 13462- 13489. doi: 10.1007/s11227-023-05201-7
6	LIU W G, ZENG L F, FENG D. CASS: a cooperative hybrid storage system consisting of an SSD and a SMR drive[C]//Proceedings of the 6th International Conference on Advanced Cloud and Big Data (CBD). Washington D.C., USA: IEEE Press, 2018: 24-29.
7	Seagate. ST2000VX015 SMR 2 TB[EB/OL]. [2024-06-05]. https://item.jd.com/10079214604944.
8	Western Digital. WD_BLACK AN1500 SSD 1 TB[EB/OL]. [2024-06-05]. https://item.jd.com/10096241359618.
9	XIE X C , XIAO L Q , GE X Z , et al. SMRC: an endurable SSD cache for host-aware shingled magnetic recording drives. IEEE Access, 2018, 6, 20916- 20928. doi: 10.1109/ACCESS.2018.2825109
10	XIE X C, YANG T Y, LI Q, et al. Duchy: achieving both SSD durability and controllable SMR cleaning overhead in hybrid storage systems[C]//Proceedings of the 47th International Conference on Parallel Processing. New York, USA: ACM Press, 2018: 1-9.
11	COOPER B F, SILBERSTEIN A, TAM E, et al. Benchmarking cloud serving systems with YCSB[C]//Proceedings of the 1st ACM Symposium on Cloud Computing. New York, USA: ACM Press, 2010: 143-154.
12	WANG C, WANG D, CHAI Y, et al. Larger cheaper but faster: SSD-SMR hybrid storage boosted by a new SMR-oriented cache framework[EB/OL]. [2024-06-05]. https://msstconference.org/MSST-history/2017/Papers/SSD-SMRHybridStorage.pdf.
13	XIAO W J, DONG H Q, MA L Y, et al. HS-BAS: a hybrid storage system based on band awareness of shingled write disk[C]//Proceedings of the 34th IEEE International Conference on Computer Design (ICCD). Washington D.C., USA: IEEE Press, 2016: 64-71.
14	MA C L, SHEN Z Y, HAN L, et al. FC: built-in flash-cache with fast cleaning for SMR storage[C]//Proceedings of the IEEE International Conference on Embedded Software and Systems (ICESS). Washington D.C., USA: IEEE Press, 2019: 1-7.
15	MA C L , SHEN Z Y , HAN L , et al. RMW-F: a design of RMW-free cache using built-in NAND-flash for SMR storage. ACM Transactions on Embedded Computing Systems, 2019, 18 (5): 1- 18. doi: 10.1145/3358210
16	LUO D , WAN J G , ZHU Y F , et al. Design and implementation of a hybrid shingled write disk system. IEEE Transactions on Parallel and Distributed Systems, 2016, 27 (4): 1017- 1029. doi: 10.1109/TPDS.2015.2425402
17	LIU W G , ZENG L F , FENG D , et al. ROCO: using a solid state drive cache to improve the performance of a host-aware shingled magnetic recording drive. Journal of Computer Science and Technology, 2019, 34 (1): 61- 76. doi: 10.1007/s11390-019-1899-7
18	ZHENG X D, ZHANG C, DUAN K Q, et al. SLA: a cache algorithm for SSD-SMR storage system with minimum RMWs[EB/OL]. [2024-06-05]. https://link.springer.com/chapter/10.1007/978-3-030-95391-1_37.
19	MA C L, ZHOU Z K, WANG Y P, et al. MU-RMW: minimizing unnecessary RMW operations in the embedded flash with SMR disk[C]//Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE). Washington D.C., USA: IEEE Press, 2022: 490-495.
20	SUN D S, CHAI Y P. SAC: a co-design cache algorithm for emerging SMR-based high-density disks[C]//Proceedings of the 25th International Conference on Architectural Support for Programming Languages and Operating Systems. Washington D.C., USA: IEEE Press, 2020: 1047-1061.
21	SUN D S , TAN R X , CHAI Y P . A universal SMR-aware cache framework with deep optimization for DM-SMR and HM-SMR disks. ACM Transactions on Storage, 2023, 19 (3): 1- 35. doi: 10.1145/3588442
22	FAN W, ZHANG Y J, SONG W N, et al. Q-Learning based backup for energy harvesting powered embedded systems[C]//Proceedings of the Design, Automation & Test in Europe Conference & Exhibition (DATE). Washington D.C., USA: IEEE Press, 2020: 1247-1252.
23	KANG W , SHIN D , YOO S . Reinforcement learning-assisted garbage collection to mitigate long-tail latency in SSD. ACM Transactions on Embedded Computing Systems, 2017, 16 (5): 1- 20. doi: 10.1145/3126537
24	KOBER J , BAGNELL J A , PETERS J . Reinforcement learning in robotics: a survey. The International Journal of Robotics Research, 2013, 32 (11): 1238- 1274. doi: 10.1177/0278364913495721
25	KAELBLING L P , LITTMAN M L , MOORE A W . Reinforcement learning: a survey. Journal of Artificial Intelligence Research, 1996, 4, 237- 285. doi: 10.1613/jair.301
26	NARAYANAN D , DONNELLY A , ROWSTRON A . Write off-loading: practical power management for enterprise storage. ACM Transactions on Storage, 2008, 4 (3): 1- 23. doi: 10.1145/1416944.1416949

[1]	DU Songlin, WU Dakui, YU Yuntao, LIU Ya, ZHOU Wenju. Distributed Assembly Workshop Scheduling Based on Collaborative Optimization Algorithm [J]. Computer Engineering, 2025, 51(3): 274-282.
[2]	JIANG Baoqing, CHEN Hongbin. Trajectory Planning for Unmanned Aerial Vehicle Assisted WSN Data Collection Based on Q-Learning [J]. Computer Engineering, 2021, 47(4): 127-134,165.
[3]	ZHOU Yunteng, ZHANG Xueying, LI Fenglian, LIU Shuchang, JIAO Jiangli, TIAN Dou. SVDPP Recommendation Algorithm Optimized by Q-learning Algorithm [J]. Computer Engineering, 2021, 47(2): 46-51.
[4]	SHI Zhao, SUN Changyin, JIANG Fan. Block-aware Power Allocation Based on Q-Learning in Millimeter-Wave Network [J]. Computer Engineering, 2020, 46(12): 185-192.
[5]	YU Jinliang, TU Shanshan, MENG Yuan. Impersonation Attack Detection Algorithm Based on Reinforcement Learning in Mobile Fog Computing [J]. Computer Engineering, 2020, 46(1): 38-44.
[6]	TU Xuezhen,TU Yaofeng,CHEN Xiaoqiang. An optimized Key-Value NoSQL system [J]. Computer Engineering, 2019, 45(6): 52-59.
[7]	CHEN Chao,CHAI Yunpeng. A Flash-based Hybrid Storage Simulation System [J]. Computer Engineering, 2015, 41(5): 45-49.
[8]	ZHOU Shimin,CHAI Yunpeng,WANG Liang,WANG Xin. Data Layout Optimization Algorithm for Database of Hybrid Storage with Solid State Drive [J]. Computer Engineering, 2015, 41(4): 55-59.
[9]	WANG Zi-qiang, WU Ji-gang. Mobile Robot Path Planning Based on RDC-Q Learning Algorithm [J]. Computer Engineering, 2014, 40(6): 211-214.
[10]	LAN Li. Data Block Replacement Optimization Algorithm in Multi-level Cache Mode [J]. Computer Engineering, 2013, 39(4): 78-81.
[11]	XU Dong-Mei, HAN Xiao-Xin, LI Ding, JIA Min. Research on Computable Model of Emotional Interaction Based on Q-learning Algorithm [J]. Computer Engineering, 2012, 38(10): 277-279.
[12]	ZHANG Yan; SHI Lei; WEI Lin;. Study on Optimal Model of Web Cache [J]. Computer Engineering, 2009, 35(8): 85-87.
[13]	WANG Xiao-yan. High Effective Stream Media Proxy Cache Replacement Algorithm [J]. Computer Engineering, 2009, 35(14): 72-74.
[14]	YANG Chuandong; YU Zhenwei; WANG Xinggang; ZHANG Huanyuan. Proxy Cache Replacement Algorithm Based on Popularity Prediction of Streaming Media File [J]. Computer Engineering, 2007, 33(07): 99-100.

Please choose a citation manager

Content to export