基于Chord算法的可扩展高性能元数据存储环设计

doi:10.3969/j.issn.1000-3428.2016.08.001

计算机工程

基于Chord算法的可扩展高性能元数据存储环设计

杜然^1,2,黄秋兰¹,阚文枭^1,2,王聪^1,2,徐琪^1,2,陈刚¹

(1.中国科学院高能物理研究所,北京 100049; 2.中国科学院大学,北京 100049)

收稿日期:2015-08-28 出版日期:2016-08-15 发布日期:2016-08-15
作者简介:杜然(1986－),女,博士研究生,主研方向为海量存储系统;黄秋兰,助理研究员;阚文枭、王聪、徐琪,博士研究生;陈刚,研究员、博士生导师。
基金资助:
国家自然科学基金资助项目(11305192,11205179,11205177);中国科学院科研信息化基金资助项目“高能物理科技领域云的建设与应用”(Y2293C00S5)。

Design of Scalable Highperformance Metadata Storage Ring Based on Chord Algorithm

DU Ran^1,2,HUANG Qiulan ¹,KAN Wenxiao ^1,2,WANG Cong ^1,2,XU Qi ^1,2,CHEN Gang ¹

(1.Institute of High Energy Physics,Chinese Academy of Sciences,Beijing 100049,China; 2.University of Chinese Academy of Sciences,Beijing 100049,China)

Received:2015-08-28 Online:2016-08-15 Published:2016-08-15

摘要/Abstract

摘要： HazelNut是一个基于块的分级存储系统,为满足该系统元数据内存需求量大、访问速度快、动态可增长的要求,设计并实现一种可扩展高性能的元数据存储环SCRing。SCRing由元数据存储环shell ring、映射环chord ring和缓存表组成,shell ring与chord ring分别用于存储与定位数据块元数据信息,缓存表用于缓存数据块元数据在内存中的位置。通过消息摘要算法计算得到SCRing双环节点间的映射关系。依据SCRing对消息摘要算法的实际需求,结合Linux内核版本,测试并比较Linux内核支持的消息摘要算法的速度、均匀性和稳定性,得出SCRing所应采用的算法种类和使用模式,并在此基础上对SCRing读写性能进行验证,结果表明SCRing具备可扩展性和高性能,满足HazelNut系统元数据访问需求。

关键词: 分级存储系统, Linux内核, Chord算法, 消息摘要算法, 元数据

Abstract: HazelNut is a blockbased hierarchical storage system,which has special metadata requirements,including huge memory cost,fast access speed and everlasting data growth.To meet those requirements,a scalable highperformance metadata storage ring,named SCRing,is presented.SCRing consists of double rings and a cache table.One of the rings is named as shell ring to store the metadata information of data block,and the other is named as chord ring to locate the metadata information of data block.The cache table is used to cache the memory address of block metadata.The mapping relationship between the two rings are got through message digest algorithms.According to the SCRing demands on message digest algorithms and the Linux kernel version,a series of tests is made on digest generation speed,uniformity and constancy to get algorithm category and usage mode.Besides,I/O performance test is done on the basis of the above message digest algorithm tests.The results prove that SCRing is featured with high performance and scalability,which meets the metadata access requirements of HazelNut system.

Key words: hierarchical storage system, Linux kernel, Chord algorithm, message digest algorithm, metadata

中图分类号:

TP316.4

杜然,黄秋兰,阚文枭,王聪,徐琪,陈刚. 基于Chord算法的可扩展高性能元数据存储环设计[J]. 计算机工程, doi: 10.3969/j.issn.1000-3428.2016.08.001.

DU Ran,HUANG Qiulan,KAN Wenxiao,WANG Cong,XU Qi,CHEN Gang. Design of Scalable Highperformance Metadata Storage Ring Based on Chord Algorithm[J]. Computer Engineering, doi: 10.3969/j.issn.1000-3428.2016.08.001.

https://www.ecice06.com/CN/Y2016/V42/I8/1

参考文献

参考文献［1］Stoica I,Morris R,Karger D,et al.Chord:A Scalable PeertoPeer Lookup Service for Internet Applications［C］//Proceedings of ACM SIGCOMM’01.New York,USA:ACM Press,2001:149160. ［2］Xu Weixia,Lu Yutong,Li Qiong,et al.Hybrid Hierarchy Storage System in Milkyway2 Supercomputer［J］.Frontiers of Computer Science,2014,8(3):367377. ［3］王笑冉,胡晓勤,黄传波.基于磁带分级存储的文件备份系统设计与实现［J］.计算机安全,2014(3):1217. ［4］Zhang Gong,Chiu L,Clem D,et al.Automated Lookahead Data Migration in SSDenabled Multitiered Storage Systems［C］//Proceedings of IEEE Symposium on Mass Storage Systems & Technologies.Washington D.C.,USA:IEEE Press,2010:16. ［5］吕帅,刘光明,徐凯,等.海量信息分级存储数据迁移策略研究［J］.计算机工程与科学,2009,31(S1):163167. ［6］黄冬梅,杜艳玲,王振华.海洋大数据分级存储中迁移模型的研究［J］.计算机应用与软件,2014,31(9):4547. ［7］施光源,张宇.基于模糊逻辑的数据分级存储模型研究［J］.计算机科学,2013,40(11):284287. ［8］Xiao Weijun,Lei Xiaoqiang,Li Ruixuan,et al.PASS:A Hybrid Storage System for Performancesynchronization Tradeoffs Using SSDs［C］//Proceedings of the 10th International Symposium on Parallel and Distributed Processing with Applications.Washington D.C.,USA:IEEE Press,2012:403410. ［9］Wang A I,Reiher P L,Popek G J,et al.Conquest:Better Performance Through a Disk/PersistentRAM Hybrid File System［C］//Proceedings of USENIX Annual Technical Conference.Berkeley,USA:USENIX Press,2002:1528. ［10］Qiu Sheng.NVMFS:A Hybrid File System for Improving Random Write in Nandflash SSD［C］//Proceedings of IEEE Symposium on Mass Storage Systems & Technologies.Washington D.C.,USA:IEEE Press,2013:15. ［11］敖莉,于得水,舒继武,等.一种海量数据分级存储系统THTS［J］.计算机研究与发展,2011,48(6):10891100. ［12］Guerra J,Pucha H,Glider J,et al.Cost Effective Storage Using Extent Based Dynamic Tiering［C］//Proceedings of USENIX Conference on File and Storage Technologies.Berkeley,USA:USENIX Press,2011:273286. ［13］Wilkes J,Golding R,Staelin C,et al.The HP Autoraid Hierarchical Storage System［J］.ACM SIGOPS Operating Systems Review,1995,29(5):96108. ［14］Kim Y,Gupta A,Urgaonkar B,et al.Hybridstore:A Costefficient,Highperformance Storage System Combining SSDS and HDDS［C］//Proceedings of IEEE International Symposium on Modeling,Analysis & Simulation of Computer & Telecommunication Systems.Washington D.C.,USA:IEEE Press,2011:227236. ［15］Chen Feng,Koufaty D A,Zhang Xiaodong.Hystor:Making the Best Use of Solid State Drives in High Performance Storage Systems［C］//Proceedings of International Conference on Supercomputing.New York,USA:ACM Press,2011:2232. ［16］Wu Xiaojian.Managing Storage Space in a Flash and Disk Hybrid Storage System［C］//Proceedings of IEEE International Symposium on Modeling,Analysis & Simulation of Computer and Telecommunication Systems.Washington D.C.,USA:IEEE Press,2009:14. ［17］Btier［EB/OL］.［20150829］.https://sourceforge.net/projects/tier. ［18］王迪,舒继武,薛巍,等.基于块级别的SAN系统自适应分级存储［J］.高技术通讯,2007,17(2):111115. ［19］Jacobs A.The Pathologies of Big Data［J］.Communications of the ACM,2009,52(8):3644. ［20］Gorman M.Understanding the Linux Virtual Memory Manager［M］.Upper Saddle River,USA:Prentice Hall,2004. ［21］Bovet D.Understanding the Linux Kernel［M］.3rd ed.Sebastopol,USA:O’Reilly Media,2005. ［22］Corbet J,Rubini A,KroahHartman G.Linux Device Drivers［M］.3rd ed.Sebastopol,USA:O’Reilly Media,2005. ［23］Love R.Linux Kernel Development［M］.3rd ed.［S.l.］:AddisonWesley Professional,2010. ［24］Linux Cross Reference［EB/OL］.［20150829］.https://lxr.freeelectrons.com/. ［25］Tanenbaum A S,Wetherall D J.Computer Networks［M］.5th ed.Upper Saddle River,USA:Prentice Hall,2010. ［26］Trappe W.Introduction to Cryptography with Coding Theory［M］.2nd ed.Upper Saddle River,USA:Prentice Hall,2005. ［27］Lkddb［EB/OL］.［20150829］.https://cateee.net/lkddb/. 编辑陆燕菲

[1]	殷文君, 黄建华, 纪元法. 基于改进密集卷积网络的皮肤肿瘤分类方法[J]. 计算机工程, 2023, 49(7): 288-294.
[2]	杨珂, 张帆, 郭威, 赵博, 穆清. 一种拟态存储元数据随机性问题解决方法[J]. 计算机工程, 2022, 48(2): 140-146,155.
[3]	刘宇廷,毕海滨,郭强,倪颖杰. 基于网络拓扑与节点元数据的社团检测算法[J]. 计算机工程, 2018, 44(11): 178-183.
[4]	陈曦,朱建涛,何晓斌. 一种面向高性能计算的分布式对象存储系统[J]. 计算机工程, 2017, 43(8): 69-73.
[5]	詹玲,方协云,李大平,万继光. 基于Ceph文件系统的元数据缓存备份[J]. 计算机工程, 2017, 43(4): 67-72,83.
[6]	阚文枭,黄秋兰,陈刚. 桌面网格环境下虚拟化技术的应用研究[J]. 计算机工程, 2017, 43(3): 11-17.
[7]	吴国锦,胡程. 一种基于起源信息的元数据预取策略[J]. 计算机工程, 2016, 42(6): 1-6.
[8]	阚文枭,王聪,徐琪,杜然,Andrei Tsaregorodtsev,陈刚. 基于桌面网格技术的高能物理计算系统[J]. 计算机工程, 2016, 42(6): 48-54.
[9]	张静,李国庆,于文洋. 基于OpenSearch的对地观测数据统一集成发现[J]. 计算机工程, 2016, 42(6): 60-67.
[10]	许艳艳,雷迎春,龚奕利. 基于可变长分块的分布式文件系统设计与实现[J]. 计算机工程, 2016, 42(5): 80-84,101.
[11]	杜然,黄秋兰,程耀东,陈刚. 基于块的分级存储系统多样性机制设计与研究[J]. 计算机工程, 2016, 42(12): 50-59.
[12]	张影,杨麦顺,张兴军,郭庆伟. 一种指令级Linux内核瞬时故障注入方法研究[J]. 计算机工程, 2015, 41(8): 32-36.
[13]	石京燕,陈德清. 基于数据库的文件系统管理工具设计与实现[J]. 计算机工程, 2015, 41(5): 1-5.
[14]	黄秋兰,程耀东,杜然,陈刚. 可扩展的分布式元数据管理系统设计[J]. 计算机工程, 2015, 41(5): 26-32,37.
[15]	李力，鄢田云. 一种基于LT码的数据云存储方案[J]. 计算机工程, 2014, 40(4): 7-13.

选择文件类型/文献管理软件名称

选择包含的内容

基于Chord算法的可扩展高性能元数据存储环设计

Design of Scalable Highperformance Metadata Storage Ring Based on Chord Algorithm

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于Chord算法的可扩展高性能元数据存储环设计

Design of Scalable Highperformance Metadata Storage Ring Based on Chord Algorithm

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价