基于任务分配和数据集副本的科学工作流数据布局

doi:10.19678/j.issn.1000-3428.0055397

摘要/Abstract

摘要： 云环境下科学工作流的数据布局成为当前工作流研究中的一个热点问题，对科学工作流中任务和数据之间多对多关系进行分析，可以发现不同数据布局方案在数据传输上的费用各不相同，在很大程度上影响工作流的运行成本。为降低科学工作流数据集传输费用，提出一种基于任务分配和数据集副本的科学工作流数据布局方法。该方法从任务分配开始，在定量计算任务依赖度的基础上进行任务分配，根据分配结果给出一个基于数据集副本的两阶段数据布局方法，以实现科学工作流运行中传输费用的优化。实例结果表明，与工作流层方法相比，该方法可以有效降低科学工作流的运行成本。

关键词: 云环境, 科学工作流, 任务分配, 数据集副本, 数据布局, 传输费用

Abstract: The Data Layout(DL) for Scientific Workflow(SW) in cloud environment becomes a hot issue in current workflow research.Considering the many-to-many relationship between tasks and data in scientific workflows,it can be found that the data transmission costs of different data layout schemes are different,which can greatly affect the running cost of workflow.In order to reduce the data transmission costs in SW,this paper proposes a SW DL method based on task assignment and dataset replicas.The method starts with task assignment,assigning these tasks based on quantitative calculation of task dependencies,and then proposes a two-stage DL method based on the dataset replicas according to the assignment result,so as to achieve the optimization of transmission costs in running scientific workflows.Sample results show that this method can effectively reduce the running cost of scientific workflows compared with the workflow layer method.

Key words: cloud environment, Scientific Workflow(SW), task assignment, dataset replicas, Data Layout(DL), transmission cost

中图分类号:

TP391

尚蕾, 刘茜萍. 基于任务分配和数据集副本的科学工作流数据布局[J]. 计算机工程, 2020, 46(5): 122-130,138.

SHANG Lei, LIU Xiping. Scientific Workflow Dataset Layout Based on Task Assignment and Dataset Replicas[J]. Computer Engineering, 2020, 46(5): 122-130,138.

https://www.ecice06.com/CN/Y2020/V46/I5/122

图/表 14

20200513201733

20200513201738

20200513201743

20200513201748

20200513201753

20200513201758

20200513201802

20200513201806

20200513201812

20200513201817

20200513201821

20200513201826

20200513201835

20200513201844

参考文献

[1] LI G,MUTHUSAMY V,JACOBSEN H A.A distri-buted service-oriented architecture for business process execution[J].ACM Transactions on the Web,2010,4(1):1-11.
[2] DEELMAN E,BLYTHE J,GIL Y,et al.Pegasus:mapping scientific workflows onto the grid[C]//Proceedings of 2004 European Across Grids Conference.Berlin,Germany:Springer,2004:11-20.
[3] LUDÄSCHER B,ALTINTAS I,BERKLEY C,et al.Scientific workflow management and the Kepler system[J].Concurrencyand Computation Practice and Experience,2006,18(10):1039-1065.
[4] AMS02[EB/OL].[2019-06-01].http://www.ams02.org/.
[5] OINN T,ADDIS M,FERRIS J,et al.Taverna:a tool for the composition and enactment of bioinformatics workflows[J].Bioinformatics,2004,20(17):3045-3054.
[6] LONG Hao,WANG Hao.A scheduling heuristic of scientific workflow under distributed computing environment[J].Journal of Chinese Computer Systems,2015,36(2):226-230.(in Chinese)龙浩,汪浩.异构环境下科学工作流的启发式调度算法[J].小型微型计算机系统,2015,36(2):226-230.
[7] WU Zhangjun,NI Zhiwei,GU Lichuan,et al.A revised discrete particle swarm optimization for cloud workflow scheduling[C]//Proceedings of International Con-ference on Computational Intelligence and Security.Washington D.C.,USA:IEEE Press,2011:184-188.
[8] YUAN Dong,YANG Yun,LIU Xiao,et al.On-demand minimum cost benchmarking for intermediate dataset storage in scientific cloud workflow systems[J].Journal of Parallel and Distributed Computing,2011,71(2):316-332.
[9] DURAO F,CARVALHO J F S,FONSEKA A,et al.A systematic review on cloud computing[J].Journal of Supercomputing,2014,68(3):1321-1346.
[10] HWANG K,DONGARRA J,FOX G C.Distributed andcloud computing:from parallel processing to the Internet of things[EB/OL].[2019-06-01].https://dl.acm.org/doi/book/10.5555/2060077.
[11] YUAN Dong,YANG Yun,LIU Xiao,et al.A data placement strategy in scientific cloud workflows[J].Future Generation Computer Systems,2010,26(8):1200-1214.
[12] ZHENG Pai,CUI Lizhen,WANG Haiyang,et al.A data placement strategy for data-intensive applications in cloud[J].Chinese Journal of Computers,2010,33(8):1472-1480.(in Chinese)郑湃,崔立真,王海洋,等.云计算环境下面向数据密集型应用的数据布局策略与方法[J].计算机学报,2010,33(8):1472-1480.
[13] LI Xuejun,WU Yang,LIU Xiao,et al.Workflow data placement for data center in hybrid cloud[J].Journal of Software,2016,27(7):1861-1875.(in Chinese)李学俊,吴洋,刘晓,等.混合云中面向数据中心的工作流数据布局方法[J].软件学报,2016,27(7):1861-1875.
[14] EBRAHIMI M,MOHAN A,KASHLEV A,et al.BDAP:a big data placement strategy for cloud-based scientific workflows[C]//Proceedings of the 1st IEEE International Conference on Big Data Computing Service and Applications.Washington D.C.,USA:IEEE Computer Society,2015:105-114.
[15] ZHAO Qing,XIONG Congcong,ZHAO Xi,et al.A data placement strategy for data-intensive scientific workflows in cloud[C]//Proceedings of IEEE/ACM International Symposium on Cluster,Cloud and Grid Computing.Washington D.C.,USA:IEEE Press,2015:928-934.
[16] IKKEN S,RENAULT E,BARKAT A,et al.Efficient intermediate data placement in federated cloud data centers storage[C]//Proceedings of International Conference on Mobile,Secure and Programmable Networking.Berlin,Germany:Springer,2016:1-15.
[17] CHEN Jian,ZHANG Jinghui,SONG Aibo.Efficient data and task co-scheduling for scientific workflow in geo-distributed datacenters[C]//Proceedings of International Conference on Advanced Cloud and Big Data.Washington D.C.,USA:IEEE Press,2017:63-68.
[18] LIN Bing,XIANG Tao,CHEN Guolong,et al.Time-driven data placement strategy for scientific workflows in hybrid cloud[J].Computer Integrated Manufacturing Systems,2019,25(4):123-133.(in Chinese)林兵,项滔,陈国龙,等.混合云环境下面向时延优化的科学工作流数据布局策略[J].计算机集成制造系统,2019,25(4):123-133.
[19] CHENG Huimin,LI Xuejun,WU Yang,et al.Data placement strategy based on multi-objective optimization for scientific workflows in cloud computing environment[J].Computer Applications and Software,2017,34(3):1-6.(in Chinese)程慧敏,李学俊,吴洋,等.云环境下基于多目标优化的科学工作流数据布局策略[J].计算机应用与软件,2017,34(3):1-6.
[20] LI Xuejun,ZHANG Lei,WU Yang,et al.A novel workflow-level data placement strategy for data-sharing scientific cloud workflows[J].IEEE Transactions on Services Computing,2019,12(3):370-383.

选择文件类型/文献管理软件名称

选择包含的内容