计算机工程 ›› 2020, Vol. 46 ›› Issue (5): 122-130,138.doi: 10.19678/j.issn.1000-3428.0055397

• 先进计算与数据处理 • 上一篇    下一篇

基于任务分配和数据集副本的科学工作流数据布局

尚蕾, 刘茜萍   

  1. 南京邮电大学 计算机学院 江苏省大数据安全与智能处理重点实验室, 南京 210023
  • 收稿日期:2019-07-05 修回日期:2019-09-27 发布日期:2019-10-11
  • 作者简介:尚蕾(1994-),女,硕士,主研方向为工作流技术、服务计算;刘茜萍(通信作者),副教授、博士。
  • 基金项目:
    国家自然科学基金(61602260)。

Scientific Workflow Dataset Layout Based on Task Assignment and Dataset Replicas

SHANG Lei, LIU Xiping   

  1. Jiangsu Key Laboratory of Big Data Security and Intelligent Processing, School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
  • Received:2019-07-05 Revised:2019-09-27 Published:2019-10-11

摘要: 云环境下科学工作流的数据布局成为当前工作流研究中的一个热点问题,对科学工作流中任务和数据之间多对多关系进行分析,可以发现不同数据布局方案在数据传输上的费用各不相同,在很大程度上影响工作流的运行成本。为降低科学工作流数据集传输费用,提出一种基于任务分配和数据集副本的科学工作流数据布局方法。该方法从任务分配开始,在定量计算任务依赖度的基础上进行任务分配,根据分配结果给出一个基于数据集副本的两阶段数据布局方法,以实现科学工作流运行中传输费用的优化。实例结果表明,与工作流层方法相比,该方法可以有效降低科学工作流的运行成本。

关键词: 云环境, 科学工作流, 任务分配, 数据集副本, 数据布局, 传输费用

Abstract: The Data Layout(DL) for Scientific Workflow(SW) in cloud environment becomes a hot issue in current workflow research.Considering the many-to-many relationship between tasks and data in scientific workflows,it can be found that the data transmission costs of different data layout schemes are different,which can greatly affect the running cost of workflow.In order to reduce the data transmission costs in SW,this paper proposes a SW DL method based on task assignment and dataset replicas.The method starts with task assignment,assigning these tasks based on quantitative calculation of task dependencies,and then proposes a two-stage DL method based on the dataset replicas according to the assignment result,so as to achieve the optimization of transmission costs in running scientific workflows.Sample results show that this method can effectively reduce the running cost of scientific workflows compared with the workflow layer method.

Key words: cloud environment, Scientific Workflow(SW), task assignment, dataset replicas, Data Layout(DL), transmission cost

中图分类号: