计算机工程

• 专栏 • 上一篇    下一篇

一种自适应的分布式数据流处理调整技术

马元文,王鹏,周之敏,汪卫   

  1. (复旦大学计算机科学技术学院上海市数据科学重点实验室,上海 201203)
  • 收稿日期:2014-12-19 出版日期:2015-12-15 发布日期:2015-12-15
  • 作者简介:马元文(1990-),男,硕士研究生,主研方向:分布式计算,数据流处理;王鹏,副教授、博士;周之敏,硕士研究生;汪卫,教授、博士。
  • 基金项目:
    国家自然科学基金资助项目(61103009);上海市科委大数据专项基金资助项目(13511504800)。

An Adaptive Adjustment Technology of Distributed Data Stream Processing

MA Yuanwen,WANG Peng,ZHOU Zhimin,WANG Wei   

  1. (Shanghai Key Laboratory of Data Science,School of Computer Science,Fudan University,Shanghai 201203,China)
  • Received:2014-12-19 Online:2015-12-15 Published:2015-12-15

摘要: 由于数据流速率的不稳定特性,将数据流查询部署在固定数量的结点上会导致分布式数据流管理系统(DDSMS)难以高效地利用计算资源。为此,针对目前支持一个结点运行多个实例模式的DDSMS,提出一种基于操作符组的自适应动态调整技术,包含桶机制的路由管理、生成调 整策略和调整执行计划等方法。通过监控当前执行计划的运行状态,DDSMS生成代价较小的调整策略,并对当前执行计划进行调整。在数据流处理的基准测试数据上进行实验,结果表明,该调整技术可以更加细粒度地制定调整策略,灵活应对数据流速率的变化。

关键词: 数据流查询, 分布式, 动态调整, 状态管理, 调整策略

Abstract: As the rate of stream data is various,it is hard to effectively use computing resource when deploying data stream query on a fixed number of nodes.For the multi-instance-per-node execution model adopted by the current Distributed Data Stream Management System(DDSMS),this paper presents an operator group based adaptive dynamic adjustment technology which contains the approaches of bucket based routing management,adjustment strategy making and adjustment execution.Based on executing status of current execution plan,DDSMS makes adjustment strategy and adjusts current execution plan according to the adjustment strategy.Through testing the adjustment technology using benchmark of stream processing,results show the effectiveness of the proposed technology.

Key words: data stream query, distributed, dynamical adjustment, state management, adjustment strategy

中图分类号: