计算机工程

• 先进计算与数据处理 • 上一篇    下一篇

基于Spark的时态查询扩展与时态索引优化研究

周亮1,李格非1,邰伟鹏2,郑啸2   

  1. (1.上海交通大学 计算机科学与工程系,上海 200240; 2.安徽工业大学 计算机科学与技术学院,安徽 马鞍山243032)
  • 收稿日期:2016-06-20 出版日期:2017-07-15 发布日期:2017-07-15
  • 作者简介:周亮(1992—),男,硕士研究生,主研方向为大数据、云计算;李格非,硕士研究生;邰伟鹏(通信作者),副教授、博士;郑啸,教授、博士。
  • 基金项目:
    安徽省高校自然科学研究重点项目“基于关键字的大规模地理数据查询方法研究”(KJ2015A310)。

Research on Temporal Query Expansion and Temporal Index Optimization Based on Spark

ZHOU Liang 1,LI Gefei 1,TAI Weipeng 2,ZHENG Xiao 2   

  1. (1.Department of Computer Science and Engineering,Shanghai Jiaotong University,Shanghai 200240,China; 2.School of Computer Science and Technology,Anhui University of Technology,Maanshan,Anhui 243032,China)
  • Received:2016-06-20 Online:2017-07-15 Published:2017-07-15

摘要: 时空数据库和基于集群计算的时间分析工具大多基于外存,将其应用在大数据处理场景下系统性能将迅速降低。为此,基于Spark构建一个易用且高可扩展的时态大数据查询分析系统。通过扩展Spark SQL解析器,使其能够支持类SQL形式的时态操作,运用SIMBA开源项目的方法,引入全局过滤和局部时态索引2种优化策略,使得系统能以高吞吐量及低延迟执行时态查询操作。基于时态查询效率的评估实验结果表明,在不同影响参数下,该系统的时态查询性能优于原生的Spark SQL查询处理方案。

关键词: 时态大数据, Spark系统, Spark SQL组件, 时态查询, 时态索引, 高吞吐量, 低延迟

Abstract: There exists some temporal databases and temporal analysis tools based on cluster-based computing systems.However,most of them are disk-oriented and performance degenerate rapidly when processing big data.This paper proposes a system which is based on Spark,and provides accessible and scalable temporal query scheme with large temporal data for users.Specifically,it extends Spark SQL parser to support SQL-like temporal operations.Besides,it uses the index manager based on Spark SQL which is proposed by SIMBA,and embeds optimization strategies in two aspects:global filtering and local temporal index.Depending on these optimization rules,the system achieves high throughput and low latency in temporal operations.Evaluation experiment results on temporal query efficiency and effectiveness show this system has improved temporal query performance over original Spark SQL in different factors.

Key words: temporal big data, Spark system, Spark SQL component, temporal query, temporal index, high throughput, low latency

中图分类号: