摘要：

针对传统关系型数据库在对大数据进行操作时,系统性能严重下降、计算效率提升有限以及可扩展性差等问题,引入MapReduce并行计算模型,提出一种大数据上基于列存储的MapReduce并行连接算法。设计面向大数据的分布式计算模型,包括MapReduce分布式环境下的列存储文件格式MCF,采用协同定位策略实现对分布式存储的优化。使用分片聚集和子连接启发式优化方法,实现大数据在MapReduce分布式环境下并行连接算法。实验结果证明,在大数据分析处理中,该算法在执行时间和负载能力上有着较好的优化性能,同时具有良好的可扩展性。

关键词: 大数据, 列存储, MapReduce模型, MCF存储格式, 并行连接, 启发式优化方法

Abstract:

The character of big data are large scale,depth,velocity,common hardware and open source.Aiming at the system’s inefficiency and scalability problem of traditional relational database in big data analysis,this paper presents an algorithm of parallel join in a MapReduce environment based on column-store by introducing MapReduce computing model.The design of large data-oriented distributed computing models is proposed.It designs the MapReduce column-store file,and achieves optimization by cooperative localization strategy.Secondly,and the partition aggregation and the heuristic optimization strategy to realize the implementation of parallel join algorithm are proposed.Experimental results show that the algorithm has the high performance and scalability in execution time and load capacity. 

Key words: big data, column-store, MapReduce model, MCF storage form, parallel join;heuristic optimization method

中图分类号: 

• TP181