摘要：

通过对Spark采用的弹性分布式数据集及任务调度等关键技术进行分析,发现数据处理I/O时间是影响Spark计算性能的主要瓶颈。为此,研究Spark合并文件运行模式,该模式能够减少缓存文件数量,提高Spark的I/O效率,但存在内存开销较高的缺点。在此基础上,给出改进的Spark Shuffle过程,即通过设计一种使每个Mapper只生成一个缓存文件的运行模式,并且每个Mapper共享同一个内存缓冲区,从而提高I/O效率和减少内存开销。仿真结果表明,与Spark默认模式相比,该运行模式宽依赖计算过程的I/O时间缩短42.9%,可有效提高内存利用率和Spark平台运算效率。

关键词: 分布式计算, Spark平台, Shuffle过程, 磁盘I/O, 任务调度

Abstract: Based on the analysis of the key technologies of Spark,such as flexible distributed data set and Spark task scheduling,it is concluded that time of I/O in data processing has a great effect on the computing performance of Spark. Aiming at this problem,this paper studies the run mode of Spark consolidating files that can reduce the number of cache files and improve the I/O efficiency of Spark to some extent,but it still has the disadvantage of high memory cost. Further more,the paper proposes an improved process of Spark Shuffle which designs a mode that every Mapper only generates one cache file,and every Mapper’s bucket shares the same memory buffer,thus these improve I/O efficiency and reduce the memory overhead. Simulation results show that,compared with the default mode of Spark,the I/O time of a wide dependent process is shortened by 42.9%,which improves the memory utilization and the efficiency of the Spark platform.

Key words: distributed computing, Spark platform, Shuffle process, disk I/O, task scheduling

中图分类号: 

• TP316.4