Abstract: Remote image data parallel processing system basically relies on foreign commercial products,while domestic independent parallel processing system the task process support capability and parallel processing cannot meet the need of scale production.Therefore,based on HDFS,MapReduce cluster parallel architecture,CPU/GPU cooperative parallel processing,memory mapping,BMP and so on,this paper proposes the architecture and realization of a high-performance distributed parallel processing platform with process driven execution.Experimental results show that the multi-granularity mixing parallel processing architecture of cluster and workstation magnificently increases the parallel performance of the platform,which proposes an independent resolution to the scale production of massive remote image data products.

Key words: big data, Hadoop architecture, Hadoop Distributed File System(HDFS), MapReduce framework, GPU parallel computing

摘要： 遥感影像数据并行处理系统大多依赖于国外商用产品,而国内自主化并行计算处理系统的任务流程化支撑能力以及并行计算性能难以适应规模化生产。为此,基于Hadoop的HDFS,MapReduce集群并行架构、CPU和GPU协同并行处理、内存映像、BMP等技术,提出流程驱动执行的高性能分布式并行计算处理平台体系架构。实验结果表明,工作站集群和工作站内多粒度混合的并行计算架构提高了平台并行处理性能,为海量遥感影像数据产品的批量生产提供一种自主化解决方案。

关键词: 大数据, Hadoop架构, Hadoop分布式文件系统, MapReduce框架, GPU并行计算

CLC Number: 

• TP391