摘要： 分布式网络爬虫的广泛应用使得搜索引擎的数据规模呈几何式增长，面对数以TB甚至PB量级的数据，单机模式下的PageRank算法由于CPU、I/O和内存的开销过大导致效率低下。为此，提出一种基于MapReduce框架的并行PageRank算法。在算法的一次迭代过程中，利用Map函数对网页拓扑信息文件进行解析，使用Reduce函数计算网页得分，从而并行化PageRank算法的中间迭代过程。通过计算全局网页得分控制迭代次数，得到较精确的网页排序结果。实验结果表明，该算法在保持原有单机PageRank算法整体网页排序精度的基础上，具有较好的集群性能和较快的执行速度。
关键词:
搜索引擎,
PageRank算法,
MapReduce框架,
并行计算,
Hadoop平台
Abstract: The emergence of distributed Web crawl largely expands the scale of related Web information. Since PageRank needs to process the topology of entire existed page set, the limitation of CPU, I/O and memory becomes the big issue when it confronts the data in TB or PB level. Aiming at these problems, this paper proposes a parallel PageRank algorithm based on MapReduce. In a certain iteration of algorithm, it processes the files containing the topology of Web page graph by Map function and calculates the pages’ scores by Reduce function. Using the global Web page score as convergence to control iterations and get more precise Web page sorting result. Experimental result shows that the improved algorithm has better clustering performance and faster execution speed on the basis of keeping the overall Web page sorting accuracy of single machine PageRank algorithm.
Key words:
search engine,
PageRank algorithm,
MapReduce framework,
parallel computing,
Hadoop platform
中图分类号: