计算机工程 ›› 2011, Vol. 37 ›› Issue (11): 37-39.doi: 10.3969/j.issn.1000-3428.2011.11.013

所属专题: 云计算专题

• 云计算专题 • 上一篇    下一篇

基于Hadoop的Web日志挖掘

程 苗a,陈华平b   

  1. (中国科学技术大学 a. 管理学院;b. 计算机科学与技术学院,合肥 230026)
  • 收稿日期:2011-04-20 出版日期:2011-06-05 发布日期:2011-06-05
  • 作者简介:程 苗(1986-),女,硕士研究生,主研方向:云计算,商务智能;陈华平,教授、博士生导师
  • 基金项目:
    博士点基金资助项目(200803580024);创新研究群体科学基金资助项目(70821001)

Weblog Mining Based on Hadoop

CHENG Miao  a, CHEN Hua-ping  b   

  1. (a. College of Management; b. College of Computer Science and Technology, University of Science and Technology of China, Hefei 230026, China)
  • Received:2011-04-20 Online:2011-06-05 Published:2011-06-05

摘要: 基于单一节点的数据挖掘系统在挖掘Web海量数据源时存在计算瓶颈,针对该问题,利用云计算的分布式处理和虚拟化技术的优势,设计一种基于云计算的Hadoop集群框架的Web日志分析平台,提出一种能够在云计算环境中进行分布式处理的混合算法。为进一步验证该平台的高效性,在该平台上利用改进后的算法挖掘Web日志中用户的偏爱访问路径。实验结果表明,在集群中运用分布式算法处理大量的Web日志文件,可以明显提高Web数据挖掘的效率。

关键词: 云计算, Hadoop架构, Map/Reduce编程模式, Web日志挖掘, 遗传算法, 偏爱访问路径

Abstract: The mass data from Web are distributed, heterogeneous and dynamic, so the current data mining system based on single node has developed to a bottleneck. Using the advantage of cloud computing——distributed processing and virtualization, this paper presents a Weblog analysis platform under the Hadoop’s cluster framework based on cloud computing, it also presents a hybrid algorithm which can distributed process in the cloud computing environment. To further verify the effectiveness and efficiency of the platform, it uses the improved algorithm to mine users’ preferred access path in Weblog on the platform. Experimental results show that, using distributed algorithm to process large number of Weblog files in the cluster, can significantly improve the efficiency of Web data mining.

Key words: cloud computing, Hadoop frame, Map/Reduce, Weblog mining, genetic algorithm, preferred browsing path

中图分类号: