计算机工程

• 先进计算与数据处理 • 上一篇    下一篇

基于Hadoop的贝叶斯过滤MapReduce模型

曾青华,袁家斌,张云洲   

  1. (南京航空航天大学计算机科学与技术学院,南京 210016)
  • 收稿日期:2012-10-15 出版日期:2013-11-15 发布日期:2013-11-13
  • 作者简介:曾青华(1987-),女,硕士,主研方向:云计算,并行计算;袁家斌,教授、博士、博士生导师;张云洲,硕士
  • 基金项目:
    国家“863”计划基金资助项目(2009AA044601);国家自然科学基金资助重点项目(61139002);南京航空航天大学基本科研业务费专项基金资助项目(NS2010230)

Hadoop-based MapReduce Model of Bayesian Filtering

ZENG Qing-hua, YUAN Jia-bin, ZHANG Yun-zhou   

  1. (School of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 210016, China)
  • Received:2012-10-15 Online:2013-11-15 Published:2013-11-13

摘要: 传统分布式大型邮件系统对海量邮件的过滤存在编程难、效率低、前期训练耗用资源大等缺点,为此,对传统贝叶斯过滤算法进行并行化改进,利用云计算MapReduce模型在海量数据处理方面的优势,设计一种基于Hadoop开源云架构的贝叶斯邮件过滤MapReduce模型,优化邮件的训练和过滤过程。实验结果表明,与传统分布式计算模型相比,该模型在召回率、查准率和精确率方面性能较好,同时可降低邮件过滤成本,提高系统执行效率。

关键词: 云计算, MapReduce模型, Hadoop架构, 贝叶斯算法, 垃圾邮件, 反垃圾邮件过滤

Abstract: There are some disadvantages of mass mail filtering for large mail systems on the traditional distributed system including programming difficulties, low efficiency, mass system and network resources consumed. Taking advantage of the high performance of the cloud computing in processing data processing effectively, a MapReduce model of Bayesian mail filtering based on Hadoop is proposed. It improves the traditional Bayesian filtering algorithms and optimizes the mail training and filtering processes. Experimental results show that, compared with traditional distributed computing model, the Hadoop-based MapReduce model of Bayesian anti-spam mail filtering performs better in recall, precision and accuracy, reduces the cost of mail learning and classifying and improves the system efficiency.

Key words: cloud computing, MapReduce model, Hadoop framework, Bayesian algorithm, spam mail, anti-spam mail filtering

中图分类号: