摘要： 通过分析基于新闻要素的在线新事件检测算法的时间消耗,提出一种面向大规模数据环境的在线新事件 检测算法。该算法利用基于倒排索引的高效相似报道搜索机制,有效减少单路径聚类算法中的相似度比较次数。通过对报道预处理、报道与事件比较以及索引搜索这3 个过程的并行化,提高算法在多机环境下的运行效率和可 伸缩性。实验结果表明,该算法在不影响漏检率和误检率的基础上,提高了新事件检测的速度,并且在千万到亿级 别的报道规模下,其吞吐量达到150 条/ s ~200 条/ s。

关键词: 新事件检测, 单路径聚类, 大规模数据, 并行计算, 倒排索引, MapReduce 架构

Abstract: Through analyzing the time consumption of the existing online New Event Detection(NED) algorithm based on news elements,this paper improves an online NED algorithm for large-scale data environment． The algorithm uses efficient reported similar search mechanism based on inverted index to reduce the similarity comparison of single path clustering algorithms． Through parallelization of report pretreatment, report and event comparison, index search, it improves the efficiency and scalability of the algorithm in multimachine. Experimental result shows that the algorithm can greatly improve new event detection speed without affecting the miss probability and false-alarm probability,and its throughput reaches 150 ~200 reports / s at the scale of 10 ~100 million reports.

Key words: New Event Detection(NED), single-pass clustering, large-scale data, parallel computing, inverted index, MapReduce architecture

中图分类号: 

• TP391