检索
E-mail
RSS
作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2010, Vol. 36 ›› Issue (9): 285-287.doi: 10.3969/j.issn.1000-3428.2010.09.101

• 开发研究与设计技术 • 上一篇    下一篇

Web论坛数据源增量爬虫的研究

蔡欣宝，郭若飞，赵朋朋，崔志明   

  1. (苏州大学智能信息处理及应用研究所，苏州 215006)
  • 收稿日期:1900-01-01 修回日期:1900-01-01 出版日期:2010-05-05 发布日期:2010-05-05

Research on Web Forum Data Source Incremental Crawler

CAI Xin-bao, GUO Ruo-fei, ZHAO Peng-peng, CUI Zhi-ming   

  1. (Institute of Intelligent Information Processing and Application, Soochow University, Suzhou 215006)
  • Received:1900-01-01 Revised:1900-01-01 Online:2010-05-05 Published:2010-05-05

PDF

1297

被引次数

11 | 1

摘要： 针对Web论坛站点结构复杂、内容更新快等特点，提出一种针对论坛的增量信息采集算法，使用站点地图重建技术及网页更新频繁度估计方法，根据站点地图选择有效的链接，按照网页更新频度确定网页的采集频度。实验结果表明，该方法是有效的。

关键词: Web论坛, 增量爬虫, 站点地图, 泊松模型

Abstract: According to the characters of Web forum site such as the complex structure and quickly updating contents, an algorithm of forum incremental information sampling is presented. The technologies of site map rebuilding and estimating the frequency of page update are used. According to the site map, the crawler selects effective links. According to the frequency of Web page update, the crawler determines the crawling frequency of the Web page. Experimental results indicate this method is effective.

Key words: Web forum, incremental crawler, site map, Poisson model

中图分类号: