Abstract: According to the characters of Web forum site such as the complex structure and quickly updating contents, an algorithm of forum incremental information sampling is presented. The technologies of site map rebuilding and estimating the frequency of page update are used. According to the site map, the crawler selects effective links. According to the frequency of Web page update, the crawler determines the crawling frequency of the Web page. Experimental results indicate this method is effective.

Key words: Web forum, incremental crawler, site map, Poisson model

摘要： 针对Web论坛站点结构复杂、内容更新快等特点，提出一种针对论坛的增量信息采集算法，使用站点地图重建技术及网页更新频繁度估计方法，根据站点地图选择有效的链接，按照网页更新频度确定网页的采集频度。实验结果表明，该方法是有效的。

关键词: Web论坛, 增量爬虫, 站点地图, 泊松模型

CLC Number: 

• TP311.52