摘要： 基于关键词匹配检索的传统搜索引擎爬全率和爬准率较低，而使用基于语义检索的主题爬虫方法容易偏离主题与陷入局部最优。针对该问题，提出一种采用多目标蚁群优化算法的主题爬虫方法。构建主题爬虫领域本体和主题向量，以链接的锚文本相关度、链接所在网页主题相关度以及链接指向网页主题相关度为指标判断链接是否与主题相关，并建立链接主题相关度的多目标优化模型，将基于多目标优化的蚁群算法引入主题爬虫的链接选择过程，采用非支配排序和最近最远候选解法选取Pareto最优链接，以引导主题爬虫搜索方向并提升全局搜索性能。实验结果表明，与FCSA、WSE等传统主题爬虫方法相比，该方法爬准率更高，并且能更快抓取到主题相关度高的网页。

关键词: 主题爬虫, 蚁群算法, 多目标优化, 暴雨灾害, 本体构建

Abstract: The traditional search engine based on keyword matching retrieval often fail to ensure the completeness and accuracy of the scraped data,but using the focused crawler method based on semantic retrieval tend to deviate from the focuse and fall into local optimum.To solve the problem,this paper proposes a focused crawler method based on multi-objective ant colony optimization algorithm.The method constructs a focused crawler domain ontology and focused vector.Then whether the link is relevant to the focuse is determined based on the anchor text relevance of the link,the focused relevance of the Web page where the link is located and the focused relevance of the page pointed to by the link.The multi-objective optimization model for the focused relevance degree of the link is established.The ant colony algorithm based on multi-objective optimization is introduced into the link selection proess of the focused crawler,and the non-dominated sorting and the Nearest and Farthest Candidate Solution(NFCS) is adopted to select the Pareto optimal link in order to guide the search direction of the focused crawler and improve the global search performance.Experimental results show that compared with FCSA,WSE and other traditional focused crawler methods,the proposed method improves the completeness of scraped data and can capture the Web pages with high relevance to the focuse more quickly.

Key words: focused crawler, ant colony algorithm, multi-objective optimization, rainstorm disaster, ontology construction

中图分类号: 

• TP391.3