摘要： 为高效精确地提取存在于互联网中的灾害主题网页文本信息,引入本体语义,提出一种新的灾害主题爬虫策略。给出本体语义支持的灾害主题爬虫框架和流程,改进本体概念语义相似度计算方法,利用语义相似度计算主题语义向量,通过HTML位置加权获取网页文本特征向量,并进行主题相关度计算。设计URL锚文本主题相关度计算方法,分析URL链接优先度,优化爬行队列。选取地震灾害和气象灾害2个主题进行测试与分析,实验结果表明,该策略能有效提高稳定性和爬准率。

关键词: 主题爬虫, 本体, 语义相似度, 向量空间模型, 相关度计算, 锚文本

Abstract: This paper introduces ontology semantics and proposes a new strategy of disaster focused crawler to retrieve disaster theme webpages from the Internet efficiently and accurately.Firstly,the frame and process of disaster focused crawler are designed,and an improved ontology semantic similarity calculation method is proposed.Secondly,the thematic semantic vector is calculated based on semantic similarity,the webpage text feature vector is obtained based on HTML location weighting,and the thematic relevance is calculated.Then a relevance calculation method of URL anchor text is proposed,URL link priority is analyzed,and the crawling queue is optimized.Earthquake disaster and meteorologic disaster are selected to test and analyze,and the experimental results show that the proposed strategy can improve stability and accuracy.

Key words: focused crawler, ontology, semantic similarity, Vector Space Model(VSM), relevance calculation, achor text

中图分类号: 

• P208