作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2012, Vol. 38 ›› Issue (11): 284-286. doi: 10.3969/j.issn.1000-3428.2012.11.086

• 开发研究与设计技术 • 上一篇    下一篇

一种Deep Web爬虫爬行策略

刘 徽1,黄宽娜1,余建桥2   

  1. (1. 乐山师范学院数学与信息科学学院,四川 乐山 614004;2. 西南大学计算机与信息科学学院,重庆 400715)
  • 收稿日期:2011-07-25 出版日期:2012-06-05 发布日期:2012-06-05
  • 作者简介:刘 徽(1979-),男,讲师、硕士,主研方向:网络信息系统,模式识别;黄宽娜,讲师、硕士;余建桥,教授、博士
  • 基金资助:
    四川省教育厅科研基金资助项目(10ZB023)

Crawling Strategy of Deep Web Crawler

LIU Hui    1, HUANG Kuan-na    1, YU Jian-qiao    2   

  1. (1. College of Mathematics and Information Science, Leshan Normal University, Leshan 614004, China; 2. College of Computer and Information Science, Southwest University, Chongqing 400715, China)
  • Received:2011-07-25 Online:2012-06-05 Published:2012-06-05

摘要: Deep Web包含丰富的、高质量的信息资源,由于没有直接指向Deep Web页面的静态链接,目前大多搜索引擎不能发现这些页 面,只能通过填写表单提交查询获取。为此,提出一种Deep Web爬虫爬行策略。用网页分类器的分层结果指导链接信息提取器提取有前途的链接,将爬行深度限定在3层,从最靠近查询表单中提取链接,且只提取属于这3个层次的链接,从而减少爬虫爬行时间,提高爬虫的准确度,并设计聚焦爬行算法的约束条件。实验结果表明,该策略可以有效地下载Deep Web页面,提高爬行效率。

关键词: Deep Web页面, 反馈机制, 爬行策略, 聚焦爬虫, 网络数据库, 分类器

Abstract: Deep Web has rich, high-quality information resources. Because it does not have a static page links which is directly point to the Deep Web, most current search engines can not find these pages, but they can submit a query by filling out the form to obtain. This paper proposes a crawling strategy of Deep Web crawler. A hierarchical classification with the results of Web links guides the information extractor to extract the link. The strategy limits the crawling depth in three, extracts from the nearest link to the query form, and only extracts the links belonging to these three levels. Thus it reduces the reptiles crawling time, and improves the accuracy of reptiles. At the same time, it designs the Deep Web data sources for focused crawling algorithm. Experimental results show that this strategy can effectively download Deep Web pages to improve crawling efficiency.

Key words: Deep Web page, feedback mechanism, crawling strategy, focused crawler, network database, classifier

中图分类号: