计算机工程 ›› 2018, Vol. 44 ›› Issue (11): 289-299.doi: 10.19678/j.issn.1000-3428.0048511

• 开发研究与工程应用 • 上一篇    下一篇

结合有监督广度优先搜索策略的通用垂直爬虫方法

高峰a,刘震a,b,高辉a,b   

  1. 电子科技大学 a.计算机科学与工程学院; b.大数据研究中心,成都 611731
  • 收稿日期:2017-09-01 出版日期:2018-11-15 发布日期:2018-11-15
  • 作者简介:高峰(1992—),男,硕士研究生,主研方向为数据库技术、数据挖掘;刘震,副教授、博士;高辉,教授。
  • 基金项目:

    国家自然科学基金(61300018)

General Vertical Crawler Method Combined with Supervised Breadth-First Search Strategy

GAO Fenga,LIU Zhena,b,GAO Huia,b   

  1. a.School of Computer Science and Engineering; b.Big Data Research Center,University of Electronic Science and Technology of China,Chengdu 611731,China
  • Received:2017-09-01 Online:2018-11-15 Published:2018-11-15

摘要:

垂直爬虫程序无法直接移植到其他网站并且程序设计需要大量人工干预。为此,提出一种高可移植性的通用型垂直爬虫设计方法。自动识别目标主题和目录页面URL,并利用URL聚类生成URL正则表达式过滤器,以解决垂直爬虫中需人工维护初始URL队列的问题。然后,利用正则表达式过滤器和解析路径模板以及有监督的广度优先与网页赋权搜索策略,实现相关页面的精确定位和数据的快速准确提取。实验结果表明,该方法能够对不同网站实现高效、快速、通用的数据爬取。

关键词: 垂直爬虫, URL聚类, 赋权网页, 路径模板解析, 有监督广度优先搜索策略

Abstract:

Aiming at the problem that the vertical crawler program can not be directly transplanted to other websites and the programming needs a lot of manual intervention,a highly portable and general vertical crawler design method is proposed.The target topic and directory page URL are automatically identified,and the URL regular expression filter is generated by URL clustering to solve the problem of manual maintenance of initial URL queue in vertical crawler.Using the regular expression filter,the parsing path template and the supervised breadth-first and Web page weighting search strategy,the accurate location of the relevant pages,the fast and accurate extraction of the data can be realized.Experimental results show that this method can achieve efficient,fast and universal data crawling for different websites.

Key words: vertical crawler, URL clustering, weighted Web page, parser of the path template, supervised breadth-first search strategy

中图分类号: