计算机工程 ›› 2019, Vol. 45 ›› Issue (2): 184-190.doi: 10.19678/j.issn.1000-3428.0052035

• 人工智能及识别技术 • 上一篇    下一篇

基于网页空间进化算法的暴雨灾害主题爬虫策略

刘景发1,2,李新1,蒋盛益2   

  1. 1.南京信息工程大学 计算机与软件学院,南京 210044; 2.广东外语外贸大学 信息科学与技术学院,广州 510006
  • 收稿日期:2018-07-06 出版日期:2019-02-15 发布日期:2019-02-15
  • 作者简介:刘景发(1972—),男,教授、博士,主研方向为智能信息检索、智能计算、网络爬虫;李新(通信作者),硕士研究生;蒋盛益,教授、博士。
  • 基金项目:

    国家自然科学基金(61373016);国家社会科学基金重大招标项目(16ZDA047);江苏省自然科学基金(BK20171458,BK20181409)。

Focused Crawler for Rainstorm Disaster Strategy Based on Web Space Evolutionary Algorithm

LIU Jingfa  1,2,LI Xin 1,JIANG Shengyi 2   

  1. 1.College of Computer and Software,Nanjing University of Information Science and Technology,Nanjing 210044,China; 2.College of Information Science and Technology,Guangdong University of Foreign Studies,Guangzhou 510006,China
  • Received:2018-07-06 Online:2019-02-15 Published:2019-02-15

摘要:

针对单目标优化算法求解爬虫问题时难以获得最优加权因子和易于陷入局部最优的缺点,将多目标优化算法引入主题爬虫,提出一种基于多目标优化的网页空间进化算法。通过计算测试链接与种子链接库中链接的最短距离,将其与种子链接库中所有链接间的平均距离进行比较来更新种子链接库。针对多目标优化中Pareto最优解的选取问题,给出一种最近最远候选解法。实验结果表明,与宽度优先搜索等算法相比,该算法具有较高的爬准率和稳定性。

关键词: 多目标优化, 主题爬虫, 网页空间进化算法, Pareto最优, 暴雨灾害

Abstract:

Aiming at the shortcomings of single target optimization algorithm to solve the problem that the crawler problem is difficult to obtain the optimal weighting factor and easy to fall into the local optimum,the multi-objective optimization algorithm is introduced into the topic crawler,and a Web Space Evolution (WSE) algorithm based on multi-objective optimization is proposed.The seed link library is updated by calculating the shortest distance between the test link and the link in the seed link library,comparing it to the average distance of all links in the seed link library.Aiming at the selection of Pareto optimal solution in multi-objective optimization,a recent farthest candidate solution is proposed.Experimental results show that compared with the algorithm of breadth-first search,the algorithm has high tracking rate and stability.

Key words: multi-objective optimization, focused crawler, Web Space Evolutionary(WSE) algorithm, Pareto optimal, rainstorm disaster

中图分类号: