Abstract: As to the problems that a significant part of information can not be crawled effectively because of the dynamic Webs, a Web spider for the deep Web based on automated test tools called WatiJ is designed. The principle of using WatiJ to imitate users to submit query forms, continued next page is described, key steps of the crawling for the dynamic Webs are introduced by examples. Proved by the experiments, this spider is an effective one for crawling dynamic Webs in authorized data source.

Key words: dynamic webpage, automated test, Web spider

摘要： Deep Web中相当一部分内容因为动态网页存在而不能进行有效抓取。为此，设计并实现一种基于Web自动化测试工具——WatiJ的Deep Web网络蜘蛛。阐述利用WatiJ实现用户提交查询表单、循环点击翻页按钮等拟人交互方式的原理，通过实例给出动态网页抓取的关键步骤。实验结果表明，该蜘蛛是针对授权数据源进行动态网页抓取的一种有效解决方案。

关键词: 动态网页, 自动化测试, 网络蜘蛛

CLC Number: 

• N945