摘要： 当前基于单机系统的传输控制协议(TCP)报文重组工具随着数据量的增大,运行效率越来越低,难以满足大数据时代的需求。为此,提出一种基于Hadoop的海量数据TCP报文重组系统。利用Hadoop分布式文件系统(HDFS)以及MapReduce并行处理框架,对TCP报文重组中的数据读取和结果输出步骤进行并行化处理。采用MapReduce辅助排序技术实现海量TCP数据流的分类。测试结果表明,与基于单机系统的报文重组工具相比,基于Hadoop的报文重组系统工作效率更高,并且当数据量达到100 GB时,运行效率可提升近84%。

关键词: 海量数据, 传输控制协议, 报文重组, 分布式系统, 并行处理, 辅助排序

Abstract: Nowadays,Transfer Control Protocol(TCP) packet reassembly technologies based on stand-alone system are getting hard to satisfy the need of processing mass data,as the larger the data amount,the lower their efficiency.Aiming at this problem,a Hadoop-based TCP packet reassembly system is proposed.It takes advantages of Hadoop Distributed File System(HDFS) and the parallel processing framework of MapReduce to parallelize data reading and result outputting in TCP packet reassembly.It also takes advantages of the secondary sort technology in MapReduce to achieve TCP traffic classification.Test result shows that the Hadoop-based packet reassembly technology works more effectively than the packet reassembly tool based on stand-alone system.The operating efficiency can be improved by 84% when the size of the data reaches 100 GB.

Key words: mass data, Transfer Control Protocol(TCP), packet reassembly, distributed system, parallel processing, secondary sort

中图分类号: 

• TP393