一种基于云计算的并行流生成方法

doi:10.3969/j.issn.1000-3428.2013.10.003

摘要/Abstract

摘要： 在高速网络中，网络设备的分组转发性能在打开流采集功能后会受其影响。为此，提出一种在网络设备外基于云计算平台的并行流生成方法。在需要监测的网络设备附近部署小型云，把分组流量复制到云中，采用云的Map-Reduce并行处理框架快速地从海量分组数据中生成流记录。设计基于Map-Reduce框架的并行流生成算法，通过配置合适数量的云节点，可分析任意大小的网络流量。用实际网络分组数据对并行流生成方法的性能进行验证，实验结果表明，在由3台、5台或7台节点构成的小型云平台上，从超过40 GB的文本分组数据中共提取了15 160 052条流，与顺序处理相比，耗费时间至少可减小85%、90%和94%。

关键词: 网络流量分析, 并行处理, Map-Reduce框架, Hadoop平台

Abstract: In high speed networks, the forwarding performance of a network device which captures flows is inevitably degraded. A parallel flow generating approach based on small size cloud computing platform outside of network device is proposed. A small size cloud can be deployed near the selected network device being monitored, and packet traffic through the network device is copied into the cloud. Then flow records can be extracted fast from the large volume packet traffic by the parallel Map-Reduce framework of cloud. A parallel flow extraction algorithm based on Map-Reduce framework is proposed. In addition, cloud scale is flexible to expand, and by configuring proper cloud workers, this approach can be adopted to analyze network traffic in arbitrary size. Real network packet traffic is used to verify the performance of the parallel flow generating approach. Analysis results show that in clouds consisting of 3, 5 or 7 worker nodes, 15 160 052 flows are recognized from more than 40 GB text packet data, and comparing against sequentially processing method, the consumed time are reduced by more than 85%, 90% and 94%, respectively.

Key words: network traffic analysis, parallel processing, Map-Reduce framework, Hadoop platform

中图分类号:

TP393.02

孙韩林. 一种基于云计算的并行流生成方法[J]. 计算机工程, doi: 10.3969/j.issn.1000-3428.2013.10.003.

SUN Han-lin. A Parallel Flow Generating Approach Based on Cloud Computing[J]. Computer Engineering, doi: 10.3969/j.issn.1000-3428.2013.10.003.

参考文献

参考文献 [1] 陈亮, 龚俭. 基于NetFlow 记录的高速应用流量分类方法[J]. 通信学报, 2012, 33(1): 145-152. [2] Tran Q A, Jiang F, Hu Jiankun. A Real-time NetFlow-based Intrusion Detection System with Improved BBNN and High-frequency Field Programmable Gate Arrays[C]//Proc. of the 11th International Conference on Trust, Security and Privacy in Computing and Communications. Liverpool, UK: IEEE Press, 2012. [3] 郑建忠, 周世杰, 王娟. 基于NetFlow的动态K层特征模型库建立[J]. 计算机工程, 2010, 36(22): 165-167. [4] 许博, 陈鸣, 胡超, 等. 基于NetFlow的P2P流分析系统[J]. 北京邮电大学学报, 2010, 33(2): 94-98. [5] Ye Wei, Luo Xuan, Xie Rui, et al. Combining sFlow and Tracker Traffic Analysis: A Novel Estimation Approach for Network-wide BitTorrent Distribution[C]//Proc. of the 13th Asia-Pacific Network Operations and Management Sympo- sium. Taipei, China: [s. n.], 2011. [6] Zhu Haiting, Zhang Xiaoguo, Ding Wei. Research on Errors of Utilized Bandwidth Measured by NetFlow[C]//Proc. of the 2nd International Conference on Networking and Distributed Computing. Beijing, China: [s. n.], 2011. [7] Lee M, Duffield N, Kompella R R. Opportunistic Flow-level Latency Estimation Using Consistent NetFlow[J]. IEEE/ACM Transactions on Networking, 2012, 20(1): 139-152. [8] 高磊, 杨家海, 张辉, 等. 基于NetFlow 的网络流量测量基础设施建设[J]. 广西大学学报: 自然科学版, 2011, 36(z1): 78-82. [9] Liu Yuhui, Sun Jinshan, Sun Rui, et al. Next Generation Internet Traffic Monitoring System Based on NetFlow[C]// Proc. of International Conference on Intelligent System Design and Engineering Application. Changsha, China: [s. n.], 2010. [10] Deri L, Chou E, Cherian Z, et al. Increasing Data Center Network Visibility with Cisco NetFlow-Lite[C]//Proc. of the 7th International Conference on Network and Service Management. Parise, France: [s. n.], 2011. [11] Quittek J, Zseby T, Claise B, et al. Requirements for IP Flow Information Export(IPFIX)[S]. RFC 3917, 2004. [12] Moore A W, Zuev D. Internet Traffic Classification Using Bayesian Analysis Techniques[C]//Proc. of ACM SIGMETRICS International Conference on Measurements and Modeling of Computer Systems. Banff, Canada: [s. n.], 2005. [13] Dean J, Ghemawat S. MapReduce: Simplified Data Processing on Large Clusters[J]. Communications of the ACM, 2008, 51(1): 107-113. [14] CAIDA. The CAIDA UCSD Anonymized Internet Traces[EB/OL]. (2012-10-18). http://www.caida.org/data/passive/passive_2012_ dataset.xml. [15] Cisco Systems, Inc.. Introduction to Cisco IOS NetFlow[EB/OL]. (2012-05-01). http://www.cisco.com/en/US/prod/collateral/ iosswrel/ps6537/ps6555/ps6601/prod_white_paper0900aecd80406232.html. 编辑任吉慧