摘要： 安全管理平台(SMP)是实现安全管理工作常态化运行的技术支撑平台,在实际应用中需要实时处理来自安全设备所产生的海量日志信息。为解决现有SMP 中海量日志查询效率低下的问题,设计基于云计算的SMP 日志存储分析系统。基于Hive 的任务转化模式,利用Hadoop 架构的分布式文件系统和MapReduce 并行编程模型,实现海量SMP 日志的有效存储与查询。实验结果表明,与基于关系数据的多表关联查询方法相比,该系统使得 SMP 日志的平均查询效率提高约90% ,并能加快SMP 集中管控的整体响应速度。

关键词: 安全管理平台, 云计算, Hadoop 架构, 海量日志, 并行处理

Abstract: With the development of cloud computing,massive data can be very easy to be stored and managed. Security Management Platform(SMP) is a support platform which realizes security management normalized operation. In a real application,this platform needs to process the massive information which generates from security device in real time.Considering the problem of low query efficiency,an efficient log analysis system based on the cloud computing for SMP is presented. It introduces the Hadoop distributed system infrastructure,and in the meantime,based on the study of transformation mission of the Hive,Hadoop Distributed File System(HDFS) and MapReduce are applied to effective storage and query of massive log. Experimental results show that,using proposed system can obtain a general increase in the query performance by about 90% compared with the existing Oracle storage method,and it can also further improve response speed of the SMP.

Key words: Security Management Platform ( SMP ), cloud computing, Hadoop architecture, massive log, parallel processing

中图分类号: 

• TP311