Improved Map-Reduce Model in Cloud Computing Environment

doi:10.3969/j.issn.1000-3428.2012.11.009

Abstract

Abstract: Map-Reduce model does not consider the computing performance in large number of heterogeneous nodes when allocating the input files in the cluster, and this leads to network data transfer increase to run the map task. Addressing this issue, this paper proposes the improved Map-Reduce model under the cloud computing environment. For a large number of nodes in the cluster calculation with different performance, it raises the maximum computational time minimal objective function model, uses Genetic Algorithm(GA) to solve the model to get the distribution plan. Simulation results prove the validity of the model.

Key words: cloud computing, Map-Reduce model, heterogeneous environment, calculated performance, Genetic Algorithm(GA)

摘要： Map-Reduce模型在分配输入文件时没有考虑集群中大量异构节点的计算性能，导致运行map任务时网络数据传送量增加。针对该问题，提出一种云计算环境下的改进型Map-Reduce模型。根据集群中大量节点计算性能不同的特点，采用最小化最大计算时间的目标函数进行建模，利用遗传算法求解该模型得到分配方案。仿真结果证明了该模型的有效性。

关键词: 云计算, Map-Reduce模型, 异构环境, 计算性能, 遗传算法

CLC Number:

TP338.8

LI Shen, DU Zhong-Jun-. Improved Map-Reduce Model in Cloud Computing Environment[J]. Computer Engineering, 2012, 38(11): 27-29,37.

李震, 杜中军. 云计算环境下的改进型Map-Reduce模型[J]. 计算机工程, 2012, 38(11): 27-29,37.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.3969/j.issn.1000-3428.2012.11.009

http://www.ecice06.com/EN/Y2012/V38/I11/27

References

[1] Foster I, Zhao Yong, Raicu I, et a1. Cloud Computing and Grid Computing 360-degree Compared[C]//Proc. of GCE’08. Washington D. C., USA: IEEE Computer Society, 2008.
[2] Ghemawat S, Gobioff H, Leung S. The Google File System[C]// Proc. of SOSP’03. New York, USA: ACM Press, 2003.
[3] Chang F, Dean J, Ghemawat S, et al. BigTable: A Distributed Storage System for Structured Data[J]. ACM Trans. on Computer Systems, 2008, 26(2): 1-26.
[4] Dean J, Ghemawat S. Map-Reduce: Simplified Data Processing on Large Clusters[C]//Proc. of the 6th Symposium on Operating System Design and Implementation. New York, USA: ACM Press, 2004. (下转第37页)
[5] Nurmi D, Wolski R, Grzegorczyk C, et a1. The Eucalyptus Open- source Cloud-computing System[C]//Proc. of the 9th IEEE/ACM International Symposium on Cluster Computing and the Grid. [S. l.]: IEEE Computer Society, 2009.
[6] Armbrust M, Fox A, Griffith R, et al. Abore the Clouds: Aberkeley View of Cloud Computing[R]. Berkeley, USA: University of California, Berkeley: Technical Report: UCB/EECS-2009-28, 2009.
[7] Yan Baoqiang, Rhodes P J. Toward Automatic Parallelization of Spatial Computation for Computing Clusters[C]//Proc. of the 17th International Symposium on High Performance Distributed Computing. New York, USA: ACM Press, 2008.
[8] Nagarajan A B, Mueller F, Engelmann C, et al. Proactive Fault Tolerance for HPC with Xen Virtualization[C]//Proc. of the 21st Annual International Conference on Supercomputing. New York, USA: ACM Press, 2007.
[9] Pinar A, Hendrickson B. Exploiting Flexibly Assignable Work to Improve Load Balance[C]//Proc. of the 14th Annual ACM Symposium on Parallel Algorithms. New York, USA: ACM Press, 2002.
[10] 刘怡, 张勘. 基于负载均衡和经验值的工作流任务分配策略[J]. 计算机工程, 2009, 35(21): 57-59.
[11] 陈国良. 遗传算法及其应用[M]. 北京: 人民邮电出版社, 1996.
[12] Liang Yanchun, Ge Hongwei, Zhou Chunguang. Solving Tra- veling Salesman Problem by Genetic Algorithms[J]. Progress in Natural Science, 2003, 13(2): 135-142.

[1]	BAI Zhixu, WANG Hengjun. Adversarial Example Generation Method Based on Improved Genetic Algorithm [J]. Computer Engineering, 2023, 49(5): 139-149.
[2]	SANG Yongxuan, WEI Jiangpo, WANG Bo, SONG Ying. Hybrid Heuristic Task Offloading Algorithm with Edge Caching Mechanism [J]. Computer Engineering, 2023, 49(4): 149-158.
[3]	LIU Zhibin, HUANG Qiulan, HU Qingbao, CHENG Yaodong, HU Yu, TIAN Haolai. Design and Implementation of Fine-Grained Scheduling Strategy for Kubernetes Heterogeneous Resources [J]. Computer Engineering, 2023, 49(2): 31-36,45.
[4]	XI Zhiwen, CAI Jingjing, YANG Wenmin, CHAI Zhilei. Concurrent Request Scheduling Mechanism for FPGA Cloud Platform Based on Microservice Architecture [J]. Computer Engineering, 2022, 48(7): 206-213.
[5]	HE Xiaowei, XU Jingjie, WANG Bin, WU Hao, ZHANG Bowen. Research on Cloud Computing Resource Load Forecasting Based on GRU-LSTM Combination Model [J]. Computer Engineering, 2022, 48(5): 11-17,34.
[6]	WU Tiezhou, ZOU Zhi, JIANG Ben, ZHANG Xiaoxing. Short-Term Load Forecasting Based on TLBGA-GRU Neural Network [J]. Computer Engineering, 2022, 48(11): 69-76.
[7]	DU Xiuli, ZHOU Min, Lü Yana, QIU Shaoming. Effectiveness Evaluation of Equipment Support System Based on Optimized RBF Neural Network [J]. Computer Engineering, 2021, 47(9): 282-287,296.
[8]	SHI Lingpeng, ZHU Zheng, ZHOU Junsong, LI Xin, LI Jing. Load Balancing Mechanism for Microservice Architecture in Cloud-based Systems [J]. Computer Engineering, 2021, 47(9): 44-50,58.
[9]	ZHAO Yufeng, LEI Sheng, ZHANG Guogang, GENG Yingsan. Design and Development of Container-based Cloud Platform for Power Equipment Simulation [J]. Computer Engineering, 2021, 47(9): 171-177,184.
[10]	WEI Xiuran, WANG Feng. Data Replication Strategy of Cloud Storage Based on Coordinator and Genetic Algorithm [J]. Computer Engineering, 2021, 47(8): 124-130,139.
[11]	ZHENG Juanyi, CUI Zhuo, SU Hailong, YIN Shuaishuai, LIU Yaoyao. Intelligent Method Based on Improved GA-Elman for Predicting the Wireless Propagation Loss [J]. Computer Engineering, 2021, 47(7): 155-160,167.
[12]	NI Siyuan, HU Hongchao, LIU Wenyan, LIANG Hao. Heterogeneous Cloud Resource Allocation Algorithm Based on Rotation Strategy [J]. Computer Engineering, 2021, 47(6): 44-51,67.
[13]	YE Peiwen, JIA Xiangdong, YANG Xiaorong, NIU Chunyu. Collaborative Edge and Cloud Offloading for Internet of Vehicles Using Multi-Agent Reinforcement Learning [J]. Computer Engineering, 2021, 47(4): 13-20.
[14]	YANG Tian, YANG Jun. Offloading Decision and Resource Allocation Strategy in Mobile Edge Computing [J]. Computer Engineering, 2021, 47(2): 19-25.
[15]	LI Lingshu, WU Jiangxing. SaaS Security Oriented Virtual Network Function Embedding Method Under Cloud-Network Integration [J]. Computer Engineering, 2021, 47(12): 30-39.

Please choose a citation manager

Content to export