Improvement Algorithm of Levenberg-Marquardt Back Propagation for Saddle Point in Neural Network

doi:10.3969/j.issn.1000-3428.2012.23.043

Computer Engineering ›› 2012, Vol. 38 ›› Issue (23): 173-176,180. doi: 10.3969/j.issn.1000-3428.2012.23.043

• Networks and Communications • Previous Articles Next Articles

Improvement Algorithm of Levenberg-Marquardt Back Propagation for Saddle Point in Neural Network

LI Jiong-cheng, XIAO Heng-hui, LI Gui-yu

(Key Wireless Network Optimization Center of Guangzhou, Guangdong Planning and Designing Institute of Telecommunications Co., Ltd., Guangzhou 510630, China)

Received:2012-01-17 Online:2012-12-05 Published:2012-12-03

神经网络中处理鞍点的LMBP改进算法

李炯城，肖恒辉，李桂愉

(广东省电信规划设计院有限公司广州市无线网络优化重点工程中心，广州 510630)

作者简介:李炯城(1972－)，男，博士，主研方向：最优化算法，无线通信技术，软件系统架构；肖恒辉，博士；李桂愉，硕士
基金资助:
广东省教育部产学研结合基金资助项目(2009B090300393)；广州市软件(动漫)产业发展基金资助项目(2060404)

Abstract

Abstract: Aiming at the Levenberg-Marquardt Back Propagation(LMBP) algorithm of neural network sometimes converges to the saddle point during training process, an improved LMBP algorithm which can overcome the saddle point effectively is proposed. All eigenvectors for all the positive eigenvalue of Jacobi matrix are calculated as new searching directions. The improved LMBP algorithm is proved that it can get out of saddle point, and it iterates to minima effectively by an example of comparing with the traditional LMBP algorithm and the improved one.

Key words: neural network, Levenberg-Marquardt Back Propagation(LMBP) algorithm, saddle poin, Jacobi matrix, Hessian matrix, Gauss- Newton algorithm

摘要： 针对目前神经网络中的Levenberg-Marquardt反向传播(LMBP)算法在训练过程中有可能迭代到鞍点的问题，提出一种能有效克服鞍点的LMBP改进算法。计算鞍点处雅克比矩阵的正特征值对应的特征向量并将其作为新的搜索方向。通过实例对比传统LMBP算法与改进LMBP算法的效果，证明改进的算法能有效地脱离鞍点并进一步收敛到极小点处。

关键词: 神经网络, Levenberg-Marquardt反向传播算法, 鞍点, 雅克比矩阵, 海森矩阵, 高斯-牛顿法

CLC Number:

TP301.6

LI Jiong-Cheng, XIAO Heng-Hui, LI Gui-Tou. Improvement Algorithm of Levenberg-Marquardt Back Propagation for Saddle Point in Neural Network[J]. Computer Engineering, 2012, 38(23): 173-176,180.

李炯城, 肖恒辉, 李桂愉. 神经网络中处理鞍点的LMBP改进算法[J]. 计算机工程, 2012, 38(23): 173-176,180.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.3969/j.issn.1000-3428.2012.23.043

http://www.ecice06.com/EN/Y2012/V38/I23/173

References

[1] McCulloch W, Pitts W. A Practical Calculus of The Ideas Immanent in Nervous Activity[J]. Bulletin of Mathematical Biophysics, 1943, 5(4): 115-133.
[2] Yu Chien-Cheng, Liu Binda. A Backpropagation Algorithm with Adaptive Learning Rate and Momentum Coefficient[C]//Proc. of International Joint Conference on Neural Networks. [S. l.]: IEEE Press, 2002: 1218-1223.
[3] Bilski J, Rutkowski L. A Fast Training Algorithm for Neural Networks[J]. IEEE Trans. on Circuits and Systems II: Analog and Digital Signal Processing, 1998, 45(6): 749-753.
[4] Meybodi M R, Beigy H. A Note on Learning Automata-based Schemes for Adaptation of BP Parameters[C]//Proc. of the 2nd International Conference on Intelligent Data Engineering and Automated Learning, Data Mining, Financial Engineering, and Intelligent Agents. London, UK: Springer-Verlag, 2000: 145-149.
[5] Adawy M I, Aboul-Wafa M E, Keshk H A, et al. A Soft- backpropagation Algorithm for Training Neural Networks[C]//Proc. of the 19th National Radio Science Conference. [S. l.]: IEEE Press, 2002: 397-404.
[6] Chen Y J, Huang T C, Hwang R C. An Effective Learning of Neural Network by Using RFBP Learning Algorithm[J]. International Journal on Informatics and Computer Science, 2004, 167(1-4): 77-86.
[7] Hagan M T, Demuth H B, Beale M H. 神经网络设计[M]. 戴葵, 宋辉, 谭明峰, 等, 译. 北京: 机械工业出版社, 2006.
[8] 薛毅. 最优化原理与方法[M]. 北京: 北京工业大学出版社, 2008.
[9] Hagan M T, Menhaj M. Training Feedforward Networks with The Marquardt Algorithm[J]. IEEE Trans. on Neural Networks, 1994, 5(6): 526-535.
[10] 王晓明, 亓学广, 王霞. 改进LMBP算法收敛速度的方法研究及仿真[J]. 电测与仪表, 2008, 45(9): 21-24.
[11] 张妮, 田学民. LMBP神经网络改进算法的研究[J]. 计算机与应用化学, 2010, 27(8): 1111-1114.
[12] 李炯城, 黄汉雄. 神经网络中LMBP算法收敛速度改进的研究[J]. 计算机工程与应用, 2006, 42(16): 46-49.
[13] 原凌燕, 韩慧莲. 一种逃离局部极小点的LMBP算法[J]. 长治学院学报, 2007, 24(z1): 25-27.
[14] Hirsch M W, Smale S, Devaney R L. Differential Equations, Dynamical Systems, and an Introduction to Chaos[M]. Boston, USA: Boston Academic Press, 2004: 39-61.
[15] Golub G H, van Loan C F. Matrix Computations[M]. Baltimore, USA: Johns Hopkins University Press, 1996: 48-60.

Please choose a citation manager

Content to export

Improvement Algorithm of Levenberg-Marquardt Back Propagation for Saddle Point in Neural Network

神经网络中处理鞍点的LMBP改进算法

PDF

Knowledge

Cited

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments

[1]	Ping CAO, Huaizhi YANG, Yijun BO, Jia YOU, Chunjie ZHANG, Danyong LI. Low-quality Crack Image Classification with Multi-Knowledge Distillation [J]. Computer Engineering, 2023, 49(7): 204-213.
[2]	Mingchang BAI. Node Embedding Method Based on Folded Path Aggregation on Attributed Network [J]. Computer Engineering, 2023, 49(7): 76-84.
[3]	Yanxia JIN, Zhiru SHI, Jing YANG, Yabian LIU, Xingyu QIAO, Ling ZHANG. Research on Collision Detection Algorithm Between Cloth and Finely Modeled Object [J]. Computer Engineering, 2023, 49(7): 269-277.
[4]	Shihao ZHAO, Guojun MAO, Baoping XIONG, Shan HUANG, Jianghong LIN. Spatio-temporal Graph Mining Model Based on Graph Wavelet Convolutional Neural Network [J]. Computer Engineering, 2023, 49(7): 85-93.
[5]	LIAO Tao, SUN Haojie, ZHANG Shunxiang. Entity-Relation Joint Extraction Model Based on Span and Feature Fusion [J]. Computer Engineering, 2023, 49(6): 107-114.
[6]	YU Haiyang, JING Peng, ZHANG Wentao, XIE Saifei, HUA Zhihua, SONG Caoyuan. Improved U-Net Model for Road Crack Detection Based on Residual and Attention Mechanism [J]. Computer Engineering, 2023, 49(6): 265-273.
[7]	DAI Zuhua, LIU Yuanyuan, DI Shilong. Semantic Enhanced Aspect-Level Text Sentiment Analysis of Graph Neural Networks [J]. Computer Engineering, 2023, 49(6): 71-80.
[8]	SHEN Xueli, TIAN Guiyuan, JIANG Yanji, MA Linlin. Time-Frequency Domain Speech Enhancement Algorithm Based on Dual-Stage Conv-Transformer [J]. Computer Engineering, 2023, 49(6): 123-130.
[9]	DING Zixuan, YU Lei, ZHANG Juan, LI Xiang, WANG Xinyu. Image Super-Resolution Reconstruction Based on Depth Residual Adaptive Attention Network [J]. Computer Engineering, 2023, 49(5): 231-238.
[10]	OU Zhanhua, LI Cuiran, YANG Qian. Relay Selection Strategy for Energy Harvesting Wireless Sensor Network Based on ANN [J]. Computer Engineering, 2023, 49(5): 215-222,230.
[11]	CHEN Zhixu, JIN Yanxia, LU Ye, YANG Jing, LIU Yabian, SHI Zhiru. Multi-Precision Clothing Modeling Method Based on Subgraph Convolutional Neural Network [J]. Computer Engineering, 2023, 49(4): 174-181.
[12]	HENG Hongjun, MIAO Jing. Joint Extraction of Binary Tagging Entity Relation for Enhanced Semantic and Syntactic Information [J]. Computer Engineering, 2023, 49(4): 77-84.
[13]	YANG Jingjing, XIE Haiyan, XUE Nini, ZHANG Aoming. Research on Underwater Image Denoising Based on Dual-Channels Residual Network [J]. Computer Engineering, 2023, 49(4): 188-198.
[14]	AN Zhiguo, PENG Zheng, YI Mancheng, LIU Jianxin, YU Sifan. Filter Competition Training of Neural Networks [J]. Computer Engineering, 2023, 49(4): 120-124.
[15]	XU Kang, LI Fei, JI Donghong. Aspect Sentiment Triple Extraction by Combining Dependency Graph Convolution and Text Span Search [J]. Computer Engineering, 2023, 49(4): 61-67.

模态框（Modal）标题

Please choose a citation manager

Content to export

Improvement Algorithm of Levenberg-Marquardt Back Propagation for Saddle Point in Neural Network

神经网络中处理鞍点的LMBP改进算法

PDF

Knowledge

Cited

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

Comments