Abstract
The Levenberg Marquardt (LM) algorithm is one of the most effective algorithms in speeding up the convergence rate of the Artificial Neural Networks (ANN) with Multilayer Perceptron (MLP) architectures. However, the LM algorithm suffers the problem of local minimum entrapment. Therefore, we introduce several improvements to the Levenberg Marquardt algorithm by training the ANNs with meta-heuristic nature inspired algorithm. This paper proposes a hybrid technique Accelerated Particle Swarm Optimization using Levenberg Marquardt (APSO_LM) to achieve faster convergence rate and to avoid local minima problem. These techniques are chosen since they provide faster training for solving pattern recognition problems using the numerical optimization technique.The performances of the proposed algorithm is evaluated using some bench mark of classification’s datasets. The results are compared with Artificial Bee Colony (ABC) Algorithm using Back Propagation Neural Network (BPNN) algorithm and other hybrid variants.Based on the experimental result, the proposed algorithms APSO_LM successfully demonstrated better performance as compared to other existing algorithms in terms of convergence speed and Mean Squared Error (MSE) by introducing the error and accuracy in network convergence.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
References
Zheng, H., Meng, W., Gong, B.: Neural Network and its Application on Machine fault Diagnosis. In: ICSYSE 1992, pp. 576–579 (1992)
Kosko, B.: Neural Network and Fuzzy Systems, 1st edn. Prentice Hall of India (1992)
Basheer, I.A., Hajmeer: Artificial Neural Networks: fundamentals, computing, design and application. Journal of Microbiological Methods 43(1), 3–31 (2000)
Krasnopolsky, V.M., Chevallier: Some Neural Network applications in environmental sciences. Part II, advancing computational efficiency of environmental numerical models. Neural Networks 16(3), 335–348 (2003)
Coppin, B.: Artificial Intelligence Illuminated. Jones and Bartlet illuminated Series, USA, pp. 291–324 (2004)
Nawi, N.M., Khan, A., Rehman, M.Z.: A new cuckoo search based levenberg-marquardt (CSLM) algorithm. In: Murgante, B., Misra, S., Carlini, M., Torre, C.M., Nguyen, H.-Q., Taniar, D., Apduhan, B.O., Gervasi, O. (eds.) ICCSA 2013, Part I. LNCS, vol. 7971, pp. 438–451. Springer, Heidelberg (2013)
Nawi, N.M., Khan, A., Rehman, M.Z.: A new back-propagation neural network optimized with cuckoo search algorithm. In: Murgante, B., Misra, S., Carlini, M., Torre, C.M., Nguyen, H.-Q., Taniar, D., Apduhan, B.O., Gervasi, O. (eds.) ICCSA 2013, Part I. LNCS, vol. 7971, pp. 413–426. Springer, Heidelberg (2013)
Contreras, J., et al.: ARIMA models to predict next-day electricity prices. IEEE Transactions on Power Systems 18, 1014–1020 (2003)
Leung, C.T., Chow, T.: A hybrid global learning algorithm based on global search and least squares techniques for back propagation networks. In: International Conference on Neural Networks, pp. 1890–1895 (1997)
Zhang, J.-R., et al.: A hybrid particle swarm optimization–back-propagation algorithm for feedforward neural network training. Applied Mathematics and Computation 185, 1026–1037 (2007)
Ahmed, W.A.M., Saad, E.S.M., Aziz, E.S.: Modified Back Propagation Algorithm for Learning Artificial Neural Networks. In: Proceedings of the Eighteenth National Radio Science Conference, pp. 345–352 (2001)
Nawi, N.M., et al.: Countering the Problem of Oscillations in Bat-BP Gradient Trajectory by Using Momentum. In: Proceedings of the First International Conference on Advanced Data and Information Engineering (DaEng 2013), pp. 103–110. Springer Singapore (2014)
Jin, W., et al.: The improvements of BP neural network learning algorithm. In: Proceedings of 5th International Conference on Signal Processing WCCC-ICSP 2000, vol. 3, pp. 1647–1649 (2000)
Nawi, N.M., Ransing, R.S., Salleh, M.N.M., Ghazali, R., Hamid, N.A.: An Improved Back Propagation Neural Network Algorithm on Classification Problems. In: Zhang, Y., Cuzzocrea, A., Ma, J., Chung, K.-i., Arslan, T., Song, X. (eds.) DTA and BSBT 2010. CCIS, vol. 118, pp. 177–188. Springer, Heidelberg (2010)
Mohd Nawi, N., et al.: BPGD-AG: A New Improvement Of Back-Propagation Neural Network Learning Algorithms With Adaptive Gain. Journal of Science and Technology 2 (2011)
Nawi, N.M., Ghazali, R., Salleh, M.N.M.: The Development of Improved Back-Propagation Neural Networks Algorithm for Predicting Patients with Heart Disease. In: Zhu, R., Zhang, Y., Liu, B., Liu, C. (eds.) ICICA 2010. LNCS, vol. 6377, pp. 317–324. Springer, Heidelberg (2010)
Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 5, 989–993 (1994)
Wilamowski, B., et al.: Neural network trainer with second order learning algorithms. In: 11th International Conference on Intelligent Engineering Systems, INES 2007, pp. 127–132 (2007)
Lourakis, M.I.: A brief description of the Levenberg-Marquardt algorithm implemented by levmar (2005)
Xue, Q., et al.: Improved LMBP algorithm in the analysis and application of simulation data. In: 2010 International Conference on Computer Application and System Modeling (ICCASM), vol. 6, pp. 545–547 (2010)
Yan, J., et al.: Levenberg-Marquardt algorithm applied to forecast the ice conditions in Ningmeng Reach of the Yellow River. In: Fifth International Conference on Natural Computation, ICNC 2009, pp. 184–188 (2009)
Ozturk, C., Karaboga, D.: Hybrid artificial bee colony algorithm for neural network training. In: 2011 IEEE Congress on Evolutionary Computation (CEC), pp. 84–88 (2011)
Nawi, A.K.N.M., Rehman, M.Z.: A New Levenberg-Marquardt based Back-propagation Algorithm trained with Cuckoo Search. In: ICEEI 2013 Procedia Technology 8C, pp. 18–24 (2013)
Bäck, T., Schwefel, H.-P.: An overview of evolutionary algorithms for parameter optimization. Evolutionary Computation 1, 1–23 (1993)
Blum, C., Socha, K.: Training feed-forward neural networks with ant colony optimization: An application to pattern classification. In: Fifth International Conference on Hybrid Intelligent Systems, HIS 2005, p. 6 (2005)
Karaboga, D., Akay, B., Ozturk, C.: Artificial bee colony (ABC) optimization algorithm for training feed-forward neural networks. In: Torra, V., Narukawa, Y., Yoshida, Y. (eds.) MDAI 2007. LNCS (LNAI), vol. 4617, pp. 318–329. Springer, Heidelberg (2007)
Montana, D.J., Davis, L.: Training Feedforward Neural Networks Using Genetic Algorithms. In: IJCAI, pp. 762–767 (1989)
Nawi, N.M., Rehman, M.Z., Khan, A.: A New Bat Based Back-Propagation (BAT-BP) Algorithm. In: Swiątek, J., Grzech, A., Swiątek, P., Tomczak, J.M. (eds.) Advances in Systems Science. AISC, vol. 240, pp. 395–404. Springer, Heidelberg (2014)
Zhang, J.Z.T.L.J., Lyu, M.: A hybrid particle swarm optimization back propagation algorithm for neural network training. Applied Mathematics and Computation 185, 1026–1037 (2007)
Yang, X.-S.: Nature-inspired metaheuristic algorithms. Luniver Press (2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Nawi, N.M., khan, A., Rehman, M.Z., Aziz, M.A., Herawan, T., Abawajy, J.H. (2014). An Accelerated Particle Swarm Optimization Based Levenberg Marquardt Back Propagation Algorithm. In: Loo, C.K., Yap, K.S., Wong, K.W., Teoh, A., Huang, K. (eds) Neural Information Processing. ICONIP 2014. Lecture Notes in Computer Science, vol 8835. Springer, Cham. https://doi.org/10.1007/978-3-319-12640-1_30
Download citation
DOI: https://doi.org/10.1007/978-3-319-12640-1_30
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-12639-5
Online ISBN: 978-3-319-12640-1
eBook Packages: Computer ScienceComputer Science (R0)