Abstract
Extreme learning machine (ELM) has been extensively researched for its fast training speed and powerful learning abilities. Entering the era of big data, large-scale learning tasks, the universality of noisy data and data with distributed storage pose considerable challenges to ELM. The outlier robust ELM (OR-ELM) is an important variant of ELM that dramatically improves the robustness of the model by introducing the \(\ell _1\)-norm in the error term. Nevertheless, the solution of OR-ELM is fully dense, which requires a large amount of storage space and computational resources for massive learning tasks. In this paper, we extended OR-ELM to the sparse and outlier robust ELM (SOR-ELM) based on the elastic-net theory that can simultaneously improve the sparsity and stability of the model. We also proposed a distributed version of SOR-ELM (DSOR-ELM) for handling data with distributed storage and large-scale learning tasks. In addition, an effective iterative algorithm, the alternating direction method of multipliers (ADMM), was employed to train our proposed models. Even though extending ADMM to multi-block issues is not straightforward, its convergence can still be ensured for training SOR-ELM and DSOR-ELM. Finally, extensive numerical experiments demonstrate the superiority of SOR-ELM and DSOR-ELM in training data with outliers and distributed learning environments.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Jordan MI, Mitchell TM (2015) Machine learning: trends, perspectives, and prospects. Science 349(6245):255–260
Biamonte J, Wittek P, Pancotti N, Rebentrost P, Wiebe N, Lloyd S (2017) Quantum machine learning. Nature 549(7671):195–202
Qiu J, Wu Q, Ding G, Xu Y, Feng S (2016) A survey of machine learning for big data processing. EURASIP J Adv Signal Process 2016(1):1–16
Yu Y, Si X, Hu C, Zhang J (2019) A review of recurrent neural networks: LSTM cells and network architectures. Neural Comput 31(7):1235–1270
Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: a comprehensive review. Neural Comput 29(9):2352–2449
Ding S, Zhang N, Zhang X, Wu F (2017) Twin support vector machine: theory, algorithm and applications. Neural Comput Appl 28(11):3119–3130
Xing H, Xiao Z, Zhan D, Luo S, Dai P, Li K (2022) Selfmatch: robust semisupervised time-series classification with self-distillation. Int J Intell Syst 37(11):8583–8610
Xing H, Xiao Z, Qu R, Zhu Z, Zhao B (2022) An efficient federated distillation learning system for multi-task time series classification. IEEE Trans Instrum Meas 71:1–12
Huang GB, Zhu QY, Siew CK (2004) Extreme learning machine: a new learning scheme of feedforward neural networks. In: 2004 IEEE international joint conference on neural networks (IEEE Cat. No. 04CH37541), vol 2, pp 985–990 . IEEE
Huang G-B, Zhu Q-Y, Siew C-K (2006) Extreme learning machine: theory and applications. Neurocomputing 70(1–3):489–501
Zhu S, Wang H, Lv H, Zhang H (2021) Augmented online sequential quaternion extreme learning machine. Neural Process Lett 53(2):1161–1186
Eshtay M, Faris H, Obeid N (2019) Metaheuristic-based extreme learning machines: a review of design formulations and applications. Int J Mach Learn Cybern 10(6):1543–1561
Perales-González C (2021) Global convergence of negative correlation extreme learning machine. Neural Process Lett 53(3):2067–2080
Albadr MAA, Tiun S (2020) Spoken language identification based on particle swarm optimisation-extreme learning machine approach. Circuits Syst Signal Process 39(9):4596–4622
Xu X, Deng J, Coutinho E, Wu C, Zhao L, Schuller BW (2018) Connecting subspace learning and extreme learning machine in speech emotion recognition. IEEE Trans Multimed 21(3):795–808
Ma J, Yang L, Wen Y, Sun Q (2020) Twin minimax probability extreme learning machine for pattern recognition. Knowl Based Syst 187:104806
Wang S-J, Chen H-L, Yan W-J, Chen Y-H, Fu X (2014) Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Process Lett 39(1):25–43
Chen BL, Shen YY, Zhu GC, Yu YT, Ji M (2022) An empirical mode decomposition fuzzy forecast model for COVID-19. Neural Process Lett 1–22
Jiang F, Zhu Q, Tian T (2022) Breast cancer detection based on modified Harris Hawks optimization and extreme learning machine embedded with feature weighting. Neural Process Lett 1–24
Martínez-Martínez JM, Escandell-Montero P, Soria-Olivas E, Martín-Guerrero JD, Magdalena-Benedito R, GóMez-Sanchis J (2011) Regularized extreme learning machine for regression problems. Neurocomputing 74(17):3716–3721
Li G, Niu P (2013) An enhanced extreme learning machine based on ridge regression for regression. Neural Comput Appl 22(3):803–810
Miche Y, Van Heeswijk M, Bas P, Simula O, Lendasse A (2011) TROP-ELM: a double-regularized ELM using LARS and Tikhonov regularization. Neurocomputing 74(16):2413–2421
Fakhr MW, Youssef ENS, El-Mahallawy MS (2015) L1-regularized least squares sparse extreme learning machine for classification. In: 2015 International conference on information and communication technology research (ICTRC), pp 222–225. IEEE
Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodol) 58(1):267–288
Efron B, Hastie T, Johnstone I, Tibshirani R (2004) Least angle regression. Ann Stat 32(2):407–499
Shi X, Kang Q, An J, Zhou M (2021) Novel l1 regularized extreme learning machine for soft-sensing of an industrial process. IEEE Trans Ind Inf 18(2):1009–1017
Zhang K, Luo M (2015) Outlier-robust extreme learning machine for regression problems. Neurocomputing 151:1519–1527
Wang Z, Sui L, Xin J, Qu L, Yao Y (2020) A survey of distributed and parallel extreme learning machine for big data. IEEE Access 8:201247–201258
Wang Y, Dou Y, Liu X, Lei Y (2016) PR-ELM: parallel regularized extreme learning machine based on cluster. Neurocomputing 173:1073–1081
Ming Y, Zhu E, Wang M, Ye Y, Liu X, Yin J (2018) DMP-ELMS: data and model parallel extreme learning machines for large-scale learning tasks. Neurocomputing 320:85–97
Bi X, Zhao X, Wang G, Zhang P, Wang C (2015) Distributed extreme learning machine with kernels based on mapreduce. Neurocomputing 149:456–463
Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113
Xin J, Wang Z, Qu L, Yu G, Kang Y (2016) A-ELM*: Adaptive distributed extreme learning machine with mapreduce. Neurocomputing 174:368–374
Zou H, Hastie T (2005) Regularization and variable selection via the elastic net. J R Stat Soc Ser B (Stat Methodol) 67(2):301–320
Boyd S, Parikh N, Chu E, Peleato B, Eckstein J, et al (2011) Distributed optimization and statistical learning via the alternating direction method of multipliers. Found Trends® Mach Learn 3(1):1–122
Zhang C, Li H, Chen C, Qian Y, Zhou X (2020) Enhanced group sparse regularized nonconvex regression for face recognition. IEEE Trans Pattern Anal Mach Intell
Li H, Zhang C, Jia X, Gao Y, Chen C (2021) Adaptive label correlation based asymmetric discrete hashing for cross-modal retrieval. IEEE Trans Knowl Data Eng
Li D, Tian Y (2018) Improved least squares support vector machine based on metric learning. Neural Comput Appl 30(7):2205–2215
Chen C, He B, Ye Y, Yuan X (2016) The direct extension of ADMM for multi-block convex minimization problems is not necessarily convergent. Math Program 155(1):57–79
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323:533–536
Marquardt DW (1963) An algorithm for least-squares estimation of nonlinear parameters. J Soc Ind Appl Math 11(2):431–441
Zhang Y, Dai Y, Wu Q (2022) An accelerated optimization algorithm for the elastic-net extreme learning machine. Int J Mach Learn Cybern 13(12):3993–4011
da Silva BLS, Inaba FK, Salles EOT, Ciarelli PM (2020) Outlier robust extreme machine learning for multi-target regression. Expert Syst Appl 140:112877
Luo M, Zhang L, Liu J, Guo J, Zheng Q (2017) Distributed extreme learning machine with alternating direction method of multiplier. Neurocomputing 261:164–170
Donoho DL (1995) De-noising by soft-thresholding. IEEE Trans Inf Theory 41(3):613–627
Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2:1–27
Dua D, Graff C (2017) UCI machine learning repository. http://archive.ics.uci.edu/ml
Torgo L (2017) Regression data sets. https://www.dcc.fc.up.pt/~ltorgo/Regression/DataSets.html
Yıldırım H, Revan Özkale M (2021) LL-ELM: a regularized extreme learning machine based on l_1-norm and Liu estimator. Neural Comput Appl 33(16):10469–10484
Lipu MSH, Hannan MA, Hussain A, Saad MH, Ayob A, Uddin MN (2019) Extreme learning machine model for state-of-charge estimation of lithium-ion battery using gravitational search algorithm. IEEE Trans Ind Appl 55(4):4225–4234
Guan L, Sun T, Qiao LB, Yang ZH, Li DS, Ge KS, Lu XC (2020) An efficient parallel and distributed solution to nonconvex penalized linear SVMs. Front Inf Technol Electron Eng 21(4):587–603
Funding
This paper is supported by the National Natural Science Foundation of China (Grant No. 12271479).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Zhang, Y., Dai, Y. & Wu, Q. Sparse and Outlier Robust Extreme Learning Machine Based on the Alternating Direction Method of Multipliers. Neural Process Lett 55, 9787–9809 (2023). https://doi.org/10.1007/s11063-023-11227-y
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-023-11227-y