Abstract
Recently some novel strategies have been proposed for neural network training that set randomly the weights from input to hidden layer, while weights from hidden to output layer are analytically determined by Moore-Penrose generalised inverse; such non-iterative strategies are appealing since they allow fast learning. Aim of this study is to investigate the performance variability when random projections are used for convenient setting of the input weights: we compare them with state of the art setting i.e. weights randomly chosen according to a continuous uniform distribution. We compare the solutions obtained by different methods testing this approach on some UCI datasets for both regression and classification tasks; this results in a significant performance improvement with respect to conventional method.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Rumellhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. In: Parallel Distrib. Process.: Exploration in the Microstructure of Cognition, vol. 1, pp. 318–362. MIT Press, Cambridge (1986)
LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.-R.: Efficient backProp. In: Orr, G.B., Müller, K.-R. (eds.) NIPS-WS 1996. LNCS, vol. 1524, pp. 9–50. Springer, Heidelberg (1998)
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., Bengio, Y.: An empirical evaluation of deep architectures on problems with many factors of variation. In: 24th ICML (2007)
Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: 25th ICML (2008)
Collobert, R., Weston, J.: A unified architecture for language processing: Deep neural networks with multitask learning. In: 25th ICML (2008)
Mnih, A., Hinton, G.E.: A scalable hierarchical distributed language model. In: 23rd NIPS, pp. 1081–1088 (2009)
Poggio, T., Girosi, F.: Networks for approximation and learning. IEEE 78(9), 1481–1497 (1990)
Cancelliere, R.: A High Parallel Procedure to Initialize the Output Weights of a Radial Basis Function or BP Neural Network. In: Sørevik, T., Manne, F., Moe, R., Gebremedhin, A.H. (eds.) PARA 2000. LNCS, vol. 1947, pp. 384–390. Springer, Heidelberg (2001)
Huang, G.-B., Zhu, Q.-Y., Siew, C.-K.: Extreme Learning Machine: Theory and applications. Neurocomputing 70, 489–501 (2006)
Halawa, K.: A method to improve the performance of multilayer perceptron by utilizing various activation functions in the last hidden layer and the least squares method. Neural Processing Letters 34, 293–303 (2011)
Nguyen, T.D., Pham, H.T.B., Dang, V.H.: An efficient Pseudo Inverse matrix-based solution for secure auditing. In: IEEE International Conference on Computing and Communication Technologies, Research, Innovation, and Vision for the Future (2010)
Kohno, K., Kawamoto, M., Inouye, Y.: A Matrix Pseudoinversion Lemma and Its Application to Block-Based Adaptive Blind Deconvolution for MIMO Systems. IEEE Transactions on Circuits and Systems I: Regular Papers 57(7), 1449–1462 (2010)
Ajorloo, H., Manzuri-Shalmani, M.T., Lakdashti, A.: Restoration of damaged slices in images using matrix pseudo inversion. In: 22nd International Symposium on Computer and Information Sciences (2007)
Wang, X.-Z., Wang, D., Huang, G.-B.: Special Issue on Extreme Learning Machines. Editorial. Soft Comput. 16(9), 1461–1463 (2012)
Wang, X.: Special Issue on Extreme Learning Machine with Uncertainty. Editorial. Int. J. Unc. Fuzz. Knowl. Based Syst. 21(supp. 02), v–vi (2013)
Arriaga, R.I., Vempala, S.: An algorithmic theory of learning: robust concepts and random projection. In: 40th Annual Symp. on Foundations of Computer Science, pp. 616–623. IEEE Computer Society Press (1999)
Vempala, S.: Random projection: a new approach to VLSI layout. In: 39th Annual Symp. on Foundations of Computer Science. IEEE Computer Society Press (1998)
Indyk, P., Motwani, R.: Approximate nearest neighbors: towards removing the curse of dimensionality. In: 30th Symp. on Theory of Computing, pp. 604–613. ACM (1998)
Penrose, R.: On best approximate solution of linear matrix equations. Proceedings of the Cambridge Philosophical Society 52, 17–19 (1956)
Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)
Badeva, V., Morosov, V.: Problemes incorrectements posès, thèorie et applications (in French). Masson, Paris (1991)
Cancelliere, R., De Luca, R., Gai, M., Gallinari, P., Artières, T.: Pseudoinversion for neural training: tuning the regularisation parameter. Technical report n. 149/13, Dep. of Computer Science, University of Turin (2013)
Tikhonov, A.N., Arsenin, V.Y.: Solutions of Ill-Posed Problems. Winston, Washington, DC (1977)
Tikhonov, A.N.: Solution of incorrectly formulated problems and the regularization method. Soviet Mathematics 4, 1035–1038 (1963)
Gallinari, P., Cibas, T.: Practical complexity control in multilayer perceptrons. Signal Processing 74, 29–46 (1999)
Poggio, T., Girosi, F.: Regularization algorithms that are equivalent to multilayer networks. Science 247, 978–982 (1990)
Girosi, F., Jones, M., Poggio, T.: Regularization theory and neural networks architectures. Neural Computation 7(2), 219–269 (1995)
Haykin, S.: Neural Networks, a comprehensive foundation. Prentice Hall, U.S.A. (1999)
Fuhry, M., Reichel, L.: A new Tikhonov regularization method. Numerical Algorithms 59, 433–445 (2012)
Hecht-Nielsen, R.: Context vectors: general purpose approximate meaning representations self-organized from raw data. In: Zurada, J.M., Marks II, R.J., Robinson, C.J. (eds.) Computational Intelligence: Imitating Life, pp. 43–56. IEEE Press (1994)
Bingham, E., Mannila, H.: Random projection in dimensionality reduction: Applications to image and text data. In: Conference on Knowledge Discovery and Data Mining, KDD 2001, San Francisco, CA, USA (2001)
Johnson, W.B., Lindenstrauss, J.: Extensions of Lipshitz mapping into Hilbert space. In: Conference in Modern Analysis and Probability. Contemporary Mathematics, vol. 26, pp. 189–206. Amer. Math. Soc. (1984)
Dasgupta, S., Gupta, A.: An elementary proof of the Johnson-Lindenstrauss lemma. Technical report TR-99-006, International Computer Science Institute, Berkeley, California, USA (1999)
Achlioptas, D.: Database-friendly random projections. In: ACM Symp. on the Principles of Database Systems, pp. 274–281 (2001)
Asuncion, A., Newman, D.J.: UCI Machine Learning Repository, University of California, Irvine, School of Information and Computer Sciences (2007), http://www.ics.uci.edu/~mlearn/MLRepository.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Rubini, L., Cancelliere, R., Gallinari, P., Grosso, A., Raiti, A. (2014). Computational Experience with Pseudoinversion-Based Training of Neural Networks Using Random Projection Matrices. In: Agre, G., Hitzler, P., Krisnadhi, A.A., Kuznetsov, S.O. (eds) Artificial Intelligence: Methodology, Systems, and Applications. AIMSA 2014. Lecture Notes in Computer Science(), vol 8722. Springer, Cham. https://doi.org/10.1007/978-3-319-10554-3_24
Download citation
DOI: https://doi.org/10.1007/978-3-319-10554-3_24
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-10553-6
Online ISBN: 978-3-319-10554-3
eBook Packages: Computer ScienceComputer Science (R0)