Summary
Determining linear separability is an important way to understand structures present in data.We review the behavior of several classical descent procedures for determining linear separability and training linear classifiers in the presence of linearly nonseparable input. We compare the adaptive procedures to linear programming methods using many pairwise discrimination problems from a public database. We found that the adaptive procedures have serious implementational problems that make them less preferable than linear programming.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
M. Basu, Q. Liang. The fractional correction rule: a new perspective. Neural Network, 11, 1027–1039, 1998.
C. Blake, E. Keogh, C.J. Merz. UCI repository of machine learning databases [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Irvine, CA: University of California, Department of Information and Computer Science, 1998.
H.D. Block, S.A. Levin. On the boundedness of an iterative procedure for solving a system of linear inequalities. Proc. AMS, 26, 229–235, 1970.
K.P. Bennett, O.L. Mangasarian. Robust linear programming discrimination of two linearly inseparable sets. Optimization Methods and Software, 1, 23–24, 1992.
A.R. Butz. Perceptron type learning algorithms in nonseparable situations. Journal of Mathematical Analysis and Applications, 17, 560–576, 1967.
R.O. Duda, P.E. Hart. Pattern Classification and Scene Analysis. New York: Wiley, 1973.
R. Fourer, D.M. Gay, B.W. Kernighan. AMPL: A Modeling Language for Mathematical Programming. South San Francisco: The Scientific Press, 1993.
S.I. Gallant. Neural Network Learning & Expert Systems. Cambridge, MA: MIT Press, 1993.
F. Glover. Improved linear programming models for discriminant analysis. Decision Sciences, 21(4), 771–785, 1990.
R.G. Grinold. Comment on “Pattern Classification Design by Linear Programming”. IEEE Transactions on Computers, C-18(4), 378–379, April 1969.
R.G. Grinold. Mathematical programming methods of pattern classification. Management Science, 19(3), 272–289, 1972.
M.H. Hassoun, J. Song. Adaptive Ho-Kashyap rules for perceptron training. IEEE Transactions on Neural Networks, 3, 51–61, 1992.
Y.C. Ho, R.L. Kashyap. An algorithm for linear inequalities and its applications. IEEE Transactions on Electronic Computers, 14, 683–688, 1965.
S.-C. Huang, Y.-F. Huang. Bounds on the number of hidden neurons in multilayer perceptrons. IEEE Transactions on Neural Networks, 2, 47–55, 1991.
T.Y. Kwok, D.Y. Yeung. Objective functions for training new hidden units in constructive neural networks. IEEE Transactions on Neural Networks, 8(5), 1131–1148, 1997.
O.L. Mangasarian. Linear and nonlinear separation of patterns by linear programming. Operations Research, 13, 444–452, 1965.
O.L. Mangasarian. Nonlinear Programming, New York: McGraw-Hill, 1969.
C.H. Mays. Adaptive Threshold Logic. Ph.D. thesis, Stanford Electronics Labs, Stanford, CA, 1963.
M. Minsky, S. Papert. Perceptrons, expanded edition. Cambridge, MA: MIT Press, 1988.
B.A. Murtagh, M.A. Saunders. Large-scale linearly constrained optimization. Mathematical Programming, 14, 41–72, 1978.
N.J. Nilsson. The Mathematical Foundations of Learning Machines. San Mateo, CA: Morgan Kaufmann, 1990.
R. Reed. Pruning algorithms—a survey. IEEE Transactions on Neural Networks, 4, 740–747, 1993.
F. Rosenblatt. Principles of Neurodynamics: Perceptron and the Theory of Brain Mechanism. Washington, D.C.: Spartan Press, 1962.
V.P. Roychowdhury, K.Y. Siu, T. Kailath. Classification of linearly nonseparable patterns by linear threshold elements. IEEE Transactions on Neural Networks, 6(2), 318–331, March 1995.
M.A. Sartori, P.J. Antsaklis, A simple method to derive bounds on the size and to train multilayer neural networks. IEEE Transactions on Neural Networks, 2, 467–471, 1991.
K.Y. Siu, V.P. Roychowdhury, T. Kailath. Discrete Neural Computation. Englewood Cliffs, NJ: Prentice Hall, 1995.
F.W. Smith. Pattern classifier design by linear programming. IEEE Transactions on Computers, C-17(4), 367–372, April 1968.
S. Tamura, T. Masahiko. Capabilities of a four-layered feedforward neural network: four layer versus three. IEEE Transactions on Neural Networks, 8, 251–255, 1997.
J.T. Tou, R.C. Gonzalez. Pattern Recognition Principles. Reading, MA: Addison-Wesley, 1974.
A.W. Tucker. Dual systems of homogeneous linear relations. In H.W. Kuhn, A.W. Tucker, eds., Linear Inequalities and Related Systems. Annals of Mathematics Studies Number 38, Princeton, NJ: Princeton University Press, pages 3–18, 1956.
V.N. Vapnik, A.J. Chervonenkis. Theory of Pattern Recognition (in Russian). Nauka, Moscow, 1974; German translation: W.N. Wapnik, A.J. Tschervonenkis. Theorie der Zeichenerkennung. Berlin: Akademia, 1979.
V.N. Vapnik. Statistical Learning Theory. New York: Wiley, 1998.
B. Widrow, M.E. Hoff, Jr. Adaptive switching circuits. Tech. Report 1553-1, Stanford Electronics Labs, Stanford, CA, 1960.
B. Widrow, M.A. Lehr. 30 years of adaptive neural networks: perceptron, madeline, and backpropagation. Proceedings of the IEEE, 78, 1415–1442, 1990.
B. Widrow, S.D. Stearns. Adaptive Signal Processing. Englewood Cliffs, NJ: Prentice Hall, 1985.
M.H. Wright, Interior methods for constrained optimization. In A. Iserles, ed. Acta Numerica, pages 341–407, Cambridge: Cambridge University Press, 1992.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer Verlag London Limited
About this chapter
Cite this chapter
Basu, M., Ho, T.K. (2006). Linear Separability in Descent Procedures for Linear Classifiers. In: Basu, M., Ho, T.K. (eds) Data Complexity in Pattern Recognition. Advanced Information and Knowledge Processing. Springer, London. https://doi.org/10.1007/978-1-84628-172-3_4
Download citation
DOI: https://doi.org/10.1007/978-1-84628-172-3_4
Publisher Name: Springer, London
Print ISBN: 978-1-84628-171-6
Online ISBN: 978-1-84628-172-3
eBook Packages: Computer ScienceComputer Science (R0)