Abstract
Let x ≡ (x 1,…,x n )T ≥ 0 be a nonnegative n-dimensional column vector and p ≡ (p 1,…,p n )T > 0 be a positive n-dimensional column vector. With the convention of 0 ln 0 = 0, we define the quantity \(\sum _{j = 1}^n{x_j}\ln ({x_j}/{p_j})\) to be the cross-entropy of x with respect to p, in a general sense. Note that when x and p are both probability distributions, i.e., \(\sum _{j = 1}^n{x_j} = \sum _{j = 1}^n{p_j} = 1\) this quantity becomes the commonly defined cross-entropy between the two probability distributions (see Chapter 1).
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Agmon, N., Alhassid, Y., and Levine, R.D., “An Algorithm for Determining the Lagrange Parameters in the Maximal Entropy formalism,” The Maximum Entropy Formalism, edited by R. D. Levine and M. Tribus, MIT Press, Cambridge, Massachusetts, 1979, pp. 207–209.
Anas, A., “Discrete Choice Theory, Information Theory and the Multinomial Logit and Gravity Models,” Transportation Research Part B, Vol. 17B, 1983, pp. 13–23.
Bazaraa, M. S., and Shetty, C. M., Nonlinear Programming: Theory and Algorithms, John Wiley, New York, 1979.
Bellman, R.E., Mathematical Methods in Medicine, World Scientific, Singapore, 1983.
Ben-Tal A., and Charnes, A., “A Dual Optimization Framework for Some Problems of Information Theory and Statistics,” Problems of Control and Information Theory, Vol. 8, 1979, pp. 387–401.
Ben-Tal, A., Melman, A., and Zowe, J., “Curved Search Methods for Unconstrained Optimization,” Optimization, Vol. 21, 1990, pp. 669–695.
Ben-Tal, A., Teboulle, M., and Charnes, A., “The Role of Duality in Optimization Problems involving Entropy Functionals with Applications to Information Theory,” Journal of Optimization Theory and Applications, Vol. 58, 1988, pp. 209–223.
Bishop, Y.M.M., “Full Contingency Tables, Logits, and Split Contingency Tables,” Biometrika, Vol. 25, 1969, pp. 339–383.
Borwein, J.M., and Lewis, A.S., “Duality Relationships for Entropy-Like Minimization Problems,” SIAM Journal on Control and Optimization, Vol. 29, 1991, pp. 325–338.
Borwein, J.M., and Lewis, A.S., “Partially-Finite Programming in L 1 and the Existence of Maximum Entropy Estimates,” SIAM Journal on Optimization, Vol. 3, 1993, pp. 248–267.
Bregman, L.M., “The Relaxation Method of Finding the Common Point of Convex Sets and Its Application to the Solution of Problems in Convex Programming,” U.S.S.R. Computational Mathematics and Mathematical Physics, Vol. 7, 1967, pp. 200–217.
Censor, Y., “Row-Action Methods for Huge and Sparse Systems and Their Applications,” SIAM Review, Vol. 23, 1981, pp. 444–466.
Censor, Y., “On Linearly Constrained Entropy Maximization,” Linear Algebra and Its Applications, Vol. 80, 1986, pp. 191–195.
Censor, Y., Elfving, T., and Herman, G.T., “Special Purpose Algorithms for Linearly Constrained Optimization,” Maximum-Entropy and Bayesian Spectral Analysis and Estimation Problems, edited by C.R. Smith and G.J. Erickson, D. Reidel Publishing Company, 1987, pp. 241–254.
Censor, Y., and Lent, A., “An Iterative Row-Action Method for Interval Convex Programming,” Journal of Optimization Theory and Applications, Vol. 34, 1981, pp. 321–353.
Censor, Y., De Pierro, A.R., Elfving, T., Herman, G.T., and Iusem, A.N., “On Iterative Methods for Linearly Constrained Entropy Maximization,” Banach Center Publications, Vol. 24, 1990, pp. 145–163.
Darroch, J.N., and Ratcliff, D., “Generalized Iterative Scaling for Log-linear Models,” Annals of Mathematical Statistics, Vol. 43, 1972, pp. 1470–1480.
Dennis, J.E., and Schnabel, R.B., Numerical Methods for Unconstrained Optimization and Nonlinear Equations, Prentice Hall, Englewood Cliffs, New Jersey, 1983.
Dinkel, J.J., and Kochenberger, G.A., “Constrained Entropy Models: Solvability and Sensitivity,” Management Science, Vol. 25, 1979, pp. 555–564.
Dinkel, J.J., Kochenberger, G.A., and Wong, S.N., “Entropy Optimization and Geometric Programming,” Environment and Planning, Vol. 9, 1977, pp. 419–427.
Dinkel, J.J., and Wong, D.S., “External Zones in Trip Distribution Models: Characterization and Solvability,” Transportation Science, Vol. 18, 1984, pp. 253–266.
Elfving, T., “On Some Methods for Entropy Maximization and Matrix Scaling,” Linear Algebra and its Applications, Vol. 34, 1980, pp. 321–339.
Eriksson, J.R., “A Note on Solution of Large Sparse Maximum Entropy Problems with Linear Equality Constraints,” Mathematical Programming, Vol. 18, 1980, pp. 146–154.
Erlander, S., “Accessibility, Entropy and the Distribution and Assignment of Traffic,” Transportation Research, Vol. 11, 1977, pp. 149–153.
Erlander, S., “Entropy in Linear Programming,” Mathematical Programming, Vol. 21, 1981, pp. 137–151.
Fang, S.-C., “A New Unconstrained Convex Programming Approach to Linear Programming,” Zeitschrift fur Operations Research, Vol. 36, 1992, pp. 149–161.
Fang, S.-C., Peterson, E.L., and Rajasekera, J.R., “Minimum Cross-Entropy Analysis with Entropy-Type Constraints,” Journal of Computational and Applied Mathematics, Vol. 39, 1992, pp. 165–178.
Fang, S.-C., and Puthenpura, S., Linear Optimization and Extensions: Theory and Algorithms, Prentice Hall, Englewood Cliffs, New Jersey, 1993.
Fang, S.-C., and Rajasekera, J.R., “Quadratically Constrained Minimum Cross-Entropy Analysis,” Mathematical Programming, Vol. 44, 1989, pp. 85–96.
Fang, S.-C., and Tsao, H.-S.J., “Linear Programming with Entropic Perturbation,” Zeitschrift fur Operations Research, Vol. 37, 1993, pp. 171–186.
Fang, S.-C., and Tsao, H.-S.J., “A Quadratically Convergent Global Algorithm for the Linearly-Constrained Minimum Cross-Entropy Problem,” European Journal of Operational Research, Vol. 79, 1994, pp. 369–378.
Fang, S.-C., and Wu, S.-Y., “An Inexact Approach to Solving Linear Semi-Infinite Programming Problems,” Optimization, Vol. 28, 1994, pp. 291–299.
Fiacco, A.V., and Kortanek, K.O., Semi-Infinite Programming and Applications, Lecture Notes in Economics and Mathematical Systems No. 215, Springer-Verlag, New York, 1983.
Gordon, R., Bender, R., and Herman, G.T., “Algebraic Reconstruction Techniques (ART) for Three Dimensional Electron Microscopy and X-ray Photography,” Journal of Theoretical Biology, Vol. 29, 1970, pp. 471–481.
Grandy, W.T. Jr., and Schick, L.H., editors, Proceedings of the 10th International Workshop on Maximum Entropy and Bayesian Methods, Kluwer Academic Publishers, Dordrecht, The Netherlands, 1991.
Guiasu, S., Information Theory with Applications, McGraw-Hill, New York, 1977.
Guiasu, S., “Maximum Entropy Condition in Queueing Theory,” Journal of Operational Research Society, Vol. 37, 1986, pp. 293–301.
Hoshino, K., Kumar, V., Kumar, U., “On Optimizing of Generalized Iterative Scaling Method,” Working Paper No. 89-22, School of Business, Carleton University, Ontario, Canada, 1989.
Jan, G.W., and Fang, S.-C., “A New Variant of the Primal Affine Scaling Algorithm for Linear Programs,” Optimization, Vol. 22, 1991, pp. 681–715.
Jaynes, E.T., “Information Theory and Statistical Mechanics II,” Physics Review, Vol. 108, 1957, pp. 171–190.
Jefferson, T.R., and Scott, C.H., “The Analysis of Entropy Models with Equality and Inequality Constraints,” Transportation Research, Vol. 138, 1979, pp. 123–132.
Jornsten, K.O., and Lundgren, J.T., “An Entropy-Based Model Split Model,” Transportation Research Part B, Vol. 23B, 1989, pp. 345–359.
Kapur, J.N., Maximum-Entropy Models in Science and Engineering, Wiley Eastern Ltd., New Delhi, 1989.
Kortanek, K.O., Potra, F., and Ye, Y., “On Some Efficient Interior Point Methods for Nonlinear Convex Programming,” Linear Algebra and Its Applications, Vol. 152, 1991, pp. 169–189.
Kumar, V., Hoshino, K., and Kumar, U., “An Application of the Entropy Maximization Approach in Shopping Area Planning,” International Journal of General Systems, Vol. 16, 1989, pp. 25–42.
Kullback, S., Information and Statistics, John Wiley, New York, 1959.
Kullback, S., and Libler, R.A., “On Information and Sufficiency,” Annals of Mathematical Statistics, Vol. 22, 1951, pp. 79–86.
Lamond, B., and Stewart, N.F., “Bregman’s Balancing Method,” Transportation Research Part B, Vol. 15B, 1981, pp. 239–248.
Lent, A., “Maximum Entropy and MART,” in Image Analysis and Evaluation, SPSE Conference Proceedings, edited by R. Shaw, Toronto, Canada, 1976, pp. 249–257.
Lent, A., “A Convergent Algorithm for Maximum Entropy Image Restoration with a Medical X-Ray Application,” in Image Analysis and Evaluation, edited by R. Shaw, Society of Photographic Scientists and Engineers (SPSE), Washington, D.C., 1977, pp. 249–257.
Luenberger, D.G., Linear and Nonlinear Programming, 2nd Edition, Addison-Wesley, Reading, Massachusetts, 1984.
Minoux, M., Mathematical Programming, Wiley-Interscience, New York, 1986.
Peterson, E.L., “Geometric Programming,” SIAM Review, Vol. 19, 1976, pp. 1–45.
De Pierro, A.R. and Iusem, A.N., “A Relaxed version of Bregman’s Method for Convex Programming,” Journal of Optimization Theory and Applications, Vol. 51, 1986, pp. 421–440.
Potra, F., and Ye, Y., “A Quadratically Convergent Polynomial Algorithm for Solving Entropy Optimization Problems,” SIAM Journal on Optimization, Vol. 3, 1993, pp. 843–860.
Rockafellar, R.T., Convex Analysis, Princeton University Press, Princeton, New Jersey, 1970.
Safwat, K.N.A., and Magnanti, T.L., “A Combined Trip Generation, Trip Distribution, Modal Split, and Trip Assignment Model,” Transportation Science, Vol. 22, 1988, pp. 14–30.
Shore, J.E., “Minimum Cross-entropy Spectral Analysis,” IEEE Transactions on Acoustics, Speech, and Signal Processing, Vol. ASSP-29, 1981, pp. 230–237.
Sheu, R.L., and Fang, S.-C., “On the Generalized Path-Following Methods for Linear Programming,” Optimization, Vol. 30, 1994, pp. 235–249.
Smith, C.R., and Grandy, W.T., editors, Maximum-Entropy and Bayesian Methods in Inverse Problems, D. Reidel Publishing Company, Dordrecht, Holland, 1985.
Todd, M.J., and Ye, Y., “A Centered Projective Algorithm for Linear Programming,” Mathematics of Operations Research, Vol. 15, 1990, pp. 508–529.
Tomlin, J.A., “A Mathematical Programming Model for the Combined Distribution-Assignment of Traffic,” Transportation Science, Vol. 5, 1971, pp. 122–140.
Tomlin, J.A., and Tomlin, S.G., “Traffic Distribution and Entropy,” Nature, Vol. 220, 1968, pp. 974–976.
Tsao, H.-S.J., and Fang, S.-C., “Linear Programming with Inequality Constraints via Entropic Perturbation,” International Journal of Mathematics and Mathematical Sciences, Vol. 19, 1996, pp. 177–184.
Tsao, H.-S.J., Fang, S.-C., and Lee, D.N., “On the Optimal Entropy Analysis,” European Journal of Operational Research, Vol. 59, 1992, pp. 324–329.
Tsao, H.-S.J., Fang, S.-C., and Lee, D.N., “A Bayesian Interpretation of the Linearly-Constrained Cross-Entropy Minimization Problem,” Engineering Optimization, Vol. 22, 1993, pp. 65–75.
Wilson, A.G., “A Statistical Theory of Spatial Distribution Models,” Transportation Research, Vol. 1, 1967, pp. 253–269.
Wu, J.-S., and Chan, W.C., “Maximum Entropy Analysis of Multiple-Server Queueing Systems,” Journal of Operational Research Society, Vol. 40, 1989, pp. 815–826.
Zhang, J., and Brockett, P.L., “Quadratically Constrained Information Theoretic Analysis,” SIAM Journal on Applied Mathematics, Vol. 47, 1987, pp. 871–885.
Zhu, J., and Ye, Y., “A Path-Following Algorithm for a Class of Convex Programming Problems,” Working Paper No. 90-14, College of Business Administration, The University of Iowa, Iowa City, Iowa, 1990.
Author information
Authors and Affiliations
Rights and permissions
Copyright information
© 1997 Springer Science+Business Media New York
About this chapter
Cite this chapter
Fang, SC., Rajasekera, J.R., Tsao, HS.J. (1997). Entropy Optimization Methods: Linear Case. In: Entropy Optimization and Mathematical Programming. International Series in Operations Research & Management Science, vol 8. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-6131-6_3
Download citation
DOI: https://doi.org/10.1007/978-1-4615-6131-6_3
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-7810-5
Online ISBN: 978-1-4615-6131-6
eBook Packages: Springer Book Archive