Abstract
In a remarkable series of papers beginning in 1957, E. T. Jaynes (1957) began a revolution in inductive thinking with his principle of maximum entropy. He defined probability as a degree of plausibility, a much more general and useful definition than the frequentist definition as the limit of the ratio of two frequencies in some imaginary experiment. He then used Shannon’s definition of entropy and stated that in any situation in which we have incomplete information, the probability assignment which expresses all known information and is maximally non-committal with respect to all unknown information is that unique probability distribution with maximum entropy (ME). It is also a combinatorial theorem that the unique ME probability distribution is the one which can be realized in the greatest number of ways. The ME principle also provides the fairest description of our state of knowledge. When further information is obtained, if that information is pertinent then a new ME calculation can be performed with a consequent reduction in entropy and an increase in our total information. It must be emphasized that the ME solution is not necessarily the “correct” solution; it is simply the best that can be done with whatever data are available. There is no one “correct solution”, but an infinity of possible solutions. These ideas will now be made quite concrete and expressed mathematically.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Cox. R., (1974). Probability, Frequency and Reasonable Expectation, Am. J. Physics, 17, 1.
Cox, R., (1961). The Algebra of Probable Inference, Johns Hopkins University Press, Baltimore, MD.
Czuber, E., (1908) Wahrscheinlichkeitsrechnung.
Hobson, A. (1972). The Interpretation of Inductive Probabilities, J. Stat. Phys 6, 189.
Frieden, B. Roy (1985). Dice Entropy and Likelihood, Proc. IEEE 73, 1764.
Friedman K., (1973), Replies to Tribus and Motroni and to Gage and Hestenes, J. Stat. Phys 2, 265.
Friedman K. and A. Shimony, (1971). Jaynes Maximum Entropy Prescription and Probability Theory, J. Stat. Phys. 1, 193.
Gage, D. W. and D. Hestenes, (1973). Comments on the paper “Jaynes Maximum Entropy Prescription and Probability Theory”, J. Stat. Phys 7, 89.
Jaynes, E. T., (1957). Information Theory and Statistical Mechanics, Part I, Phys. Rev., 106, 620; Part II; ibid, 108,171.
Jaynes, E. T. (1963a), “Brandeis Lectures” in E. T. Jaynes Papers on Probability, Statistics and Statistical Physics, R. D. Rosenkrantz, Ed. D. Reidel Publishing Co., Boston, Mass.
Jaynes, E. T., (1968). “Prior Probabilities”, IEEE Trans Syst. Sci Cybern., SSC4, 227.
Jaynes, E. T., (1978). Where do we stand on Maximum Entropy, in the Maximum Entropy Formalism, R. D. Levine and M. Tribus,Editors, MIT Press, Cambridge, Mass.
Jaynes, E. T., (1979). “Concentration of Distributions at Entropy Maxima” in E. T. Jaynes: Papers on Probability, Statistics and Statistical Physics, R. D. Rosenkrntz, Ed., D. Reidel Publishing Co. Boston, Mass.
Jaynes E. T. (1982). “On The Rationale of Maximum — Entropy Methods”, Proc. IEEE, 70 939.
Keynes, J.M., (1952). A treatise on Probability. MacMillam & Co, London.
deLaplace, Pierre Simon, (1951). A Philosphical Essay on Probabilities, Dover, New York.
Makhoul, J. (1986), “Maximum Confusion Spectral Analysis”, Proc. Third ASSP Workshop on Spectrum Estimation and Modelling; Boston, Mass.
Macqueen J. and J. Marschak, (1975) “Partial Knowledge, Entropy and Estimation”, Proc. Nat. Acad. Sci., Vol 72, pp. 3819–3824.
Rowlinson, J. S., (1970). Probability, Information and Entropy, Nature 225, 1196.
Shannon, C. E. and W. Weaver, (1949). The Mathematical Theory of Communication, The University of Illinois Press: Urbana.
Shimony, A., (1973). Comment on the interpretation of inductive probabilities, J. Stat. Phys 9, 187.
Teubners, B. G., Sammlung Von Lehr Buchern Auf Dem Gebiete Der Mathematischen Wissenschaften, Band IX p. 149, Berlin.
Tribus, Myron (1961), Thermostatics and Thermodynamics, D Van Nostrand Co., Princeton, N. J.
Tribus, Myron (1969), Rational Descriptions, Decisions and Designs. Pergamon Press, Oxford.
Tribus, Myron and H. Motroni, (1977) Comments on the Paper, Jaynes Maximum Entropy Prescription and Probability Theory”, J. Stat. Phys 4, 227.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Additional information
To Ed Jaynes, who started it 30 years ago and whose clarity of exposition is an inspiration to us all.
Rights and permissions
Copyright information
© 1988 Kluwer Academic Publishers
About this chapter
Cite this chapter
Fougere, P.F. (1988). Maximum Entropy Calculations on a Discrete Probability Space. In: Erickson, G.J., Smith, C.R. (eds) Maximum-Entropy and Bayesian Methods in Science and Engineering. Fundamental Theories of Physics, vol 31-32. Springer, Dordrecht. https://doi.org/10.1007/978-94-009-3049-0_10
Download citation
DOI: https://doi.org/10.1007/978-94-009-3049-0_10
Publisher Name: Springer, Dordrecht
Print ISBN: 978-94-010-7871-9
Online ISBN: 978-94-009-3049-0
eBook Packages: Springer Book Archive