Summary
A class of numerical measures of informativity of observation channels or statistical experiments is defined by the aid off-divergences introduced by the author as measures of difference of two probability distributions. For observation channels with given prior probabilities, thef-informativity measures are generalizations of Shannon's mutual information and include Gallager's functionE 0(ϱQ) appearing in the derivation of error exponent for noisy channels, as well. For observation channels without prior probabilities, the suggested informativity measures have the geometric interpretation of a radius.
Thef-informativity defined for the Bayesian case shares several useful properties of the mutual information, such as e. g. the “data processing theorem”. Its maximum with respect to all possible prior distributions is shown by a minimax argument to be just thef-radius, thus the latter is a generalization of channel capacity. Thef-informativity measures can also be used to characterize the statistical sufficiency of indirect observations.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
U. Augustin, Error estimates for low rate codes,Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 14 (1969), 61–88.
D. Blackwell, Comparison of experiments,Proc. 2nd Berkeley Sympos. Math. Statist. and Probability, Vol. 1, Berkeley, 1951, 93–102.
D. Blackwell, Equivalent comparison of experiments,Ann. Math. Statist. 24 (1953), 265–272.
D. Blackwell andM. A. Girshick,Theory of Games and Statistical Decisions, New York, 1954.
L. L. Campbell, A coding theorem and Rényi's entropy,Information and Control 8 (1965), 423–429.
H. Chernoff, A measure of asymptotic efficiency for tests of a hypothesis based on a sum of observations,Ann. Math. Statist. 23 (1952), 493–507.
I. Csiszár, Eine Informationstheoretische Ungleichung und ihre Anwendung auf den Beweis der Ergodizität von Markoffschen Ketten,Magyar. Tud. Akad. Mat. Kutató Int. Közl. 8 (1963), 85–108.
I. Csiszár, Information-type measures of difference of probability distributions and indirect observations,Studia Sci. Math. Hungar. 2 (1967), 299–318.
R. G. Gallager,Information Theory and Reliable Communication, New York, 1968.
H. Heyer, Erschöpftheit und Invarianz beim Vergleich von Experimenten,Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 12 (1969), 21–55.
F. Jelinek, Buffer overflow in variable length coding of fixed rate sources,IEEE Trans. Information Theory 14 (1968), 490–501.
S. Kullback,Information Theory and Statistics, New York, 1959
D. V. Lindley, On a measure of information provided by an experiment,Ann. Math. Statist. 27 (1956), 986–1005.
N. Morse andR. Sackstedter, Statistical isomorphism,Ann. Math. Statist. 37 (1966), 203–214.
A. Perez, Sur la théorie de l'information dans le cas d'un alphabet abstrait,Trans. First Prague Confer. on Information Theory Statist. Decision Functions, Random Process, Prague, 1957, 209–243.
A. Perez, Information-theoretic risk estimates in statistical decision,Kybernetika (Prague)3 (1967), 1–21.
A. Rényi, On measures of entropy and information,Proc. 4th Berkeley Sympos. Math. Statist. and Probability, Vol. 1, Berkeley 1960, 547–561.
A. Rényi, On the amount of information concerning an unknown parameter in a sequence of observations,Magyar Tud. Akad. Mat. Kutató Int. Közl. 9 (1964), 617–624.
A. Rényi, On some problems of statistics from the point of view of information theory,Proc. Colloq. on Information Theory, Debrecen, 1967, 343–347.
R. Sackstedter, A note on statistical equivalence,Ann. Math. Statist. 38 (1967), 787–794.
R. Sibson, Information radius,Z. Wahrscheinlichkeitstheorie und Verw. Gebiete 14 (1969), 149–161.
F. Topsoe, An information theoretical identity and a problem involving capacity,Studia Sci. Math. Hungar. 2 (1967), 291–292.
I. Vajda, On the convergence of information contained in a sequence of observations,Proc. Colloq. on Information Theory, Vol. 2, Debrecen, 1967, 489–501.
I. Vajda, A contribution to the informational analysis of pattern,Proc. Fourth Hawaii Internat. Confer. on System Sci., Western Period. Co.
Author information
Authors and Affiliations
Additional information
Dedicated to the memory of Alfréd Rényi
This work was done while the author was visiting professor at The Catholic University of America, Washington, D.C., sponsored by National Science Foundation Grant No. GP-9396.
Rights and permissions
About this article
Cite this article
Csiszár, I. A class of measures of informativity of observation channels. Period Math Hung 2, 191–213 (1972). https://doi.org/10.1007/BF02018661
Received:
Issue Date:
DOI: https://doi.org/10.1007/BF02018661