Abstract
The past century has seen a steady increase in the need of estimating and predicting complex systems and making (possibly critical) decisions with limited information. Although computers have made possible the numerical evaluation of sophisticated statistical models, these models are still designed by humans because there is currently no known recipe or algorithm for dividing the design of a statistical model into a sequence of arithmetic operations. Indeed enabling computers to think as humans, especially when faced with uncertainty, is challenging in several major ways: (1) Finding optimal statistical models remains to be formulated as a well-posed problem when information on the system of interest is incomplete and comes in the form of a complex combination of sample data, partial knowledge of constitutive relations and a limited description of the distribution of input random variables. (2) The space of admissible scenarios along with the space of relevant information, assumptions, and/or beliefs, tends to be infinite dimensional, whereas calculus on a computer is necessarily discrete and finite. With this purpose, this paper explores the foundations of a rigorous framework for the scientific computation of optimal statistical estimators/models and reviews their connections with decision theory, machine learning, Bayesian inference, stochastic optimization, robust optimization, optimal uncertainty quantification, and information-based complexity.
Similar content being viewed by others
References
Richardson, L.F.: Weather Prediction by Numerical Process. Cambridge Mathematical Library. Cambridge University Press, Cambridge (1922)
Ackerman, N.L., Freer, C.E., Roy, D.M.: On the computability of conditional probability. arXiv:1005.3014 (2010)
Adams, M., Lashgari, A., Li, B., McKerns, M., Mihaly, J.M., Ortiz, M., Owhadi, H., Rosakis, A.J., Stalzer, M., Sullivan, T.J.: Rigorous model-based uncertainty quantification with application to terminal ballistics. Part II: systems with uncontrollable inputs and large scatter. J. Mech. Phys. Solids 60(5), 1002–1019 (2012)
Aliprantis, C.D., Border, K.C.: Infinite Dimensional Analysis: A Hitchhiker’s Guide, 3rd edn. Springer, Berlin (2006)
Anderson, T.W.: The integral of a symmetric unimodal function over a symmetric convex set and some probability inequalities. Proc. Am. Math. Soc. 6(2), 170–176 (1955)
Belot, G.: Bayesian orgulity. Philos. Sci. 80(4), 483–503 (2013)
Ben-Tal, A., El Ghaoui, L., Nemirovski, A.: Robust Optimization. Princeton Series in Applied Mathematics. Princeton University Press, Princeton (2009)
Ben-Tal, A., Hochman, E.: More bounds on the expectation of a convex function of a random variable. J. Appl. Probab. 9, 803–812 (1972)
Ben-Tal, A., Nemirovski, A.: Robust convex optimization. Math. Oper. Res. 23(4), 769–805 (1998)
Bentkus, V.: A remark on the inequalities of Bernstein, Prokhorov, Bennett, Hoeffding, and Talagrand. Liet. Mat. Rink. 42(3), 332–342 (2002)
Bentkus, V.: On Hoeffding’s inequalities. Ann. Probab. 32(2), 1650–1673 (2004)
Bentkus, V., Geuze, G.D.C., van Zuijlen, M.C.A.: Optimal Hoeffding-like inequalities under a symmetry assumption. Statistics 40(2), 159–164 (2006)
Bernstein, S.N.: Collected Works. Izdat. “Nauka”, Moscow (1964)
Bertsimas, D., Brown, D.B., Caramanis, C.: Theory and applications of robust optimization. SIAM Rev. 53(3), 464–501 (2011)
Bertsimas, D., Popescu, I.: Optimal inequalities in probability theory: a convex optimization approach. SIAM J. Optim. 15(3), 780–804 (electronic) (2005)
Birge, J.R., Wets, R.J.-B.: Designing approximation schemes for stochastic optimization problems, in particular for stochastic programs with recourse. Math. Prog. Stud. 27, 54–102 (1986)
Blackwell, D.: Equivalent comparisons of experiments. Ann. Math. Stat. 24(2), 265–272 (1953)
Bogachev, V.I.: Measure Theory, vol. II. Springer, Berlin (2007)
Boţ, R.I., Lorenz, N., Wanka, G.: Duality for linear chance-constrained optimization problems. J. Korean Math. Soc. 47(1), 17–28 (2010)
Boucheron, S., Lugosi, G., Massart, P.: A sharp concentration inequality with applications. Random Struct. Algorithms 16(3), 277–292 (2000)
Brown, L.D.: Minimaxity, more or less. In: Gupta, S.S., Berger, J.O. (eds.) Statistical Decision Theory and Related Topics V, pp. 1–18. Springer, New York (1994)
Brown, L.D.: An essay on statistical decision theory. J. Am. Stat. Assoc. 95(452), 1277–1281 (2000)
Castillo, I., Nickl, R.: Nonparametric Bernstein–von Mises theorems in Gaussian white noise. Ann. Stat. 41(4), 1999–2028 (2013)
Chen, W., Sim, M., Sun, J., Teo, C.-P.: From CVaR to uncertainty set: implications in joint chance-constrained optimization. Oper. Res. 58(2), 470–485 (2010)
Daley, D.J., Vere-Jones, D.: An Introduction to the Theory of Point Processes.: General Theory and Structure. Probability and its Applications (New York), vol. II, 2nd edn. Springer, New York (2008)
Dantzig, G.B.: Linear programming under uncertainty. Manag. Sci. 1, 197–206 (1955)
Diaconis, P., Freedman, D.A.: On the consistency of Bayes estimates. Ann. Stat. 14(1), 1–67 (1986). With a discussion and a rejoinder by the authors
Doob, J.L.: Application of the theory of martingales. In: Le Calcul des Probabilités et ses Applications, Colloques Internationaux du Centre National de la Recherche Scientifique, vol. 13, pp. 23–27. Centre National de la Recherche Scientifique, Paris (1949)
Doob, J.L.: Measure Theory. Graduate Texts in Mathematics, vol. 143. Springer, New York (1994)
Drenick, R.F.: Aseismic design by way of critical excitation. J. Eng. Mech. Div. Am. Soc. Civ. Eng. 99(4), 649–667 (1973)
Dubins, L.E.: On extreme points of convex sets. J. Math. Anal. Appl. 5(2), 237–244 (1962)
Dudley, R.M.: Real Analysis and Probability. Cambridge Studies in Advanced Mathematics, vol. 74. Cambridge University Press, Cambridge (2002). Revised reprint of the 1989 original
Dvoretzky, A., Wald, A., Wolfowitz, J.: Elimination of randomization in certain statistical decision procedures and zero-sum two-person games. Ann. Math. Stat. 22(1), 1–21 (1951)
Edmundson, H.P.: Bounds on the expectation of a convex function of a random variable. Technical report, DTIC Document (1957)
Elishakoff, I., Ohsaki, M.: Optimization and Anti-optimization of Structures Under Uncertainty. World Scientific, London (2010)
Ermoliev, Y., Gaivoronski, A., Nedeva, C.: Stochastic optimization problems with incomplete information on distribution functions. SIAM J. Control Optim. 23(5), 697–716 (1985)
Fisher, R.: The Design of Experiments. Oliver and Boyd, Edinburgh (1935)
Fisher, R.: Statistical methods and scientific induction. J. R. Stat. Soc. Ser. B. 17, 69–78 (1955)
Fisher, R.A.: On the mathematical foundations of theoretical statistics. Philos. Trans. R. Soc. Lond. Ser. A 222, 309–368 (1922)
Fisher, R.A.: “Student”. Ann. Eugen. 9(1), 1–9 (1939)
Frauendorfer, K.: Solving SLP recourse problems with arbitrary multivariate distributions-the dependent case. Math. Oper. Res. 13(3), 377–394 (1988)
Freedman, D.A.: On the asymptotic behavior of Bayes’ estimates in the discrete case. Ann. Math. Stat. 34, 1386–1403 (1963)
Freedman, D.A.: On the Bernstein-von Mises theorem with infinite-dimensional parameters. Ann. Stat. 27(4), 1119–1140 (1999)
Gaivoronski, A.A.: A numerical method for solving stochastic programming problems with moment constraints on a distribution function. Ann. Oper. Res. 31(1), 347–369 (1991)
Gassmann, H., Ziemba, W.T.: A tight upper bound for the expectation of a convex function of a multivariate random variable. In: Stochastic Programming 84 Part I. Mathematical Programming Study, vol. 27, pp. 39–53. Springer, Berlin (1986)
Geoffrion, A.M.: Generalized Benders decomposition. JOTA 10(4), 237–260 (1972)
Gilboa, I., Schmeidler, D.: Maxmin expected utility with non-unique prior. J. Math. Econ. 18(2), 141–153 (1989)
Godwin, H.J.: On generalizations of Tchebychef’s inequality. J. Am. Stat. Assoc. 50(271), 923–945 (1955)
Goh, J., Sim, M.: Distributionally robust optimization and its tractable approximations. Oper. Res. 58(4, part 1), 902–917 (2010)
Halmos, P.R., Savage, L.J.: Application of the Radon-Nikodym theorem to the theory of sufficient statistics. Ann. Math. Stat. 20(2), 225–241 (1949)
Han, S., Tao, M., Topcu, U., Owhadi, H., Murray, R.M.: Convex optimal uncertainty quantification. SIAM J. Optim. 25(23), 1368–1387 (2015). arXiv:1311.7130
Han, S., Topcu, U., Tao, M., Owhadi, H., Murray, R.: Convex optimal uncertainty quantification: algorithms and a case study in energy storage placement for power grids. In: American Control Conference (ACC), 2013, Washington, DC, pp. 1130–1137. IEEE (2013)
Hanasusanto, G.A., Roitch, V., Kuhn, D., Wiesemann, W.: A distributionally robust perspective on uncertainty quantification and chance constrained programming. Math. Program. 151(1), 35–62 (2015)
Hoeffding, W.: On the distribution of the number of successes in independent trials. Ann. Math. Stat. 27(3), 713–721 (1956)
Hotelling, H.: Abraham Wald. Am. Stat. 5(1), 18–19 (1951)
Huang, C.C., Vertinsky, I., Ziemba, W.T.: Sharp bounds on the value of perfect information. Oper. Res. 25(1), 128–139 (1977)
Huang, C.C., Ziemba, W.T., Ben-Tal, A.: Bounds on the expectation of a convex function of a random variable: with applications to stochastic programming. Oper. Res. 25(2), 315–325 (1977)
Huber, P.J.: Robust estimation of a location parameter. Ann. Math. Stat. 35, 73–101 (1964)
Huber, P.J.: The 1972 Wald lecture- Robust statistics: a review. Ann. Math. Stat. 1041–1067 (1972)
Isii, K.: On a method for generalizations of Tchebycheff’s inequality. Ann. Inst. Stat. Math. Tokyo 10(2), 65–88 (1959)
Isii, K.: The extrema of probability determined by generalized moments. I. Bounded random variables. Ann. Inst. Stat. Math. 12(2), 119–134; errata, 280 (1960)
Isii, K.: On sharpness of Tchebycheff-type inequalities. Ann. Inst. Stat. Math. 14(1):185–197, 1962/1963.
Jaynes, E.T.: Probability Theory. Cambridge University Press, Cambridge (2003)
Joe, H.: Majorization, randomness and dependence for multivariate distributions. Ann. Probab. 15(3), 1217–1225 (1987)
Johnstone, I.M.: High dimensional Bernstein–von Mises: simple examples. In Borrowing Strength: Theory Powering Applications—A Festschrift for Lawrence D. Brown, volume 6 of Inst. Math. Stat. Collect., pages 87–98. Inst. Math. Statist., Beachwood, OH (2010)
Kac, M., Slepian, D.: Large excursions of Gaussian processes. Ann. Math. Stat. 30, 1215–1228 (1959)
Kall, P.: Stochastric programming with recourse: upper bounds and moment problems: a review. Math. Res. 45, 86–103 (1988)
Kallenberg, O.: Random Measures. Akademie-Verlag, Berlin (1975) Schriftenreihe des Zentralinstituts für Mathematik und Mechanik bei der Akademie der Wissenschaften der DDR, Heft 23.
Kamga, P.-H.T., Li, B., McKerns, M., Nguyen, L.H., Ortiz, M., Owhadi, H., Sullivan, T.J.: Optimal uncertainty quantification with model uncertainty and legacy data. J. Mech. Phys. Solids 72, 1–19 (2014)
Karlin, S., Studden, W.J.: Tchebycheff Systems: With Applications in Analysis and Statistics. Pure and Applied Mathematics, vol. XV. Interscience Publishers/Wiley, New York/London/Sydney (1966)
Kendall, D.G.: Simplexes and vector lattices. J. Lond. Math. Soc. 37(1), 365–371 (1962)
Kidane, A.A., Lashgari, A., Li, B., McKerns, M., Ortiz, M., Owhadi, H., Ravichandran, G., Stalzer, M., Sullivan, T.J.: Rigorous model-based uncertainty quantification with application to terminal ballistics. Part I: Systems with controllable inputs and small scatter. J. Mech. Phys. Solids 60(5), 983–1001 (2012)
Kiefer, J.: Optimum experimental designs. J. R. Stat. Soc. Ser. B 21, 272–319 (1959)
Kiefer, J.: Collected Works, vol. III. Springer, New York (1985)
Kleijn, B.J.K., van der Vaart, A.W.: The Bernstein-Von-Mises theorem under misspecification. Electron. J. Stat. 6, 354–381 (2012)
Kolmogorov, A.N.: Foundations of the Theory of Probability. Chelsea Publishing Co., New York (1956). Translation edited by Nathan Morrison, with an added bibliography by A. T. Bharucha-Reid
Kreĭn, M.G.: The ideas of P. L. C̆ebys̆ev and A. A. Markov in the theory of limiting values of integrals and their further development. In: Dynkin, E.B. (ed.) Eleven Papers on Analysis, Probability, and Topology, American Mathematical Society Translations, Series 2, vol. 12, pp. 1–122. American Mathematical Society, New York (1959)
Kurz, H.D., Salvadori, N.: Understanding ‘Classical’ Economics: Studies in Long Period Theory. Routledge, London/New York (2002)
Laird, N.M.: A conversation with F. N. David. Stat. Sci. 4, 235–246 (1989)
Le Cam, L.: On some asymptotic properties of maximum likelihood estimates and related Bayes’ estimates. Univ. Calif. Publ. Stat. 1, 277–329 (1953)
Le Cam, L.: An extension of Wald’s theory of statistical decision functions. Ann. Math. Stat. 26, 69–81 (1955)
Le Cam, L.: Sufficiency and approximate sufficiency. Ann. Math. Stat. 35, 1419–1455 (1964)
Le Cam, L.: Asymptotic Methods in Statistical Decision Theory. Springer, New York (1986)
Leahu, H.: On the Bernstein–von Mises phenomenon in the Gaussian white noise model. Electron. J. Stat. 5, 373–404 (2011)
Lehmann, E.L.: “Student” and small-sample theory. Stat. Sci. 14(4), 418–426 (1999)
Lehmann, E.L.: Optimality and symposia: some history. Lect. Notes Monogr. Ser. 44, 1–10 (2004)
Lehmann, E.L.: Some history of optimality. Lect. Notes Monogr. Ser. 57, 11–17 (2009)
Lenhard, J.: Models and statistical inference: the controversy between Fisher and Neyman–Pearson. Br. J. Philos. Sci. 57(1), 69–91 (2006)
Leonard, R.: Von Neumann, Morgenstern, and the Creation of Game Theory: From Chess to Social Science, 1900–1960. Cambridge University Press, New York (2010)
Lynch, P.: The origins of computer weather prediction and climate modeling. J. Comput. Phys. 227(7), 3431–3444 (2008)
Madansky, A.: Bounds on the expectation of a convex function of a multivariate random variable. Ann. Math. Stat. 743–746 (1959)
Madansky, A.: Inequalities for stochastic linear programming problems. Manag. Sci. 6(2), 197–204 (1960)
Mangel, M., Samaniego, F.J.: Abraham Wald’s work on aircraft survivability. J. A. S. A. 79(386), 259–267 (1984)
Marshall, A.W., Olkin, I.: Multivariate Chebyshev inequalities. Ann. Math. Stat. 31(4), 1001–1014 (1960)
Marshall, A.W., Olkin, I.: Inequalities: Theory of Majorization and Its Applications. Mathematics in Science and Engineering, vol. 143. Academic [Harcourt Brace Jovanovich Publishers], New York (1979)
McKerns, M.M., Strand, L., Sullivan, T.J., Fang, A., Aivazis, M.A.G.: Building a framework for predictive science. In: Proceedings of the 10th Python in Science Conference (SciPy 2011) (2011)
Morgenstern, O.: Abraham Wald, 1902–1950. Econometrica: J. Econom. Soci. 361–367 (1951)
Mulholland, H.P., Rogers, C.A.: Representation theorems for distribution functions. Proc. Lond. Math. Soc. (3) 8(2), 177–223 (1958)
Nash, J.: Non-cooperative games. Ann. Math. (2) 54, 286–295 (1951)
Nash, J.F. Jr.: Equilibrium points in n-person games. Proc. Natl. Acad. Sci. U. S. A. 36, 48–49 (1950)
Nemirovsky, A.S.: Information-based complexity of linear operator equations. J. Complex. 8(2), 153–175 (1992)
Neyman, J.: Outline of a theory of statistical estimation based on the classical theory of probability. Philos. Trans. R. Soc. Lond. Ser. A 236(767), 333–380 (1937)
Neyman, J.: A Selection of Early Statistical Papers of J. Neyman. University of California Press, Berkeley (1967)
Neyman, J., Pearson, E.S.: On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika 20A, 175–240, 263–294 (1928)
Neyman, J., Pearson, E.S.: On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. Ser. A 231, 289–337 (1933)
Olkin, I., Pratt, J.W.: A multivariate Tchebycheff inequality. Ann. Math. Stat. 29(1), 226–234 (1958)
Owhadi, H.: Multigrid with rough coefficients and multiresolution operator decomposition from hierarchical information games. SIAM Rev. (Research spotlights) (2016, to appear). arXiv:1503.03467
Owhadi, H., Scovel, C.: Qualitative robustness in Bayesian inference. arXiv:1411.3984 (2014)
Owhadi, H., Scovel, C.: Brittleness of Bayesian inference and new Selberg formulas. Commun. Math. Sci. 14(1), 83–145 (2016)
Owhadi, H., Scovel, C.: Extreme points of a ball about a measure with finite support. Commun. Math. Sci. (2015, to appear). arXiv:1504.06745
Owhadi, H., Scovel, C.: Separability of reproducing kernel Hilbert spaces. Proc. Am. Math. Soc. (2015, to appear). arXiv:1506.04288
Owhadi, H., Scovel, C., Sullivan, T.J.: Brittleness of Bayesian inference under finite information in a continuous world. Electron. J. Stat. 9, 1–79 (2015)
Owhadi, H., Scovel, C., Sullivan, T.J.: On the Brittleness of Bayesian Inference. SIAM Rev. (Research Spotlights) (2015)
Owhadi, H., Scovel, C., Sullivan, T.J., McKerns, M., Ortiz, M.: Optimal Uncertainty Quantification. SIAM Rev. 55(2), 271–345 (2013)
Packel, E.W.: The algorithm designer versus nature: a game-theoretic approach to information-based complexity. J. Complex. 3(3), 244–257 (1987)
Pearson, E.S.: ‘Student’ A Statistical Biography of William Sealy Gosset. Clarendon Press, Oxford (1990)
Pfanzagl, J.: Conditional distributions as derivatives. Ann. Probab. 7(6), 1046–1050 (1979)
Pinelis, I.: Exact inequalities for sums of asymmetric random variables, with applications. Probab. Theory Relat. Fields 139(3-4):605–635 (2007)
Pinelis, I.: On inequalities for sums of bounded random variables. J. Math. Inequal. 2(1), 1–7 (2008)
Platzman, G.W.: The ENIAC computations of 1950-gateway to numerical weather prediction. Bull. Am. Meteorol. Soc. 60, 302–312 (1979)
Ressel, P.: Some continuity and measurability results on spaces of measures. Mathematica Scandinavica 40, 69–78 (1977)
Rikun, A.D.: A convex envelope formula for multilinear functions. J. Global Optim. 10(4), 425–437 (1997)
Rockafellar, R.T.: Augmented Lagrange multiplier functions and duality in nonconvex programming. SIAM J. Control 12(2), 268–285 (1974)
Rojo, J.: Optimality: The Second Erich L. Lehmann Symposium. IMS, Beachwood (2006)
Rojo, J.: Optimality: The Third Erich L. Lehmann Symposium. IMS, Beachwood (2009)
Rojo, J., Pérez-Abreu, V.: The First Erich L. Lehmann Symposium: Optimality. IMS, Beachwood (2004)
Rustem, B., Howe, M.: Algorithms for Worst-Case Design and Applications to Risk Management. Princeton University Press, Princeton (2002)
Savage, L.J.: The theory of statistical decision. J. Am. Stat. Assoc. 46, 55–67 (1951)
Scovel, C., Hush, D., Steinwart, I.: Approximate duality. J. Optim. Theory Appl. 135(3), 429–443 (2007)
Shapiro, A., Kleywegt, A.: Minimax analysis of stochastic problems. Optim. Methods Softw. 17(3), 523–542 (2002)
Sherali, H.D.: Convex envelopes of multilinear functions over a unit hypercube and over special discrete sets. Acta Math. Vietnam. 22(1), 245–270 (1997)
Singpurwalla, N.D., Swift, A.: Network reliability and Borel’s paradox. Am. Stat. 55(3), 213–218 (2001)
Smith, J.E.: Generalized Chebychev inequalities: theory and applications in decision analysis. Oper. Res. 43(5), 807–825 (1995)
Sniedovich, M.: The art and science of modeling decision-making under severe uncertainty. Decis. Mak. Manuf. Serv. 1(1–2), 111–136 (2007)
Sniedovich, M.: A classical decision theoretic perspective on worst-case analysis. Appl. Math. 56(5), 499–509 (2011)
Sniedovich, M.: Black Swans, new Nostradamuses, Voodoo decision theories, and the science of decision making in the face of severe uncertainty. Int. Trans. Oper. Res. 19(1–2), 253–281 (2012)
Spanos, A.: Why the Decision-Theoretic Perspective Misrepresents Frequentist Inference (2014). https://secure.hosting.vt.edu/www.econ.vt.edu/directory/spanos/spanos10.pdf
Stein, C.: Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In: Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, 1954–1955, vol. I, pp. 197–206. University of California Press, Berkeley/Los Angeles (1956)
Strasser, H.: Mathematical Theory of Statistics: Statistical Experiments and Asymptotic Decision Theory, vol. 7. Walter de Gruyter, Berlin/New York (1985)
Stuart, A.M.: Inverse problems: a Bayesian perspective. Acta Numer. 19, 451–559 (2010)
Student: The probable error of a mean. Biometrika 1–25 (1908)
Sullivan, T.J., McKerns, M., Meyer, D., Theil, F., Owhadi, H., Ortiz, M.: Optimal uncertainty quantification for legacy data observations of Lipschitz functions. ESAIM Math. Model. Numer. Anal. 47(6), 1657–1689 (2013)
Tintner, G.: Abraham Wald’s contributions to econometrics. Ann. Math. Stat. 23, 21–28 (1952)
Tjur, T.: Conditional Probability Distributions, Lecture Notes, No. 2. Institute of Mathematical Statistics, University of Copenhagen, Copenhagen (1974)
Tjur, T.: Probability Based on Radon Measures. Wiley Series in Probability and Mathematical Statistics. Wiley, Chichester (1980)
Traub, J.F., Wasilkowski, G.W., Woźniakowski, H.: Information-Based Complexity. Computer Science and Scientific Computing. Academic, Boston (1988). With contributions by A. G. Werschulz and T. Boult
Tukey, J.W.: Statistical and Quantitative Methodology. Trends in Social Science, pp. 84–136. Philisophical Library, New York (1961)
Tukey, J.W.: The future of data analysis. Ann. Math. Stat. 33, 1–67 (1962)
Valiant, L.G.: A theory of the learnable. Commun. ACM 27(11), 1134–1142 (1984)
Vandenberghe, L., Boyd, S., Comanor, K.: Generalized Chebyshev bounds via semidefinite programming. SIAM Rev. 49(1), 52–64 (electronic) (2007)
Varadarajan, V.S.: Groups of automorphisms of Borel spaces. Trans. Am. Math. Soc. 109(2), 191–220 (1963)
von Mises, R.: Mathematical Theory of Probability and Statistics. Edited and Complemented by Hilda Geiringer. Academic, New York (1964)
Von Neumann, J.: Zur Theorie der Gesellschaftsspiele. Math. Ann. 100(1), 295–320 (1928)
Von Neumann, J., Goldstine, H.H.: Numerical inverting of matrices of high order. Bull. Am. Math. Soc. 53, 1021–1099 (1947)
Von Neumann, J., Morgenstern, O.: Theory of Games and Economic Behavior. Princeton University Press, Princeton (1944)
Wald, A.: Contributions to the theory of statistical estimation and testing hypotheses. Ann. Math. Stat. 10(4), 299–326 (1939)
Wald, A.: Statistical decision functions which minimize the maximum risk. Ann. Math. (2) 46, 265–280 (1945)
Wald, A.: An essentially complete class of admissible decision functions. Ann. Math. Stat. 18, 549–555 (1947)
Wald, A.: Sequential Analysis. 1947.
Wald, A.: Statistical decision functions. Ann. Math. Stat. 20, 165–205 (1949)
Wald, A.: Statistical Decision Functions. Wiley, New York (1950)
Wald, A., Wolfowitz, J.: Optimum character of the sequential probability ratio test. Ann. Math. Stat. 19(3), 326–339 (1948)
Wald, A., Wolfowitz, J.: Characterization of the minimal complete class of decision functions when the number of distributions and decisions is finite. In: Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, pp. 149–157. University of California Press, Berkeley (1951)
Wasserman, L.: Rise of the Machines. Past, Present and Future of Statistical Science. CRC Press, Boca Raton (2013)
Wasserman, L., Lavine, M., Wolpert, R.L.: Linearization of Bayesian robustness problems. J. Stat. Plann. Inference 37(3), 307–316 (1993)
Wiesemann, W., Kuhn, D., Sim, M.: Distributionally robust convex optimization. Oper. Res. 62(6), 1358–1376 (2014)
Wilson, M.: How a story from World War II shapes Facebook today. IBM Watson (2012). http://www.fastcodesign.com/1671172/how-a-story-from-world-war-ii-shapes-facebook-today.
Winkler, G.: On the integral representation in convex noncompact sets of tight measures. Mathematische Zeitschrift 158(1), 71–77 (1978)
Winkler, G.: Extreme points of moment sets. Math. Oper. Res. 13(4), 581–587 (1988)
Wolfowitz, J.: Abraham Wald, 1902–1950. Ann. Math. Stat. 23, 1–13 (1952)
Woźniakowski, H.: Probabilistic setting of information-based complexity. J. Complex. 2(3), 255–269 (1986)
Woźniakowski, H.: What is information-based complexity? In Essays on the Complexity of Continuous Problems, pp. 89–95. European Mathematical Society, Zürich (2009)
Wynn, H.P.: Introduction to Kiefer (1959) Optimum Experimental Designs. In Breakthroughs in Statistics, pp. 395–399. Springer, New York (1992)
Xu, L., Yu, B., Liu, W.: The distributionally robust optimization reformulation for stochastic complementarity problems. Abstr. Appl. Anal. 2014, Art. ID 469587, (2014)
Žáčková, J.: On minimax solutions of stochastic linear programming problems. Časopis Pěst. Mat. 91, 423–430 (1966)
Zhou, K., Doyle, J.C., Glover, K.: Robust and Optimal Control. Prentice Hall, Upper Saddle River (1996)
Zymler, S., Kuhn, D., Rustem, B.: Distributionally robust joint chance constraints with second-order moment information. Math. Program. 137(1-2, Ser. A), 167–198 (2013)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix
Appendix
1.1 Construction of \(\pi \odot \mathbb{D}\)
The below construction works when \(\mathcal{A}\subseteq \mathcal{G}\times \mathcal{M}(\mathcal{X})\) for some Polish subset \(\mathcal{G}\subset \mathcal{F}(\mathcal{X})\) and \(\mathcal{X}\) is Polish. Observe that since \(\mathcal{D}\) is metrizable, it follows from [4, Thm. 15.13], that, for any \(B \in \mathcal{B}(\mathcal{D})\), the evaluation \(\nu \mapsto \nu (B)\), \(\nu \in \mathcal{M}(\mathcal{D})\), is measurable. Consequently, the measurability of \(\mathbb{D}\) implies that the mapping
defined by
is a transition function in the sense that, for fixed \((f,\mu ) \in \mathcal{A}\), \(\widehat{\mathbb{D}}{\bigl ((f,\mu ),.\bigr )}\) is a probability measure, and, for fixed \(B \in \mathcal{B}(\mathcal{D})\), \(\widehat{\mathbb{D}}{\bigl (.,B\bigr )}\) is Borel measurable. Therefore, by [18, Thm. 10.7.2], any \(\pi \in \mathcal{M}(\mathcal{A})\) defines a probability measure
through
where \(\mathbb{1}_{A}\) is the indicator function of the set A:
It is easy to see that π is the \(\mathcal{A}\)-marginal of \(\pi \odot \mathbb{D}\). Moreover, when \(\mathcal{X}\) is Polish, [4, Thm. 15.15] implies that \(\mathcal{M}(\mathcal{X})\) is Polish, and when \(\mathcal{G}\) is Polish, it follows that \(\mathcal{A}\subseteq \mathcal{G}\times \mathcal{M}(\mathcal{X})\) is second countable. Consequently, since \(\mathcal{D}\) is Suslin and hence second countable, it follows from [32, Prop. 4.1.7] that
and hence \(\pi \odot \mathbb{D}\) is a probability measure on \(\mathcal{A}\times \mathcal{D}\). That is,
Henceforth denote \(\pi \cdot \mathbb{D}\) the corresponding Bayes’ sampling distribution defined by the \(\mathcal{D}\)-marginal of \(\pi \odot \mathbb{D}\), and note that by (21), one has
Since both \(\mathcal{D}\) and \(\mathcal{A}\) are Suslin, it follows that the product \(\mathcal{A}\times \mathcal{D}\) is Suslin. Consequently, [18, Cor. 10.4.6] asserts that regular conditional probabilities exist for any sub-σ-algebra of \(\mathcal{B}{\bigl (\mathcal{A}\times \mathcal{D}\bigr )}\). In particular, the product theorem of [18, Thm. 10.4.11] asserts that product regular conditional probabilities
exist and that they are \(\pi \cdot \mathbb{D}\)-a.e. unique.
1.2 Proof of Theorem 2
If \(\pi ^{\dag }\cdot \mathbb{D}\) is not absolutely continuous with respect to \(\pi \cdot \mathbb{D}\), then there exists \(B \in \mathcal{B}(\mathcal{D})\) such that \((\pi \cdot \mathbb{D})[B] = 0\) and \((\pi ^{\dag }\cdot \mathbb{D})[B] > 0\). Let \(\theta \in \Theta (\pi )\). Define
Then it is easy to see that if y is in the range of \(\Phi \), then \(\theta _{y} \in \Theta (\pi )\). Now observe that for \(y,z \in Image(\Phi )\),
Hence, for V (x) = x 2, it holds true that
with
which proves
and,
To obtain the right hand side of (19) observe that (see for instance [29, Sec. 5]) there exists \(B^{{\ast}}\in \mathcal{B}(\mathcal{D})\) such that
and (since \(\theta _{2} =\theta _{1}\) on the complement of B ∗)
We conclude by observing that for V (x) = x 2,
1.3 Conditional Expectation as an Orthogonal Projection
It easily follows from Tonelli’s Theorem that
By considering the sub σ-algebra \(\mathcal{A}\times \mathcal{B}(\mathcal{D}) \subset \mathcal{B}(\mathcal{A}\times \mathcal{D}) = \mathcal{B}(\mathcal{A}) \times \mathcal{B}(\mathcal{D})\), it follows from, e.g., Theorem 10.2.9 of [32], that \(L_{\pi \cdot \mathbb{D}}^{2}(\mathcal{D})\) is a closed Hilbert subspace of the Hilbert space \(L_{\pi \odot \mathbb{D}}^{2}(\mathcal{A}\times \mathcal{D})\) and the conditional expectation of \(\Phi \) given the random variable D is the orthogonal projection from \(L_{\pi \odot \mathbb{D}}^{2}(\mathcal{A}\times \mathcal{D})\) to \(L_{\pi \cdot \mathbb{D}}^{2}(\mathcal{D})\).
Rights and permissions
Copyright information
© 2015 Springer International Publishing Switzerland
About this entry
Cite this entry
Owhadi, H., Scovel, C. (2015). Toward Machine Wald. In: Ghanem, R., Higdon, D., Owhadi, H. (eds) Handbook of Uncertainty Quantification. Springer, Cham. https://doi.org/10.1007/978-3-319-11259-6_3-1
Download citation
DOI: https://doi.org/10.1007/978-3-319-11259-6_3-1
Received:
Accepted:
Published:
Publisher Name: Springer, Cham
Online ISBN: 978-3-319-11259-6
eBook Packages: Springer Reference MathematicsReference Module Computer Science and Engineering