Abstract
Although theoretical results for several algorithms in many application domains were presented during the last decades, not all algorithms can be analyzed fully theoretically. Experimentation is necessary. The analysis of algorithms should follow the same principles and standards of other empirical sciences. This article focuses on stochastic search algorithms, such as evolutionary algorithms or particle swarm optimization. Stochastic search algorithms tackle hard real-world optimization problems, e.g., problems from chemical engineering, airfoil optimization, or bio-informatics, where classical methods from mathematical optimization fail. Nowadays statistical tools that are able to cope with problems like small sample sizes, non-normal distributions, noisy results, etc. are developed for the analysis of algorithms. Although there are adequate tools to discuss the statistical significance of experimental data, statistical significance is not scientifically meaningful per se. It is necessary to bridge the gap between the statistical significance of an experimental result and its scientific meaning. We will propose some ideas on how to accomplish this task based on Mayo’s learning model (NPT*).
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Bartz-Beielstein T. (2006). Experimental research in evolutionary computation—The new experimentalism. Berlin, Heidelberg, New York: Springer.
Bartz-Beielstein, T., & Preuss, M. (2004). CEC tutorial on experimental research in evolutionary computation. In IEEE Congress on Evolutionary Computation, Tutorial Program. Tutorials given at CEC in 2004 and 2005.
Bartz-Beielstein, T., & Preuss, M. (2005). GECCO tutorial on experimental research in evolutionary computation. In 2005 Genetic and Evolutionary Computation Conference, Tutorial Program. Tutorials given at GECCO in 2005, 2006, and 2007.
Bartz-Beielstein, T., & Preuss, M. (2006a). Considerations of budget allocation for sequential parameter pptimization (SPO). In Workshop on Empirical Methods for the Analysis of Algorithms (EMAA), Tutorial Program. Held in conjunction with the International Conference on Parallel Problem Solving From Nature (PPSN IX) (pp. 35–40).
Bartz-Beielstein, T., & Preuss, M. (2006b). Sequential parameter optimization (SPO) and the role of tuning in experimental analysis. In Workshop on Empirical Methods for the Analysis of Algorithms (EMAA). Held in conjunction with the International Conference on Parallel Problem Solving From Nature (PPSN IX) (pp. 5–6). Invited talk
Clerc, M., & Kennedy, J. (2006). Standard PSO version 2006. http://www.particleswarm.info/Standard_PSO_2006.c. Cited 11 August 2007.
Gregoire, T. (2001). Biometry in the 21st Century: Whither statistical inference? (Invited Keynote). Proceedings of the Forest Biometry and Information Science Conference held at the University of Greenwich, June 2001. http://cms1.gre.ac.uk/conferences/iufro/proceedings/gregoire.pdf. Cited 19 May 2004.
Kleijnen J.P.C. (1987). Statistical tools for simulation practitioners. New York, NY: Marcel Dekker.
Mayo D.G. (1983). An objective theory of statistical testing. Synthese 57: 297–340
Mayo D.G. (1996). Error and the growth of experimental knowledge. Chicago, IL: The University of Chicago Press.
Mayo D.G., Spanos A. (2004). Methodology in practice: Statistical misspecification testing. Philosophy of Science 71: 1007–1025
Mayo D.G., Spanos A. (2006). Severe testing as a basic concept in a Neyman–Pearson philosophy of induction. British Journal for the Philosophy of Science 57: 323–357
Montgomery D.C. (2001). Design and analysis of experiments (5th ed.). New York, NY: Wiley.
Morrison D.E., Henkel R.E. (eds) (1970). The significance test controversy—A reader. London, UK: Butterworths.
Wolpert D., Macready W. (1997). No free lunch theorems for optimization. IEEE Transactions on Evolutionary Computation 1(1): 67–82
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Bartz-Beielstein, T. How experimental algorithmics can benefit from Mayo’s extensions to Neyman–Pearson theory of testing. Synthese 163, 385–396 (2008). https://doi.org/10.1007/s11229-007-9297-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11229-007-9297-z