Abstract
Inductive conformal predictors have been designed to overcome the computational inefficiency exhibited by conformal predictors for many underlying prediction algorithms. Whereas computationally efficient, inductive conformal predictors sacrifice different parts of the training set at different stages of prediction, which affects their informational efficiency. This paper introduces the method of cross-conformal prediction, which is a hybrid of the methods of inductive conformal prediction and cross-validation, and studies its validity and informational efficiency empirically. The computational efficiency of cross-conformal predictors is comparable to that of inductive conformal predictors, and they produce valid predictions in our empirical studies.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Balasubramanian, V.N.: Conformal predictions in multimedia pattern recognition. Ph.D. thesis, Arizona State University (2010)
Balasubramanian, V.N., Chakraborty, S., Panchanathan, S.: Conformal predictions for information fusion: a comparative study of p-value combination methods. Ann. Math. Artif. Intell. (2013, this issue)
Breiman, L., Spector, P.: Submodel selection and evaluation in regression: the X-random case. Int. Stat. Rev. 60, 291–319 (1992)
Efron, B.: Bootstrap methods: another look at the jackknife. Ann. Stat. 7, 1–26 (1979)
Efron, B.: Estimating the error rate of a prediction rule: some improvements on cross-validation. J. Am. Stat. Assoc. 78, 316–331 (1983)
Fisher, R.A.: Combining independent tests of significance. Am. Stat. 2, 30 (1948)
Fraser, D.A.S.: Nonparametric Methods in Statistics. Wiley, New York (1957)
Freund, Y., Schapire, R.E.: A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997)
Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Data Anal. 29, 1189–1232 (2001)
Friedman, J.H.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38, 367–378 (2002)
Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd edn. Springer, New York (2009)
Kohavi, R.: A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), pp. 1137–1143. Morgan Kaufmann, San Mateo, CA (1995)
Lei, J., Rinaldo, A., Wasserman, L.: A conformal prediction approach to explore functional data. Ann. Math. Artif. Intell. (2013). doi:10.1007/s10472-013-9366-6
Lei, J., Wasserman, L.: Distribution free prediction bands. Tech. Rep. arXiv: 1203.5422 [stat.ME]. e-Print archive (2012, to appear in JRRSB)
Mosteller, F., Tukey, J.W.: Data analysis, including statistics. In: Lindzey, G., Aronson, E. (eds.) Handbook of Social Psychology, 2nd edn., vol. 2, pp. 80–203. Addison-Wesley, Reading, MA (1968)
Papadopoulos, H., Vovk, V., Gammerman, A.: Qualified predictions for large data sets in the case of pattern recognition. In: Proceedings of the First International Conference on Machine Learning and Applications (ICMLA), pp. 159–163. CSREA Press, Las Vegas, NV (2002)
Simard, P., LeCun, Y., Denker, J.: Efficient pattern recognition using a new transformation distance. In: Hanson, S., Cowan, J., Giles, C. (eds.) Advances in Neural Information Processing Systems, vol. 5, pp. 50–58. Morgan Kaufmann, San Mateo, CA (1993)
Stone, M.: Cross-validatory choice and assessment of statistical predictions (with discussion). J. R. Stat. Soc. Ser. B 36, 111–147 (1974)
Vanderlooy, S., van der Maaten, L., Sprinkhuizen-Kuyper, I.: Off-line learning with transductive confidence machines: an empirical evaluation. In: Perner, P. (ed.) Proceedings of the Fifth International Conference on Machine Learning and Data Mining in Pattern Recognition. Lecture Notes in Artificial Intelligence, vol. 4571, pp. 310–323. Springer, Berlin (2007)
Vovk, V.: Conditional validity of inductive conformal predictors. In: Hoi, S.C.H., Buntine, W. (eds.) JMLR: Workshop and Conference Proceedings, vol. 25 (Asian Conference on Machine Learning), pp. 475–490 (2012). The journal version to appear in Machine Learning (ACML 2012 Special Issue)
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)
Author information
Authors and Affiliations
Corresponding author
Additional information
An extended abstract of an early version of this paper was published in the Proceedings of the Fifth Workshop on Information Theoretic Methods in Science and Engineering (WITMSE 2012, Amsterdam, August 2012, http://event.cwi.nl/witmse2012/proc.pdf). This work was partially supported by the Cyprus Research Promotion Foundation.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Vovk, V. Cross-conformal predictors. Ann Math Artif Intell 74, 9–28 (2015). https://doi.org/10.1007/s10472-013-9368-4
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-013-9368-4