Abstract
The Conformal Predictions framework is a new game-theoretic approach to reliable machine learning, which provides a methodology to obtain error calibration under classification and regression settings. The framework combines principles of transductive inference, algorithmic randomness and hypothesis testing to provide guaranteed error calibration in online settings (and calibration in offline settings supported by empirical studies). As the framework is being increasingly used in a variety of machine learning settings such as active learning, anomaly detection, feature selection, and change detection, there is a need to develop algorithmic implementations of the framework that can be used and further improved by researchers and practitioners. In this paper, we introduce PyCP, an open-source implementation of the Conformal Predictions framework that currently provides support for classification problems within transductive and Mondrian settings. PyCP is modular, extensible and intended for community sharing and development.
Chapter PDF
Similar content being viewed by others
References
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)
Vovk, V.: On-line confidence machines are well-calibrated. In: 43rd Symposium on Foundations of Computer Science, Washington, DC, USA, pp. 187–196 (2002)
Vanderlooy, S., van der Maaten, L., Sprinkhuizen-Kuyper, I.: Off-line learning with transductive confidence machines: An empirical evaluation. In: Perner, P. (ed.) MLDM 2007. LNCS (LNAI), vol. 4571, pp. 310–323. Springer, Heidelberg (2007)
Ho, S.S., Wechsler, H.: Query by transduction. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(9), 1557–1571 (2008)
Laxhammar, R., Falkman, G.: Conformal prediction for distribution-independent anomaly detection in streaming vessel data. In: Proceedings of the First International Workshop on Novel Data Stream Pattern Mining Techniques, StreamKDD 2010, pp. 47–55. ACM, New York (2010)
Bellotti, T., Luo, Z., Gammerman, A.: Strangeness minimisation feature selection with confidence machines. In: Corchado, E., Yin, H., Botti, V., Fyfe, C. (eds.) IDEAL 2006. LNCS, vol. 4224, pp. 978–985. Springer, Heidelberg (2006)
Ho, S.S., Wechsler, H.: A martingale framework for detecting changes in data streams by testing exchangeability. IEEE Transactions on Pattern Analysis and Machine Intelligence 32(12), 2113–2127 (2010)
Kukar, M.: Quality assessment of individual classifications in machine learning and data mining. Knowledge Information Systems 9(3), 364–384 (2006)
Lambrou, A., Papadopoulos, H., Gammerman, A.: Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE Transactions on Information Technology in Biomedicine 15(1), 93–99 (2011)
Shao, H., Yu, B., Nadeau, J.H.: Strangeness-based feature weighting and classification of gene expression profiles. In: Wainwright, R.L., Haddad, H. (eds.) SAC, pp. 1292–1296. ACM (2008)
Li, F., Wechsler, H.: Open set face recognition using transduction. IEEE Trans. Pattern Anal. Mach. Intell. 27(11), 1686–1697 (2005)
Dashevskiy, M., Luo, Z.: Network traffic demand prediction with confidence. In: GLOBECOM, pp. 1453–1457 (2008)
Sonnenburg, S., Braun, M.L., Ong, C.S., Bengio, S., Bottou, L., Holmes, G., LeCun, Y., Müller, K.R., Pereira, F., Rasmussen, C.E., Rätsch, G., Schölkopf, B., Smola, A., Vincent, P., Weston, J., Williamson, R.: The need for open source software in machine learning. Journal of Machine Learning Research 8, 2443–2466 (2007)
Schaul, T., Bayer, J., Wierstra, D., Sun, Y., Felder, M., Sehnke, F., Rückstieß, T., Schmidhuber, J.: PyBrain. J. Mach. Learn. Res. 11, 743–746 (2010)
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, D.: Scikit-learn: Machine learning in python. Journal of Machine Learning Research 12, 2825–2830 (2011)
Balasubramanian, V., Gouripeddi, R., Panchanathan, S., Vermillion, J., Bhaskaran, A., Siegel, R.: Support vector machine based conformal predictors for risk of complications following a coronary drug eluting stent procedure. Computers in Cardiology, 5–8 (2009)
Bache, K., Lichman, M.: UCI machine learning repository (2013)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 IFIP International Federation for Information Processing
About this paper
Cite this paper
Balasubramanian, V.N., Baker, A., Yanez, M., Chakraborty, S., Panchanathan, S. (2013). PyCP: An Open-Source Conformal Predictions Toolkit. In: Papadopoulos, H., Andreou, A.S., Iliadis, L., Maglogiannis, I. (eds) Artificial Intelligence Applications and Innovations. AIAI 2013. IFIP Advances in Information and Communication Technology, vol 412. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-41142-7_37
Download citation
DOI: https://doi.org/10.1007/978-3-642-41142-7_37
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-41141-0
Online ISBN: 978-3-642-41142-7
eBook Packages: Computer ScienceComputer Science (R0)