Abstract
In this paper we present a comprehensive comparison of the performance of a number of text categorization methods in two different data sets. In particular, we evaluate the Vector and Latent Semantic Analysis (LSA) methods, a classifier based on Support Vector Machines (SVM) and the k-Nearest Neighbor variations of the Vector and LSA models.
We report the results obtained using the Mean Reciprocal Rank as a measure of overall performance, a commonly used evaluation measure for question answering tasks. We argue that this evaluation measure is also very well suited for text categorization tasks.
Our results show that overall, SVMs and k-NN LSA perform better than the other methods, in a statistically significant way.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison-Wesley, Reading (1999)
Berger, A., Caruana, R., Cohn, D., Freitag, D., Mittal, V.O.: Bridging the lexical chasm: statistical approaches to answer-finding. In: Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2000, Athens, Greece, pp. 192–199 (2000)
Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2(2), 121–167 (1998)
Caron, J.: Experiments with LSA scoring: Optimal rank and basis. In: Proceedings of SIAM Computational Information Retrieval Workshop, Raleigh, NC, USA (October 2000)
Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines (2001), Software, available at http://www.csie.ntu.edu.tw/~cjlin/libsvm
Cooper, W.S.: Expected search length: a single measure of retrieval effectiveness based on weak ordering action of retrieval systems. Journal of the American Society for Information Science 19(1), 30–41 (1968)
Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995)
Creecy, R.M., Masand, B.M., Smith, S.J., Waltz, D.L.: Trading MIPS and memory for knowledge engineering: classifying census returns on the Connection Machine. Communications of the ACM 39(1), 48–63 (1996)
Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)
Deerwester, S.C., Dumais, S.T., Landauer, T.K., Furnas, G.W., Harshman, R.A.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)
Furnas, G.W., Deerwester, S.C., Dumais, S.T., Landauer, T.K., Harshman, R.A., Streeter, L.A., Lochbaum, K.E.: Information retrieval using a singular value decomposition model of latent semantic structure. In: Proceedings of the 11th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Grassau, France, June 1988, pp. 465–480 (1988)
Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998)
Joachims, T.: Transductive inference for text classification using support vector machines. In: Bratko, I., Dzeroski, S. (eds.) Proceedings of the 16th International Conference on Machine Learning, Bled, Slovenia, 1999, pp. 200–209. Morgan Kaufmann Publishers, Inc., San Francisco (1999)
Jones, K.S., Willett, P. (eds.): Readings in Information Retrieval. Morgan Kaufmann Publishers, Inc., Los Altos (1997)
Masand, B., Linoff, G., Waltz, D.: Classifying news stories using memory-based reasoning. In: Belkin, N.J., Ingwersen, P., Pejtersen, A.M. (eds.) Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Copenhagen, Denmark, pp. 59–65. ACM Press, New York (1992)
Salton, G.: The SMART Retrieval System. Prentice-Hall, Inc., New Jersey (1971)
Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval. Information Processing and Management 24(5), 513–523 (1988); Also reprinted in [14, pp. 323–328]
Salton, G., Lesk, M.: Computer evaluation of indexing and text processing. Journal of the ACM 15(1), 8–36 (1968); Also reprinted in [14, pp. 60–84]
Sebastiani, F.: Machine learning in automated text categorization. ACM. Computing Surveys 34(1), 1–47 (2002)
Vapnik, V.: The Nature of Statistical Learning Theory. Springer, Heidelberg (1995)
Voorhees, E.M.: The TREC-8 question answering track report. In: Voorhees, E.M., Harman, D.K. (eds.) Proceedings of the 8th Text Retrieval Conference, Gaithersburg, Maryland, USA, November 1999, pp. 77–82 (1999)
Yang, Y.: Expert network: effective and efficient learning from human decisions in text categorisation and retrieval. In: Bruce Croft, W., van Rijsbergen, C.J. (eds.) Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Dublin, Ireland, pp. 13–22. Springer, Heidelberg (1994)
Yang, Y., Liu, X.: A re-examination of text categorization methods. In: Hearst, M.A., Gey, F., Tong, R. (eds.) Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, CA, USA, August 1999, pp. 42–49 (1999)
Yang, Y., Pedersen, J.O.: A comparative study on feature selection in text categorization. In: Fisher, D.H. (ed.) Proceedings of the 14th International Conference on Machine Learning, Nashville, TN, USA, pp. 412–420. Morgan Kaufmann Publishers, Inc., San Francisco (1997)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2003 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cardoso-Cachopo, A., Oliveira, A.L. (2003). An Empirical Comparison of Text Categorization Methods. In: Nascimento, M.A., de Moura, E.S., Oliveira, A.L. (eds) String Processing and Information Retrieval. SPIRE 2003. Lecture Notes in Computer Science, vol 2857. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-39984-1_14
Download citation
DOI: https://doi.org/10.1007/978-3-540-39984-1_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-20177-9
Online ISBN: 978-3-540-39984-1
eBook Packages: Springer Book Archive