Abstract
Multilingual information retrieval tasks such as Topic Tracking have yielded high-quality results simply using word-by-word translation approaches. However, the construction of translation dictionaries for new languages is expensive and time-consuming. We show that an appropriate metric for term selection in a monolingual English corpus allows us to define a fairly small list, containing about ten thousand inflected forms or about 7500 lemmas, which works essentially as well (for a particular monolingual document classification evaluation) as an unlimited vocabulary of more than 300,000 word forms does. We suggest that such a list can be taken to form the English axis of a sort of “universal dictionary” for document classification tasks, providing a much more efficient path to the addition of new languages.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Kenneth W. Church, William A. Gale, “Inverse Document Frequency (IDF): A Measure of Deviations from Poisson,” Third Workshop on Very Large Corpora, 1995.
G. Doddington, “The 1999 Topic Detection and Tracking (TDT3) Task Definition and Evaluation Plan” http://www.nist.gov Available at, 1999.
G. Doddington, “The Topic Detection and Tracking Phase 2 (TDT2) Evaluation Plan,” http://www.nist.gov/speech/tdt_98.htm Available at, 1998.
G. Doddington, “The TDT Pilot Study Corpus Documentation,” http://www.ldc.upenn.edu/TDT/Pilot/TDT.Study.Corpus.v1.3.psAvailable at, 1997.
Linguistic Data Consortium, “North American News Text Copus (Suppliment) and AP Worldstream English,” http://www.ldc.upenn.edu Available from, 1997.
A. Martin, G. Doddington, T. Kamm, M. Ordowski, M. Przybocki, “The DET Curve in Assessment of Detection Task Performance,” EuroSpeech 1997 Proceedings Volume 4, 1997.
G. Salton and M.J. McGill, “Introduction to Modern Information Retrieval,” McGraw Hill Book Co., New York, 1983.
J. Michael Schultz, Mark Liberman, “Topic Detection and Tracking using idf-weighted Cosine Coefficient,” DARPA Broadcast News Workshop Proceedings, 1999.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer Science+Business Media New York
About this chapter
Cite this chapter
Schultz, J.M., Liberman, M.Y. (2002). Towards a “Universal Dictionary” for Multi-Language Information Retrieval Applications. In: Allan, J. (eds) Topic Detection and Tracking. The Information Retrieval Series, vol 12. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-0933-2_11
Download citation
DOI: https://doi.org/10.1007/978-1-4615-0933-2_11
Publisher Name: Springer, Boston, MA
Print ISBN: 978-1-4613-5311-9
Online ISBN: 978-1-4615-0933-2
eBook Packages: Springer Book Archive