Abstract
TREC-like evaluations do not consider topic ease and difficulty. However, it seems reasonable to reward good effectiveness on difficult topics more than good effectiveness on easy topics, and to penalize bad effectiveness on easy topics more than bad effectiveness on difficult topics. This paper shows how this approach leads to evaluation results that could be more reasonable, and that are different to some extent. I provide a general analysis of this issue, propose a novel framework, and experimentally validate a part of it.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
References
Mizzaro, S., Robertson, S.: HITS Hits TREC - Exploring IR Evaluation Results with Network Analysis. In: 30th SIGIR, pp. 479–486 (2007)
Robertson, S.: On GMAP – and other transformations. In: 13th CIKM, pp. 78–83 (2006)
Voorhees, E.M.: Overview of the TREC 2005 Robust Retrieval Track. In: TREC 2005 Proceedings (2005)
Voorhees, E.M., Harman, D.K.: TREC — Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Mizzaro, S. (2008). The Good, the Bad, the Difficult, and the Easy: Something Wrong with Information Retrieval Evaluation?. In: Macdonald, C., Ounis, I., Plachouras, V., Ruthven, I., White, R.W. (eds) Advances in Information Retrieval. ECIR 2008. Lecture Notes in Computer Science, vol 4956. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-78646-7_71
Download citation
DOI: https://doi.org/10.1007/978-3-540-78646-7_71
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-78645-0
Online ISBN: 978-3-540-78646-7
eBook Packages: Computer ScienceComputer Science (R0)