Abstract
Face datasets are considered a primary tool for evaluating the efficacy of face recognition methods. Here we show that in many of the commonly used face datasets, face images can be recognized accurately at a rate significantly higher than random even when no face, hair or clothes features appear in the image. The experiments were done by cutting a small background area from each face image, so that each face dataset provided a new image dataset which included only seemingly blank images. Then, an image classification method was used in order to check the classification accuracy. Experimental results show that the classification accuracy ranged between 13.5% (color FERET) to 99% (YaleB). These results indicate that the performance of face recognition methods measured using face image datasets may be biased. Compilable source code used for this experiment is freely available for download via the Internet.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Bishop, C. M. (2006). Pattern recognition and machine learning. New York: Springer.
Chen, L. F., Liao, H. Y., Lin, J. C., & Han, C. C. (2001). Why recognition in a statistics-based face recognition system should be based on the pure face portion: a probabilistic decision-based proof. Pattern Recognition, 34, 1393–1403.
Fei-Fei, L., Fergus, R., & Perona, P. (2006). One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 594–611.
Gabor, D. (1946). Theory of communication. Journal of IEEE, 93, 429–457.
Georghiades, A. S., Belhumeur, P. N., & Kriegman, D. J. (2001). From few to many: illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis Machine Intelligence, 23, 643–660.
Goldberg, I. G., Allan, C., Burel, J. M., Creager, D., Falconi, A., Hochheiser, H., Johnston, J., Mellen, J., Sorger, P. K., & Swedlow, J. R. (2005). The Open Microscopy Environment (OME) Data Model and XML file: open tools for informatics and quantitative analysis in biological imaging. Genome Biology, 6, R47.
Gradshtein, I., & Ryzhik, I. (1994). Table of integrals, series and products (5th edn., p. 1054). New York: Academic Press.
Gray, S. B. (1978). Local properties of binary images in two dimensions. IEEE Transactions on Computers, 20, 551–561.
Gregorescu, C., Petkov, N., & Kruizinga, P. (2002). Comparison of texture features based on Gabor filters. IEEE Transactions on Image Processing, 11, 1160–1167.
Gross, R., Baker, S., Matthews, I., & Kanade, T. (2004). Face recognition across pose and illumination. In S. Z. Lin & A. K. Jain (Eds.), Handbook of face recognition. Berlin: Springer.
Gurevich, I. B., & Koryabkina, I. V. (2006). Comparative analysis and classification of features for image models. Pattern Recognition and Image Analysis, 16, 265–297.
Hadjidementriou, E., Grossberg, M., & Nayar, S. (2001). Spatial information in multiresolution histograms. In IEEE conf. on computer vision and pattern recognition (Vol. 1, p. 702).
Haralick, R. M., Shanmugam, K., & Dimstein, I. (1973). Textural features for image classification. IEEE Transaction on System, Man and Cybernetics, 6, 269–285.
Jain, V., & Mukherjee, A. (2002). http://vis-www.cs.umass.edu/%7Evidit/IndianFaceDatabase.
Kong, S. G., Heo, J., Abidi, B. R., Paik, J., & Abidi, M. A. (2005). Recent advances in visual and infrared face recognition: a review. Computer Vision and Image Understanding, 97, 103–135.
Lim, J. S. (1990). Two-dimensional signal and image processing. In Signals, systems, and the Fourier transform (pp. 42–45). Englewood Cliffs: Prentice Hall.
Lynos, M., Akamatsu, S., Kamachi, M., & Gyboa, J. (1998). Coding facial expressions with Gabor wavelets. In Proceedings of the third IEEE international conference on automatic face and facial recognition (pp. 200–205).
Murphy, R. F., Velliste, M., Yao, J., & Porreca, G. (2001). Searching online journals for fluorescence microscopy images depicting protein subcellular location patterns. In Proceedings of the second IEEE international symposium on bioinformatics and biomedical engeering (pp. 119–128).
Orlov, N., Johnston, J., Macura, T., Shamir, L., & Goldberg, I. G. (2007). Computer vision for microscopy applications. In G. Obinata & A. Dutta (Eds.), Vision systems—segmentation and pattern recognition (pp. 221–242). Vienna: ARS.
Orlov, N., Shamir, L., Johnston, J., Macura, T., Eckley, D. M., & Goldberg, I. G. (2008, in press). WND-CHARM: Multi-purpose image classification using compound image transforms. Pattern Recognition Letters.
Otsu, N. (1979). A threshold selection method from gray level histograms. IEEE Transactions on System, Man and Cybernetics, 9, 62–66.
Phillips, P. J., Wechsler, H., Huang, J., & Rauss, P. (1998). The FERET database and evaluation procedure for face recognition algorithms. Journal of Image and Vision Computing, 16, 295–306.
Phillips, P. J., Moon, H., Rizvi, S. A., & Rauss, P. J. (2000). The FERET evaluation methodology for face recognition algorithms. IEEE Transactions Pattern Analysis and Machine Intelligence, 22, 1090–1104.
Pinto, N., Cox, D. D., & DiCarlo, J. J. (2008). Why is real-world visual object recognition hard? PLoS Computational Biology, 4, e27.
Prewitt, J. M. (1970). Object enhancement and extraction. In B. S. Lipkin & A. Rosenfeld (Eds.), Picture processing and psychopictoris (pp. 75–149). New York: Academic Press.
Rodenacker, K., & Bengsson, E. (2003). A feature set for cytometry on digitized microscopic images. Analytical Cellular Pathology, 25, 1–36.
Samaria, F., & Harter, A. C. (1994). Parameterisation of a stochastic model for human face identification. In Proceedings of the second IEEE workshop on applications of computer vision.
Hond, D., & Spacek, L. (1997). Distinctive descriptions for face processing. In Proceedings of 8th BMVC (pp. 320–329).
Spacek, L. (2002), University of Essex face database. http://dces.essex.ac.uk/mv/allfaces/index.html.
Swedlow, J. R., Goldberg, I., Brauner, E., & Sorger, P. K. (2003). Image informatics and quantitative analysis of biological images. Science, 300, 100–102.
Tamura, H., Mori, S., & Yamavaki, T. (1978). Textural features corresponding to visual perception. IEEE Transactions on System, Man and Cybernetics, 8, 460–472.
Teague, M. R. (1979). Image analysis via the general theory of moments. Journal of the Optical Society of America, 70, 920.
Zhao, W., Chellappa, R., Phillips, P. J., & Rosenfeld, A. (2005). Face recognition: A literature survey. ACM Computing Surveys, 35, 399–458.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Shamir, L. Evaluation of Face Datasets as Tools for Assessing the Performance of Face Recognition Methods. Int J Comput Vis 79, 225–230 (2008). https://doi.org/10.1007/s11263-008-0143-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-008-0143-7