Abstract
The problem of computing low rank approximations of matrices is considered. The novel aspect of our approach is that the low rank approximations are on a collection of matrices. We formulate this as an optimization problem, which aims to minimize the reconstruction (approximation) error. To the best of our knowledge, the optimization problem proposed in this paper does not admit a closed form solution. We thus derive an iterative algorithm, namely GLRAM, which stands for the Generalized Low Rank Approximations of Matrices. GLRAM reduces the reconstruction error sequentially, and the resulting approximation is thus improved during successive iterations. Experimental results show that the algorithm converges rapidly.
We have conducted extensive experiments on image data to evaluate the effectiveness of the proposed algorithm and compare the computed low rank approximations with those obtained from traditional Singular Value Decomposition (SVD) based methods. The comparison is based on the reconstruction error, misclassification error rate, and computation time. Results show that GLRAM is competitive with SVD for classification, while it has a much lower computation cost. However, GLRAM results in a larger reconstruction error than SVD. To further reduce the reconstruction error, we study the combination of GLRAM and SVD, namely GLRAM + SVD, where SVD is preceded by GLRAM. Results show that when using the same number of reduced dimensions, GLRAM + SVD achieves significant reduction of the reconstruction error as compared to GLRAM, while keeping the computation cost low.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Achlioptas, D., & McSherry, F. (2001). Fast computation of low rank matrix approximations. In ACM STOC Conference Proceedings (pp. 611–618).
Aggarwal, C. C. (2001). On the effects of dimensionality reduction on high dimensional similarity search. In ACM PODS Conference Proceedings (pp. 256–266).
Averbuch, A., Lazar, D., & Israeli, M. (1996). Image compression using wavelet transform and multiresolution decomposition. IEEE Transactions on Image Processing, 5:1, 4–15.
Berry, M., Dumais, S., & O'Brie, G. (1995). Using linear algebra for intelligent information retrieval. SIAM Review, 37, 573–595.
Bjork, A., & Golub, G. (1973). Numerical methods for computing angles between linear subspaces. Mathematics of Computation, 27:123, 579–594.
Brand, M. (2002). Incremental singular value decomposition of uncertain data with missing values. In ECCV Conference Proceedings (pp. 707–720).
Castelli, V., Thomasian, A., & Li, C.-S. (2003). CSVD: Clustering and singular value decomposition for approximate similarity searches in high dimensional space. IEEE Transactions on Knowledge and Data Engineering, 15:3, 671–685.
Deerwester, S., Dumais, S., Furnas, G., Landauer, T., & Harshman, R. (1990). Indexing by latent semantic analysis. Journal of the Society for Information Science, 41, 391–407.
Dhillon, I., & Modha, D. (2001). Concept decompositions for large sparse text data using clustering. Machine Learning, 42, 143–175.
Drineas, P., Frieze, A., Kannan, R., Vempala, S., & Vinay, V. (1999). Clustering in large graphs and matrices. In ACM SODA Conference Proceedings (pp. 291–299).
Duda, R., Hart, P., & Stork, D. (2000). Pattern classification. Wiley.
Edelman, A., Arias, T. A., & Smith, S. T. (1998). The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20:2, 303–353.
Frieze, A., Kannan, R., & Vempala, S. (1998). Fast monte-carlo algorithms for finding low-rank approximations. In ACM FOCS Conference Proceedings (pp. 370–378).
Fukunaga, K. (1990). Introduction to statistical pattern classification. San Diego, California, USA: Academic Press.
Golub, G. H., & Van Loan, C. F. (1996). Matrix computations, 3rd edition. Baltimore, MD, USA: The Johns Hopkins University Press.
Gu, M., & Eisenstat, S. C. (1993). A fast and stable algorithm for updating the singular value decomposition. Technical Report YALEU/DCS/RR-966, Department of Computer Science, Yale University.
Kanth, K. V. R., Agrawal, D., Abbadi, A. E., & Singh, A. (1998). Dimensionality reduction for similarity searching in dynamic databases. In ACM SIGMOD Conference Proceedings (pp. 166–176).
Kleinberg, J., & Tomkins, A. (1999). Applications of linear algebra in information retrieval and hypertext analysis. In ACM PODS Conference Proceedings (pp. 185–193).
Kolda, T. G. (2001). Orthogonal tensor decompositions. SIAM Journal on Matrix Analysis and Applications, 23:1, 243–255.
Martinez, A., & Benavente, R. (1998). The AR face database. Technical Report CVC Tech. Report No. 24.
Papadimitriou, C. H., Tamaki, H., Raghavan, P., & Vempala, S. (1998). Latent semantic indexing: A probabilistic analysis. In PODS Conference Proceedings (pp. 159–168).
Samaria, F., & Harter, A. (1994). Parameterisation of a stochastic model for human face identification. In Proceedings of 2nd IEEE Workshop on Applications of Computer Vision, Sarasota FL (pp. 138–142).
Shashua, A., & Levin, A. (2001). Linear image coding for regression and classification using the tensor-rank principle. In CVPR Conference Proceedings (pp. 42–49).
Sim, T., Baker, S., & Bsat, M. (2004). The CMU pose, illumination, and expression (PIE) database. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25:12, 1615–1618.
Srebro, N., & Jaakkola, T. (2003). Weighted low-rank approximations. In ICML Conference Proceedings (pp. 720–727).
Vasilescu, M. A. O., & Terzopoulos, D. (2002). Multilinear analysis of image ensembles: Tensorfaces. In ECCV Conference Proceedings (pp. 447–460).
Yang, J., Zhang, D., Frangi, A., & Yang, J. (2004). Two-dimensional PCA: A new approach to appearance-based face representation and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5:1, 131–137.
Zhang, T., & Golub, G. H. (2001). Rank-one approximation to high order tensors. SIAM Journal on Matrix Analysis and Applications, 5:2, 534–550.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor: Peter Flach
Rights and permissions
About this article
Cite this article
Ye, J. Generalized Low Rank Approximations of Matrices. Mach Learn 61, 167–191 (2005). https://doi.org/10.1007/s10994-005-3561-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-005-3561-6