Summary
This paper deals with the best low multilinear rank approximation of higher-order tensors. Given a tensor, we are looking for another tensor, as close as possible to the given one and with bounded multilinear rank. Higher-order tensors are used in higher-order statistics, signal processing, telecommunications and many other fields. In particular, the best low multilinear rank approximation is used as a tool for dimensionality reduction and signal subspace estimation.
Computing the best low multilinear rank approximation is a nontrivial task. Higher-order generalizations of the singular value decomposition lead to suboptimal solutions. The higher-order orthogonal iteration is a widely used linearly convergent algorithm for further refinement. We aim for conceptually faster algorithms. However, applying standard optimization algorithms directly is not a good idea since there are infinitely many equivalent solutions. Nice convergence properties are observed when the solutions are isolated. The present invariance can be removed by working on quotient manifolds. We discuss three algorithms, based on Newton’s method, the trust-region scheme and conjugate gradients. We also comment on the local minima of the problem.
* Research supported by: (1) The Belgian Federal Science Policy Office: IUAP P6/04 (DYSCO, “Dynamical systems, control and optimization”, 2007–2011), (2) Communauté française de Belgique - Actions de Recherche Concertées, (3) Research Council K.U.Leuven: GOA-AMBioRICS, GOA-MaNet, CoE EF/05/006 Optimization in Engineering (OPTEC), (4) F.W.O. project G.0321.06, “Numerical tensor methods for spectral analysis”. (5) “Impulsfinanciering Campus Kortrijk (2007–2012)(CIF1)” and STRT1/08/023. Part of this research was carried out while M. Ishteva was with K.U.Leuven, supported by OE/06/25, OE/07/17, OE/08/007, OE/09/004. The scientific responsibility rests with the authors.
Access provided by Autonomous University of Puebla. Download to read the full chapter text
Chapter PDF
Similar content being viewed by others
Keywords
- Matrix Anal
- Multilinear Algebra
- Steep Descent Direction
- Nonlinear Conjugate Gradient Method
- Quotient Manifold
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
References
P.-A. Absil, C. G. Baker, K. A. Gallivan. Trust-region methods on Riemannian manifolds. Found. Comput. Math., 7(3):303–330, 2007.
P.-A. Absil, M. Ishteva, L. De Lathauwer, S. Van Huffel. A geometric Newton method for Oja’s vector field. Neural Comput., 21(5):1415–1433, 2009.
P.-A. Absil, R. Mahony, R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008.
E. Acar, C. A. Bingol, H. Bingol, R. Bro, B. Yener. Multiway analysis of epilepsy tensors. ISMB 2007 Conference Proc., Bioinformatics, 23(13):i10–i18, 2007.
R. L. Adler, J.-P. Dedieu, J. Y. Margulies, M. Martens, M. Shub. Newton’s method on Riemannian manifolds and a geometric model for the human spine. IMA J. Numer. Anal., 22(3):359–390, 2002.
C. A. Andersson, R. Bro. The N-way toolbox for matlab. Chemometrics and Intelligent Laboratory Systems, 52 (1):1–4, 2000. See also http://www.models.kvl.dk/source/nwaytoolbox/.
R. Badeau, R. Boyer. Fast multilinear singular value decomposition for structured tensors. SIAM J. Matrix Anal. Appl., 30(3):1008–1021, 2008.
C. Caiafa, A. Cichocki. Reconstructing matrices and tensors from few rows and columns. In Proc. of 2009 International Symposium on Nonlinear Theory and its Applications, 2009. In press.
J. Carroll, J. Chang. Analysis of individual differences in multidimensional scaling via an N-way generalization of “Eckart-Young” decomposition. Psychometrika, 35(3):283–319, 1970.
J. Chen, Y. Saad. On the tensor SVD and the optimal low rank orthogonal approximation of tensors. SIAM J. Matrix Anal. Appl., 30(4):1709–1734, 2009.
A. Cichocki, R. Zdunek, A. H. Phan, and S.-I. Amari. Nonnegative Matrix and Tensor Factorizations. Wiley, 2009.
P. Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287–314, 1994.
P. Comon. Tensor decompositions. In J. G. McWhirter, I. K. Proudler (eds), Mathematics in Signal Processing V, pp. 1–24. Clarendon Press, Oxford, 2002.
P. Comon, G. Golub, L.-H. Lim, B. Mourrain. Symmetric tensors and symmetric tensor rank. SIAM J. Matrix Anal. Appl., 30(3):1254–1279, 2008.
A. R. Conn, N. I. M. Gould, P. L. Toint. Trust-Region Methods. MPS-SIAM Series on Optimization. SIAM, Philadelphia, PA, 2000.
L. De Lathauwer. Signal Processing Based on Multilinear Algebra. PhD thesis, Dept. of Electrical Engineering, Katholieke Universiteit Leuven, 1997.
L. De Lathauwer. A link between the canonical decomposition in multilinear algebra and simultaneous matrix diagonalization. SIAM J. Matrix Anal. Appl., 28(3):642–666, 2006.
L. De Lathauwer. Decompositions of a higher-order tensor in block terms — Part I: Lemmas for partitioned matrices. SIAM J. Matrix Anal. Appl., 30(3):1022–1032, 2008.
L. De Lathauwer. Decompositions of a higher-order tensor in block terms — Part II: Definitions and uniqueness. SIAM J. Matrix Anal. Appl., 30(3):1033– 1066, 2008.
L. De Lathauwer. A survey of tensor methods. In Proc. of the 2009 IEEE International Symposium on Circuits and Systems (ISCAS 2009), pp. 2773– 2776, Taipei, Taiwan, 2009.
L. De Lathauwer, J. Castaing. Tensor-based techniques for the blind separation of DS-CDMA signals. Signal Processing, 87(2):322–336, 2007.
L. De Lathauwer, B. De Moor, J. Vandewalle. A multilinear singular value decomposition. SIAM J. Matrix Anal. Appl., 21(4):1253–1278, 2000.
L. De Lathauwer, B. De Moor, J. Vandewalle. On the best rank-1 and rank-(R 1,R 2,…,R N ) approximation of higher-order tensors. SIAM J. Matrix Anal. Appl., 21 (4):1324–1342, 2000.
L. De Lathauwer, B. De Moor, J. Vandewalle. Independent component analysis and (simultaneous) third-order tensor diagonalization. IEEE Trans. Signal Process., 49(10):2262–2271, 2001.
L. De Lathauwer, B. De Moor, J. Vandewalle. Computation of the canonical decomposition by means of a simultaneous generalized Schur decomposition.SIAM J. Matrix Anal. Appl., 26(2):295–327, 2004.
L. De Lathauwer, D. Nion. Decompositions of a higher-order tensor in block terms — Part III: Alternating least squares algorithms. SIAM J. Matrix Anal. Appl., 30(3):1067–1083, 2008.
L. De Lathauwer, J. Vandewalle. Dimensionality reduction in higher-order signal processing and rank-(R 1,R 2,…,R N ) reduction in multilinear algebra. Linear Algebra Appl., 391:31–55, 2004.
V. de Silva, L.-H. Lim. Tensor rank and the ill-posedness of the best low-rank approximation problem. SIAM J. Matrix Anal. Appl., 30(3):1084–1127, 2008.
M. De Vos, L. De Lathauwer, B. Vanrumste, S. Van Huffel, W. Van Paesschen.Canonical decomposition of ictal scalp EEG and accurate source localization:Principles and simulation study. Journal of Computational Intelligence and Neuroscience, 2007(Article ID 58253):1–10, 2007.
M. De Vos, A. Vergult, L. De Lathauwer, W. De Clercq, S. Van Huffel, P. Dupont, A. Palmini, W. Van Paesschen. Canonical decomposition of ictal scalp EEG reliably detects the seizure onset zone. NeuroImage, 37(3):844–854, 2007.
M. Elad, P. Milanfar, G. H. Golub. Shape from moments — an estimation theory perspective. IEEE Trans. on Signal Processing, 52(7):1814–1829, 2004.
L. Eldén, B. Savas. A Newton–Grassmann method for computing the best multi-linear rank-(r 1,r 2,…,r 3) approximation of a tensor. SIAM J. Matrix Anal. Appl., 31(2):248–271, 2009.
R. Fletcher, C. M. Reeves. Function minimization by conjugate gradients. Comput. J., 7:149–154, 1964.
G. H. Golub, C. F. Van Loan. Matrix Computations. Johns Hopkins University Press, Baltimore, Maryland, 3rd edition, 1996.
M. Haardt, F. Roemer, G. Del Galdo. Higher-order SVD-based subspace estimation to improve the parameter estimation accuracy in multidimensional harmonic retrieval problems. IEEE Trans. on Signal Processing, 56(7):3198– 3213, 2008.
W. W. Hager, H. Zhang. A new conjugate gradient method with guaranteed descent and an efficient line search. SIAM Journal on Optimization, 16(1):170– 192, 2005.
R. A. Harshman. Foundations of the PARAFAC procedure: Model and conditions for an “explanatory” multi-mode factor analysis. UCLA Working Papers in Phonetics, 16(1):1–84, 1970.
U. Helmke, J. B. Moore. Optimization and Dynamical Systems. Springer-Verlag, 1993.
M. R. Hestenes, E. Stiefel. Methods of conjugate gradients for solving linear systems. J. Research Nat. Bur. Standards, 49:409–436 (1953), 1952.
F. L. Hitchcock. The expression of a tensor or a polyadic as a sum of products. Journal of Mathematical Physics, 6(1):164–189, 1927.
F. L. Hitchcock. Multiple invariants and generalized rank of a p-way matrix or tensor. Journal of Mathematical Physics, 7(1):39–79, 1927.
M. Ishteva. Numerical methods for the best low multilinear rank approximation of higher-order tensors. PhD thesis, Dept. of Electrical Engineering, Katholieke Universiteit Leuven, 2009.
M. Ishteva, P.-A. Absil, S. Van Huffel, L. De Lathauwer. Best low multilinear rank approximation with conjugate gradients. Tech. Rep. 09-246, ESAT-SISTA, K.U.Leuven, Belgium, 2009.
M. Ishteva, P.-A. Absil, S. Van Huffel, L. De Lathauwer. Tucker compression and local optima. Tech. Rep. UCL-INMA-2010.012, Université catholique de Louvain and 09-247, ESAT-SISTA, K.U.Leuven, Belgium, 2010.
M. Ishteva, L. De Lathauwer, P.-A. Absil, S. Van Huffel. Dimensionality reduction for higher-order tensors: algorithms and applications. International Journal of Pure and Applied Mathematics, 42(3):337–343, 2008.
M. Ishteva, L. De Lathauwer, P.-A. Absil, S. Van Huffel. Best low multilinear rank approximation of higher-order tensors, based on the Riemannian trustregion scheme. Tech. Rep. 09-142, ESAT-SISTA, K.U.Leuven, Belgium, 2009.
M. Ishteva, L. De Lathauwer, P.-A. Absil, S. Van Huffel. Differential-geometric Newton method for the best rank-(R 1, R 2, R 3) approximation of tensors. Numerical Algorithms, 51(2):179–194, 2009.
M. Ishteva, L. De Lathauwer, S. Van Huffel. Comparison of the performance of matrix and tensor based multi-channel harmonic analysis. In 7th International Conf. on Mathematics in Signal Processing, Cirencester, UK, pp. 77–80, 2006.
E. Kofidis, P. A. Regalia. On the best rank-1 approximation of higher-order supersymmetric tensors. SIAM J. Matrix Anal. Appl, 23(3):863–884, 2002.
T. Kolda. Orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl., 23:243–255, 2001.
T. G. Kolda, B. W. Bader. Tensor decompositions and applications. SIAM Review, 51(3):455–500, 2009.
P. M. Kroonenberg. Applied Multiway Data Analysis. Wiley, 2008.
P. M. Kroonenberg, J. de Leeuw. Principal component analysis of three-mode data by means of alternating least squares algorithms. Psychometrika, 45(1):69– 97, 1980.
M. W. Mahoney, M. Maggioni, P. Drineas. Tensor-CUR decompositions for tensor-based data. SIAM J. Matrix Anal. Appl., 30(3):957–987, 2008.
J. H. Manton. Optimization algorithms exploiting unitary constraints. IEEE Trans. Signal Process., 50(3):635–650, 2002.
C. D. Moravitz Martin, C. F. Van Loan. A Jacobi-type method for computing orthogonal tensor decompositions. SIAM J. Matrix Anal. Appl., 30(3):1219– 1232, 2008.
J. J. Moré, D. C. Sorensen. Computing a trust region step. SIAM J. Sci. Statist. Comput., 4(3):553–572, 1983.
J. Nocedal, S. J. Wright. Numerical Optimization. Springer Verlag, New York, 2nd edition, 2006. Springer Series in Operations Research.
I. V. Oseledets, D. V. Savostianov, E. E. Tyrtyshnikov. Tucker dimensionality reduction of three-dimensional arrays in linear time. SIAM J. Matrix Anal. Appl., 30(3):939–956, 2008.
J.-M. Papy, L. De Lathauwer, S. Van Huffel. Exponential data fitting using multilinear algebra: The single-channel and the multichannel case. Numer. Linear Algebra Appl., 12(8):809–826, 2005.
J.-M. Papy, L. De Lathauwer, S. Van Huffel. Exponential data fitting using multilinear algebra: The decimative case. J. Chemometrics, 23(7–8):341–351, 2009.
E. Polak, G. Ribière. Note sur la convergence de méthodes de directions conjugu ées. Rev. Française Informat. Recherche Opérationnelle, 3(16):35–43, 1969.
B. Savas, L. Eldén. Krylov subspace methods for tensor computations. Tech. Rep. LITH-MAT-R-2009-02-SE, Dept. of Mathematics, Link¨oping University, 2009.
B. Savas, L.-H. Lim. Best multilinear rank approximation of tensors with quasi- Newton methods on Grassmannians. Tech. Rep. LITH-MAT-R-2008-01-SE, Dept. of Mathematics, Link¨oping University, 2008.
M. Shub. Some remarks on dynamical systems and numerical analysis. In L. Lara-Carrero, J. Lewowicz (eds), Proc. VII ELAM., pp. 69–92. Equinoccio, U. Sim´on Bol´ıvar, Caracas, 1986.
N. Sidiropoulos, R. Bro, G. Giannakis. Parallel factor analysis in sensor array processing. IEEE Trans. Signal Process., 48:2377–2388, 2000.
A. Smilde, R. Bro, P. Geladi. Multi-way Analysis. Applications in the Chemical Sciences. John Wiley and Sons, Chichester, U.K., 2004.
S. T. Smith. Geometric Optimization Methods for Adaptive Filtering. PhD thesis, Division of Applied Sciences, Harvard University, Cambridge, MA, 1993.
S. T. Smith. Optimization techniques on Riemannian manifolds. In A. Bloch (ed), Hamiltonian and gradient flows, algorithms and control, volume 3 of Fields Inst. Commun., pp. 113–136. Amer. Math. Soc., Providence, RI, 1994.
T. Steihaug. The conjugate gradient method and trust regions in large scale optimization. SIAM J. Numer. Anal., 20(3):626–637, 1983.
P. L. Toint. Towards an efficient sparsity exploiting Newton method for minimization. In I. S. Duff (ed), Sparse Matrices and Their Uses, pp. 57–88. Academic Press, London, 1981.
L. R. Tucker. The extension of factor analysis to three-dimensional matrices. In H. Gulliksen, N. Frederiksen (eds), Contributions to mathematical psychology, pp. 109–127. Holt, Rinehart & Winston, NY, 1964.
L. R. Tucker. Some mathematical notes on three-mode factor analysis. Psychometrika, 31:279–311, 1966.
L. Vanhamme, S. Van Huffel. Multichannel quantification of biomedical magnetic resonance spectroscopy signals. In F. Luk (ed), Proc. of SPIE, Advanced Signal Processing Algorithms, Architectures, and Implementations VIII, volume 3461, pp. 237–248, San Diego, California, 1998.
T. Zhang, G. H. Golub. Rank-one approximation to high order tensors. SIAM J. Matrix Anal. Appl., 23:534–550, 2001.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer -Verlag Berlin Heidelberg
About this paper
Cite this paper
Ishteva, M., Absil, PA., Van Huffel, S., De Lathauwer, L. (2010). On the Best Low Multilinear Rank Approximation of Higher-order Tensors*. In: Diehl, M., Glineur, F., Jarlebring, E., Michiels, W. (eds) Recent Advances in Optimization and its Applications in Engineering. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12598-0_13
Download citation
DOI: https://doi.org/10.1007/978-3-642-12598-0_13
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-12597-3
Online ISBN: 978-3-642-12598-0
eBook Packages: EngineeringEngineering (R0)