Abstract
We derived a convergence result for a sequential procedure known as alternating maximization (minimization) to the maximum likelihood estimator for a pretty large family of models - Generalized Linear Models. Alternating procedure for linear regression becomes to the well-known algorithm of Alternating Least Squares, because of the quadraticity of log-likelihood function L(υ). In Generalized Linear Models framework we lose quadraticity of L(υ), but still have concavity due to the fact that error-distribution is from exponential family. Concentration property makes the Taylor approximation of L(υ) up to the second order accurate and makes possible the use of alternating minimization (maximization) technique. Examples and experiments confirm convergence result followed by the discussion of the importance of initial guess.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
A. Andresen, “Finite sample analysis of profile m-estimation in the single index model”, Electronic Journal of Statistics, 9(2), 2528–2641, 2015.
A. Andresen, V. Spokoiny, “Two convergence results for an alternation maximization”, arXiv: 1501.01525, 2015.
J. Bezdek, R. Hathaway, “Convergence of Alternating Optimization” Neural, Parallel & Pacific Computations, 11, 351–368, 2003.
L. Cavalier, G. K. Golubev, D. Picard and A. B. Tsybakov, “Oracle inequialities for inverse problems”, The Annals of Statistics, 30(3), 843–874, 2002.
X. Chen, “Large sample sieve estimation of semi-nonparametric models”, Handbook of Econometrics, 6, 5549–5632, 2007.
V. Chernozhukov, D. Chetverikov and K. Kato, “Anti-concentration and honest, adaptive confidence bands”, The Annals of Statistics, 34(4), 1653–1677, 2014.
A. P. Dempster, N. M. Laird and D. B. Rubin, “Maximum likelihood from incomplete data via the EM algorithm”, Journal of the Royal Statistical Society, Series B, 39, 1–38, 1977.
I. A. Ibragimov and R. Z. Khas’minskij, Statistical Estimation. Asymptotic Theory (Springer, Berlin, 1981).
A. Kneip, “Ordered linear smoothers”, The Annals of Statistics, 22(2), 835–866, 1994.
M. R. Kosorok, Introduction to Empirical Processes and Semiparametric Inference (Springer in Statistics, Berlin, 2005).
G. J. McLachlan and T. Krishman, The EM Algorithm and Extensions (Wiley, New York, 1997).
V. Spokoiny, T. Dickhaus, Basics of Modern Mathematical Statistics (Springer in Statistics, Berlin, 2015).
C. F. J. Wu, “On the convergence properties of the EM algorithm”, Annals of Statistics, 11, 95–103, 1983.
Author information
Authors and Affiliations
Corresponding author
Additional information
Russian Text © The Author(s), 2019, published in Izvestiya Natsional’noi Akademii Nauk Armenii, Matematika, 2019, No. 5, pp. 53–69.
About this article
Cite this article
Minasyan, A.G. Alternating Least Squares in Generalized Linear Models. J. Contemp. Mathemat. Anal. 54, 302–312 (2019). https://doi.org/10.3103/S1068362319050078
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.3103/S1068362319050078