Abstract
In a Hilbert framework ℌ, we study the convergence properties of a Newton-like inertial dynamical system governed by a general maximally monotone operator A ℌ: → 2ℌ. When A is equal to the subdifferential of a convex lower semicontinuous proper function, the dynamic corresponds to the introduction of the Hessian-driven damping in the continuous version of the accelerated gradient method of Nesterov. As a result, the oscillations are significantly attenuated. According to the technique introduced by Attouch-Peypouquet (Math. Prog. 2019), the maximally monotone operator is replaced by its Yosida approximation with an appropriate adjustment of the regularization parameter. The introduction into the dynamic of the Newton-like correction term (corresponding to the Hessian driven term in the case of convex minimization) provides a well-posed evolution system for which we will obtain the weak convergence of the generated trajectories towards the zeroes of A. We also obtain the fast convergence of the velocities towards zero. The results tolerate the presence of errors, perturbations. Then, we specialize our results to the case where the operator A is the subdifferential of a convex lower semicontinuous function, and obtain fast optimization results.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Abbas, B., Attouch, H., Svaiter, B.F.: Newton-like dynamics and forward-backward methods for structured monotone inclusions in Hilbert spaces. J. Optim. Theory Appl. 161(2), 331–360 (2014)
Alvarez, F., Attouch, H.: An inertial proximal method for maximal monotone operators via discretization of a nonlinear oscillator with damping. Set-Valued Analysis 9(1-2), 3–11 (2001)
Alvarez, F., Attouch, H., Bolte, J., Redont, P.: A second-order gradient-like dissipative dynamical system with Hessian-driven damping. Application to optimization and mechanics. J. Math. Pures Appl. 81(8), 747–779 (2002)
Apidopoulos, V., Aujol, J.-F., Dossal, C h: Convergence rate of inertial Forward-Backward algorithm beyond Nesterov’s rule. Math. Program. 180, 137–156 (2020)
Attouch, H., Cabot, A.: Convergence of a relaxed inertial proximal algorithm for maximally monotone operators. Math Program. https://doi.org/10.1007/s10107-019-01412-0 (2019)
Attouch, H., Cabot, A.: Convergence rates of inertial forward-backward algorithms. SIAM J. Optim. 28(1), 849–874 (2018)
Attouch, H., Chbani, Z., Fadili, J., Riahi, H.: First-order algorithms via inertial systems with Hessian driven damping. HAL-02193846 (2019)
Attouch, H., Chbani, Z., Peypouquet, J., Redont, P.: Fast convergence of inertial dynamics and algorithms with asymptotic vanishing viscosity. Math. Program. B 168, 123–175 (2018)
Attouch, H., Chbani, Z., Riahi, H.: Fast proximal methods via time scaling of damped inertial dynamics. SIAM J. Optim. 29(3), 2227–2256 (2019)
Attouch, H., Chbani, Z., Riahi, H.: Rate of convergence of the Nesterov accelerated gradient method in the subcritical case α ≤ 3. ESAIM-COCV 25, Article number 2 (2019)
Attouch, H., Goudou, X., Redont, P.: The heavy ball with friction method. The continuous dynamical system, global exploration of the local minima of a real-valued function by asymptotical analysis of a dissipative dynamical system. Commun. Contemp. Math. 2(1), 1–34 (2000)
Attouch, H., László, S.C.: Newton-like inertial dynamics and proximal algorithms governed by maximally monotone operators. HAL-02549730 (2020)
Attouch, H., Maingé, P.E.: Asymptotic behavior of second order dissipative evolution equations combining potential with non-potential effects. ESAIM Control Optim. Calc. Var. 17(3), 836–857 (2011)
Attouch, H., Maingé, P.E., Redont, P.: A second-order differential system with Hessian-driven damping; Application to non-elastic shock laws. Differ. Equ. Appl. 4(1), 27–65 (2012)
Attouch, H., Marques Alves, M., Svaiter, B.F.: A dynamic approach to a proximal-Newton method for monotone inclusions in Hilbert Spaces, with complexity \(\mathcal {O}(1/n^{2})\). J. Convex Anal. 23(1), 139–180 (2016)
Attouch, H., Peypouquet, J.: Convergence of inertial dynamics and proximal algorithms governed by maximal monotone operators. Math. Program. 174(1-2), 391–432 (2019)
Attouch, H., Peypouquet, J.: The rate of convergence of Nesterov’s accelerated forward-backward method is actually faster than 1/k2. SIAM J. Optim. 26(3), 1824–1834 (2016)
Attouch, H., Peypouquet, J., Redont, P.: Fast convex minimization via inertial dynamics with Hessian driven damping. J. Differ. Equ. 261(10), 5734–5783 (2016)
Attouch, H., Redont, P., Svaiter, B.F.: Global convergence of a closed-loop regularized Newton method for solving monotone inclusions in Hilbert spaces. J. Optim. Theory Appl. 157(3), 624–650 (2013)
Attouch, H., Svaiter, B.F.: A continuous dynamical Newton-Like approach to solving monotone inclusions. SIAM J. Control Optim. 49(2), 574–598 (2011)
Bauschke, H., Combettes, P.L.: Convex Analysis and Monotone Operator Theory in Hilbert spaces. CMS Books in Mathematics. Springer (2011)
Brézis, H.: Opérateurs maximaux monotones dans les espaces de Hilbert et équations d’évolution. Lecture Notes 5 North Holland (1972)
Boţ, R.I., Csetnek, E.R.: Second order forward-backward dynamical systems for monotone inclusion problems. SIAM J. Control Optim. 54, 1423–1443 (2016)
Boţ, R. I., Csetnek, E.R., László, S.C.: A second-order dynamical approach with variable damping to nonconvex smooth minimization. Appl. Anal. 99(3), 361–378 (2020)
Boţ, R.I., Csetnek, E.R., László, S.C.: Tikhonov regularization of a second order dynamical system with Hessian damping. Mathematical Programming. https://doi.org/10.1007/s10107-020-01528-8 (2020)
Castera, C., Bolte, J., Févotte, C., Pauwels, E.: An Inertial Newton Algorithm for Deep Learning. HAL-02140748 (2019)
Chambolle, A., Dossal, C h: On the convergence of the iterates of the Fast Iterative Shrinkage Thresholding Algorithm. J. Optim. Theory Appl. 166, 968–982 (2015)
Haraux, A.: Systèmes dynamiques dissipatifs et applications. RMA 17 Masson (1991)
Kim, D.: Accelerated Proximal Point Method for Maximally Monotone Operators. arXiv:1905.05149v3 (2020)
László, S.C.: Convergence rates for an inertial algorithm of gradient type associated to a smooth nonconvex minimization. Mathematical Programming. https://doi.org/10.1007/s10107-020-01534-w (2020)
Lin, T., Jordan, M.I.: A Control-Theoretic Perspective on Optimal High-Order Optimization. arXiv:1912.07168v1 (2019)
May, R.: Asymptotic for a second-order evolution equation with convex potential and vanishing damping term. Turkish J. Math. 41(3), 681–685 (2017)
Nesterov, Y.: A method for solving the convex programming problem with convergence rate o(1/k2). (Russian) Dokl. Akad. Nauk SSSR 269(3), 543–547 (1983)
Polyak, B.T.: Some methods of speeding up the convergence of iterative methods. Z. Vylist. Math. Fiz. 4, 1–17 (1964)
Shi, B., Du, S.S., Jordan, M.I., Su, W.J.: Understanding the acceleration phenomenon via high-resolution differential equations. arXiv:submit/2440124[cs.LG] (2018)
Su, W.J., Boyd, S., Candès, E.J.: A differential equation for modeling Nesterov’s accelerated gradient method: theory and insights. Neural Information Processing Systems 27, 2510–2518 (2014)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix: Auxiliary results
Appendix: Auxiliary results
In the proof of Theorem 2, we use the following straightforward result.
Lemma 2
Let A, B, C ∈ ℝ. The inequality
is satisfied for all \(X,Y\in {\mathcal{H}}\), if and only if C2 − AB ≤ 0 and A, B ≥ 0.
Rights and permissions
About this article
Cite this article
Attouch, H., László, S.C. Continuous Newton-like Inertial Dynamics for Monotone Inclusions. Set-Valued Var. Anal 29, 555–581 (2021). https://doi.org/10.1007/s11228-020-00564-y
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11228-020-00564-y
Keywords
- Damped inertial dynamics
- Hessian damping
- Maximally monotone operators
- Newton method
- Vanishing viscosity
- Yosida regularization